Platforms Thoughtworks Technology Radar of our concerns previously was explainability. Earlier this year, BigQuery Explainable AI was announced for general availability, taking a step in addressing this. We can also export BigQuery ML models to Cloud Storage as a Tensorflow SavedModel and use them for online prediction. There remain trade-offs like ease of “continuous delivery for machine learning” but with its low barrier to entry, BigQuery ML remains an attractive option, particularly when the data already resides in BigQuery. 31. Google Cloud Dataflow Trial Google Cloud Dataflow is a cloud-based data-processing service for both batch and real-time data- streaming applications. Our teams are using Dataflow to create processing pipelines for integrating, preparing and analyzing large data sets, with Apache Beam’s unified programming model on top to ease manageability. We first featured Dataflow in 2018, and its stability, performance and rich feature set make us confident to move it to Trial in this edition of the Radar. 32. Reusable workflows in Github Actions Trial We’ve seen increased interest in GitHub Actions since we first blipped it two Radars ago. With the release of reusable workflows, GitHub continues to evolve the product in a way that addresses some of its early shortcomings. Reusable workflows in Github Actions bring modularity to pipeline design, allowing parameterized reuse even across repositories (as long as the workflow repository is public). They support explicit passing of confidential values as secrets and can pass outputs to the calling job. With a few lines of YAML, GitHub Actions now gives you the type of flexibility you see with CircleCI Orbs or Azure Pipeline Templates, but without having to leave GitHub as a platform. 33. Sealed Secrets Trial Kubernetes natively supports a key-value object known as a secret. However, by default, Kubernetes secrets aren’t really secret. They’re handled separately from other key-value data so that precautions or access control can be applied separately. There is support for encrypting secrets before they are stored in etcd, but the secrets start out as plain text fields in configuration files. Sealed Secrets is a combination operator and command-line utility that uses asymmetric keys to encrypt secrets so that they can only be decrypted by the controller in the cluster. This process ensures that the secrets won’t be compromised while they sit in the configuration files that define a Kubernetes deployment. Once encrypted, these files can be safely shared or stored alongside other deployment artifacts. 34. VerneMQ Trial VerneMQ is an open-source, high-performance, distributed MQTT broker. We’ve blipped other MQTT brokers in the past like Mosquitto and EMQ. Like EMQ and RabbitMQ, VerneMQ is also based on Erlang/OTP which makes it highly scalable. It scales horizontally and vertically on commodity hardware to support a high number of concurrent publishers and consumers while maintaining low latency and fault tolerance. In our internal benchmarks, we’ve been able to achieve a few million concurrent connections in a single cluster. While it’s not new, we’ve used it in production for some time now, and it has worked well for us. © Thoughtworks, Inc. All Rights Reserved. 22
Vol 26 | Technology Radar Page 21 Page 23