Tools Thoughtworks Technology Radar 51. Great Expectations Adopt Great Expectations has become a sensible default for our teams in the data quality space, which is why we recommend adopting it — not only for the lack of better alternatives but also because our teams have reported great results in several client projects. Great Expectations is a framework that allows you to craft built-in controls that flag anomalies or quality issues in data pipelines. Just as unit tests run in a build pipeline, Great Expectations makes assertions during the execution of a data pipeline. We like its simplicity and ease of use — the rules stored in JSON can be modified by our data domain experts without necessarily needing data engineering skills. 52. k6 Adopt Since we first mentioned it in the Radar, k6 has become a go-to tool for performance testing. We continue to be fans of how easy it is to write JavaScript code for tests, but k6 also has a low-code test builder to make playing with the tool even easier. The documentation shows how easy it is to add performance testing to a pipeline across multiple CI/CD tools. Our teams find it easy to integrate visualization tools like Grafana and New Relic, which help them tune both infrastructure and applications. The developer friendliness and ecosystem make k6 a compelling option for investigating a system’s behavior under heavy load. 53. Apache Superset Trial Apache Superset is a great business intelligence (BI) tool for data exploration and visualization to work with large data lake and data warehouse setups. It supports several data sources — including AWS Redshift, BigQuery, Azure MS SQL, Snowflake and ClickHouse. Moreover, you don’t have to be a data engineer to use it; it’s meant to benefit all engineers exploring data in their everyday work. For demanding use cases, we found it easy to scale Superset by deploying it in a Kubernetes cluster. Since we last talked about it in the Radar, Superset has graduated as an Apache product, and we’ve seen great success in several projects. 54. AWS Backup Vault Lock Trial When implementing robust, secure and reliable disaster recovery, it’s necessary to ensure that backups can’t be deleted or altered before their expiry, either maliciously or accidentally. Previously, with AWS Backup, these policies and guarantees had to be implemented by hand. Recently, AWS has added the Vault Lock feature to ensure backups are immutable and untamperable. AWS Backup Vault Lock enforces retention and deletion policies and prevents even those with administrator privileges from altering or deleting backup files. This has proved to be a valuable addition and fills a previously empty space. 55. AWS Control Tower Trial Multi-team account management is a challenge in AWS, especially in setup and governance; AWS Control Tower is an attempt to address this challenge. Our team has reported good results using it to manage accounts and access control for multiple teams in the organization through a single, centralized place. © Thoughtworks, Inc. All Rights Reserved. 28