TLDR: Our partner Stitch is introducing Singer: an open source project for simple, composable ETL. Singer enables any data source to be analyzed in Redash — regardless of whether or not you’re a Stitch customer.
Redash is a powerful tool for analyzing and visualizing all kinds of data, and our users use us to gain insight into many different aspects of their business. The most interesting insights are the result of consolidating data from different tools and sources. While it’s possible in Redash to query multiple data sources, it’s much easier and performant when the data consolidated into a data warehouse. Many of our users use our partner Stitch to handle data consolidation. While Stitch integrates with more than 50 different data sources, we know that there are many more places where your data might live.
When our users need a data source that isn’t supported by commercial ETL tools, their only option has been to write a custom script to pull the data themselves. While writing a script for a one-time data pull isn’t too hard, we’ve found that over the long term it creates a big maintenance burden. APIs change, a different data warehouse is chosen, servers go down, cron jobs stop running, API rate limits get hit, and so on. This is why we recommend commercial ETL tools in the first place, but the problem isn’t totally solved if only five of your six data sources is supported.
Stitch is introducing Singer to solve that problem. Singer is a better way to write and collaborate on ETL scripts, and it includes three parts:
- Taps, which pull data from sources
- Targets, which send data to destinations
- A JSON-based format for communication between taps and targets
All taps and targets can be mixed and matched, so changing the destination you’re loading data into is easy. Since it’s all open source, the community can leverage each other’s improvements.
Singer integrations can be run on servers that you manage, but Stitch can run any integration written in the Singer format. When you use Stitch for your Singer integrations, you’ll get auto-scaling, a secure infrastructure, credential management, monitoring, and alerting.