Collibra’s Telemetry Backbone - OpenTelemetry and Apache Beam

At a certain time, when you’re a company that’s building out a successful SaaS platform there comes a time when classic observability tools don’t cut it anymore. Or they don’t scale as well as you grow or they become too expensive. That’s why the SRE team started to look at observability as a big data problem: In combination with a specification for telemetry (OpenTelemetry) and big data tools like Apache Beam they build the Telemetry Backbone, the streaming hub for all of Collibra’s telemetry data. This is the story and how Apache Beam is an important factor in its success.