Horizontally scaling Kafka consumers with rendezvous hashing

Upstash provides Serverless Redis and Kafka as a service.

The native Kafka connector included with Tinybird lets you connect to a Kafka cluster, whether that’s a self-hosted Apache Kafka instance or any one of the hosted/self-hosted variants out there (Confluent, Redpanda, Upstash, etc.).

Apache Kafka is an open-source message broker project developed by the Apache Software Foundation written in Scala.

Apache Kafka is the foundational architecture most developers choose when building streaming applications. It’s incredibly scalable, fault-tolerant, and dependable. And its popularity has yielded vast knowledge with an active community to support developers when they build applications. Naturally, Kafka is now a table-stakes data platform component for nearly every major enterprise that deals with events data.

The Apache Kafka C/C++ library. Contribute to edenhill/librdkafka development by creating an account on GitHub.

We could have made some changes at the librdkafka level (see this), but we didn’t really want to pursue this (at least not yet).