Controlling Elixir supervisors at runtime with feature flags

1

LaunchDarkly

At Knock, we use LaunchDarkly to power feature flags. Feature flags are powerful because they enable us to control, at runtime, the behavior of different parts of the system. Most of the time, this means controlling access to features on our client-side application, or controlling the rollout of new features across our application. However, we recently adopted a circuit breaker pattern built around feature flags. This pattern helps our services be more reliable when things fail.

#Feature Flags #A/B Testing #Developer Tools 36 social mentions

2

Amazon Kinesis

Like many applications, our infrastructure relies on queues to decouple various components. In our system we use AWS Kinesis as a data stream, consumed by Broadway consumers for some critical parts of our infrastructure. We have found that sometimes our Broadway consumers for AWS Kinesis fail in ways that do not gracefully recover when they crash. For example, each Kinesis shard has its own supervision tree managed by the Kinesis Broadway consumer. We found that if a shard consumer experienced a crash-inducing error, the shard would not restart and the crash would not cascade up to the Broadway producer. While we have worked on contributing to this consumer library, we decided that it would be important to have runtime control over stopping and starting consumers to respond to such failures just in case.

#Stream Processing #Data Management #Analytics 23 social mentions