Configuration as the interface
The design decision the whole service rests on is that a topic is data, not code. Topics, their Avro schemas, and the mapping into the pricing platform are all declared in a single JSON configuration. Onboarding a new topic is an edit to that file, which means it is reviewable, revertible, and does not queue behind a release.
The alternative — one consumer per topic — was the status quo, and its real cost was not the writing. It was that every topic carried its own deployment, its own dashboards, and its own retry semantics, so an operational lesson learned on one topic had to be re-learned on the rest.
Resilience
Retry handling is owned inside the service rather than left to the caller. Consumption runs against Confluent Cloud with Avro and Schema Registry, so a schema that moves is caught at deserialisation rather than downstream in the pricing data.
Operability
Datadog monitoring and alerting is end-to-end and was built alongside the service, not bolted on after. Jenkins pipelines carry it into non-production and production. The load characteristics were established rather than assumed: multi-phase Kafka load testing with JMeter and Groovy, driven through AWS SSM.