I was the third engineer at Ethos, and one of the few people left who had written large parts of the monolith. As the company moved toward service ownership, service calls grew in every direction: sync requests, ad hoc queues, one-off retry logic, and unclear ownership after failure.
I wrote the design proposal, compared options, presented to engineering leads, got CTO approval, and built the shared mechanics teams would depend on. The decision criteria were build cost, maintenance cost, technology cost, scale, reversibility, ownership, and whether product teams would actually use it.
Turned prior incidents and operational pain into explicit requirements.
Shared mechanics lived in one place, instead of being reimplemented by each service.
Defined the ownership split: product teams kept their business logic, while the platform owned the delivery substrate.
A DB write and an emitted event cannot disagree after a crash.
One domain event should feed many consumers without the producer knowing each one.
Teams need event IDs, offsets, partitions, dashboards, and a way to answer where delivery stopped.
Network partitions and crashes cannot create a committed entity with no event.
Without a schema contract, the bus just moves breaking changes faster.
The middle can be complex. The producer and consumer APIs cannot be.
The tempting design is write state, then publish an event. It looks simple until the service dies between those two operations.
Local state commits.
A crash here leaves the system globally wrong.
At service boundaries, "the request succeeded" is not enough. The durable business fact and the durable integration fact need the same transaction boundary. Otherwise every downstream incident starts with reconstructing whether the producer lied, the broker lost work, or the consumer failed.
The producer API became boring on purpose: write your business row and an outbox row in one Postgres transaction. A CDC service publishes committed outbox rows to the broker.
Parsed outbox configs and tailed committed rows for multiple tables.
Used context cancellation and grouped workers so shutdown did not duplicate or strand work.
The service could publish through either backend while teams migrated.
Schema validation was the adoption pain point teams understood. The design moved from JSON schema in the middle toward Avro and a schema registry so compatibility became an explicit producer and consumer contract.
Moving validation closer to producers improved correctness, but it required teams to own schemas as first-class artifacts. That was the recurring tradeoff: a stronger platform contract only works if the ergonomic path makes compliance cheaper than bypassing it.
Kafka fit the scaling requirements: fanout, partitions, replay, offsets, and better debugging. But making every product team write Kafka consumers would have moved complexity to the worst place.
Good for specialized teams. Too much offset, retry, and client responsibility as a default.
Familiar, but not the best long-term fit for ordering, replay, and broker-level debugging.
Less flexible than direct Kafka, but far more likely to be operated correctly by normal product teams.
A stalled CDC consumer can hold a replication slot open and grow WAL until the database is at risk. That makes replication lag a product reliability issue, not just an infra metric.
The first idea inserted synthetic outbox heartbeat events and had the dispatcher convert observations into service checks. We dropped it because not all teams used the dispatcher. A monitor that only covers one consumer path creates false confidence.
MSK was not available through localstack, so integration tests used local Kafka containers.
The complex Go CDC service needed direct tests around config parsing and worker behavior.
Validated the generalized queue dispatcher as a reusable product-team primitive.
The strongest lesson from the project: platform systems get adopted when they solve a felt pain and do not ask every team to learn the platform's hardest technology. Kafka partitions and offsets improved debugging. The dispatcher kept that power from becoming every team's daily job.
State and event commit together. CDC takes the complexity after commit.
Events become shared APIs. Schema compatibility is part of the platform.
Most teams consume over HTTP through a dispatcher. The platform owns broker complexity.