Back to portfolio
detail
00 The problem

A monolith split into services.
The communication model didn't.

I was the third engineer at Ethos, and one of the few people left who had written large parts of the monolith. As the company moved toward service ownership, service calls grew in every direction: sync requests, ad hoc queues, one-off retry logic, and unclear ownership after failure.

Before
Distributed monolith
producer knows downstream services
network failure creates partial state
debugging crosses team boundaries
needed one contract
After
Event bus contract
producer owns events
consumers own handlers
platform owns the bus mechanics
Grounding · ownership
I owned the architecture proposal and the platform substrate that made event ownership work.

I wrote the design proposal, compared options, presented to engineering leads, got CTO approval, and built the shared mechanics teams would depend on. The decision criteria were build cost, maintenance cost, technology cost, scale, reversibility, ownership, and whether product teams would actually use it.

architecture
Design proposal
requirements and tradeoffs

Turned prior incidents and operational pain into explicit requirements.

platform
Platform pieces
outbox monitor, modules, dispatcher

Shared mechanics lived in one place, instead of being reimplemented by each service.

operating model
Operating model
domain events and handlers

Defined the ownership split: product teams kept their business logic, while the platform owned the delivery substrate.

detail
01 Requirements

The requirements were really failure modes.

Producer

State and event
commit together

A DB write and an emitted event cannot disagree after a crash.

Broker

Fanout without
producer coupling

One domain event should feed many consumers without the producer knowing each one.

Operations

Replay, order,
and debug

Teams need event IDs, offsets, partitions, dashboards, and a way to answer where delivery stopped.

The product constraint The platform had to scale to more services and more messages, but it also had to be easy enough that teams would adopt it under normal delivery pressure.
Grounding · proposal checklist
Every option got scored against scale, cost, reversibility, and ownership.
must have
strong consistency
local transaction boundary

Network partitions and crashes cannot create a committed entity with no event.

must have
schema validation
contract compatibility

Without a schema contract, the bus just moves breaking changes faster.

must have
developer productivity
simple edge interfaces

The middle can be complex. The producer and consumer APIs cannot be.

detail
02 Producer failure

The obvious publish path creates split brain.

The tempting design is write state, then publish an event. It looks simple until the service dies between those two operations.

  • DB commit succeeds. The service's local state is now correct.
  • Publish fails or never runs. Every downstream system is stale.
  • Retry is ambiguous. You do not know whether the event was never sent, sent once, or sent and lost downstream.
bad path · two independent writes

1 · write database

Service ADB

Local state commits.

2 · publish event

Service ABroker

A crash here leaves the system globally wrong.

Grounding · why this mattered
The reliability target was global state, not message delivery in isolation.

At service boundaries, "the request succeeded" is not enough. The durable business fact and the durable integration fact need the same transaction boundary. Otherwise every downstream incident starts with reconstructing whether the producer lied, the broker lost work, or the consumer failed.

Rejected A producer library that performs DB write plus broker publish still has the same crash window unless it moves the event into the database transaction.
detail
03 Producer design

Put the event inside the same transaction.

The producer API became boring on purpose: write your business row and an outbox row in one Postgres transaction. A CDC service publishes committed outbox rows to the broker.

  • If the transaction commits: state and event both exist.
  • If the transaction rolls back: neither state nor event exists.
  • If publishing stalls: the outbox remains durable until the monitor catches up.
outbox pattern · consistency boundary
Service A
business code
one db transaction
domain row
outbox row
CDC publisher
event broker
Why Postgres Every service already used it. Every engineer understood transactions. The platform got consistency without forcing producers onto a new write path.
Grounding · what the platform inherited
The outbox made producers simple by moving complexity into CDC.
go service
Outbox Monitor
logical replication consumer

Parsed outbox configs and tailed committed rows for multiple tables.

shutdown
signal-aware workers
Kubernetes termination

Used context cancellation and grouped workers so shutdown did not duplicate or strand work.

compat
shared publisher interface
SNS/SQS and Kafka

The service could publish through either backend while teams migrated.

detail
04 Schema contract

An event bus without schemas is an outage amplifier.

Schema validation was the adoption pain point teams understood. The design moved from JSON schema in the middle toward Avro and a schema registry so compatibility became an explicit producer and consumer contract.

  • v1: JSON schema validation in the bus layer. Safer than nothing, but too centralized.
  • v2: Avro schemas in a registry. Better contracts, harder adoption.
  • Lesson: the right contract still needs examples, defaults, and docs.
contract path
ProducerAvro schemaSchema Registry
EventCompatible consumerHandler runs
Breaking eventRejected before damage
Grounding · adoption failure mode
The technically cleaner schema model was also harder to get used.

Moving validation closer to producers improved correctness, but it required teams to own schemas as first-class artifacts. That was the recurring tradeoff: a stronger platform contract only works if the ergonomic path makes compliance cheaper than bypassing it.

What failed Central validation was easier for teams, but left too much responsibility in the platform. Producer-owned schemas were better architecture, but needed better tooling and examples to get adoption.
detail
05 Consumer design

Use Kafka in the middle.
Expose HTTP at the edge.

Kafka fit the scaling requirements: fanout, partitions, replay, offsets, and better debugging. But making every product team write Kafka consumers would have moved complexity to the worst place.

  • v1: SNS/SQS. Familiar, but weaker for replay and stream debugging.
  • v2: Kafka/MSK. Better broker semantics, harder client model.
  • Interface fix: a generalized dispatcher pulls from Kafka and POSTs to the service.
sidecar dispatcher
Kafka
topic
consume
Kubernetes Pod
Dispatcher sidecar
consume, retry, POST
App container
normal HTTP handler
POST
Service handler
/events
The interface bet Teams could consume events by implementing an HTTP endpoint. Kafka remained the broker, not the default application API.
Grounding · options considered
The shared dispatcher optimized for adoption, not maximum control.
option
Direct Kafka consumer
maximum control

Good for specialized teams. Too much offset, retry, and client responsibility as a default.

option
SNS/SQS only
simple start

Familiar, but not the best long-term fit for ordering, replay, and broker-level debugging.

chosen default
Dispatcher sidecar
Kafka power, HTTP surface

Less flexible than direct Kafka, but far more likely to be operated correctly by normal product teams.

detail
06 Reliability

The scary part was not Kafka.
It was WAL growth.

A stalled CDC consumer can hold a replication slot open and grow WAL until the database is at risk. That makes replication lag a product reliability issue, not just an infra metric.

  • Monitor the CDC service. Alert the platform owner.
  • Monitor WAL growth. Alert before database health is at risk.
  • Monitor dispatchers by team. The handler owner owns its consumer path.
replication lag monitor
Grounding · what changed after testing
One heartbeat idea was dropped because it covered the wrong surface.

The first idea inserted synthetic outbox heartbeat events and had the dispatcher convert observations into service checks. We dropped it because not all teams used the dispatcher. A monitor that only covers one consumer path creates false confidence.

local dev
Confluent Docker images
Kafka without local cloud emulation

MSK was not available through localstack, so integration tests used local Kafka containers.

cdc
Unit and integration tests
outbox monitor

The complex Go CDC service needed direct tests around config parsing and worker behavior.

dispatcher
Image-level integration test
consume then POST

Validated the generalized queue dispatcher as a reusable product-team primitive.

detail
07 Topology

Flexible enough for direct Kafka.
Simple enough for HTTP consumers.

overall system shape
Service A Service B Service C DB A + outbox DB B + outbox Outbox Monitor
Topic A
Topic B
Topic C
Dispatcher B
POST /events
Dispatcher A
POST /events
Service C direct
Grounding · result and learning
The adoption win was hiding the hard part without hiding the operating model.

The strongest lesson from the project: platform systems get adopted when they solve a felt pain and do not ask every team to learn the platform's hardest technology. Kafka partitions and offsets improved debugging. The dispatcher kept that power from becoming every team's daily job.

1
producer pattern: transactional outbox through Postgres.
2
consumer paths: direct Kafka for power users, dispatcher for most teams.
3
owner layers: platform monitors, team dispatchers, team handlers.
N
services could join without custom service-to-service contracts.
detail
Recap consistency · contract · adoption

Outbox for consistency.
Kafka for scale.
HTTP for adoption.

Decision 1

Produce from
Postgres

State and event commit together. CDC takes the complexity after commit.

Decision 2

Validate the
contract

Events become shared APIs. Schema compatibility is part of the platform.

Decision 3

Keep Kafka
off the edge

Most teams consume over HTTP through a dispatcher. The platform owns broker complexity.

The through-line The event bus was not about adding a broker. It was about changing service ownership: producers publish durable facts, consumers own their reactions, and the platform owns the delivery substrate.
Sources
Primary references used for this public-safe deck.
deck
Event Bus Architecture
Google Slides

Original presentation source.

pattern
Transactional Outbox
microservices.io

Pattern reference for the consistency mechanism.

docs
Postgres, Kafka, Avro
primary docs

Logical replication, Kafka, and Avro.

← → advance · detail toggle / space flips detail
1/9