Back to portfolio
detail
00 The problem

Event Bus Platform

I was the third engineer at Ethos, so I had seen the system grow from a monolith into team-owned microservices as the engineering org scaled past roughly 50 people. But service-to-service communication was still ad hoc: direct calls, custom payloads, and team-specific retry logic made the system feel less like independent services and more like a distributed monolith.

Why it hurt
Distributed monolith
payload contract drift
partial failure created inconsistent state
custom retries were unsafe
new consumers required producer changes
operational upgrades were risky
→
needed one contract
After
Event bus contract
producer owns events
consumers own handlers
platform owns the bus mechanics
Grounding · ownership
I led the architecture and built the foundation every team adopted.

I wrote the design proposal, compared options, presented to engineering leads, got CTO approval, and built the shared mechanics teams would depend on. Two services had already caused recurring incidents when producer and consumer payloads drifted. The decision criteria were build cost, maintenance cost, scale, reversibility, ownership, and whether product teams would actually use the result.

architecture
Design proposal
requirements and tradeoffs

Turned prior incidents and operational pain into explicit requirements.

platform
Platform pieces
outbox monitor, Terraform registry, dispatcher

Shared mechanics lived in one place, instead of being reimplemented by each service.

operating model
Operating model
domain events and handlers

Defined the ownership split: product teams kept their business logic, while the platform owned the delivery substrate.

detail
Grounding · failure modes

We designed around the failure modes we had already hit.

Producer

State and event
commit together

A DB write and an emitted event cannot disagree after a crash.

Broker

Fanout without
producer coupling

One domain event should feed many consumers without the producer knowing each one.

Operations

Replay, order,
and debug

Teams need event IDs, offsets, partitions, dashboards, and a way to answer where delivery stopped.

The adoption constraint Beyond the failure modes, we emphasized developer experience so teams could adopt the platform under normal delivery pressure.
01 Requirements
I defined the requirements in collaboration with engineering team leads.

Every option got scored against scale, cost, reversibility, and ownership.

must have
strong consistency
local transaction boundary

Network partitions and crashes cannot create a committed entity with no event.

must have
schema validation
contract compatibility

Without a schema contract, the bus just moves breaking changes faster.

must have
developer productivity
simple edge interfaces

The middle can be complex. The producer and consumer APIs cannot be.

detail
02 Producer interface

The producer API had to be easy to adopt.

We needed event production to be easy to adopt, high-throughput, ordered where needed, and strongly consistent. Since every service already used Postgres, and every engineer understood database transactions, we decided services would produce events through Postgres.

Consistency

Commit state and
event together

Postgres gave producers a transaction boundary they already understood.

Throughput

Publish after
commit

Product code records the event. CDC handles broker publishing asynchronously.

Ordering

Group related
events

Ordering keys let the platform preserve sequence where business workflows needed it.

Why Postgres Every service already used it. Every engineer knew how to use it. The platform got consistency without forcing producers onto a new broker-specific write path.
Grounding · rejected producer path
A two-step write could leave the system inconsistent.

The naive approach was: write the business row, then publish an event. If the service crashed between those steps, the database was correct locally, but downstream services never learned that anything changed.

  • DB commit succeeds. The service's local state is correct.
  • Publish fails or never runs. The event may never exist.
  • Downstream systems stay stale. Consumers have no durable signal to react to.
  • Retry cannot prove what happened. The producer cannot distinguish never sent from sent and failed downstream.
Naive producer writes a customer record to the database and separately sends a customer created event to an event stream.
Rejected A producer library that performs DB write plus broker publish still has the same crash window unless it moves the event into the database transaction.
detail
03 Producer design

The outbox pattern made the event part of the transaction.

The fix was to write the business row and the outbox row in the same Postgres transaction. If the transaction committed, both existed. If it rolled back, neither existed. A CDC monitor could then publish durable outbox rows to the event stream.

  • One transaction writes business state and event record.
  • Outbox row survives crashes and deploys.
  • CDC monitor publishes after commit.
  • delivered_at tracks publisher progress, not exactly-once delivery.
  • partition_key preserves order for related events.
Outbox pattern writes customer and outbox rows in one database transaction, then forwards outbox events to the event stream.
Grounding · what the platform inherited
The normal contract was at-least-once delivery, so handlers had to be idempotent.

The producer-facing API created a normal Postgres row. The platform-owned monitor tailed committed rows, validated payloads, published them, and recorded progress. A crash after publish but before completion bookkeeping could produce a duplicate. Product teams therefore built handlers that could safely process the same event again.

CREATE TABLE outbox_events (
  id UUID NOT NULL DEFAULT gen_random_uuid() PRIMARY KEY,
  name text NOT NULL,
  created_at timestamp with time zone DEFAULT CURRENT_TIMESTAMP NOT NULL,
  partition_key text,
  payload jsonb NOT NULL,
  source text NOT NULL,
  failure_strategy text CHECK (failure_strategy IN ('retry', 'drop', 'dlq')),
  delivered_at timestamp with time zone
);
detail
04 Schema contract

Outbox made events reliable. Schemas made them safe to consume.

We needed producers and consumers to agree on event shape, versioning, and compatibility before messages reached production. Each producer owned the schema for the events it emitted.

  • Avro schemas defined the payload contract.
  • Kafka Schema Registry stored versions and compatibility rules.
  • Producer validation caught breaking changes before publish.
  • Consumer compatibility let teams evolve independently.
contract path
Producer→Avro schema→Schema Registry
Event→Compatible consumer→Handler runs
Breaking event→
DLQinfinite retrydrop
Grounding · adoption failure mode
Schemas moved the contract out of tribal knowledge.

Before schemas, the contract lived in code, docs, Slack threads, and assumptions between teams. Two services repeatedly caused incidents when their payload expectations drifted. With Avro and Schema Registry, the event shape became a versioned artifact. Producers owned the schema, consumers could validate compatibility, and the recurring drift failure dropped sharply.

Tradeoff Producer-owned schemas were better architecture, but only worked if examples, defaults, and tooling made the correct path easy.
detail
05 Consumer design

Use Kafka in the middle.
Expose HTTP at the edge.

Kafka fit the scaling requirements: fanout, partitions, replay, offsets, and better debugging. But making every product team write Kafka consumers would have moved complexity to the worst place.

  • v1: SNS/SQS. Familiar, but weaker for replay and stream debugging.
  • v2: Kafka/MSK. Better broker semantics, harder client model.
  • Interface fix: a generalized dispatcher pulls from Kafka and POSTs to the service.
sidecar dispatcher
Kafka
topic
consume
Kubernetes Pod
Dispatcher sidecar
consume, retry, POST
App container
normal HTTP handler
POST
Service handler
/events
The interface bet Teams could consume directly from Kafka if they needed the control, but most chose the dispatcher sidecar: implement an HTTP endpoint and let the platform own the broker mechanics.
Grounding · options considered
The shared dispatcher optimized for adoption, not maximum control.
option
SNS/SQS only
simple start

Familiar, but not the best long-term fit for ordering, replay, and broker-level debugging.

chosen default
Dispatcher sidecar
Kafka power, HTTP surface

Most teams chose this path: less flexible than direct Kafka, but easier to operate correctly.

option
Direct Kafka consumer
maximum control

Good for specialized teams. Too much offset, retry, and client responsibility as a default.

dispatcher sidecar config
input:
  kafka:
    addresses: [ ... ]
    topics: [ policy_events ]
    consumer_group: policy_service

pipeline:
  processors:
    - for_each:
      - mapping: |
          root.event_id = this.id
          root.name = this.name
          root.payload = this.payload

output:
  http_client:
    url: http://localhost:8080/events
    verb: POST
detail
06 Developer enablement

Terraform module for self-serve topics, with assistance.

The platform needed more than a broker and a dispatcher. Teams also needed a clear path for creating topics and making the few choices that really mattered.

  • Terraform registry gave teams a standard way to create topics.
  • Partition docs explained how to choose partition counts, ordering keys, and throughput assumptions.
  • Review support let teams sanity-check topic design before production traffic depended on it.
topic creation path
Product team→Terraform module→Kafka topic
Partition guide→Ordering key→Review
Paved path Topic creation was self-service, but not guesswork. The defaults, docs, and review loop kept Kafka decisions from becoming scattered team folklore.
Grounding · production evolution
I built the first publisher against the WAL, then moved the platform toward Debezium.

The first version used a custom WAL monitor because I did not yet know about Debezium. It proved the outbox design, but it also made replication behavior platform code we had to own. Moving toward Debezium kept the producer and consumer contracts intact while replacing custom CDC machinery with a purpose-built tool.

first version
Custom WAL monitor
prove the contract

I built the initial replication and publishing path around committed outbox rows.

operating cost
Custom CDC ownership
replication edge cases

The platform team inherited WAL behavior, replication health, and recovery logic.

evolution
Move toward Debezium
purpose-built CDC

The product-facing outbox API stayed stable while the publishing implementation matured.

detail
07 Before eventing

Before: The producer owned the workflow.

Creating a policy meant the producer service owned the full orchestration: call downstream services, wait for responses, and save local state. Every new consumer or dependency made that producer more coupled.

Service A
Create policy
Service B
Generate quote
Service C
Start underwriting
Database
Save policy
HTTP request
response: quote details
HTTP request
response: underwriting status
Grounding · ownership shift
Eventing moved orchestration out of the producer.

Before the bus, Service A had to know which services cared about a policy change, how to call them, and what to do when one dependency failed. The event bus changed that ownership model: Service A emitted a durable business fact, and each consumer owned its reaction.

producer before
Downstream orchestration
calls, retries, responses

Owns downstream calls, retries, and response contracts.

producer after
Durable business fact
state change and event

Owns the state change and emitted event.

consumer after
Independent reaction
handler and failure policy

Owns its handler, failure policy, and operational behavior.

detail
08 After eventing

After: Flexible enough for direct Kafka.
Simple enough for HTTP consumers.

create policy event flow
Policy
Create policy Quote
other events Underwriting
direct publish Policy DB
+ outbox Quote DB
+ outbox
Policy
Events
Quote
Events
Underwriting
Events
Quote handler
POST /events
Underwriting handler
POST /events
Analytics
direct Kafka
Grounding · result and learning
The platform reached multiple teams without making Kafka every team's daily job.

Approximately 5–10 services adopted the platform across roughly 10–20 topics. Kafka partitions and offsets improved replay and debugging. The dispatcher let most teams keep a normal HTTP handler while specialized consumers retained direct Kafka access.

5–10
services adopted the shared event path.
10–20
topics formed the approximate production footprint.
2
consumer paths: dispatcher or direct Kafka.
1
producer contract: transactional outbox through Postgres.
detail
09 Operating reliability

CDC was the toughest reliability problem.

A stalled CDC consumer can hold a replication slot open and grow WAL until the database is at risk. That makes replication lag a product reliability issue, not just an infra metric.

  • Monitor the CDC service. Alert the platform owner.
  • Monitor WAL growth. Alert before database health is at risk.
  • Monitor dispatchers by team. The handler owner owns its consumer path.
replication lag monitor
Grounding · delivery contract
The platform owned delivery. Teams owned idempotency.

The normal delivery contract was at least once. Product teams had to make handlers safe for duplicates and select a failure policy for each path. Infinite retry preserved partition order by blocking later events. Other consumers could explicitly drop a failed event or move it to a DLQ.

platform-owned
CDC pipeline health
replication and outbox lag

Replication slot health, WAL growth, and publisher lag belonged to the platform.

team-owned
Idempotent handlers
duplicate-safe effects

A repeated event could not create an incorrect second business operation.

configured per path
Failure strategy
retry, DLQ, or drop

Teams chose whether a poison event blocked its partition, moved aside, or was discarded.

10 Retrospective

2021 → 2026: What might be different?

Developer experience should now be designed more for agents than humans. In 2021, I built a system that any product engineer could use safely without becoming an infrastructure expert. In 2026, I would give an agent the context and guardrails to help a strong engineer own more of the system.

CDC

Keep the custom CDC outbox monitor instead of migrating to Debezium

I migrated mainly to reduce bus-factor risk: the high-throughput WAL processing and Kafka queueing code was complex and deeply technical. My Go implementation also outperformed Debezium in our workload. Today, an agent could quickly explain it to a strong engineer, making the custom system maintainable without me available for questions.

Consumption

Let teams own their Kafka consumers

In 2021, asking the average product engineer to learn Kafka well enough to build a safe consumer was too large a lift, so Benthos delivered events to HTTP endpoints. In 2026, I would keep that sidecar as the default but use a skill to teach MSK IAM auth and scaffold a direct consumer when a team wants more control.

Provisioning

Replace the onboarding meeting with an agent

In 2021, engineers read my Notion guide, then worked with me to choose partitions and other topic requirements; I applied the Terraform module. In 2026, a Slack agent should run that interview, open the Terraform PR, send it to the engineer, and merge after approval. I only join for edits or deletes.

Incident response

Event bus incident agent

With access to Datadog and the codebase, an agent knows which monitors and code paths map to each part of the bus. It can correlate WAL growth, CDC and consumer lag, DLQs, and recent deploys—then guide the owning engineer to the likely failure.

Schema evolution

Schema change agent

When a schema changes, an agent can find every repository and service that produces or consumes it, check compatibility, propose the coordinated code changes, and sequence the rollout.

detail
Recap consistency · contract · adoption

Outbox for consistency.
Kafka for scale.
HTTP for adoption.

Decision 1

Produce from
Postgres

State and event commit together. CDC takes the complexity after commit.

Decision 2

Keep Kafka
off the edge

Most teams consume over HTTP through a dispatcher. The platform owns broker complexity.

Decision 3

Validate the
contract

Events become shared APIs. Schema compatibility is part of the platform.

The through-line The event bus was not about adding a broker. It was about changing service ownership: producers publish durable facts, consumers own their reactions, and the platform owns the delivery substrate.
Sources
Primary references used for this public-safe deck.
pattern
Transactional Outbox
microservices.io

Pattern reference for the consistency mechanism.

docs
Postgres, Kafka, Avro
primary docs

Logical replication, Kafka, and Avro.

← → advance · detail toggle / space flips detail
1/9