Event-driven architecture (EDA) is seductive.
It promises:
- Loose coupling
- Independent scaling
- Reactive workflows
- Clean separation of concerns
But the production reality is harsher:
Most event-driven systems fail not because of throughput, but because of operational chaos.
- Duplicate deliveries.
- Out-of-order events.
- Poison messages.
- Replays that double-charge customers.
- “Invisible” side effects.
- Debugging that feels like ghost hunting.
This article is about building event-driven systems that survive production: retries, rebalances, network partitions, deployments, and humans.
The core misconception
Many teams think event-driven means:
“Publish events, subscribe to them, done.”
No.
Event-driven means:
“You are building a distributed state machine over unreliable transport.”
If you don’t design for that explicitly, your system will eventually eat itself.
1) The failure modes you must assume will happen
Duplicate delivery (at-least-once is normal)
Kafka, SQS, Pub/Sub, RabbitMQ most real-world setups are at-least-once in practice.
You will see duplicates.
Out-of-order delivery
Retries + parallelism + partitions + consumer group rebalances = reordering.
You cannot assume sequence unless you enforce it.
Partial side effects
A consumer updates the DB, then crashes before committing the offset. Or it calls a payment API, then times out, then retries.
This is where chaos begins.
Replays
Backfills. Topic compaction changes. Bug fixes. New consumers. Disaster recovery.
If replaying events can break your system, your system is fragile.
2) The naïve architecture that breaks
Each consumer does:
- Read event
- Do side effect
- Commit offset
Looks fine until:
- Payment runs twice → double charge
- Inventory decrements twice → negative stock
- Email sends twice → angry customers
- “Fix” involves manual DB edits → worse
Chaos.
3) The stability rules
Rule 1: Every event must be uniquely identifiable and deduplicable
Every event should carry:
- eventId (unique per emission)
- aggregateId (business entity key)
- version (monotonic per aggregate) or a strict ordering key
- occurredAt
Example payload:
{
"eventId": "evt_92af3b2a",
"type": "OrderPaid",
"aggregateId": "order_123",
"version": 3,
"occurredAt": "2026-02-28T12:00:00Z",
"data": {
"paymentId": "pay_987",
"amount": 4999,
"currency": "EUR"
}
}
A consumer must be able to answer: “Have I already processed this exact event?”
Rule 2: State changes and "I processed this event" must be atomic
If you update your DB and later mark the offset (or eventId) as processed, you risk double-processing.
You need a single atomic unit: apply + record.
Rule 3: External side effects must be guarded
Anything outside your DB (emails, payments, SMS, webhooks, third-party calls) must be protected against retries.
Rule 4: Events are immutable facts
Never edit old events. Emit new ones.
Rule 5: Replay safety is the real test
Ask:
If I replay the last 30 days of events, does the system converge to the same state?
If not, you don’t have a reliable system.
4) Publishing without lies: the Transactional Outbox
The classic failure:
- You write to DB
- You publish to Kafka
- Publish fails
- Now DB says it happened, stream doesn’t
Or vice versa.
The correct pattern
Outbox table (conceptually):
- id (eventId)
- aggregateId
- type
- payload
- status (NEW/SENT)
- created_at
- sent_at
- retry_count
This makes your database and your event stream consistent by design.
5) Consuming without chaos: the production-grade consumer loop
The “safe consumer” algorithm
Key points:
- Dedup check must be fast (often a table keyed by eventId)
- State transition must be deterministic
- “Processed eventId” must be committed in the same transaction as the state change
6) Exactly-once: what you can and can’t claim
“Exactly-once semantics” is overloaded marketing.
In practice:
- Kafka EOS can give you exactly-once between Kafka and Kafka in specific setups
- Your business still isn’t exactly-once unless your side effects are idempotent
The pragmatic, production truth:
Build for at-least-once delivery + idempotent processing + deterministic state.
That is “exactly-once outcomes”.
7) Handling ordering: versioning beats timestamps
Out-of-order is unavoidable. Your strategy depends on what matters.
Option A: Per-aggregate version
Best when events are state transitions.
Consumer rule:
- Accept only
version == current_version + 1 - If higher version arrives early → buffer or park
- If lower version arrives late → ignore (already applied)
Option B: Partition by aggregateId
You can ensure ordering within a partition if you partition by aggregateId.
This helps, but rebalances and retries still mean: duplicates remain.
8) The “poison message” problem: DLQ is not a trash can
Most teams treat DLQ as a graveyard. That’s operational malpractice.
DLQ must be:
- Alerting (page someone when it grows)
- Inspectable (payload, headers, failure cause)
- Replayable (after fix)
- Rate-limited (avoid floods)
- Auditable (especially in regulated domains)
A sane message lifecycle
Rule of thumb:
- Transient failures → retry with backoff
- Permanent failures (schema mismatch, invariant violation) → DLQ fast
9) Side effects: don’t do them inline unless you can afford pain
External calls are the #1 chaos amplifier.
If you do: consume → call provider → then commit
You’ll eventually cause:
- double charges
- repeated emails
- inconsistent third-party state
The stable pattern: “persist intent, then execute”
This makes side effects:
- retryable
- observable
- independently scalable
- safe on replays
10) Replay safety: design for time travel
Replays will happen. Plan for them.
Your replay contract
A replay should:
- not re-send emails unless explicitly configured
- not re-charge cards
- not re-decrement inventory
- converge to the same final state
Two replay modes (be explicit)
- State rebuild: recompute projections from event stream (safe default)
- Action replay: re-trigger external effects (dangerous, requires strict idempotency + explicit operator intent)
If you don’t separate these modes, you’ll replay into disaster.
11) Schema evolution: how teams silently break production
You will change event payloads.
If you do it casually, consumers will crash, DLQ will fill, and you’ll be in incident mode.
Compatibility rules
- Add fields freely (consumers ignore what they don’t know)
- Never remove or rename fields without versioning
- Treat required → optional carefully
- Use a schema registry where possible
Versioning strategies
Either:
- consumers handle multiple versions, or
- a translation layer normalizes events
12) Observability: the layer that prevents panic
Event-driven systems become operational chaos when you can’t see them.
You must track:
- consumer lag per group
- processing time percentiles
- retry counts
- DLQ rate and age
- dedup hits (duplicate rate is a signal)
- outbox backlog (publisher health)
The basic “where is it stuck?” map
If you cannot answer “where is it stuck?” in under 60 seconds, you will suffer during incidents.
13) The stable reference architecture
Here’s the architecture that scales without chaos:
Properties:
- Outbox on the producer
- Idempotent consumer with eventId dedup
- Deterministic transitions with ordering checks
- Side effects executed via persistent tasks
- DLQ is monitored, replayable, and disciplined
- Replay safety is a first-class design goal
14) When not to use event-driven
Don’t adopt EDA because it "feels modern".
Avoid event-driven if:
- You need strict synchronous consistency
- You have 1–2 services and no scaling pain
- You can’t commit to observability + on-call readiness
- Your team is new to distributed failure modes
Event-driven doesn’t reduce work.
It moves work from product code to operations and correctness.
The mental model to keep you sane
Treat your EDA system as:
A replicated asynchronous state machine with unreliable delivery and infinite retries.
If you design for:
- duplicates
- reordering
- retries
- replays
- schema evolution
- observable operations
you get the benefits of EDA without the operational nightmare. If you assume "it won’t happen": It will.