← All blogs

FullStack Engineering CourseLesson 9 of 20 22 min

Event-Driven Design: Messaging, Outbox, Sagas and CQRS

Why event-driven

When service A calls service B over HTTP and waits for the answer, A depends on B being up and fast at that moment. If B is down, A fails too. Messaging removes that dependency: A puts a message on a broker (a service that stores and delivers messages, such as Azure Service Bus), carries on, and B processes the message when it can. If B is down for an hour, the messages wait, and B catches up when it returns.

There are three kinds of message, and naming them correctly makes a design clearer:

KindMeaningNamed asReceivers
CommandA request for one specific service to do somethingAn instruction: CheckEligibility, GenerateInvoiceExactly one handler
EventA fact: something has already happenedPast tense: ClaimSubmitted, PaymentReceivedZero, one or many subscribers; the sender does not know them
QueryA request for dataGetClaimStatusOne handler, and the sender waits for the answer (usually HTTP, not messaging)

Events give the loosest coupling. The Claims service publishes ClaimSubmitted and does not know or care that Eligibility, Analytics and Notifications all react to it. A new consumer can be added without changing the publisher.

The price is complexity. Data is updated a little later in other services (eventual consistency), messages can arrive twice or out of order, and a request is harder to trace across services. The rest of this lesson is about handling those costs.

Choosing a broker

Azure Service BusAzure Event Hubs / KafkaAzure Event Grid
ModelQueues and topics with subscriptionsA partitioned, append-only log (a stream)Push delivery of event notifications
Best forBusiness commands and events that must each be handled reliablyHigh-volume streams: telemetry, logs, clickstreams, change feedsReacting to events, especially from Azure resources (a blob was uploaded)
Message is removedWhen a consumer completes itNever by consumers; kept for a retention period, and each consumer tracks its own positionAfter delivery to each subscriber
OrderingPer session (a group of related messages)Per partitionNot guaranteed
Built-in featuresDead-letter queue, sessions, duplicate detection, scheduled messages, transactionsReplay from any point, many independent consumer groups, very high throughputFiltering, retries, dead-lettering to storage

A simple rule: use Service Bus for business messages where every message matters, Event Hubs (or Kafka) for streams of data you analyse or replay, and Event Grid to react to things happening in Azure. Event Hubs also accepts Kafka clients, so Kafka-based code can run on it without a Kafka cluster.

Delivery guarantees

  • At most once: a message is delivered once or not at all. Messages can be lost. Rarely acceptable for business data.
  • At least once: a message is delivered one or more times. Nothing is lost, but duplicates happen. This is what Service Bus, Event Hubs and Kafka give you in practice.
  • Exactly once: each message has its effect exactly once. Brokers cannot guarantee this end to end on their own. You get it by combining at-least-once delivery with consumers that ignore duplicates (idempotent consumers, below).

So the working assumption is: every message may arrive more than once, and design every consumer for that.

Azure Service Bus in practice

A topic delivers a copy of each message to every subscription; a queue delivers each message to one consumer. Service Bus uses peek-lock: a consumer receives a message, and the message is locked (hidden from other consumers) while it is processed. Then the consumer does one of three things:

  • Complete: processing succeeded; the message is removed.
  • Abandon (or let the lock expire): processing failed; the message becomes available again and is retried.
  • Dead-letter: the message can never succeed (for example, it is malformed); it is moved to the dead-letter queue (DLQ), a side queue where failed messages wait for someone to inspect them.

If a message keeps failing, Service Bus moves it to the dead-letter queue automatically after MaxDeliveryCount attempts (10 by default). Such a message is called a poison message: one that fails every time and would otherwise block or slow the queue for ever.

C#
// Publishing an event. MessageId enables duplicate detection; SessionId keeps one claim's events in order.
await using var client = new ServiceBusClient("claims-bus.servicebus.windows.net", new DefaultAzureCredential());
ServiceBusSender sender = client.CreateSender("claim-events");

var message = new ServiceBusMessage(BinaryData.FromObjectAsJson(evt))
{
    MessageId = evt.EventId.ToString(),
    SessionId = evt.ClaimId.ToString(),
    Subject = nameof(ClaimSubmitted),
    ContentType = "application/json"
};
await sender.SendMessageAsync(message, ct);

// Consuming from the "eligibility" subscription, with sessions for per-claim ordering.
ServiceBusSessionProcessor processor = client.CreateSessionProcessor("claim-events", "eligibility",
    new ServiceBusSessionProcessorOptions { MaxConcurrentSessions = 8, AutoCompleteMessages = false });

processor.ProcessMessageAsync += async args =>
{
    try
    {
        var submitted = args.Message.Body.ToObjectFromJson<ClaimSubmitted>()!;
        await handler.HandleAsync(submitted, args.Message.MessageId, args.CancellationToken);
        await args.CompleteMessageAsync(args.Message);                       // success: remove it
    }
    catch (JsonException e)
    {
        await args.DeadLetterMessageAsync(args.Message, "InvalidPayload", e.Message);  // will never succeed
    }
    // Any other exception: not completed, so the lock expires and the message is retried,
    // and after MaxDeliveryCount attempts Service Bus dead-letters it.
};
processor.ProcessErrorAsync += args =>
{
    logger.LogError(args.Exception, "Service Bus error in {Entity}", args.EntityPath);
    return Task.CompletedTask;
};
await processor.StartProcessingAsync(ct);

Dead-letter queues need an owner. Monitor their length, alert when messages arrive, and have a tool or runbook to inspect, fix and resubmit them. A dead-letter queue that nobody watches is just a place where data is quietly lost.

Ordering costs throughput. Sessions guarantee order within one session (one claim) by giving each session to one consumer at a time, while different sessions run in parallel. Only ask for ordering where it matters, and use a key that spreads the load, such as the claim ID rather than the payer ID.

The Outbox pattern

A service often needs to save a change and publish an event about it. These are two separate systems, so doing both is a dual write, and it can fail half-way:

  • Save to the database, then publish: if the process crashes in between, the change is saved but the event is lost, and other services never hear about it.
  • Publish, then save: if the save fails, other services react to an event for a change that never happened.

There is no distributed transaction across the database and the broker that you should rely on. The Outbox pattern solves this: write the event into an outbox table in the same database transaction as the business change. Either both are saved or neither is. A separate background process (the relay) then reads unpublished rows from the outbox, publishes them, and marks them as sent:

C#
public sealed class OutboxMessage
{
    public Guid Id { get; init; }
    public string Type { get; init; } = "";
    public string Payload { get; init; } = "";
    public DateTimeOffset OccurredAt { get; init; }
    public DateTimeOffset? PublishedAt { get; set; }
}

// In the use case: the claim change and the event are saved in ONE transaction.
claim.Submit();
db.Outbox.Add(new OutboxMessage
{
    Id = Guid.NewGuid(),
    Type = nameof(ClaimSubmitted),
    Payload = JsonSerializer.Serialize(new ClaimSubmitted(claim.Id, claim.Total)),
    OccurredAt = clock.GetUtcNow()
});
await db.SaveChangesAsync(ct);

// The relay: a BackgroundService that publishes what is in the outbox.
public sealed class OutboxRelay(IServiceScopeFactory scopes, ServiceBusSender sender) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        using var timer = new PeriodicTimer(TimeSpan.FromSeconds(2));
        while (await timer.WaitForNextTickAsync(stoppingToken))
        {
            await using var scope = scopes.CreateAsyncScope();
            var db = scope.ServiceProvider.GetRequiredService<ClaimsDbContext>();

            var batch = await db.Outbox.Where(m => m.PublishedAt == null)
                                       .OrderBy(m => m.OccurredAt).Take(100)
                                       .ToListAsync(stoppingToken);
            foreach (var m in batch)
            {
                await sender.SendMessageAsync(new ServiceBusMessage(m.Payload)
                {
                    MessageId = m.Id.ToString(),   // if we crash after sending but before marking, the resend is a duplicate
                    Subject = m.Type               // that duplicate detection and idempotent consumers absorb
                }, stoppingToken);
                m.PublishedAt = DateTimeOffset.UtcNow;
            }
            await db.SaveChangesAsync(stoppingToken);
        }
    }
}

The relay can still publish a message twice (it sends, then crashes before marking it as published), so the Outbox gives at-least-once publishing. That is fine, because consumers are idempotent. In production, run the relay as a single instance or claim rows with a conditional update (lesson 5), and clean up old published rows. Libraries such as MassTransit, NServiceBus and Wolverine include a ready-made outbox, and change data capture (lesson 13) is another way to turn database changes into events.

The Inbox pattern: idempotent consumers

Because messages can arrive more than once, each consumer must make sure that handling a message twice has the same effect as handling it once. The Inbox pattern records every processed message ID in the consumer's database, in the same transaction as the work it did. A unique key on the message ID makes the database reject a second attempt:

C#
public async Task HandleAsync(ClaimSubmitted evt, string messageId, CancellationToken ct)
{
    db.ProcessedMessages.Add(new ProcessedMessage(messageId, Consumer: "eligibility"));   // unique (MessageId, Consumer)
    db.EligibilityChecks.Add(EligibilityCheck.Start(evt.ClaimId));

    try
    {
        await db.SaveChangesAsync(ct);       // the record and the work are saved together, or not at all
    }
    catch (DbUpdateException e) when (e.InnerException is SqlException { Number: 2601 or 2627 })
    {
        // Already processed: do nothing, so the caller completes the message.
    }
}

Some handlers are naturally idempotent, such as "set the claim status to Eligible". For those, the inbox table is not needed. Handlers that add, increase or send something always need protection.

Sagas: transactions across services

A business process that spans several services cannot use one database transaction. For example, "process a claim" might check eligibility in one service, price the claim in another, and schedule payment in a third. A saga splits the process into local transactions, one per service. If a later step fails, the saga runs compensating actions (steps that undo the effect of earlier steps, such as "release the reserved payment"), because the earlier transactions are already committed and cannot be rolled back.

There are two ways to coordinate a saga:

ChoreographyOrchestration
HowEach service listens for events and publishes its own; no central coordinatorAn orchestrator (a state machine) sends commands to each service and decides the next step from their replies
ExampleClaimSubmitted → Eligibility publishes EligibilityConfirmed → Pricing publishes ClaimPriced → Payments schedulesClaimProcessingSaga: send CheckEligibility, wait; send PriceClaim, wait; on failure, send compensations
StrengthsLoose coupling, no single point of controlThe whole flow is in one place: easy to read, monitor and change
WeaknessesThe flow is spread across services and hard to follow; cycles creep inThe orchestrator is extra code to build and run, and must not take over business logic
FitsShort flows with two or three stepsLonger flows, flows with many failure paths, or flows that need clear status

Design compensations together with the forward steps. Some actions cannot be undone (an email already sent), so put them at the end of the saga, after the steps that can still fail. Tools that help include MassTransit state machines, NServiceBus sagas and Azure Durable Functions orchestrations, which save the saga's state for you.

CQRS

CQRS (Command Query Responsibility Segregation) separates the model used for changing data (commands) from the model used for reading it (queries). Lesson 3 showed the smallest step: separate interfaces for reads and writes. The full version goes further:

  • The write side uses the domain model (aggregates and their rules) and a schema designed for consistent updates.
  • The read side uses read models: tables, documents or a search index shaped exactly for each screen or query, often denormalised (with data copied and combined in advance, so no joins are needed).
  • Events from the write side update the read models, usually a moment later.

CQRS pays off when reads and writes are very different: many more reads than writes, complex screens that need data from several aggregates, or read and write sides that must scale separately. It costs extra moving parts and eventual consistency between the two sides, so it is not a default for every service.

Event sourcing is a related but separate idea. Instead of storing the current state, the system stores every event that ever happened to an aggregate (ClaimCreated, LineAdded, ClaimSubmitted), and rebuilds the state by replaying them. It gives a complete audit history and the ability to rebuild read models at any time, but it is a big step in complexity. Use it only where the history itself is a core requirement.

Eventual consistency and the CAP theorem

With events, other services see a change a little later, usually within milliseconds or seconds. This is eventual consistency: if no new changes happen, all copies of the data eventually agree. Design for the delay:

  • Return 202 Accepted with a status URL for work that continues in the background (lesson 4).
  • Show the user what is happening ("Eligibility check in progress") instead of pretending the work is done.
  • Read your own writes: after a change, show the user the data from the write side, not from a read model that may not have caught up.

The CAP theorem explains why this trade-off cannot be avoided. A distributed data system can guarantee only two of these three at the same time:

  • Consistency (C): every read sees the latest write.
  • Availability (A): every request gets an answer, even if some nodes are unreachable.
  • Partition tolerance (P): the system keeps working when the network splits nodes apart.

Network partitions will happen, so in practice the choice is between C and A during a partition. A CP system refuses some requests to stay correct (for example, a payments ledger). An AP system keeps answering, possibly with slightly old data (for example, a product catalogue or a claim-status dashboard). Choose per kind of data, not once for the whole system. PACELC extends CAP with the everyday case: even when there is no partition, you trade latency against consistency, which is exactly what caching and read replicas do.

Designing events that last

  • Include an event ID, the time it happened, its type and the ID of the thing it is about. The CloudEvents specification is a standard envelope with these fields.
  • Decide how much to include. A thin event (ClaimSubmitted with only the claim ID) forces consumers to call back for details. A fat event (with all the data consumers need) avoids that callback, but couples them to more of your schema. A common middle ground is the IDs plus the fields most consumers need.
  • Treat events as a public contract. Only make additive changes, never rename or remove fields, and publish a new event type for breaking changes (the same rules as API versioning in lesson 4).
  • Do not put secrets or unneeded personal data in events; they are stored and copied widely.

Check your understanding

QuestionShort answer
Command or event?A command asks one service to do something; an event states that something happened, for any number of subscribers.
Service Bus or Event Hubs?Service Bus for business messages that must each be handled; Event Hubs for high-volume streams that are kept and replayed.
What delivery guarantee should consumers assume?At least once: every message may arrive more than once.
What is a dead-letter queue?A side queue for messages that cannot be processed, either rejected explicitly or after too many failed attempts.
How do you keep one claim's messages in order?Use Service Bus sessions (or a partition key) with the claim ID; order is kept within a session.
What problem does the Outbox pattern solve?The dual write: saving a change and publishing its event atomically, by writing the event to an outbox table in the same transaction.
What is the Inbox pattern?Recording processed message IDs in the consumer's database, in the same transaction as the work, so duplicates are ignored.
What is a compensating action?A step that undoes the effect of an earlier saga step that is already committed.
Choreography or orchestration?Choreography for short, simple flows; orchestration when the flow is long or has many failure paths.
Is CQRS the same as event sourcing?No. CQRS separates read and write models; event sourcing stores events instead of current state. Either can be used without the other.
What does CAP say?During a network partition, a system must choose between consistency and availability.

Summary

  • Use messaging to decouple services in time: the sender does not wait for the receiver to be up.
  • Name messages as commands or events, and choose the broker by the job: Service Bus, Event Hubs or Event Grid.
  • Assume at-least-once delivery. Make every consumer idempotent with the Inbox pattern or natural idempotency.
  • Publish events with the Outbox pattern, never with a dual write.
  • Watch and act on dead-letter queues, and ask for ordering only where it matters.
  • Coordinate multi-service processes with sagas and compensations, choosing choreography or orchestration by complexity.
  • Use CQRS and event sourcing only where their benefits outweigh their cost.
  • Design the user experience for eventual consistency, and choose consistency or availability per kind of data.

The next lesson covers integrating with and replacing older systems: the Strangler Fig and Anti-Corruption Layer patterns, multi-tenancy, and real-time updates with WebSockets, SignalR and Server-Sent Events.

Found this useful? Share it: LinkedIn X

Comments