← All blogs

FullStack Engineering CourseLesson 7 of 20 18 min

Resilience: Timeouts, Retries, Circuit Breakers and Health Checks

Failures are normal

In a distributed system (an application made of several services talking over a network), something is always failing somewhere: a database fails over, a network link drops packets, a payer API restarts, a pod is moved to another node. Resilience means the system keeps working, perhaps with reduced features, when its dependencies (the services and stores it calls) misbehave.

Failures come in two kinds, and they need different handling:

  • Transient failures last a moment: a dropped connection, a timeout during a failover, a 503 while a service restarts. Trying again shortly afterwards usually works.
  • Persistent failures last minutes or hours: a service is down, a certificate expired, a bug returns 500 for every call. Trying again only adds load and wastes the caller's time.

A slow dependency is often worse than one that is down. A failed call returns at once, but a slow call holds a thread, a connection and memory while it waits. If enough requests wait on the same slow dependency, the caller runs out of resources and fails too. That is how one slow service brings down the services that call it (a cascading failure).

Each pattern in this lesson addresses part of this:

PatternProtects against
TimeoutWaiting for ever on a slow dependency
Retry with backoffTransient failures
Circuit breakerHammering a dependency that is down, and waiting on it
BulkheadOne slow dependency using up all threads or connections
FallbackUsers seeing an error when a reduced answer would do
Health checksTraffic being sent to an instance that cannot serve it
Load sheddingOverload: too many requests for the capacity available

Timeouts

Every call that leaves the process needs a timeout: HTTP calls, database queries, Redis calls and message sends. Without one, a call can wait for minutes. HttpClient's default timeout is 100 seconds, which is far too long for an API that should answer in under a second.

Choose timeouts from measurements, not guesses. Look at the dependency's normal response times (for example, the 99th percentile: the time that 99% of calls stay under) and set the timeout a little above it. There are two levels:

  • Attempt timeout: how long one try may take.
  • Total timeout: how long the whole operation may take, including all retries. It must fit inside the time your own caller is willing to wait.

Pass a CancellationToken everywhere (lesson 1), so that when a timeout fires, the work actually stops instead of carrying on in the background.

Retries with exponential backoff and jitter

Retry only when it can help:

  • Retry: network errors, timeouts, 408 Request Timeout, 429 Too Many Requests, 500, 502, 503 and 504.
  • Do not retry: 400, 401, 403, 404, 409, 422. The same request will fail the same way.
  • Retry only idempotent operations (GET, PUT, DELETE, or a POST with an idempotency key from lesson 5). Retrying a plain POST can create duplicates.

Wait between attempts, and wait longer each time. This is exponential backoff: for example 0.5, 1, 2 and 4 seconds. It gives the dependency time to recover. Also add jitter: a random variation in each delay. Without it, all the clients that failed at the same moment retry at the same moment too, and hit the recovering service in synchronised waves.

Keep the number of retries small, usually two or three. Retries multiply across layers. If the gateway, the API and the service behind it each make up to 4 attempts (1 try plus 3 retries), one user request can turn into 4 × 4 × 4 = 64 calls to the struggling service. This is a retry storm. Retry at one layer only, normally the one closest to the failing dependency, and honour Retry-After headers when the server sends them.

Circuit breakers

A circuit breaker watches the calls to one dependency. When too many fail, it stops sending calls for a while and fails them at once instead. This saves the caller from waiting on a dependency that is down, and gives the dependency room to recover. It has three states:

  1. Closed: normal operation. Calls pass through, and failures are counted.
  2. Open: the failure rate went over the threshold. Calls fail immediately (with BrokenCircuitException in Polly), without touching the dependency, for a set break duration.
  3. Half-open: after the break, a few trial calls are allowed through. If they succeed, the breaker closes again. If they fail, it opens again.

Configure it with a failure ratio (for example 50% of calls), a minimum number of calls before it can trip (so two failures out of three calls at night do not open it), a sampling window (for example the last 30 seconds) and a break duration. While the circuit is open, return a fallback or a clear 503, not a long wait.

Bulkheads

A ship's hull is divided into watertight compartments called bulkheads, so one leak cannot sink the whole ship. In software, a bulkhead gives each dependency its own limited share of resources, so one slow dependency cannot use up everything:

  • Limit how many calls to each dependency can run at once (a concurrency limiter). If the payer API hangs, at most 20 requests wait on it, and the rest of the API keeps working.
  • Use separate HttpClient instances (and so separate connection pools) for separate dependencies.
  • Run different kinds of work in different places: slow report generation in a separate worker or queue, not in the same pool as user-facing requests.

Polly and Microsoft.Extensions.Http.Resilience

Polly is the standard .NET resilience library. Version 8 builds a resilience pipeline: a chain of strategies (timeout, retry, circuit breaker and so on) that wraps each call. For HttpClient, the Microsoft.Extensions.Http.Resilience package adds a ready-made pipeline in one line:

C#
builder.Services.AddHttpClient<PayerApiClient>(c => c.BaseAddress = new Uri("https://payer.example.com"))
    .AddStandardResilienceHandler(o =>
    {
        o.AttemptTimeout.Timeout = TimeSpan.FromSeconds(2);         // each try
        o.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(10);   // all tries together
        o.Retry.MaxRetryAttempts = 3;                               // exponential backoff with jitter by default
        o.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(30);
    });

The standard handler chains five strategies, from outside to inside: a rate limiter, the total timeout, retry, the circuit breaker and the attempt timeout. It only retries transient HTTP failures, and it respects Retry-After. The order matters:

  • The total timeout is outermost, so it limits the whole operation, retries included.
  • Retry wraps the circuit breaker, so each attempt is counted by the breaker, and an open breaker stops further retries quickly.
  • The attempt timeout is innermost, so a single slow attempt is cut short and can be retried.

When you need different settings, build the pipeline yourself:

C#
builder.Services.AddHttpClient<EligibilityClient>(c => c.BaseAddress = new Uri("https://eligibility.example.com"))
    .AddResilienceHandler("eligibility", pipeline =>
    {
        pipeline.AddTimeout(TimeSpan.FromSeconds(10));                 // total
        pipeline.AddRetry(new HttpRetryStrategyOptions
        {
            MaxRetryAttempts = 3,
            BackoffType = DelayBackoffType.Exponential,
            Delay = TimeSpan.FromMilliseconds(500),
            UseJitter = true
        });
        pipeline.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
        {
            FailureRatio = 0.5,                                        // open at 50% failures...
            MinimumThroughput = 20,                                    // ...once at least 20 calls were made...
            SamplingDuration = TimeSpan.FromSeconds(30),               // ...in the last 30 seconds
            BreakDuration = TimeSpan.FromSeconds(15)                   // stay open for 15 seconds
        });
        pipeline.AddConcurrencyLimiter(permitLimit: 20, queueLimit: 10); // bulkhead
        pipeline.AddTimeout(TimeSpan.FromSeconds(2));                  // per attempt
    });

The same pipelines work for anything, not only HTTP. This one protects Redis calls, with a short timeout so a slow cache cannot slow down every request:

C#
builder.Services.AddResiliencePipeline("redis", b => b
    .AddRetry(new RetryStrategyOptions
    {
        ShouldHandle = new PredicateBuilder().Handle<RedisConnectionException>().Handle<RedisTimeoutException>(),
        MaxRetryAttempts = 2,
        Delay = TimeSpan.FromMilliseconds(100),
        BackoffType = DelayBackoffType.Exponential,
        UseJitter = true
    })
    .AddTimeout(TimeSpan.FromMilliseconds(300)));

public sealed class CachedPayerReader(ResiliencePipelineProvider<string> pipelines, IConnectionMultiplexer redis,
                                      IPayerRepository repository)
{
    private readonly ResiliencePipeline _redis = pipelines.GetPipeline("redis");

    public async Task<string?> GetPayerJsonAsync(string code, CancellationToken ct)
    {
        try
        {
            RedisValue cached = await _redis.ExecuteAsync(
                async token => await redis.GetDatabase().StringGetAsync($"payer:{code}"), ct);
            if (cached.HasValue) return cached;
        }
        catch (Exception e) when (e is RedisException or TimeoutRejectedException)
        {
            // Fallback: the cache is optional, so carry on to the database.
        }
        return await repository.GetJsonAsync(code, ct);
    }
}

Database retries in EF Core

Azure SQL and other cloud databases have short transient failures during failovers and updates. EF Core can retry these automatically with an execution strategy:

C#
builder.Services.AddDbContext<ClaimsDbContext>(o => o.UseSqlServer(connectionString, sql =>
    sql.EnableRetryOnFailure(maxRetryCount: 5, maxRetryDelay: TimeSpan.FromSeconds(10), errorNumbersToAdd: null)));

There is one catch. If you open your own transaction, EF Core cannot retry part of it, so it throws an error asking you to use the strategy. Wrap the whole transaction in the strategy, so it can be retried from the start as one unit:

C#
var strategy = db.Database.CreateExecutionStrategy();
await strategy.ExecuteAsync(async () =>
{
    await using var tx = await db.Database.BeginTransactionAsync(ct);
    db.Claims.Add(claim);
    db.AuditEntries.Add(new AuditEntry("ClaimSubmitted", claim.ExternalReference));
    await db.SaveChangesAsync(ct);
    await tx.CommitAsync(ct);
});

Because the whole block may run more than once, keep it free of side effects outside the database, such as sending an email. Do those afterwards, or through the Outbox pattern (covered in the event-driven lesson).

Fallbacks and graceful degradation

When a dependency fails, an error is not the only possible answer. A fallback returns something useful instead:

  • Serve the last known good value from a cache, marked as possibly out of date.
  • Return a default: an empty recommendations list instead of a failed page.
  • Accept the work and finish it later: queue the claim and return 202, instead of failing because the eligibility service is down.
  • Turn off a non-essential feature with a feature flag while its dependency is unhealthy.

Graceful degradation means planning this in advance. List the features of the system, mark which ones are essential, and decide what each non-essential feature does when its dependency is down. A claims portal might keep submission and status working while hiding analytics charts.

Load shedding is the last line of defence. When a service is overloaded, rejecting some requests quickly (with 503 or 429) is better than accepting all of them and timing out on every one. The rate limiters and concurrency limiters from lesson 4 do this. Shed the least important work first, such as reports and bulk exports, before user-facing requests.

Health checks

A health check is an endpoint that reports whether an instance can do its job. Load balancers, Kubernetes and Azure App Service call it, and stop sending traffic to instances that fail. There are three kinds, and mixing them up causes outages:

ProbeQuestion it answersWhat happens on failureWhat to check
LivenessIs the process stuck and in need of a restart?The platform restarts the instanceOnly the process itself. Never external dependencies
ReadinessCan this instance serve traffic right now?Traffic is taken away until it passes again; no restartCritical dependencies: database, cache, required configuration
StartupHas the app finished starting (migrations, cache warm-up)?Liveness and readiness checks wait until it passesStart-up work only

The most common mistake is checking the database in the liveness probe. When the database has a short outage, every instance fails liveness, the platform restarts all of them at once, and they all reconnect at the same moment, which makes the outage longer. A database problem should fail readiness, not liveness.

C#
builder.Services.AddHealthChecks()
    .AddCheck("self", () => HealthCheckResult.Healthy(), tags: ["live"])
    .AddSqlServer(connectionString, tags: ["ready"])        // package AspNetCore.HealthChecks.SqlServer
    .AddRedis(redisConnectionString, tags: ["ready"]);      // package AspNetCore.HealthChecks.Redis

app.MapHealthChecks("/health/live",  new HealthCheckOptions { Predicate = c => c.Tags.Contains("live") });
app.MapHealthChecks("/health/ready", new HealthCheckOptions { Predicate = c => c.Tags.Contains("ready") });

Keep health endpoints fast and cheap (they are called every few seconds), give each check its own short timeout, and do not expose detailed results publicly, because they reveal internal structure. If Redis is optional (as in the fallback above), report it as Degraded rather than Unhealthy, so a cache outage does not remove every instance from the load balancer.

Putting it together

A resilient call to an external dependency usually has all of these:

  1. A total timeout that fits inside the caller's own deadline.
  2. A per-attempt timeout based on measured response times.
  3. Two or three retries with exponential backoff and jitter, only for transient errors and idempotent operations, at one layer only.
  4. A circuit breaker per dependency.
  5. A concurrency limit per dependency (a bulkhead).
  6. A planned fallback, or a clear and fast error.
  7. Metrics and logs for retries, timeouts and breaker state changes, so you can see problems before users report them.

Test it. Run the system with a dependency that is slow, failing or completely down, and check that it behaves as planned. Chaos engineering (deliberately injecting failures, for example with Azure Chaos Studio) is covered in the testing lesson.

Check your understanding

QuestionShort answer
Why is a slow dependency often worse than one that is down?Slow calls hold threads, connections and memory while waiting, so the caller runs out of resources and fails too.
Attempt timeout or total timeout?Both: the attempt timeout limits one try, and the total timeout limits all tries together and must fit the caller's deadline.
Which errors should be retried?Network errors, timeouts, 408, 429 and 5xx, and only for idempotent operations. Never 4xx errors such as 400 or 404.
Why add jitter to backoff?So clients that failed together do not retry together in synchronised waves.
What is a retry storm?Retries at several layers multiplying the load on a struggling service. Retry at one layer only.
What are the three circuit breaker states?Closed (normal), open (fail fast for a break period), half-open (let trial calls through to test recovery).
What is a bulkhead?A separate, limited share of resources per dependency, so one slow dependency cannot exhaust everything.
What does AddStandardResilienceHandler add?A rate limiter, a total timeout, retry, a circuit breaker and an attempt timeout, in that order from outside to inside.
Why must EF Core transactions run inside the execution strategy?So the whole transaction can be retried as one unit after a transient failure.
Liveness or readiness for a database check?Readiness. A database check in liveness restarts every instance during a short database outage.
What is graceful degradation?Keeping essential features working and switching off or simplifying the rest when dependencies fail.

Summary

  • Expect failures, and treat slowness as a failure.
  • Put a timeout on every outbound call, at attempt and total level.
  • Retry only transient errors on idempotent operations, two or three times, with exponential backoff and jitter, at one layer.
  • Use a circuit breaker and a concurrency limit for each dependency.
  • Plan fallbacks and graceful degradation, and shed load before you are overwhelmed.
  • Use Polly pipelines (AddStandardResilienceHandler for HttpClient) and EF Core's execution strategy rather than hand-written retry loops.
  • Separate liveness, readiness and startup checks, and never check dependencies in liveness.

The next lesson moves up a level to architecture: domain-driven design, clean and hexagonal architecture, microservices and modular monoliths, the Backend-for-Frontend pattern and micro-frontends.

Found this useful? Share it: LinkedIn X

Comments