← All blogs

FullStack Engineering CourseLesson 5 of 20 35 min

Idempotency, Concurrency and Distributed Locks

Why APIs break under retries and concurrent users

An API that works in testing can still give wrong results in production. Two things cause most of these problems:

  • Retries: networks drop responses. A client that gets no answer cannot tell whether the server did the work, so it sends the request again. Gateways, HTTP client libraries and message brokers retry on their own too. Without protection, a retried POST creates a second claim or charges a payment twice.
  • Concurrent users: two people, or two instances of a service, change the same data at the same time. Without protection, one change silently overwrites the other (a lost update), or a job runs twice.

This lesson covers the patterns that handle both: idempotency keys, optimistic concurrency, conditional updates, ETags, distributed locks and atomic Redis operations.

Idempotency keys

An operation is idempotent when doing it twice has the same effect as doing it once. GET, PUT and DELETE are idempotent by design. POST is not: each call creates something new. An idempotency key makes a POST safe to retry.

The client creates a unique key (usually a GUID) for each operation, such as "submit this claim", and sends it in an Idempotency-Key header. If it has to retry, it sends the same key again. The server remembers each key and the response it gave:

  1. First request with a key: the server marks the key as "processing", does the work, stores the response, and returns it.
  2. Retry with the same key, after the first one finished: the server does not repeat the work. It returns the stored response.
  3. Retry while the first one is still running: the server returns 409 Conflict, and the client tries again a little later.
  4. Same key with a different request body: this is a client bug. Return 422, so the client does not think a different operation succeeded.
  5. The first request failed with a server error: the server deletes the key, so a retry can run the work again.

Redis is a good place to store the keys, because setting a key only if it does not exist is a single atomic operation (done as one indivisible step). Only one of two simultaneous requests can win. This endpoint filter (code that runs around an endpoint, like middleware for one route) adds idempotency to any minimal API endpoint:

C#
public sealed record StoredResponse(int Status, string Body);

public sealed class IdempotencyFilter(IConnectionMultiplexer redis) : IEndpointFilter
{
    private static readonly TimeSpan ProcessingTtl = TimeSpan.FromMinutes(5);  // frees the key if the server crashes mid-request
    private static readonly TimeSpan ResultTtl = TimeSpan.FromHours(24);       // how long retries get the stored answer

    public async ValueTask<object?> InvokeAsync(EndpointFilterInvocationContext context, EndpointFilterDelegate next)
    {
        HttpContext http = context.HttpContext;
        string? key = http.Request.Headers["Idempotency-Key"];
        if (string.IsNullOrWhiteSpace(key))
            return TypedResults.Problem("The Idempotency-Key header is required.", statusCode: 400);

        IDatabase db = redis.GetDatabase();
        string caller = http.User.FindFirstValue(ClaimTypes.NameIdentifier) ?? "anonymous";
        var redisKey = new RedisKey($"idem:{caller}:{key}");      // keys are per caller, so clients cannot collide

        // 1. Try to claim the key. SET ... NX succeeds for exactly one request.
        bool claimed = await db.StringSetAsync(redisKey, "processing", ProcessingTtl, When.NotExists);
        if (!claimed)
        {
            RedisValue saved = await db.StringGetAsync(redisKey);
            if (saved == "processing")
                return TypedResults.Problem("A request with this key is still in progress.", statusCode: 409);

            var stored = JsonSerializer.Deserialize<StoredResponse>(saved.ToString())!;
            return Results.Content(stored.Body, "application/json", statusCode: stored.Status);  // 2. replay
        }

        try
        {
            object? result = await next(context);                   // 3. do the real work once
            int status = (result as IStatusCodeHttpResult)?.StatusCode ?? 200;
            if (status >= 500)
            {
                await db.KeyDeleteAsync(redisKey);                   // server error: let the client retry
                return result;
            }

            object? value = (result as IValueHttpResult)?.Value;
            var response = new StoredResponse(status, JsonSerializer.Serialize(value, JsonSerializerOptions.Web));
            await db.StringSetAsync(redisKey, JsonSerializer.Serialize(response), ResultTtl);  // 4. remember the answer
            return result;
        }
        catch
        {
            await db.KeyDeleteAsync(redisKey);                       // exception: free the key for a retry
            throw;
        }
    }
}

// Apply it to the endpoints that create things:
claims.MapPost("/", SubmitClaim).AddEndpointFilter<IdempotencyFilter>();

Some details to get right:

  • Scope keys per caller (here by user ID), so two clients that happen to choose the same key do not see each other's responses.
  • Use a short TTL (time to live: how long Redis keeps the key) for "processing" and a long one for the result. If the server crashes in the middle of a request, the short TTL frees the key, so the client can retry later.
  • To catch a key reused with a different body, store a hash of the request body next to the response, and compare it on every retry.
  • Keep results long enough to cover the client's whole retry period. 24 hours is common.

Redis gives fast protection, but it is a separate store from your database. If the claim is saved and the server crashes before the response is stored in Redis, a retry can create a duplicate. Where a duplicate is truly unacceptable (payments, claims), also put a unique constraint in the database. Store the idempotency key, or a natural business key such as the hospital's claim reference, in a column with a unique index. The database then rejects a second insert, even when everything else fails:

SQL
-- The database itself refuses a second claim with the same reference from the same provider.
CREATE UNIQUE INDEX UX_Claims_Provider_ExternalRef ON Claims (ProviderId, ExternalReference);
C#
try
{
    db.Claims.Add(claim);
    await db.SaveChangesAsync(ct);
}
catch (DbUpdateException e) when (e.InnerException is SqlException { Number: 2601 or 2627 })  // unique index violation
{
    var existing = await db.Claims.SingleAsync(c => c.ProviderId == claim.ProviderId &&
                                                    c.ExternalReference == claim.ExternalReference, ct);
    return TypedResults.Ok(ClaimDto.From(existing));   // it already exists: return it instead of failing
}

Some operations can be made idempotent by design, without any keys. Let the client choose the ID and use PUT (PUT /claims/{clientGeneratedId} creates the claim or replaces it with the same content). Or write "set status to Approved" instead of "toggle status". Prefer this when it fits, because there is nothing extra to store.

The lost update problem

Two claims officers open claim 42 at the same time. Both see the amount 1,000. One corrects the amount to 1,200 and saves. A minute later, the other approves the claim and saves the whole record, still holding the old amount of 1,000. The correction is gone, and nobody gets an error. This is a lost update.

There are two ways to prevent it:

Pessimistic concurrencyOptimistic concurrency
IdeaLock the row when you read it, so nobody else can change it until you finishDo not lock. When saving, check that nobody changed the row since you read it
HowSELECT ... WITH (UPDLOCK), or a lock flag on the recordA version column compared in the UPDATE's WHERE clause
CostOthers wait; long locks block the system; risk of deadlocksNo waiting; a conflict is detected and reported instead
Works across HTTP requestsNo: you cannot hold a database lock while a user edits a formYes: the version travels to the client and back
Best forShort, high-contention updates inside one transactionWeb apps and APIs, where conflicts are rare

APIs almost always use optimistic concurrency, because a user may take minutes to edit a form, and a database lock cannot stay open that long.

Optimistic concurrency with EF Core

Add a version column to the table. In SQL Server, the rowversion type changes automatically on every update. EF Core then adds the version to the WHERE clause of every UPDATE and DELETE. If someone changed the row since you read it, the version no longer matches, no row is updated, and EF Core throws DbUpdateConcurrencyException:

C#
public class Claim
{
    public int Id { get; set; }
    public ClaimStatus Status { get; set; }
    public decimal Amount { get; set; }

    [Timestamp]                                  // SQL Server rowversion: changes on every update
    public byte[] RowVersion { get; set; } = [];
}

var claim = await db.Claims.SingleAsync(c => c.Id == 42, ct);   // RowVersion read along with the data
claim.Status = ClaimStatus.Approved;

try
{
    await db.SaveChangesAsync(ct);
    // EF Core runs: UPDATE Claims SET Status = @p0 WHERE Id = @p1 AND RowVersion = @p2
}
catch (DbUpdateConcurrencyException)
{
    // 0 rows matched: someone changed claim 42 after we read it.
    return TypedResults.Conflict("The claim was changed by someone else. Reload it and try again.");
}

When a conflict happens, you have three choices:

  • Report it (most common): return 409 or 412, and let the user reload and decide. This is the only safe default when both changes matter.
  • Last write wins: reload the current version into the entity and save again. Only use this when overwriting is truly acceptable.
  • Merge: reload the database values, combine them with the user's changes field by field, and save. This needs business rules for each field, so use it only where it is worth the effort.

PostgreSQL has no rowversion type. With Npgsql, map a uint property to PostgreSQL's hidden xmin column, which changes on every update and works the same way. Without either, an integer Version column that you increase on every update also works.

Conditional updates (compare-and-swap)

Many changes are state transitions: Pending to Approved, Approved to Paid. These need a simpler check than a full row version: "change it only if it is still in the state I expect". Put the expected state in the WHERE clause, and check how many rows were updated. This is called compare-and-swap (change a value only if it still equals what you compared it with), and it needs no lock and no extra read:

SQL
-- Approve claim 42, but only if it is still Pending.
UPDATE Claims
SET    Status = 'Approved', ApprovedBy = @userId, ApprovedAt = SYSUTCDATETIME()
WHERE  Id = 42 AND Status = 'Pending';
-- 1 row affected: we approved it. 0 rows: someone else changed it first.
C#
int updated = await db.Claims
    .Where(c => c.Id == id && c.Status == ClaimStatus.Pending)
    .ExecuteUpdateAsync(s => s.SetProperty(c => c.Status, ClaimStatus.Approved)
                              .SetProperty(c => c.ApprovedBy, userId), ct);

if (updated == 0)
    return TypedResults.Conflict("The claim is no longer pending.");

Two officers clicking Approve at the same moment now get one success and one clear conflict, instead of two approvals. The same idea works for counters ("reduce stock by 1 only if stock > 0") and for claiming a job from a table ("set Owner = me where Owner is null").

ETags and If-Match: concurrency across HTTP

The row version check only helps if the version the user saw travels with their change. Between a GET and a later PUT or PATCH there may be minutes, and the server keeps no state in between. HTTP solves this with ETags:

  1. GET returns the resource with an ETag header that holds its version (here, the rowversion encoded as base64).
  2. The client sends that value back in an If-Match header when it changes the resource.
  3. If the version in If-Match is still current, the update runs, and the response carries the new ETag.
  4. If the resource changed in the meantime, the server returns 412 Precondition Failed, and the client reloads.
  5. If the client sends no If-Match header at all, the server can require one with 428 Precondition Required.
C#
static async Task<IResult> ApproveClaim(int id, HttpContext http, ClaimsDbContext db, CancellationToken ct)
{
    string? ifMatch = http.Request.Headers.IfMatch;
    if (string.IsNullOrEmpty(ifMatch))
        return TypedResults.Problem("Send the claim's ETag in an If-Match header.", statusCode: 428);

    var claim = await db.Claims.FindAsync([id], ct);
    if (claim is null) return TypedResults.NotFound();

    byte[] versionSeenByClient;
    try { versionSeenByClient = Convert.FromBase64String(ifMatch.Trim('"')); }
    catch (FormatException) { return TypedResults.Problem("Invalid ETag.", statusCode: 412); }

    // Tell EF Core which version the client saw. The UPDATE only succeeds if it is still current.
    db.Entry(claim).Property(c => c.RowVersion).OriginalValue = versionSeenByClient;
    claim.Status = ClaimStatus.Approved;

    try
    {
        await db.SaveChangesAsync(ct);
    }
    catch (DbUpdateConcurrencyException)
    {
        return TypedResults.Problem("The claim changed since you read it. Reload and try again.", statusCode: 412);
    }

    http.Response.Headers.ETag = $"\"{Convert.ToBase64String(claim.RowVersion)}\"";   // the new version
    return TypedResults.NoContent();
}

This is the same ETag that lesson 4 used for 304 Not Modified. On a read, If-None-Match asks "send it only if it changed". On a write, If-Match says "change it only if it did not change".

Distributed locks

Sometimes only one instance of a service may do a piece of work at a time. Examples:

  • A nightly import that runs on every instance at 2 a.m., but must run only once.
  • Refreshing a shared access token that all instances use.
  • Processing a hospital's claim file, where two instances must not import the same file.

lock and SemaphoreSlim (lesson 1) only work inside one process. Across instances you need a distributed lock: a lock held in a shared store that every instance can see. Redis is the usual choice. A Redis lock is a key that exists only while someone holds the lock:

  • Acquire: SET lock:import <owner-id> NX PX 30000. NX means "only if the key does not exist", so only one instance can succeed. PX 30000 gives the key a 30-second expiry.
  • Release: delete the key, but only if it still holds your owner ID. Otherwise you could delete a lock that another instance took after yours expired.
  • The expiry is a lease: if the holder crashes, the lock frees itself after 30 seconds instead of staying stuck for ever.

StackExchange.Redis has these operations built in: LockTakeAsync, LockExtendAsync and LockReleaseAsync. When the work may take longer than the lease, keep extending the lease while you work, and stop the work if you lose the lock:

C#
public sealed class DailyImportJob(IConnectionMultiplexer redis, ILogger<DailyImportJob> log)
{
    private static readonly RedisKey LockKey = "lock:daily-claims-import";
    private static readonly TimeSpan Lease = TimeSpan.FromSeconds(30);

    public async Task RunAsync(CancellationToken ct)
    {
        IDatabase db = redis.GetDatabase();
        string owner = Guid.NewGuid().ToString();              // proves which instance holds the lock

        if (!await db.LockTakeAsync(LockKey, owner, Lease))
        {
            log.LogInformation("Another instance is running the import. Skipping.");
            return;
        }

        using var work = CancellationTokenSource.CreateLinkedTokenSource(ct);
        Task renewal = KeepLeaseAsync(db, owner, work);       // extend the lease every 10 seconds
        try
        {
            await ImportClaimsAsync(work.Token);              // stops if the lock is lost
        }
        finally
        {
            work.Cancel();
            try { await renewal; } catch (OperationCanceledException) { }
            await db.LockReleaseAsync(LockKey, owner);        // deletes the key only if we still own it
        }
    }

    private async Task KeepLeaseAsync(IDatabase db, string owner, CancellationTokenSource work)
    {
        using var timer = new PeriodicTimer(TimeSpan.FromSeconds(10));
        while (await timer.WaitForNextTickAsync(work.Token))
        {
            if (!await db.LockExtendAsync(LockKey, owner, Lease))
            {
                log.LogWarning("Lost the import lock. Stopping the import.");
                work.Cancel();                                // someone else may hold it now
                return;
            }
        }
    }
}

A lease-based lock has a known weakness. If the holder pauses for longer than the lease (a long garbage collection, a stalled VM, a network split), the lock expires and a second instance takes it. For a moment both think they hold the lock. When correctness really depends on the lock, add a fencing token: a number that grows every time the lock is taken. Every write carries the token, and the storage rejects writes with an older token:

C#
long fence = await db.StringIncrementAsync("fence:daily-claims-import");   // a bigger number for every new holder
SQL
-- The old holder (smaller fence) can no longer write once a new holder has written.
UPDATE ImportState
SET    LastProcessedRow = @row, Fence = @fence
WHERE  Job = 'daily-claims' AND Fence <= @fence;

Other ways to get a distributed lock:

  • Azure Blob Storage lease: lease a blob for 15 to 60 seconds and renew it. It suits teams already on Azure, with no Redis needed.
  • SQL Server sp_getapplock: a named lock held by a transaction or a session in the database you already use.
  • A lock table: a row with an owner and an expiry time, taken with a conditional update (the compare-and-swap above).

The best lock is often no lock. Before adding one, check whether the work can be made idempotent so running it twice is harmless, or split so each item has one owner (for example, Azure Service Bus sessions deliver all messages for one claim to one consumer, in order), or given to a single scheduled worker.

Atomic operations in Redis and Lua scripts

Redis runs commands one at a time, so each single command is atomic. INCR, DECRBY, SET with NX, HINCRBY and LPUSH can never be interrupted halfway by another client. The danger is a sequence of commands. A GET followed by a SET is two steps, and another client can change the value in between, which is the same race as count++ in lesson 1.

Redis offers two ways to make several steps atomic:

  • Transactions (MULTI and EXEC, used through CreateTransaction in StackExchange.Redis): queue several commands and run them together. Add conditions (WATCH in Redis, AddCondition in the client) to make the transaction fail if a key changed since you read it. This is optimistic concurrency again.
  • Lua scripts (EVAL): send a small script that Redis runs as one atomic step. No other command runs until the script finishes. This is the simplest way to write "check, then change" logic.

For example, a payer has a daily budget for automatic approvals. Reserving an amount must check the remaining budget and reduce it in one step, or two requests could both see enough budget and both spend it:

-- KEYS[1] = the payer's remaining budget for today (in cents)
-- ARGV[1] = the amount to reserve (in cents)
local remaining = tonumber(redis.call('GET', KEYS[1]) or '0')
local amount = tonumber(ARGV[1])
if remaining < amount then
  return -1                                      -- not enough budget: change nothing
end
return redis.call('DECRBY', KEYS[1], amount)     -- reserve it and return what is left
C#
private const string ReserveScript = """
    local remaining = tonumber(redis.call('GET', KEYS[1]) or '0')
    local amount = tonumber(ARGV[1])
    if remaining < amount then return -1 end
    return redis.call('DECRBY', KEYS[1], amount)
    """;

public async Task<bool> TryReserveAsync(string payerId, long amountCents)
{
    IDatabase db = redis.GetDatabase();
    var result = await db.ScriptEvaluateAsync(ReserveScript,
        keys: [new RedisKey($"budget:{payerId}:{DateTime.UtcNow:yyyyMMdd}")],
        values: [amountCents]);
    return (long)result >= 0;    // -1 means the budget was too low
}

The safe lock release from the previous section is also a Lua script under the hood. It compares the owner and deletes the key in one step:

if redis.call('GET', KEYS[1]) == ARGV[1] then
  return redis.call('DEL', KEYS[1])
end
return 0

Rules for Lua scripts:

  • Keep them short. Redis does nothing else while a script runs, so a slow script blocks every client.
  • Pass every key the script uses in KEYS, never build key names inside the script. Redis Cluster needs this to send the script to the right node.
  • In Redis Cluster, all keys used by one script must live on the same node. Put the shared part of the name in braces, such as budget:{payer-7}:today and spent:{payer-7}:today, so they hash to the same slot.
  • Store numbers as whole units (cents), because Lua numbers are floating point.

Choosing the right pattern

ProblemPattern
A client retry creates a duplicateIdempotency key, plus a unique constraint where duplicates are unacceptable
Two users overwrite each other's changesOptimistic concurrency: a row version, with ETag and If-Match over HTTP
A state change must happen only onceConditional update (compare-and-swap) on the current state
Only one instance may run a jobMake it idempotent or give it one owner; otherwise a distributed lock with a lease (and a fencing token if correctness depends on it)
Check-then-change on shared data in RedisAn atomic command, a transaction with a condition, or a Lua script

Check your understanding

QuestionShort answer
Why does a POST need an idempotency key?A client that gets no response retries, and without a key the server cannot tell a retry from a new request, so it creates duplicates.
What does the server store for each idempotency key?A processing marker while the work runs, then the status and body of the response, for about 24 hours.
Why delete the key after a 500 error?So the client's retry can run the work again instead of replaying a failure.
Why add a unique constraint as well as Redis keys?Redis and the database can disagree after a crash; the unique index stops duplicates even then.
What is a lost update?Two users read the same data, and the later save silently overwrites the earlier change.
Optimistic or pessimistic concurrency for APIs?Optimistic: no locks are held between requests; conflicts are detected when saving.
What does [Timestamp] do in EF Core?Maps a rowversion column that EF Core adds to the WHERE clause of updates; a mismatch throws DbUpdateConcurrencyException.
What is a conditional update?An UPDATE whose WHERE clause includes the expected current state; 0 rows updated means someone else changed it first.
If-Match or If-None-Match?If-Match on writes (412 if the resource changed); If-None-Match on reads (304 if it did not change).
Why does a Redis lock need an expiry?So a crashed holder cannot keep the lock for ever; the expiry is a lease that must be extended during long work.
Why release a lock only if you own it?Your lease may have expired and another instance may hold the lock now; deleting it would let a third instance in.
What is a fencing token?A number that grows with every new lock holder; storage rejects writes with an older number, so a paused old holder cannot do damage.
Why use a Lua script in Redis?To run a check and a change as one atomic step, so no other client can act in between.

Summary

  • Assume every request can arrive twice. Make POSTs safe to retry with idempotency keys, and back them with a unique constraint where duplicates are unacceptable.
  • Prefer operations that are idempotent by design: client-generated IDs with PUT, and "set" instead of "toggle".
  • Prevent lost updates with optimistic concurrency: a row version in the WHERE clause, reported as 409 or 412.
  • Use conditional updates for state transitions, and check the number of rows updated.
  • Carry the version across HTTP with ETag and If-Match, and require If-Match with 428 where it matters.
  • Use a distributed lock only when nothing simpler works. Give it a lease, extend it during long work, release it only if you own it, and add a fencing token when correctness depends on it.
  • In Redis, rely on single atomic commands, and use Lua scripts or conditional transactions for check-then-change logic.

The next lesson covers caching: cache-aside and write-through, Redis and in-memory caches, keeping caches correct when data changes, and protecting the database from a stampede when a popular entry expires.

Found this useful? Share it: LinkedIn X

Comments