Idempotency Keys Under Distributed Retry Pressure: Key Scoping, Storage Contracts, and the Deduplication Boundary

Idempotency keys are a solved problem in the abstract and a continuous failure mode in production. The concept is straightforward: a caller supplies a stable key, the server uses it to detect duplicate requests, and it returns the same response without re-executing side effects. What makes idempotency hard is not the idea — it is the contract between key scoping, storage semantics, retry timing, and failure recovery. Get any one of those wrong and you either duplicate the operation or falsely deduplicate it.

This article focuses on the mechanics that matter in a Go microservice under real retry pressure: where the deduplication boundary sits, what storage backend choices actually guarantee, and where the most common scoping mistakes create silent correctness holes.

Where Idempotency Lives in the Stack

Most teams bolt idempotency onto their HTTP or gRPC handler layer as if it were a middleware concern. It is not, or at least not entirely. The deduplication boundary must sit at the boundary of the side effect, not the boundary of the network call.

Consider a payment service that charges a card, writes a ledger entry, and publishes a payment.completed event. A middleware that deduplicates at the handler boundary can short-circuit all three of those steps on a retry — which is correct. But if the payment charge succeeds and the service crashes before the ledger write, the next retry must not be deduplicated. The key must still be in an uncommitted state because the operation did not complete.

This leads to the first architectural constraint: idempotency keys must track operation state, not just request receipt.

type IdempotencyRecord struct {
    Key       string
    Status    RecordStatus // Pending | Complete | Failed
    Response  []byte       // serialized response, nil if Pending
    CreatedAt time.Time
    ExpiresAt time.Time
}

type RecordStatus int

const (
    StatusPending  RecordStatus = iota
    StatusComplete
    StatusFailed
)

A Pending record means a previous attempt started but has not committed a result. That is fundamentally different from a missing record, and your retry path must handle it differently.

Key Scoping: The Most Underspecified Part

A common mistake is treating the idempotency key as globally unique. It should be scoped to the operation type. A key like req-abc123 that a client reuses across different endpoints will produce false deduplication if your store does not namespace by operation.

More subtle is scoping across service versions. If you deploy a new version of the payment handler that changes what the response body contains, a deduplicated response from an old record may return a shape the caller cannot parse. This is not a theoretical concern — it surfaces during rolling deployments when old records are still in TTL window.

The storage key should encode at minimum: {service}.{operation}.{caller-supplied-key}. In Go:

func idempotencyStoreKey(service, operation, clientKey string) string {
    return fmt.Sprintf("%s:%s:%s", service, operation, clientKey)
}

For multi-tenant systems, tenant ID must also be part of the namespace. A key that one tenant generates is not transferable to another tenant's operation space, and failure to namespace there is a data isolation bug, not just a correctness one.

Storage Backend Contracts and What They Actually Guarantee

The storage layer for idempotency records is where most production failures originate. The two common choices are Redis and a relational or document store. Each has a distinct failure profile.

Redis is fast and TTL-native, but it offers no transactional guarantee across the idempotency write and the actual side-effect operation. The classic failure sequence:

1. Request arrives. Redis key absent — proceed.
2. Write Pending record to Redis.
3. Execute payment charge.
4. Redis crashes before writing Complete.
5. Retry arrives. Redis key absent (evicted or lost). Operation executes again.

This is not a Redis-specific bug — it is a consequence of using a separate store for idempotency state without a two-phase commit between the side-effect store and the idempotency store. Redis also silently evicts keys under memory pressure unless you configure maxmemory-policy noeviction, which most clusters do not do for performance reasons.

The mitigation is to write the idempotency record in the same transaction as the side effect when the side-effect store supports transactions. If your ledger is in MongoDB, write the idempotency record as a document in the same database within a multi-document transaction. The idempotency record and the business record commit atomically or both roll back.

session, err := mongoClient.StartSession()
if err != nil {
    return err
}
defer session.EndSession(ctx)

_, err = session.WithTransaction(ctx, func(sc mongo.SessionContext) (interface{}, error) {
    // Check and insert idempotency record atomically
    _, err := idempotencyCol.InsertOne(sc, IdempotencyRecord{
        Key:       storeKey,
        Status:    StatusPending,
        CreatedAt: time.Now(),
        ExpiresAt: time.Now().Add(24 * time.Hour),
    })
    if mongo.IsDuplicateKeyError(err) {
        existing := &IdempotencyRecord{}
        if findErr := idempotencyCol.FindOne(sc, bson.M{"_id": storeKey}).Decode(existing); findErr != nil {
            return nil, findErr
        }
        if existing.Status == StatusComplete {
            return existing.Response, ErrAlreadyProcessed
        }
        // Pending: a concurrent attempt is in-flight or a prior attempt crashed.
        return nil, ErrConcurrentRequest
    }
    if err != nil {
        return nil, err
    }

    // Execute the side effect
    result, err := chargeCard(sc, req)
    if err != nil {
        return nil, err
    }

    // Mark complete
    encoded, _ := json.Marshal(result)
    _, err = idempotencyCol.UpdateOne(sc,
        bson.M{"_id": storeKey},
        bson.M{"$set": bson.M{"status": StatusComplete, "response": encoded}},
    )
    return encoded, err
})

The duplicate key error on insert is the atomic check-and-set. MongoDB's unique index on _id makes this safe under concurrent retries.

The Pending State Is a Correctness Boundary, Not an Edge Case

When a record is Pending, you have two valid behaviors: wait with a timeout, or return a 409 Conflict with a Retry-After header. The wrong behavior is to let the second request proceed as if the key did not exist. That produces a concurrent execution with two threads operating on the same logical operation, which is exactly what idempotency is supposed to prevent.

A Pending record older than a plausible request timeout (say, 30 seconds beyond your p99 handler latency) can be treated as a crashed operation. At that point, the retry policy should allow the new request to claim the key and attempt the operation again — but only after the Pending record is explicitly transitioned to Failed or deleted. This requires a background reaper or a compare-and-swap transition.

In Go, that reaper is a goroutine with a ticker that scans for stale Pending records:

func (s *IdempotencyStore) ReapStale(ctx context.Context, staleness time.Duration) {
    ticker := time.NewTicker(30 * time.Second)
    defer ticker.Stop()
    for {
        select {
        case <-ctx.Done():
            return
        case <-ticker.C:
            cutoff := time.Now().Add(-staleness)
            _, _ = s.col.DeleteMany(ctx, bson.M{
                "status":     StatusPending,
                "created_at": bson.M{"$lt": cutoff},
            })
        }
    }
}

The staleness value must be calibrated against your upstream timeout chain. If your downstream payment processor has a 10-second timeout and your handler has a 15-second deadline, a 20-second staleness threshold is defensible.

TTL Contracts and the Replay Attack Boundary

Idempotency records cannot be kept forever. A 24-hour TTL is common for payment-style operations; a 7-day TTL appears in some batch processing designs. The TTL is not just an operational cleanup concern — it defines the window inside which a caller can safely retry. A caller retrying after the TTL has expired gets no deduplication guarantee.

This creates a decision: do you document the TTL as part of your API contract, or do you absorb the operational cost of longer retention? For any operation with financial consequence, the TTL window should be explicit in API documentation and enforced server-side, not left as an undocumented implementation detail.

The TTL also bounds replay attack surface. An idempotency key that lives indefinitely is a key that an attacker can replay indefinitely to get a cached successful response. This is especially relevant if your response body contains sensitive data (a charge receipt, a token). TTL expiry forces re-authentication of the operation, which is the correct security posture.

Decision Framework

When designing idempotency into a service, answer these questions in order:

1. Where does the side effect commit? Your idempotency record must commit in the same atomic scope as the side effect, or your deduplication guarantee is probabilistic.

2. What is your key namespace? Include service, operation, and caller-defined key at minimum. Add tenant ID for multi-tenant systems.

3. What does Pending mean for your retry policy? Define a maximum in-flight window beyond which a Pending record is treated as failed, and implement a reaper.

4. What is your TTL, and is it part of the API contract? Document it. Callers need to know how long their retry window is valid.

5. What does your storage backend actually guarantee under failure? Redis is fast but requires explicit noeviction policy and does not solve the two-phase commit problem. Transactional document or relational stores give stronger guarantees at the cost of throughput.

6. What happens when you return a cached response after a schema change? Version your response schema or scope your idempotency keys to include a version segment if your response structure evolves.

Idempotency is not middleware. It is a distributed protocol between the caller's retry logic, the service's state machine, and the storage backend's consistency guarantees. The failures that surface in production are almost always at the seams between those three — not inside any one of them.