SQS Backpressure Against ECS: Queue Depth Scaling, Visibility Timeout Mechanics, and the Overload Boundary

The Pressure Problem Nobody Draws on the Architecture Diagram

When you wire SQS to an ECS consumer fleet, the standard advice is: emit ApproximateNumberOfMessagesVisible as a CloudWatch metric, attach a target-tracking policy, and let autoscaling do the rest. That advice is incomplete in a way that produces real production incidents.

The queue depth metric tells you how many messages are waiting. It says nothing about how fast your consumers are processing, whether they're crashing mid-flight, or whether the messages they've already received are accumulating invisibility debt that will reappear as a phantom spike. Understanding the mechanics underneath each of those signals—and how ECS scaling policy evaluation races against SQS redelivery—is what separates a system that degrades gracefully from one that enters a thundering-herd loop.

---

SQS Visibility: The Inflight Lease

When an ECS task calls ReceiveMessage, SQS moves each returned message into an inflight state for a duration equal to the configured VisibilityTimeout. During that window, no other consumer sees the message. If the consumer does not call DeleteMessage before the timeout expires, the message becomes visible again—without any signal to the producer, without a dead-letter record, and without the queue depth metric incrementing first.

This is the inflight lease model. Its failure mode: if your consumer is slow but not dead—processing a message in 55 seconds against a 60-second visibility timeout—you'll get partial redeliveries. The consumer finishes, calls DeleteMessage, and the call succeeds because the message is still technically within the lease by milliseconds. But if you've tuned for median processing time and then deployed a batch with a regression that causes p99 latency to spike to 90 seconds, you start seeing duplicate processing without any alarm firing on the queue side.

The correct operational posture is to extend visibility mid-flight using ChangeMessageVisibility. A Go worker loop that does this correctly:

func processWithHeartbeat(ctx context.Context, client *sqs.Client, queueURL string, msg types.Message) error {
    visibilityCtx, cancel := context.WithCancel(ctx)
    defer cancel()

    go func() {
        ticker := time.NewTicker(30 * time.Second)
        defer ticker.Stop()
        for {
            select {
            case <-ticker.C:
                _, err := client.ChangeMessageVisibility(visibilityCtx, &sqs.ChangeMessageVisibilityInput{
                    QueueUrl:          &queueURL,
                    ReceiptHandle:     msg.ReceiptHandle,
                    VisibilityTimeout: 60,
                })
                if err != nil {
                    // Log and let the outer processing race the original timeout.
                    return
                }
            case <-visibilityCtx.Done():
                return
            }
        }
    }()

    return doWork(ctx, msg)
}

The heartbeat goroutine extends the lease every 30 seconds with a 60-second window, giving a 30-second margin before any extension call must succeed. The cancel() deferred from the outer function tears down the goroutine regardless of processing outcome.

What this does not solve: if the ECS task itself is OOM-killed or receives SIGKILL—common under memory pressure during a scaling surge—the heartbeat goroutine dies too, and the message sits invisible until the current window expires.

---

Queue Depth Metric Lag and ECS Scaling Drift

ApproximateNumberOfMessagesVisible is published by SQS to CloudWatch at approximately one-minute intervals. ECS autoscaling evaluates that metric on a CloudWatch alarm, which itself aggregates over a period you configure (commonly two to three data points). This means your scaling decision lags actual queue growth by two to four minutes in the happy path.

During that lag, if your producer is ingesting at 500 messages per second and your existing fleet processes 400 per second, the queue grows by 60,000 messages before the first scale-out action completes—ECS task launch latency for a pre-warmed container is 20–40 seconds, longer if ECR image pull is required.

Three calibration moves that reduce this drift:

1. Scale on ApproximateNumberOfMessagesNotVisible as a secondary signal. This metric reflects inflight volume. A sudden rise without a corresponding rise in visible messages means consumers are receiving work but not finishing it—early warning of a processing slowdown, not a producer spike.

2. Publish a derived metric: messages per task. A Lambda function or a CloudWatch metric math expression computes ApproximateNumberOfMessagesVisible / RunningTaskCount. Target-tracking against this ratio—rather than raw depth—prevents over-scaling when your current fleet is actually processing at capacity and the queue will drain without new tasks.

3. Use step scaling with a short cooldown for the first step only. Target-tracking has a built-in cooldown that prevents thrashing but also dampens aggressive response to genuine spikes. A hybrid policy—step scaling to add two tasks immediately when depth crosses a low threshold, then target-tracking for sustained load—gives fast initial response without the oscillation risk of purely reactive scaling.

---

The Overload Boundary: Where Backpressure Actually Breaks

Backpressure in an SQS-ECS system is implicit. Unlike gRPC streaming or a socket-level flow control mechanism, SQS does not slow the producer when the consumer is behind. The queue absorbs indefinitely up to the account limit. This is usually framed as a feature—decoupling—but it means the consumer bears the entire burden of managing overload.

The boundary breaks at three specific points:

Dead-letter queue saturation. If messages fail processing and exhaust their maxReceiveCount, they move to the DLQ. A DLQ spike is often the first observable signal of a processing regression. But by the time DLQ depth is high enough to alarm, the main queue has already absorbed thousands of messages that will never reprocess without a manual redrive. Redrive, when triggered against a large DLQ, temporarily doubles the inflight pressure on a fleet that is already impaired.

Memory pressure from over-polling. ECS tasks polling SQS with MaxNumberOfMessages: 10 and a short WaitTimeSeconds can accumulate in-process buffers faster than they can process. Under Go's garbage collector, a worker holding 200 partially-decoded message bodies during a GC pause extends that pause and delays downstream DeleteMessage calls, which defers lease renewal, which risks redelivery. Tuning WaitTimeSeconds to 20 (long polling) reduces empty receives and gives the GC more idle time between batches.

ECS task replacement during scaling. When ECS replaces tasks due to a rolling deployment or a scale-in event during active processing, messages held by the terminating task become visible again after their visibility timeout—not immediately. If your StopTimeout in the ECS task definition is shorter than your visibility timeout, the task is killed before it can either complete or return messages explicitly. The result is redelivery after a delay equal to the remaining visibility window, which often lands during the next scaling interval when the new fleet is still warming.

The fix is mechanical: set StopTimeout to at least your visibility timeout, and implement a SIGTERM handler that stops polling, finishes in-flight work, and calls ChangeMessageVisibility with VisibilityTimeout: 0 to immediately re-enqueue messages the task cannot complete.

func handleShutdown(ctx context.Context, client *sqs.Client, queueURL string, active []activeMessage) {
    for _, m := range active {
        client.ChangeMessageVisibility(ctx, &sqs.ChangeMessageVisibilityInput{
            QueueUrl:          &queueURL,
            ReceiptHandle:     m.ReceiptHandle,
            VisibilityTimeout: 0, // immediately visible
        })
    }
}

This requires the task to track which messages are currently in-flight before graceful shutdown—a straightforward sync.Map or a slice guarded by a mutex is sufficient.

---

Decision Framework

When designing or auditing an SQS-to-ECS consumer system, evaluate against these checkpoints in order:

1. Visibility timeout vs. p99 processing latency. Your timeout must exceed p99 with margin. If it doesn't, set up heartbeat extension. If your p99 is variable enough that no static timeout is safe, the processing logic has an unbounded latency problem that autoscaling will not fix.

2. Scaling signal fidelity. Are you scaling on raw depth alone? Add the messages-per-task derived metric. Instrument ApproximateNumberOfMessagesNotVisible as a secondary alarm. Ensure your CloudWatch period and evaluation periods are tuned for your producer burst characteristics, not left at defaults.

3. Graceful drain at scale-in. Does your SIGTERM handler exist, and does ECS StopTimeout give it enough time? A task that is killed mid-work is not a clean scale-in—it's an uncontrolled redelivery event.

4. DLQ redrive pressure. Before triggering a redrive against a large DLQ, calculate the additional inflight volume against your current fleet size. A redrive is a controlled backpressure event; treat it as a load test against an already-strained system and scale the fleet before initiating it.

5. Inflight cap awareness. SQS limits inflight messages to 120,000 for standard queues and 20,000 for FIFO. A fleet that scales aggressively and holds long-visibility leases can hit this cap, at which point ReceiveMessage returns empty responses even though the queue depth is high. This is invisible to most monitoring setups unless you explicitly alert on NumberOfMessagesReceived dropping relative to expected polling rate.

Each of these is an operational invariant, not a configuration preference. Violating any one of them under load produces incidents that look like queue or scaling failures but are fundamentally lease-management and policy-evaluation timing problems.