Go Scheduler Preemption Under Goroutine Saturation: P Starvation, Work Stealing, and the Latency Cliff

The Preemption Model Most Engineers Misread

Go's scheduler is documented as an M:N cooperative scheduler with asynchronous preemption bolted on since Go 1.14. The practical reading most engineers carry is: "goroutines yield at function calls, the runtime preempts long-running ones via signals." That reading is correct but incomplete in ways that produce real production incidents.

The gap becomes visible at saturation: when your service is handling a traffic spike, GOMAXPROCS Ps are all occupied, and goroutines are queuing faster than they are completing. At that point the scheduler's internal mechanics—local run queues, the global run queue, work stealing, and the preemption signal delivery pathway—interact in ways that produce latency cliffs that no amount of profiling at steady state would have predicted.

Scheduler Topology in One Precise Pass

The Go scheduler operates on three abstractions: G (goroutine), M (OS thread), and P (logical processor). GOMAXPROCS bounds the number of Ps. Each P holds a local run queue (LRQ) capped at 256 Gs. An M must hold a P to execute Go code.

When a goroutine is created via go f(), the runtime attempts to put the new G into the current P's LRQ. If the LRQ is full, half of it is moved to the global run queue (GRQ). Ms without a P become idle and park; when a P has work, it either wakes an idle M or the creating M continues. Work stealing: when a P's LRQ is empty, it checks the GRQ, then steals from another P's LRQ—specifically, it steals half of the victim P's queue.

Preemption happens at two layers:

1. Cooperative: at safe points—function call prologues check the stackguard0 sentinel; if it is set to stackPreempt, the goroutine yields.
2. Asynchronous (signal-based): the sysmon goroutine (which runs outside any P) sends SIGURG to an M running a goroutine that has been scheduled for more than 10 ms. The signal handler inserts an async preemption at the current instruction if it is a safe point.

The 10 ms threshold and the safe-point requirement are load-bearing details.

Where Saturation Breaks the Model

LRQ Overflow Into the Global Queue

When traffic spikes, goroutines are spawned at a rate exceeding P throughput. The LRQ fills to 256. The runtime moves half to the GRQ. GRQ access requires a global mutex (sched.lock). Under sustained load, multiple Ps simultaneously contend on that lock during both GRQ drains and overflow pushes. This is not theoretical—GODEBUG=schedtrace=100 will show gomaxprocs Ps with near-full LRQs and a growing GRQ count, and runtime/metrics will surface scheduler latency.

In a microservice handling inbound gRPC requests, each accepted stream spawns at least one goroutine. At 10k RPS with P=8, if average goroutine lifetime exceeds ~80 µs, LRQs saturate. GRQ contention at that point directly widens P99 because waiting Gs in the GRQ are deprioritized relative to LRQ Gs.

Work Stealing Amplifying Tail Latency

Work stealing is efficient at balancing load across Ps but adds non-trivial cost under contention. The stealing path calls runqsteal, which acquires the victim P's run queue lock (p.runqlock). If every P is simultaneously stealing from every other P because all LRQs are cycling rapidly, the locking overhead compounds.

More critically: a stolen G resumes on a different P than the one where it was created. For a goroutine that touches per-P caches (e.g., the mcache for small allocations), migration flushes the effective cache locality. Under saturation, Gs migrate frequently, mcache miss rate rises, mcentral contention increases, and allocation latency climbs—separately from scheduling latency but coincident with it.

Sysmon Preemption Latency Under All-P Busy

Sysmon runs without holding a P. Its preemption loop wakes every 20 µs (under light load) up to 10 ms (under zero preemption demand), then detects goroutines exceeding 10 ms of continuous scheduling on any M and sends SIGURG. Under saturation:

• Sysmon itself may be delayed because the OS scheduler is fully occupied servicing the M threads running Go Ps. SIGURG delivery is subject to OS scheduling latency.
• A goroutine doing tight CPU work without function calls (e.g., tight numeric loops in an AI embedding pipeline or a custom HMAC batch processor) may not be at an async-safe point when SIGURG fires, causing the preemption attempt to be deferred.
• The 10 ms threshold means any goroutine that legitimately needs more than 10 ms of CPU will be preempted at least once, adding at minimum one context switch per such goroutine.

This creates a latency floor for CPU-bound goroutines and a latency ceiling problem for goroutines waiting in the run queue behind CPU-bound ones.

A Concrete Failure Mode: The P Starvation Pattern

Consider a Go service fronting an AI inference backend. Request handling is I/O-light; the expensive operation is a synchronous call to a local embedding model via a C library through cgo. cgo calls release the P (entersyscall), so the M detaches from P and the P is free. But: the cgo call returns quickly for small inputs and holds M for 2–15 ms for large inputs.

During a traffic spike:

1. Many Ms enter cgo, releasing their Ps—Ps are briefly available.
2. The scheduler assigns those Ps to queued Gs immediately.
3. When cgo returns, those Ms need a P to resume. If all Ps are occupied, the M blocks waiting for a P (handoffp path).
4. The scheduler may spin a new thread (up to GOMAXPROCS + thread limit), increasing M count.
5. More Ms than Ps exist; OS scheduling overhead climbs; latency widens.

The operational signal is runtime/metrics: /sched/latencies:seconds histogram widening at the 95th–99th percentile while CPU is not fully saturated by Go code—it is saturated by the OS scheduler managing surplus threads.

Measuring Scheduler Behavior in Production

import (
    "runtime/metrics"
    "time"
)

func collectSchedulerMetrics() map[string]any {
    descs := []metrics.Sample{
        {Name: "/sched/latencies:seconds"},
        {Name: "/sched/goroutines:goroutines"},
        {Name: "/godebug/non-default-behavior/asynctimerchan:events/sec"},
    }
    metrics.Read(descs)
    result := make(map[string]any, len(descs))
    for _, d := range descs {
        switch d.Value.Kind() {
        case metrics.KindFloat64Histogram:
            h := d.Value.Float64Histogram()
            result[d.Name] = p99(h)
        case metrics.KindUint64:
            result[d.Name] = d.Value.Uint64()
        }
    }
    return result
}

func p99(h *metrics.Float64Histogram) float64 {
    total := uint64(0)
    for _, c := range h.Counts {
        total += c
    }
    target := total * 99 / 100
    cumulative := uint64(0)
    for i, c := range h.Counts {
        cumulative += c
        if cumulative >= target {
            return h.Buckets[i+1]
        }
    }
    return h.Buckets[len(h.Buckets)-1]
}

Expose this through your metrics pipeline (Prometheus gauge or histogram via OpenTelemetry bridge). /sched/latencies:seconds P99 above 1 ms in a latency-sensitive service is a signal worth acting on before the traffic cliff hits.

Tradeoffs and Mitigations

Bound goroutine fan-out. A semaphore-based worker pool limits the number of in-flight Gs to a value the P count can actually schedule without LRQ overflow. This converts scheduling jitter into bounded queuing delay, which is easier to reason about and surface in metrics.

type BoundedPool struct {
    sem chan struct{}
}

func NewBoundedPool(concurrency int) *BoundedPool {
    return &BoundedPool{sem: make(chan struct{}, concurrency)}
}

func (p *BoundedPool) Submit(ctx context.Context, fn func()) error {
    select {
    case p.sem <- struct{}{}:
        go func() {
            defer func() { <-p.sem }()
            fn()
        }()
        return nil
    case <-ctx.Done():
        return ctx.Err()
    }
}

Set concurrency to GOMAXPROCS * N where N is empirically tuned to keep LRQs below ~50% capacity under peak load.

GOMAXPROCS tuning in containers. The Go runtime reads GOMAXPROCS from the OS CPU count by default, which in a containerized environment returns the node CPU count, not the cgroup limit. Use go.uber.org/automaxprocs or set GOMAXPROCS explicitly in your container entrypoint. Overprovisioned Ps generate more threads than CPUs available, worsening context-switch overhead under saturation.

Isolate CPU-bound work. If a service mixes I/O goroutines with CPU-bound goroutines (batch embedding, crypto, compression), the CPU-bound goroutines hold Ps for long stretches, delaying I/O goroutines even after their I/O completes. Run CPU-bound work in a dedicated goroutine pool with its own concurrency bound, or offload to a separate process/service to prevent scheduler interference.

Profile with execution tracer at saturation, not at steady state. runtime/trace captures scheduler events per G. Running a 5-second trace during a controlled load spike will show run-queue wait times per goroutine, P idle gaps, and work-stealing events. go tool trace renders this as a timeline. Schedule latency problems are invisible in CPU profiles and in steady-state traces.

Decision Framework

When diagnosing latency anomalies in a Go service that worsen non-linearly with load:

1. Check /sched/latencies:seconds P99 first. If it is climbing proportionally with request rate but CPU is not pegged, the bottleneck is scheduling, not computation.
2. Check goroutine count trend. Unbounded go spawning under a spike is the common cause. Apply a bounded pool.
3. Verify GOMAXPROCS matches cgroup limit. Misconfigured P count is the highest-leverage fix when running in containers.
4. Separate CPU-bound and I/O-bound goroutine populations. Do not let embedding, hashing, or compression goroutines share Ps with request-handling goroutines under the same scheduler.
5. Use execution tracing at synthetic peak load, not profiling at steady state. The artifacts that matter—run queue depth, P idle time, steal frequency—are invisible without the tracer.
6. Treat GRQ growth as a leading indicator. GODEBUG=schedtrace=1000 emits GRQ length; a growing GRQ means LRQs are overflowing and global lock contention is accumulating.

The Go scheduler is sophisticated enough to handle a wide range of workloads without tuning. The cases where it fails predictably are bounded: goroutine fan-out without backpressure, GOMAXPROCS misconfiguration, and CPU/I/O workload mixing without isolation. Recognizing which of the three is active during a saturation event determines whether the correct response is a pool, a configuration change, or an architectural separation.