Trace Context in Multi-Threaded Environments

Q: Does Python's contextvars work across os.fork() and multiprocessing?

No. contextvars are not propagated across os.fork() or multiprocessing.Pool. Serialize the active context to a dict using the W3C TraceContext propagator's inject() method, pass it as an argument to the child process, and call extract() to reconstitute it on the other side.

Q: What causes stale context to bleed between unrelated requests in a thread pool?

Thread-local storage that is not explicitly cleared between task executions will retain the previous request's context. Use the SDK's Context.makeCurrent() in a try-with-resources block so the scope is always closed, or call Context.root().makeCurrent() at the start of each task to guarantee a clean slate before attaching the correct captured context.

Q: How do I prevent unclosed spans when an async task is cancelled?

Wrap the coroutine body in a try/finally block and call span.end() with status CANCELLED in the finally clause. In Python asyncio, use asyncio.shield() around critical span-closing logic so a CancelledError does not prevent the finally block from running.

When a single inbound request fans out across a thread pool or async runtime, the spans that describe each parallel unit of work frequently appear as disconnected root spans in your tracing backend rather than as children of the originating request. Engineers chasing this bug see inflated root-span counts in Jaeger, latency histograms that do not reflect actual end-to-end request time, and service maps with phantom entry points that should never exist. The underlying cause is always the same: the execution scope carrying the active trace ID was not transferred across the thread or event-loop boundary.

Prerequisites

Before applying the patterns on this page, make sure you have:

OpenTelemetry SDK initialized with at least one configured propagator and a working OTLP exporter
W3C TraceContext propagation enabled as the default header format across all services involved
A basic understanding of span lifecycle and parent-child relationships, particularly how parent_span_id links spans into a tree
Language runtime version: Java 11+, Python 3.7+ (contextvars), or Node.js 12.17+ (AsyncLocalStorage)

How Context Gets Lost at Thread Boundaries

The OpenTelemetry SDK stores the active span in an execution-scoped context object, not in a global variable. For synchronous, blocking threads this context lives in a thread-local slot. When your application dispatches work to a thread pool — via ExecutorService.submit() in Java, ThreadPoolExecutor in Python, or worker_threads in Node.js — the new thread starts with an empty context by default. The SDK has no mechanism to automatically clone the caller’s active scope into the worker, so the first tracer.startSpan() call inside the worker creates a new root span with a fresh, unrelated trace ID.

The diagram below traces the propagation lifecycle from a web handler through a thread pool dispatch, showing where context is captured, where it would be lost without explicit wrapping, and where it is restored inside the worker.

The same failure pattern applies to message queue consumers, background job runners, and any other mechanism that separates work submission from work execution.

Step-by-Step Implementation

Step 1 — Initialize with Explicit Propagation Formats

Configure the SDK to enforce deterministic header parsing rather than relying on auto-negotiation. In mixed-protocol environments, silent propagator mismatches are a common source of context loss.

# opentelemetry-sdk-config.yaml
otel.traces.propagators: tracecontext,b3multi
otel.traces.sampler: parentbased_traceidratio
otel.traces.sampler.arg: "1.0"
otel.traces.exporter: otlp
otel.exporter.otlp.endpoint: http://collector:4317
otel.bsp.max_export_batch_size: 512
otel.bsp.schedule_delay: 5000

The parentbased_traceidratio sampler is critical here: it ensures that if the inbound request was sampled, all child spans — including those spawned in worker threads — inherit that sampling decision. Without it, workers may independently sample at a different rate, producing traces with arbitrarily missing spans.

Step 2 — Java: Wrap ExecutorService and CompletableFuture

Never pass raw lambdas or Runnable instances directly to a thread pool. Capture Context.current() at the submission site and restore it inside the worker.

import io.opentelemetry.context.Context;
import io.opentelemetry.context.Scope;

public final class TracedRunnable implements Runnable {
    private final Context capturedContext;
    private final Runnable delegate;

    public TracedRunnable(Runnable delegate) {
        // Capture context at submission time, not execution time
        this.capturedContext = Context.current();
        this.delegate = delegate;
    }

    @Override
    public void run() {
        // Restore captured scope for the duration of the worker task
        try (Scope scope = capturedContext.makeCurrent()) {
            delegate.run();
        }
        // Scope is closed here even on exception — no context leak
    }
}

// Usage
ExecutorService pool = Executors.newFixedThreadPool(10);

// Incorrect: lambda gets an empty context in the worker
pool.submit(() -> processOrder(orderId));

// Correct: context is transferred
pool.submit(new TracedRunnable(() -> processOrder(orderId)));

For CompletableFuture, wrap the executor itself using Context.taskWrapping() from the OpenTelemetry SDK. This automatically wraps every task submitted to the pool without requiring per-call boilerplate:

import io.opentelemetry.context.Context;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

ExecutorService basePool = Executors.newFixedThreadPool(10);
// Wrap once at construction time — all submits inherit context automatically
ExecutorService tracedPool = Context.taskWrapping(basePool);

CompletableFuture.supplyAsync(() -> processOrder(orderId), tracedPool)
    .thenApplyAsync(result -> enrichResult(result), tracedPool);

Context.taskWrapping() is available in opentelemetry-api 1.x and later. Use it wherever you construct a long-lived pool.

Step 3 — Python: Use copy_context() for Thread Dispatch

Python’s contextvars module provides deterministic context copying. The copy_context() call takes a snapshot of all active context variables at the call site, including the OpenTelemetry span stored in context_api._RUNTIME_CONTEXT.

from concurrent.futures import ThreadPoolExecutor
from contextvars import copy_context
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def process_item(item_id: str) -> dict:
    # This span correctly appears as a child because copy_context
    # transferred the parent span's context variable into this thread
    with tracer.start_as_current_span("process_item") as span:
        span.set_attribute("item.id", item_id)
        return fetch_and_enrich(item_id)

def handle_request(item_ids: list[str]) -> list[dict]:
    with tracer.start_as_current_span("handle_request"):
        # Capture context snapshot here, before ThreadPoolExecutor takes over
        ctx = copy_context()
        with ThreadPoolExecutor(max_workers=8) as executor:
            # Each future runs inside the copied context
            futures = [
                executor.submit(ctx.run, process_item, item_id)
                for item_id in item_ids
            ]
            return [f.result() for f in futures]

ctx.run(fn, *args) executes fn(*args) inside the copied context, so the worker sees the same Token-based span attachment that the main thread had at submission time.

Important: contextvars are not propagated across os.fork() or multiprocessing.Pool. For cross-process propagation, serialize the context using the propagator’s inject() method, pass it as a plain dict argument, and call extract() on the other side:

from opentelemetry.propagate import inject, extract

# Serialise context for cross-process transfer
carrier: dict = {}
inject(carrier)  # e.g. {"traceparent": "00-abc123-..."}

# In child process or queue consumer:
ctx = extract(carrier)
with tracer.start_as_current_span("child_process_work", context=ctx):
    do_work()

Step 4 — Node.js: AsyncLocalStorage and worker_threads

Node.js propagates context across event-loop ticks via AsyncLocalStorage. The OpenTelemetry Node.js SDK uses this internally, but you must initialize the storage before any async operation begins — typically at application startup.

const { AsyncLocalStorage } = require('async_hooks');
const { context, trace } = require('@opentelemetry/api');

// Wrap the handler in the active context so all async continuations
// automatically inherit the current span
async function handleRequest(req, res) {
  const span = trace.getTracer('my-service').startSpan('handle_request');
  const ctx = trace.setSpan(context.active(), span);

  await context.with(ctx, async () => {
    // All awaits inside here inherit ctx — no manual re-attachment needed
    const results = await Promise.all(
      req.body.items.map(id => processItem(id))
    );
    res.json(results);
    span.end();
  });
}

async function processItem(id) {
  // context.active() correctly returns the parent span context here
  const span = trace.getTracer('my-service').startSpan('process_item');
  try {
    return await fetchItem(id);
  } finally {
    span.end();
  }
}

For worker_threads, AsyncLocalStorage does not cross the thread boundary automatically. Serialize the active context via the propagator, pass it through workerData, and reconstruct it inside the worker:

const { Worker, workerData, isMainThread, parentPort } = require('worker_threads');
const { context, propagation } = require('@opentelemetry/api');

if (isMainThread) {
  // Serialize active context into a plain object
  const carrier = {};
  propagation.inject(context.active(), carrier);

  const worker = new Worker(__filename, {
    workerData: { carrier, itemId: '42' }
  });
  worker.on('message', result => console.log(result));
} else {
  // Reconstruct context from carrier in the worker thread
  const ctx = propagation.extract(context.active(), workerData.carrier);
  context.with(ctx, () => {
    const span = trace.getTracer('my-service')
      .startSpan('worker_process_item');
    // span.parentSpanId correctly references the main thread's span
    doWork(workerData.itemId);
    span.end();
    parentPort.postMessage('done');
  });
}

Verification

Use the SDK’s in-memory exporter to run a deterministic concurrency test before promoting to staging:

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from concurrent.futures import ThreadPoolExecutor
from contextvars import copy_context

exporter = InMemorySpanExporter()
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(exporter))
tracer = provider.get_tracer("test")

NUM_WORKERS = 50

with tracer.start_as_current_span("root") as root_span:
    root_trace_id = root_span.get_span_context().trace_id
    root_span_id  = root_span.get_span_context().span_id
    ctx = copy_context()
    with ThreadPoolExecutor(max_workers=NUM_WORKERS) as pool:
        futs = [pool.submit(ctx.run, lambda: tracer.start_as_current_span("child").__enter__()) 
                for _ in range(NUM_WORKERS)]
        [f.result() for f in futs]

spans = exporter.get_finished_spans()
child_spans = [s for s in spans if s.name == "child"]

assert len(child_spans) == NUM_WORKERS, "Some child spans are missing"
for s in child_spans:
    assert s.context.trace_id == root_trace_id, \
        f"Mismatched trace_id: {hex(s.context.trace_id)}"
    assert s.parent.span_id == root_span_id, \
        f"Wrong parent: {hex(s.parent.span_id)}"

print(f"PASS — all {NUM_WORKERS} child spans correctly linked to root")

You can also verify in Jaeger’s UI by searching for your traceId and confirming the waterfall view shows all worker spans as children of the ingress span with no orphaned roots.

Edge Cases and Gotchas

Capturing context inside the lambda instead of at submission time. If you call Context.current() inside the worker function body rather than at the point where you call executor.submit(), you will capture the context of whichever thread happens to execute the lambda first — which is non-deterministic under load. Always capture context before the dispatch call.
Thread pool keep-alive retaining stale context. Long-lived threads in a cached pool carry thread-local storage from previous tasks into subsequent ones. After the scope closes (via try-with-resources or try/finally), thread-local slots are cleared — but only if the SDK’s ThreadLocalContextStorage is in use. Confirm your SDK version uses scope-based cleanup, not manual Context.root() resets.
Unawaited Promises in Node.js creating orphaned spans. Calling tracer.startSpan() inside a Promise that is never awaited creates a span with no reliable end time. The event loop may flush the span processor before span.end() is called. Always await all Promises that contain tracing calls, or attach .finally(() => span.end()).
asyncio task cancellation leaving spans open. When asyncio.Task.cancel() is raised as CancelledError inside a coroutine, any span that was started but not ended will remain open in the batch processor’s buffer, consuming memory and skewing latency percentiles. Wrap coroutine bodies in try/finally with explicit span.end() calls, and use asyncio.shield() to protect the finally block from a second cancellation.
Forked process inheriting a half-open OTLP connection. After os.fork(), the child process inherits the parent’s gRPC channel to the OTLP collector in an undefined state. Always call TracerProvider.shutdown() before forking (e.g., in a gunicorn pre_fork hook) and reinitialize the SDK in the child. Otherwise the child’s span export will silently fail or corrupt in-flight batches.
CompletableFuture chaining losing context between stages. When you chain .thenApplyAsync() without passing an explicit executor, Java uses ForkJoinPool.commonPool(), which is untraced. Context captured in stage N will not automatically flow to stage N+1. Pass tracedPool explicitly to every async stage in the chain.

Performance and Scale Notes

Thread-local and contextvars-based context storage adds negligible overhead for most workloads — typically under 0.5% CPU when spans are sampled at 100%. The main cost drivers are:

Span processor flush rate. The BatchSpanProcessor defaults to a 5-second schedule delay and a 512-span batch cap. Under high concurrency (thousands of worker tasks per second), reduce otel.bsp.schedule_delay to 2000 ms and increase otel.bsp.max_queue_size to 4096 to avoid dropped spans when the queue fills.
Context copy overhead in Python. copy_context() deep-copies all ContextVar entries, not just the tracing ones. If your application stores large objects in context variables, this copy can become measurable. Prefer storing references (IDs, handles) rather than data payloads in context variables.
AsyncLocalStorage in Node.js. The overhead scales with the number of active async resources, not the number of requests. In applications with tens of thousands of concurrent open sockets or timers, AsyncLocalStorage can add 2–5% CPU. Use AsyncLocalStorage.disable() during benchmark baselines to measure the true delta.
Sampling at the thread boundary. For head-based sampling, the sampling decision is attached to the TraceFlags byte in the context object and flows with it automatically — no extra work is needed. The span itself is never started for unsampled traces, so the thread-wrapping overhead for those requests reduces to a single Context.current() read.

For auto-instrumented services, the agent handles executor wrapping transparently for known frameworks. The manual patterns above are needed only for custom thread dispatch or frameworks not covered by the agent’s bytecode instrumentation.

Handling async spans correctly in Python and Node.js is covered in depth on the async boundaries page, which includes FastAPI and Express-specific patterns.

Troubleshooting FAQ

Why do child spans appear as disconnected root spans in Jaeger?

This happens when a Runnable or coroutine is dispatched without capturing the caller’s active context. The worker thread starts a new root span because no parent context was attached at construction time. Wrap the dispatch with TracedRunnable (Java) or copy_context().run() (Python), or use Context.taskWrapping() on the executor so context is transferred automatically on every submit().

Does Python’s contextvars work across os.fork() and multiprocessing?

No. contextvars are not propagated across os.fork() or multiprocessing.Pool. Serialize the active context to a carrier dict using the propagator’s inject() method, pass it as a plain argument to the child process or pool task, and call extract() to reconstitute it on the other side.

What causes stale context to bleed between unrelated requests in a thread pool?

Thread-local storage that is not cleared between task executions will retain the previous request’s context. Use Context.makeCurrent() in a try-with-resources block so the scope is always closed when the task finishes. If stale context is still leaking, call Context.root().makeCurrent() at the start of each task to guarantee a clean baseline before attaching the correct captured context.

How much overhead does AsyncLocalStorage add in Node.js?

Benchmarks show roughly 1–2% additional CPU on high-throughput event loops. For the vast majority of services this is negligible, but profile before enabling it on latency-sensitive hot paths that already operate near hardware limits.

How do I prevent unclosed spans when an async task is cancelled?

Wrap the coroutine body in a try/finally block and call span.end() with status CANCELLED in the finally clause. In Python asyncio, use asyncio.shield() around critical span-closing logic so a CancelledError does not abort the finally block before span.end() is called.

Related

Handling Async Boundaries in Node.js and Python — coroutine-level context management for asyncio and the Node.js event loop
Auto-Instrumentation vs Manual Span Creation — when agents handle executor wrapping automatically and when you need manual control
OpenTelemetry SDK Setup for Backend Services — SDK initialization, resource attributes, and exporter configuration
Context Propagation Across Service Meshes — sidecar-level header forwarding and mTLS trust boundary considerations
Span Lifecycle and Parent-Child Relationships — how parent_span_id links spans into a trace tree

↑ Back to SDK Implementation & Context Propagation