OpenTelemetry standardizes generating and transporting traces and metrics. Spans represent related stages of an operation, helping teams locate latency and failures across services.

A trace commonly begins when a request enters a service and follows calls to databases, queues, and downstream APIs. Each span has timing, context, attributes, and a parent relationship. Those relationships reconstruct the operation across process boundaries. OpenTelemetry defines APIs, SDKs, semantic conventions, and OTLP; it is not the database or query interface for telemetry. A Collector and an observability backend fill those roles.

Separate the API, SDK, instrumentation, and Collector

The API is the contract used by application code and instrumented libraries. The SDK implements sampling, processing, and export. A reusable library should usually depend only on the API so that its host application controls the SDK and destination. An application configures the SDK once, early in startup, before concurrent work begins.

Automatic instrumentation wraps supported frameworks and clients to create spans at common boundaries. Manual instrumentation adds domain meaning around operations that matter. Use both deliberately. Inspect automatic spans before adding manual ones because two spans around the same HTTP call create noise and needless cost.

The Collector is a separate service that receives telemetry, runs processing pipelines, and exports to one or more destinations. It keeps backend credentials and routing logic out of every application and can provide batching, filtering, and memory protection. It does not remove the application's responsibility to avoid producing sensitive data.

Create a manual span

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("store.orders")


def calculate_total(items: list[int]) -> int:
    with tracer.start_as_current_span("calculate_total") as span:
        span.set_attribute("order.item_count", len(items))
        return sum(items)

Console export is useful for learning. In production, send OTLP to a Collector and use batch processing. Do not attach emails, tokens, request bodies, or sensitive order values.

An OTLP exporter over gRPC can be configured directly:

python -m pip install opentelemetry-exporter-otlp-proto-grpc
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace.export import BatchSpanProcessor

exporter = OTLPSpanExporter(endpoint="http://collector:4317", insecure=True)
provider.add_span_processor(BatchSpanProcessor(exporter))

Reserve insecure=True for a deliberately non-TLS local network, such as development. In production, authenticate the destination, validate certificates, and deliver credentials through a secrets mechanism. Standard environment variables such as OTEL_EXPORTER_OTLP_ENDPOINT are preferable when deployment configuration changes between environments.

BatchSpanProcessor exports outside the request path and suits services. SimpleSpanProcessor waits for every export, making it useful mainly in tests and debugging. Before a short-lived worker exits, shut down the provider cleanly so pending batches have a chance to leave.

Record status, exceptions, and events

A failed operation should have an error status, but not every high HTTP status means an internal failure. Follow the relevant semantic conventions. For manual spans, record exceptions without attaching secrets:

from opentelemetry.trace import Status, StatusCode

with tracer.start_as_current_span("order.reserve") as span:
    try:
        reserve_inventory()
    except OutOfStock as error:
        span.record_exception(error)
        span.set_status(Status(StatusCode.ERROR, "inventory unavailable"))
        raise

Exception messages can contain unexpected values. Review what your exception classes expose and redact before export when necessary. Span events fit point-in-time occurrences such as a retry. Do not turn every log line into an event, which duplicates volume without improving diagnosis.

Instrumentation libraries can create spans for frameworks and HTTP clients. Review the fields and compatible versions. Propagate context across service boundaries; a ContextVar is local and does not cross a network by itself. See Python contextvars.

Across the network, a propagator injects identifiers into headers and the receiver extracts them. W3C Trace Context uses traceparent and optionally tracestate. Never treat incoming trace context as authorization or identity; it supports correlation, not access control. At untrusted boundaries, constrain baggage because extra key-value pairs may travel through many services.

Queues need explicit handling. A producer injects context into message metadata, and a consumer extracts it before creating a span. Workers must not leak the context from a previous task. For batch processing, consult the applicable convention and consider links when several independent messages contribute to the new operation.

Design stable names and attributes

A span name describes an operation, not one instance. Prefer HTTP GET /orders/{id} or order.process over a complete URL or order ID. Useful attributes support grouping: method, normalized route, database system, or a result from a small set. UUIDs and other unique values increase cardinality and indexing cost. If an identifier is essential, a correlated access-controlled log may be a better home.

Do not attach:

  • authorization headers, cookies, or tokens;
  • complete request and response bodies;
  • email addresses, government identifiers, or personal addresses;
  • SQL statements with interpolated parameters;
  • file names or error messages that can contain customer data.

Define and test an attribute policy, then add defensive filtering in the Collector. Downstream filtering reduces exposure in the backend, but the value has already crossed the process and network. The strongest protection is not generating it.

Balance sampling, cost, and diagnosis

Sampling chooses which traces to record. A parent-based sampler respects an upstream decision and helps prevent fragmented traces; a ratio sampler limits volume. An initial decision cannot know whether the request will fail later, so an overly low rate can hide rare errors. Some Collectors and backends support tail sampling after more of the trace is visible, at an additional infrastructure cost.

Measure actual request rate, spans per trace, and attribute size before selecting a ratio. Keep a controlled debugging path, but do not let an external client force expensive sampling. In tests, use an in-memory exporter and assert names, relationships, and absence of prohibited data. Avoid assertions on random identifiers.

Plan for telemetry failures

Observability must not take down the application. Set queue limits and exporter timeouts, monitor dropped spans, and decide what happens when the Collector is unavailable. At the same time, silent failure leaves operators blind. Expose internal pipeline metrics and alert on sustained export loss.

Roll out instrumentation in stages. Begin with one service boundary, inspect exported data, estimate volume, and verify that propagation continues through a real downstream call. Add a dashboard or trace search workflow that answers a known incident question. This proves the signal is usable, not merely present. Document which team owns the Collector pipeline and which team owns span semantics. During package upgrades, compare span names and attributes because semantic convention changes can affect saved queries, sampling rules, and cost even when application behavior is unchanged.

Use stable names and controlled-cardinality attributes. Combine traces with structured logs and metrics because each signal answers different questions.

The official OpenTelemetry Python documentation and the official trace specification, accessed July 22, 2026, cover concepts, configuration, and signal status. Recheck component status before adopting experimental APIs and pin compatible instrumentation package versions.