Opentelemetry Net Instrumentation

OpenTelemetry .NET Instrumentation Skill

When to Use

• Adding OpenTelemetry instrumentation to .NET code (traces, metrics, logs)
• Creating or modifying ActivitySources, Meters, or ILogger usage
• Setting up the OpenTelemetry SDK, resources, exporters, or sampling
• Reviewing telemetry implementations for spec compliance
• Optimizing instrumentation performance
• Designing telemetry APIs that become part of the public surface
• Implementing context propagation across service boundaries

Architecture: .NET Is Different

CRITICAL: The .NET OpenTelemetry implementation is fundamentally different from other platforms. .NET provides tracing, metrics, and logging APIs in the framework itself. That means OTel does not provide a separate instrumentation API — it uses the built-in .NET APIs and acts as the collection/export layer.

The Three Built-in .NET APIs (Primary — Zero Dependencies)

These are the primary and only APIs library authors should use for instrumentation. They ship with the .NET runtime — no NuGet packages required.

The OTel Collection/Export Layer (Secondary — Application Root Only)

OTel NuGet packages are the collection and export layer, added only at the application composition root (not in libraries):

Package Decision Guide

Before adding ANY OpenTelemetry NuGet package, discuss the trade-off with the user:

"You're about to add an OTel NuGet package. Is this an application where you need to export telemetry to an observability backend (Jaeger, Prometheus, OTLP collector)? If you're writing a library, you likely need zero OTel packages — just use System.Diagnostics.ActivitySource / System.Diagnostics.Metrics.Meter and let the consuming application configure the export pipeline. Do you want to proceed?"

Library authors: Add nothing. Use only System.Diagnostics.* and ILogger. The consuming application wires up the SDK and exporters.

Application authors: Add OpenTelemetry.Extensions.Hosting + the exporters and instrumentation libraries you need. See sdk-resources-and-logs-reference.md for full setup patterns.

Never add OpenTelemetry.Api to a library — System.Diagnostics.* IS the API.

For SDK setup, resource configuration, exporters, sampling, and logs integration, see sdk-resources-and-logs-reference.md.

Core Principles

Resiliency First

CRITICAL: Exceptions in diagnostic/tracing/metrics logic MUST NEVER impact application processing.

• Assume Activity instances can be null. Always protect against null Activity references except in Activity extension methods (use activity?.ExtensionMethod())
• Guard all instrumentation code with appropriate null checks

API Surface Awareness

• Any telemetry emitted becomes part of the public API surface
• Changes are subject to breaking changes guidelines
• Telemetry should be emitted by default (users opt-in to collection via OpenTelemetry extensions)
• Exception: High-cardinality metric dimensions may require explicit opt-in

Standards Compliance

• Follow Microsoft best practices for distributed tracing instrumentation
• Follow OpenTelemetry semantic conventions
• Attribute values support: string, boolean, double (IEEE 754), int64, byte arrays, and homogeneous arrays of these primitive types. Null/empty values are valid and meaningful per the OTel AnyValue spec — they MUST be stored and passed to exporters.
• Attribute keys must be non-null, non-empty strings

Traces / Spans (Activities)

ActivitySource Setup

// ✅ CORRECT: Use ActivitySource, not DiagnosticSource
public class MyFeature
{
    // Primary ActivitySource - name typically matches the component or NuGet package name
    private static readonly ActivitySource ActivitySource = new("MyApp.MyComponent", "1.0.0");

    // Specialized ActivitySource for opt-in scenarios
    private static readonly ActivitySource DetailedActivitySource = new("MyApp.MyComponent.Detailed", "1.0.0");
}

Rules:

• Every component defines a primary ActivitySource for mainstream activities
• Name typically matches the component or NuGet package (e.g., "MyCompany.MyLibrary")
• Version the ActivitySource using SemVer
• Create separate ActivitySources for specialized or opt-in scenarios. Use hierarchical source names, e.g. MyCompany.MyLibrary and MyCompany.MyLibrary.Detailed, so consuming applications can subscribe only to the sources they want via AddSource(...) and backends can filter by instrumentation scope.

Creating Activities

// ✅ Check HasListeners, null-check, then guard expensive work behind IsAllDataRequested
if (ActivitySource.HasListeners())
{
    using var activity = ActivitySource.StartActivity("ProcessItem", ActivityKind.Internal);
    if (activity != null && activity.IsAllDataRequested)
    {
        activity.DisplayName = "Processing order #12345";
        activity.SetTag("app.item_id", itemId);
        activity.SetTag("app.item_type", itemType);
    }
}

// ❌ WRONG: Don't start activities in fire-and-forget tasks where the
// using scope ends before the async work completes (AsyncLocal context is lost)
async Task HelperAsync()
{
    using var activity = ActivitySource.StartActivity("Helper");
    _ = Task.Run(() => DoWorkAsync()); // ❌ activity disposed before task completes
}

Rules:

• Check ActivitySource.HasListeners() before creating (zero-allocation fast path)
• Always null-check Activity after creation (listener may filter or sample it out)
• Never start activities in async helper methods (Activity.Current uses AsyncLocal)
• Guard expensive tag computation behind activity.IsAllDataRequested
• Use W3C TraceContext. .NET Core 3.0+ / .NET 5+ uses it by default; older TFMs or .NET Framework apps may set Activity.DefaultIdFormat = ActivityIdFormat.W3C at startup and use Activity.ForceDefaultIdFormat = true to override hierarchical parents.

Activity Naming

// ✅ Unique operation name, friendly display name (null-check before accessing)
using var activity = ActivitySource.StartActivity(
    name: "ProcessItem",              // Unique, identifies class of spans
    kind: ActivityKind.Internal
);
if (activity != null)
    activity.DisplayName = "Processing order #12345"; // User-friendly, can be specific

// ❌ WRONG: Don't include runtime data in operation name
using var badActivity = ActivitySource.StartActivity($"Process_{itemId}"); // ❌

Rules:

• Each span type has unique OperationName (identifies statistically interesting class of spans)
• Operation name should NOT contain runtime data (only compile/config-time info)
• Use human-readable DisplayName for specifics
• Follow OpenTelemetry span naming conventions

SpanKind Selection

Choose the correct ActivityKind to clarify the span's role in distributed tracing:

Rules:

• A single span SHOULD NOT serve more than one purpose
• Create the outgoing span before injecting its SpanContext into the request. If you inject first, the parent's context propagates instead and the outgoing span ends up dangling (no connection to the downstream call).
• See traces-and-propagation-reference.md for detailed SpanKind guidance with examples

Span Attributes (Tags)

// ✅ Application code: use your own namespace
activity?.SetTag("myapp.order_id", orderId);
activity?.SetTag("myapp.payment.status", "confirmed");

// ✅ Manual infrastructure instrumentation: use semantic conventions
// activity?.SetTag("db.system.name", "postgresql"); // custom database client
// activity?.SetTag("http.request.method", "GET"); // custom HTTP transport

// Values can be strings, numbers, booleans, or homogeneous arrays
activity?.SetTag("app.item_count", 42);
activity?.SetTag("app.related_ids", new int[] { 1, 2, 3 });

// ❌ WRONG: PascalCase, hyphen delimiter, plural, or unrelated namespace
activity?.SetTag("MyApp.OrderId", orderId);     // ❌ Wrong case
activity?.SetTag("myapp.order-id", orderId);    // ❌ Wrong delimiter

Rules:

• Namespace prefix matching your component: myapp.*, myapp.db.*
• All lowercase, underscore (_) delimiters, singular form
• Attribute values: string, boolean, double, int64, byte arrays, homogeneous arrays (null/empty valid per AnyValue spec)
• Business/domain attributes: use your own namespace (myapp.*).
• HTTP, database, messaging, or RPC concepts you manually instrument: use semantic conventions. Do not duplicate attributes already emitted by auto-instrumentation. Do not use OTel namespaces as prefixes for custom attributes.

Activity Status and Errors

try
{
    await ProcessItemAsync(); // ✅ success: leave status Unset, do not call SetStatus(Ok) — see rules below
}
catch (Exception ex)
{
    if (activity != null)
    {
        activity.SetStatus(ActivityStatusCode.Error, ex.Message); // modern API
        activity.SetTag("error.type", ex.GetType().FullName);
    }
    throw;
}

Rules (per Recording Errors and the Set Status API spec):

• Leave span status Unset on success — do not call SetStatus(ActivityStatusCode.Ok).
• Ok is for application code, not instrumentation libraries. The trace API spec: "Instrumentation Libraries SHOULD NOT set the status code to Ok, unless explicitly configured to do so (...) Application developers and Operators may set the status code to Ok" — typically to override a library-reported Error they've decided isn't a real failure (e.g. suppressing a noisy 404). Once set, Ok is final; later calls are ignored."
• On failure: SHOULD set ActivityStatusCode.Error, SHOULD set the error.type tag, SHOULD set the status description to the exception message.
• Use SetStatus — legacy otel.status_code/otel.status_description tags are no longer needed.
• Do not record errors that were retried or handled and let the operation complete gracefully — Error status and error.type describe operations that failed, not ones that recovered.

Recording Exceptions — Keep Emitting Span Events, Add the Logs Opt-In

exceptions-spans — the convention behind Activity.AddEvent(new ActivityEvent("exception", ...)) — carries a Deprecated status in favor of exceptions-logs, but the .NET ecosystem has not followed yet: no OpenTelemetry.* .NET package implements the spec's OTEL_SEMCONV_EXCEPTION_SIGNAL_OPT_IN opt-in (verified against OpenTelemetry.Api 1.17.0), and OpenTelemetry.Instrumentation.AspNetCore 1.17.0 itself still records exceptions via Activity.AddException, i.e. span events. That means the exporters, backends, and dashboards a typical .NET user has wired up today are built to read exception span events, not log-based exceptions. Keep implementing span events as the default — dropping them because the underlying spec document is marked Deprecated will silently break exception visibility for anyone still on today's tooling. Layer the newer logs-based path on top as an opt-in, exactly as the spec's own transition guidance describes for instrumentations moving off span events:

// Mirrors the spec's OTEL_SEMCONV_EXCEPTION_SIGNAL_OPT_IN values: unset/anything else → spans only (today's default), "logs/dup" → both, "logs" → logs only.
private static readonly string? ExceptionSignalOptIn = Environment.GetEnvironmentVariable("OTEL_SEMCONV_EXCEPTION_SIGNAL_OPT_IN");
private static readonly bool EmitSpanEvents = ExceptionSignalOptIn != "logs";
private static readonly bool EmitLogs = ExceptionSignalOptIn is "logs" or "logs/dup";

try
{
    await ProcessItemAsync();
}
catch (Exception ex)
{
    activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
    activity?.SetTag("error.type", ex.GetType().FullName);

    if (EmitSpanEvents) // ✅ default — what current .NET tooling and dashboards actually consume today
    {
        activity?.AddEvent(new ActivityEvent("exception", tags: new ActivityTagsCollection
        {
            ["exception.type"] = ex.GetType().FullName,
            ["exception.message"] = ex.Message,
            ["exception.stacktrace"] = ex.ToString()
        }));
    }

    if (EmitLogs) // opt-in — the spec's forward direction; pass the exception instance while `activity` is still Activity.Current so the SDK derives trace_id/span_id and exception.type/message/stacktrace from it
    {
        logger.LogError(ex, "Item processing failed");
    }

    throw;
}

Rules:

• Don't set exception.escaped on the span event — it's deprecated outright: "no longer recommended to record exceptions that are handled and do not escape the scope of a span."
• Support OTEL_SEMCONV_EXCEPTION_SIGNAL_OPT_IN (logs / logs/dup) so consumers who are ready can opt into logs or dual-emission, but default to spans-only — this is the spec's own transition guidance, not an optional nicety, and it exists precisely so nobody has to choose one signal over the other before they're ready.
• When emitting logs (opt-in or dual), pick severity per exceptions-logs: ERROR for unhandled exceptions (especially on SERVER/CONSUMER spans), WARN for exceptions expected to be handled by the caller (especially on CLIENT/PRODUCER spans), DEBUG for exceptions that don't indicate an actual issue (e.g., a request cancelled client-side), FATAL only when the exception causes application shutdown.
• The log call MUST happen while the span it's associated with is still current — the spec requires "exception events emitted by instrumentations that also record spans for the same operation MUST be associated with the corresponding span context." Logging from a detached error-reporting callback after the using activity scope has ended silently correlates to the wrong span, or none. See Accessing Activities below.
• Only move to logs-only once you've held dual-emission for at least six months on a stable major version (the spec's own minimum) and confirmed your actual consumers read log-based exceptions — not on a fixed timeline alone.

See traces-and-propagation-reference for the full pattern, including how a library with no logging dependency in its core can still expose a callback so a separate OTel integration package can capture exception details while the right span is active.