Observability Onboarding

Observability Onboarding

Instrument an application with the Elastic Distribution of OpenTelemetry (EDOT) and prove the telemetry arrived. The instrumentation change is only half the job: an application can be configured perfectly and still emit nothing, so this skill ends by querying Elasticsearch for the service's data rather than declaring success from a config diff.

The scope is application instrumentation with the EDOT SDKs. Deploying an EDOT Collector, collecting logs from files or infrastructure, and onboarding data through Elastic Agent, Fleet, or integration packages are separate ingest paths that this skill does not cover.

Once telemetry is flowing, use the observability-sre-triage skill to assess service health, observability-k8s-investigation for Kubernetes-layer failures, observability-service-reliability to define SLOs and alerts on the new signals, and observability-llm-obs for GenAI and agentic workloads.

<!-- begin-partial: preamble -->

Environment Configuration

This skill executes Elasticsearch operations through the elastic CLI. If the elastic CLI is not installed, tell the user what it is needed for. Do not guess credentials, call the HTTP API directly, or attempt other workarounds.

This skill references operations in HTTP-shorthand form (e.g., GET /, GET /_cat/indices, GET /{index}/_mapping, GET /{index}/_settings/index.mode, POST /_query). The Operations table at the end of this document maps each shorthand to the equivalent elastic CLI command — always use the CLI rather than calling the HTTP API directly.

<!-- end-partial: preamble -->

Analysis without cluster access

The CLI check above gates querying the cluster — it does not gate analysis. When the user has already supplied the evidence in their question (metric values, counts, status reasons, log lines, alert payloads, configuration), reason from that evidence and deliver the conclusion.

When you genuinely do need data the user has not provided, still say what you would check and how — name the specific query, index, and field that would settle the question — and then ask for CLI setup. An answer that names the check is useful without a cluster; one that only asks for setup is not.

Only the verification step in this skill talks to Elasticsearch; it runs through POST /_query. Everything else edits application code and configuration in the user's workspace.

Jobs to be done

• Decide whether a service needs fresh instrumentation or a migration off the classic Elastic APM agent
• Apply the EDOT configuration for the service's language and runtime
• Point telemetry at the correct OTLP destination and reject APM Server URLs
• Verify that traces, metrics, and logs land in Elasticsearch under the expected service.name
• Diagnose an instrumented service that is reporting nothing, or reporting from the wrong agent

Output discipline

Applies to every response produced under this skill.

• Make the edit. When the language, the current agent, and the destination are known, apply the change. Do not narrate a plan and ask for approval on each file; ask only when a decision genuinely cannot be inferred.
• Do not dump documentation. Link the relevant page and state the two or three rules that apply to this service. Pasting a whole setup guide into the reply is a defect.
• Verify before reporting success. "Configured correctly" is not the finding; "telemetry is arriving" is. If verification returns no rows, say the service is not reporting and give the most likely cause.
• Do not speculate past the evidence. If the verification query is empty, that is one fact with several possible causes — name the likeliest and how to distinguish it, rather than asserting one.
• End on the result. No trailing offers such as "want me to set up dashboards next?". Follow-up work belongs in a recommendations list, phrased as a recommendation.

Process: route the request

1. Identify the language and runtime. Read the project's dependency manifest — pom.xml or build.gradle, requirements.txt or pyproject.toml, *.csproj, composer.json. The decision: which language reference to open. Also note how the process starts (a Dockerfile ENTRYPOINT, a Kubernetes pod spec, a systemd unit), because that is where the agent attaches and where the environment variables must be set.

   Runtime	Reference	How EDOT attaches
   Java	references/java.md	-javaagent: flag or JAVA_TOOL_OPTIONS
   Python	references/python.md	opentelemetry-instrument entrypoint wrapper
   .NET	references/dotnet.md	builder.AddElasticOpenTelemetry() in startup
   PHP	references/php.md	native extension via OS package, then full process restart

   For a language not listed, follow the same shape — the required configuration and the following verification step are language-independent — and use the upstream EDOT SDK documentation for the attach mechanism.

2. Detect a classic Elastic APM agent. The decision: instrument path or migrate path. Each language reference lists the markers; the common ones are ELASTIC_APM_* environment variables and a language-specific Elastic APM package. If any marker is present, take the migrate path — do not layer EDOT on top.

3. Resolve the telemetry destination. The decision: which URL goes in OTEL_EXPORTER_OTLP_ENDPOINT. Two valid answers, and one common wrong one:

   • The Elastic managed OTLP endpoint for the deployment or Serverless project. Correct default.
   • An EDOT Collector the user already runs, when telemetry must be enriched, sampled, or fanned out before it reaches Elastic.
   • Never an APM Server URL. If the candidate contains apm-server, ends in port 8200, or has an /intake/v2/events path, it is the classic ingest endpoint and EDOT cannot use it. On the migrate path this is the single most common mistake — the old ELASTIC_APM_SERVER_URL value must not be carried over.

4. Apply the shared configuration below, then the language reference. The shared rules are the same for every runtime; the reference covers only what is language-specific.

5. Verify. Do not report success from the configuration alone — run the verification process.

Required configuration

Identical across every EDOT SDK. Set exactly three environment variables:

Two rules that hold for every language:

• Do not set OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER, or OTEL_LOGS_EXPORTER. The defaults are already correct. Setting them is the usual reason a service reports traces but no metrics or logs.
• Never run a classic Elastic APM agent and EDOT together on the same process. They double-instrument and produce inconsistent traces. Removal and replacement belong in the same change.