Opentelemetry

What it does

When your agent uses it

• Instrumenting a service
• Pointing OTLP at Grafana Cloud
• Switching from Jaeger / Datadog / New Relic
• Choosing head- vs tail-sampling

Example prompts

• “spans aren”
• “— even when the user says”
• “send traces to Grafana”
• “/opentelemetry”

Requirements

• Python 3
• Node.js
• Docker
• A credential in API_KEY
• A credential in GRAFANA_CLOUD_API_KEY

Workflow steps

5 steps, taken from the step headings in SKILL.md.

1. 1Authenticate to the Grafana Cloud OTLP endpoint
2. 2Auto-instrument a Java app + verify
3. 3Auto-instrument a Python app
4. 4Add Alloy as a buffering / sampling collector
5. 5Kubernetes — auto-inject via the Operator

What it can do on your machine

Read from SKILL.md and the folder at commit 1ccacf2. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  Shell commands in SKILL.md call:

  • curl
  • java
  • pip
  • kubectl

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  Hosts in commands or code, which the agent is likely to contact:

  • otlp-gateway-prod-us-east-0.grafana.net
  • github.com

  Also links to:

  • grafana.com
  • opentelemetry.io

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names these keys or tokens, usually read from environment variables:

  • API_KEY
  • GRAFANA_CLOUD_API_KEY

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Opentelemetry loads about 1.7k tokens when it runs, and up to ~12k if it reads all its reference files. Until then it costs about 232 tokens; SKILL.md has 259 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~232

When it runs · the whole SKILL.md, loaded when a task matches

~1.7k

With references · SKILL.md plus every file in references/, read only if the agent opens them

~12k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from grafana/skills at commit 1ccacf2, republished under its Apache-2.0 licence (© grafana). 259 words, ~1,720 tokens.

name

opentelemetry

description

Instrument any app with OpenTelemetry and ship metrics / logs / traces to Grafana Cloud or self-hosted Mimir / Loki / Tempo / Pyroscope. Covers SDK auto-instrumentation for Go, Java (Grafana JVM agent), Python (`opentelemetry-instrument`), Node.js, .NET (`Grafana.OpenTelemetry`), Beyla eBPF for zero-code; Grafana Cloud OTLP gateway + Basic-auth (instanceID + API key, base64); env-var config (`OTEL_EXPORTER_OTLP_*`, `OTEL_RESOURCE_ATTRIBUTES`); Alloy / OTel-Collector pipelines; Kubernetes Operator inject-annotations; and head + tail sampling. Use when instrumenting a service, pointing OTLP at Grafana Cloud, switching from Jaeger / Datadog / New Relic, choosing head- vs tail-sampling, or debugging "spans aren't showing in Explore" — even when the user says "auto-instrument my Java app", "send traces to Grafana", "what env vars do I set", "OTLP endpoint", or "Operator inject" without naming OpenTelemetry.

license

Apache-2.0

# 1. Build the auth header
INSTANCE_ID=123456
API_KEY="glc_eyJ..."
export OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic $(echo -n "${INSTANCE_ID}:${API_KEY}" | base64)"
export OTEL_RESOURCE_ATTRIBUTES="service.name=myapp,service.namespace=myteam,deployment.environment=prod"

# 2. Smoke-test creds with a curl POST against the OTLP traces endpoint (empty body)
curl -s -o /dev/null -w "%{http_code}\n" \
  -X POST -H "Content-Type: application/x-protobuf" \
  -H "Authorization: Basic $(echo -n "${INSTANCE_ID}:${API_KEY}" | base64)" \
  "$OTEL_EXPORTER_OTLP_ENDPOINT/v1/traces" --data-binary '\n'
# Expect 400 (malformed payload) — NOT 401 (auth) or 404 (wrong endpoint).

# 1. Download the Grafana JVM agent (single jar)
curl -sLO https://github.com/grafana/grafana-opentelemetry-java/releases/latest/download/grafana-opentelemetry-java.jar

# 2. Run with the agent + env from step 1
java -javaagent:./grafana-opentelemetry-java.jar -jar myapp.jar

# 3. Generate traffic, then verify in Grafana → Explore → Tempo:
#    TraceQL: { resource.service.name = "myapp" }
#    Expect spans within ~30s. Also verify metrics:
#    PromQL: count by (service_name)({service_name="myapp"})

pip install "opentelemetry-distro[otlp]"
opentelemetry-bootstrap -a install

# Same env vars as step 1, then:
opentelemetry-instrument python app.py

# Verify the same way — Explore → Traces filter service.name=myapp.

# Application points at local Alloy (gRPC fastest)
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc

# Alloy environment for forwarding to Cloud
export GRAFANA_CLOUD_OTLP_ENDPOINT=https://otlp-gateway-prod-us-east-0.grafana.net/otlp
export GRAFANA_CLOUD_INSTANCE_ID=$INSTANCE_ID
export GRAFANA_CLOUD_API_KEY=$API_KEY
alloy run /etc/alloy/config.alloy

# Verify Alloy received and forwarded
curl -s http://localhost:12345/metrics | grep otelcol_exporter_sent_spans

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata: { name: my-instrumentation }
spec:
  exporter: { endpoint: http://otelcol:4317 }
  propagators: [tracecontext, baggage]
  java:
    image: us-docker.pkg.dev/grafanalabs-global/docker-grafana-opentelemetry-java-prod/grafana-opentelemetry-java:2.3.0-beta.1
  nodejs: {}
  python: {}

metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"
    # or: inject-nodejs, inject-python, inject-dotnet

# Verify the operator injected the agent
kubectl describe pod <pod> | grep -A2 'opentelemetry-auto-instrumentation'
# Then run the same Grafana Explore checks.

# Head sampling (cheap, decided at start; may lose rare errors)
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1   # 10%

© grafana, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 2 other files (references) in skills/grafana-core/opentelemetry of grafana/skills.

• SKILL.md
• references/collector-config.md
• references/instrumentation.md

Open the folder on GitHubat commit 1ccacf2