Observability Architecture
majiayu000/litellm-rs
LiteLLM-RS Observability Architecture. An agent skill from majiayu000/litellm-rs.
Add OpenTelemetry traces to an AG2 beta Agent via TelemetryMiddleware (autogen.beta.middleware.builtin).
$ npx skills add ag2ai/build-with-ag2 --skill ag2-telemetry -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install ag2ai/build-with-ag2 ag2-telemetry --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/ag2ai/build-with-ag2.git skills-src && mkdir -p .claude/skills && cp -r skills-src/.agents/skills/ag2-telemetry .claude/skills/ag2-telemetry && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "ag2-telemetry" agent skill from https://github.com/ag2ai/build-with-ag2/tree/main/.agents/skills/ag2-telemetry into .claude/skills/ag2-telemetry/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "ag2-telemetry", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/ag2ai/build-with-ag2/tree/main/.agents/skills/ag2-telemetryType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add ag2ai/build-with-ag2 --skill ag2-telemetry -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install ag2ai/build-with-ag2 ag2-telemetry --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/ag2ai/build-with-ag2.git skills-src && mkdir -p .agents/skills && cp -r skills-src/.agents/skills/ag2-telemetry .agents/skills/ag2-telemetry && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "ag2-telemetry" agent skill from https://github.com/ag2ai/build-with-ag2/tree/main/.agents/skills/ag2-telemetry into .agents/skills/ag2-telemetry/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "ag2-telemetry", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add ag2ai/build-with-ag2 --skill ag2-telemetry -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install ag2ai/build-with-ag2 ag2-telemetry --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/ag2ai/build-with-ag2.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/.agents/skills/ag2-telemetry .cursor/skills/ag2-telemetry && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "ag2-telemetry" agent skill from https://github.com/ag2ai/build-with-ag2/tree/main/.agents/skills/ag2-telemetry into .cursor/skills/ag2-telemetry/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "ag2-telemetry", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/ag2ai/build-with-ag2.git --path .agents/skills/ag2-telemetry--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add ag2ai/build-with-ag2 --skill ag2-telemetry -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install ag2ai/build-with-ag2 ag2-telemetry --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/ag2ai/build-with-ag2.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/.agents/skills/ag2-telemetry .gemini/skills/ag2-telemetry && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "ag2-telemetry" agent skill from https://github.com/ag2ai/build-with-ag2/tree/main/.agents/skills/ag2-telemetry into .gemini/skills/ag2-telemetry/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "ag2-telemetry", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install ag2ai/build-with-ag2 ag2-telemetryInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add ag2ai/build-with-ag2 --skill ag2-telemetry -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/ag2ai/build-with-ag2.git skills-src && mkdir -p .github/skills && cp -r skills-src/.agents/skills/ag2-telemetry .github/skills/ag2-telemetry && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "ag2-telemetry" agent skill from https://github.com/ag2ai/build-with-ag2/tree/main/.agents/skills/ag2-telemetry into .github/skills/ag2-telemetry/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "ag2-telemetry", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add ag2ai/build-with-ag2 --skill ag2-telemetry -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install ag2ai/build-with-ag2 ag2-telemetry --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/ag2ai/build-with-ag2.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/.agents/skills/ag2-telemetry .opencode/skills/ag2-telemetry && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "ag2-telemetry" agent skill from https://github.com/ag2ai/build-with-ag2/tree/main/.agents/skills/ag2-telemetry into .opencode/skills/ag2-telemetry/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "ag2-telemetry", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
ag2-telemetryAdd OpenTelemetry traces to an AG2 beta Agent via TelemetryMiddleware (autogen.beta.middleware.builtin).
Ag2 Telemetry is an agent skill from ag2ai/build-with-ag2. Add OpenTelemetry traces to an AG2 beta Agent via TelemetryMiddleware (autogen.beta.middleware.builtin). Emits spans for the full turn, each LLM call, each tool execution, and each human-input request, following the OpenTelemetry GenAI semantic conventions. Compatible with any OTLP backend — Jaeger, Grafana Tempo, Datadog, Honeycomb, Langfuse. Use when the user wants production-grade traces, latency analysis, token-usage attribution, or to ship telemetry into an existing observability stack.
Its SKILL.md is about 1.9k tokens, which your agent loads only when the skill is triggered. It is a single SKILL.md file with no bundled scripts.
It sits in DevOps & Cloud, covering Observability, Monitoring and alerting and LLM observability. It works with OpenTelemetry, Langfuse, Datadog and Grafana. The repository describes itself as: Sample code and application showcases to get you going with AG2 (formally AutoGen). The licence is Apache-2.0.
Read from SKILL.md and the folder at commit 29eeac3. It shows what the files ask for, not the result of running them.
Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.
From allowed-tools in the SKILL.md frontmatter.
Shell commands in SKILL.md call:
pipFrom the folder's file list and the shell code blocks in SKILL.md.
Links to these hosts (documentation or services it may open):
opentelemetry.ioFrom URLs in SKILL.md, links to its own repository left out.
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Ag2 Telemetry loads about 1.9k tokens when it runs. Until then it costs about 129 tokens; SKILL.md has 531 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.
The full file from ag2ai/build-with-ag2 at commit 29eeac3, republished under its Apache-2.0 licence (© ag2ai). 531 words, ~1,852 tokens.
.claude/skills/ag2-telemetry/SKILL.md (or your agent's skills folder).The user wants to:
If they just want quick stdout debugging, point them at LoggingMiddleware instead (see ag2-middleware).
pip install "ag2[openai,tracing]"from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter
from autogen.beta import Agent
from autogen.beta.config import OpenAIConfig
from autogen.beta.middleware.builtin import TelemetryMiddleware
# 1. Configure OpenTelemetry
resource = Resource.create({"service.name": "ag2-beta-quickstart"})
tracer_provider = TracerProvider(resource=resource)
tracer_provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(tracer_provider)
# 2. Wire the middleware
agent = Agent(
"assistant",
prompt="You are a helpful assistant.",
config=OpenAIConfig(model="gpt-4o-mini"),
middleware=[
TelemetryMiddleware(
tracer_provider=tracer_provider,
agent_name="assistant",
),
],
)
# 3. Run — spans emit automatically
import asyncio
asyncio.run(agent.ask("What is the capital of France?"))For production, swap ConsoleSpanExporter for OTLPSpanExporter (or your backend's exporter) and SimpleSpanProcessor for BatchSpanProcessor.
Each ask() produces a root span with children:
invoke_agent assistant
├── chat gpt-4o-mini # LLM API call
├── execute_tool get_weather # tool execution
├── chat gpt-4o-mini # LLM call after tool result
└── await_human_input assistant # human-in-the-loopEvery span has an ag2.span.type attribute:
ag2.span.type | Operation name | Hook |
|---|---|---|
agent | invoke_agent | on_turn — full turn |
llm | chat | on_llm_call — each LLM call |
tool | execute_tool | on_tool_execution — each tool |
human_input | await_human_input | on_human_input — HITL |
Spans carry standard OpenTelemetry GenAI attributes:
| Attribute | Spans | Description |
|---|---|---|
gen_ai.operation.name | All | invoke_agent / chat / execute_tool / await_human_input |
gen_ai.agent.name | agent, human_input | Agent name |
gen_ai.provider.name | agent, llm | Auto-detected (openai, anthropic, …) |
gen_ai.request.model | agent, llm | e.g. gpt-4o-mini |
gen_ai.response.model | llm | Resolved from response |
gen_ai.response.finish_reasons | llm | e.g. ["stop"], ["tool_calls"] |
gen_ai.usage.input_tokens | llm | Prompt tokens |
gen_ai.usage.output_tokens | llm | Completion tokens |
gen_ai.usage.cache_creation_input_tokens | llm | Prompt-cache writes (Anthropic) |
gen_ai.usage.cache_read_input_tokens | llm | Prompt-cache reads (Anthropic, OpenAI, Gemini) |
gen_ai.tool.name | tool | Tool function name |
gen_ai.tool.call.id | tool | Tool call ID |
gen_ai.tool.type | tool | Always function |
By default, message content, tool args, and results are included on spans. Useful for debugging but can leak sensitive data:
TelemetryMiddleware(
tracer_provider=tracer_provider,
agent_name="assistant",
capture_content=False, # omit messages, tool args, results
)When enabled, additional attributes appear:
| Attribute | Span | Content |
|---|---|---|
gen_ai.input.messages | llm | JSON request messages |
gen_ai.output.messages | llm | JSON response messages |
gen_ai.tool.call.arguments | tool | Tool args (JSON) |
gen_ai.tool.call.result | tool | Tool result |
ag2.human_input.prompt | human_input | Prompt shown to human |
ag2.human_input.response | human_input | Human's response |
For privacy-sensitive backends (or anywhere telemetry leaves your infra), set capture_content=False.
| Parameter | Type | Default | Description |
|---|---|---|---|
tracer_provider | TracerProvider | None | Global provider | OpenTelemetry TracerProvider |
capture_content | bool | True | Include message/tool content in spans |
agent_name | str | None | "unknown" | Agent name for span attributes |
provider_name | str | None | None | Provider override (auto-detected if unset) |
model_name | str | None | None | Model override (auto-detected if unset) |
TelemetryMiddleware uses standard OpenTelemetry, so any OTLP-compatible backend works:
OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")For container-orchestrated stacks, this repo includes a tracing/ directory with Docker-Compose for otel-collector + Grafana Tempo.
website/docs/beta/telemetry.mdx — full attribute table, configuration, example.tracing/ — Docker setup for local otel-collector + Tempo + Grafana.ag2-middleware.SimpleSpanProcessor + ConsoleSpanExporter in production — synchronous, blocks every span emit. Use BatchSpanProcessor and a real exporter (OTLP / Jaeger / vendor) outside of dev.capture_content=True is the default. For privacy-sensitive prompts (PII, credentials), set capture_content=False and audit what your backend retains.trace.set_tracer_provider(...) — without it, tracer_provider you pass to the middleware is fine, but third-party libraries that auto-instrument may use a different provider.gen_ai.usage.* requires the provider client to surface usage in the response. Streaming providers may emit usage only at the end; if you don't see them, check the provider's response shape.instrument_agent() / instrument_llm_wrapper() / instrument_pattern(); beta uses TelemetryMiddleware).© ag2ai, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
Just SKILL.md in .agents/skills/ag2-telemetry of ag2ai/build-with-ag2.
Open the folder on GitHubat commit 29eeac3
Ag2 Telemetry next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| Ag2 Telemetry this skillag2ai/build-with-ag2 | 252 | — | ~1.9k | Automated safety check: Pass | Apache-2.0 | |
| Observability Architecturemajiayu000/litellm-rs | 117 | — | ~1.3k | Automated safety check: Pass | MIT | |
| Frontmcp Observabilityagentfront/frontmcp | 146 | — | ~4.6k | Automated safety check: Pass | Apache-2.0 | |
| Monitoring Observabilityahmedasmar/devops-claude-skills | 203 | — | ~3.9k | Automated safety check: Pass | None | |
| App Observabilitygrafana/skills | 279 | — | ~1.8k | Automated safety check: Pass | Apache-2.0 | |
| Monitoring Observabilityyonatangross/orchestkit | 289 | — | ~2.2k | Automated safety check: Pass | MIT |
majiayu000/litellm-rs
LiteLLM-RS Observability Architecture. An agent skill from majiayu000/litellm-rs.
agentfront/frontmcp
A skill your agent uses when adding tracing, structured logging, metrics, or monitoring to a FrontMCP server.
ahmedasmar/devops-claude-skills
Monitoring and observability strategy, implementation, and troubleshooting.
grafana/skills
Get RED metrics + service maps + frontend RUM + AI/LLM monitoring out of Grafana Cloud — Application Observability (tracesspanmetrics from OTel traces, p50/p95/p99 latency, exemplar-to-trace…
yonatangross/orchestkit
Monitoring and observability patterns for Prometheus metrics, Grafana dashboards, Langfuse v4 LLM tracing (astype, scorecurrentspan, shouldexportspan, LangfuseMedia), and drift detection.
TheBeardedBearSAS/claude-craft
OpenTelemetry, distributed tracing, structured logging, metrics (Prometheus, Grafana, Datadog).
ag2ai/build-with-ag2
Add a custom Python tool to an AG2 beta Agent using the @tool decorator.
ag2ai/build-with-ag2
Intercept the AG2 beta agent loop with BaseMiddleware — wrap full turns (onturn), each LLM call (onllmcall), each tool execution (ontoolexecution), or each human-input request (onhumaninput).
ag2ai/build-with-ag2
Wire AG2 beta's shipped tools into an Agent — both provider-native server-side tools (web search, web fetch, code execution, MCP, image generation, memory) and locally-executed common toolkits…
ag2ai/build-with-ag2
Persist agent state across runs, shape what the LLM sees per turn, and cap history to fit a context window.
ag2ai/build-with-ag2
Monitor an AG2 beta agent's stream — log events, detect repeated tool calls, track token spend, build trigger-driven observers, route observer alerts to the model, and halt on FATAL conditions.
ag2ai/build-with-ag2
Build a minimal AG2 beta Agent end to end — pick a model provider, set a prompt, call agent.ask(), then continue the conversation with reply.ask() (multi-turn).
Categories
Add OpenTelemetry traces to an AG2 beta Agent via TelemetryMiddleware (autogen.beta.middleware.builtin). Ag2 Telemetry is an agent skill from ag2ai/build-with-ag2.builtin).
Ag2 Telemetry fits situations like: the user wants production-grade traces; latency analysis; token-usage attribution; ship telemetry into an existing observability stack.
Run `npx skills add ag2ai/build-with-ag2 --skill ag2-telemetry -a claude-code`. Or copy the skill folder (.agents/skills/ag2-telemetry in ag2ai/build-with-ag2) into .claude/skills/ag2-telemetry in your project. Claude Code loads it when a task matches its description.
Run `npx skills add ag2ai/build-with-ag2 --skill ag2-telemetry -a codex`. Or copy the skill folder (.agents/skills/ag2-telemetry in ag2ai/build-with-ag2) into .agents/skills/ag2-telemetry in your project. Codex loads it when a task matches its description.
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add ag2ai/build-with-ag2 --skill ag2-telemetry -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/ag2-telemetry, .gemini/skills/ag2-telemetry, .github/skills/ag2-telemetry and .opencode/skills/ag2-telemetry in your project.
Going by SKILL.md and its folder, Ag2 Telemetry needs the command-line tools its instructions call (pip). Our summary lists: Python 3; Docker.
SKILL.md names 1 domain. As links in the text: opentelemetry.io. This is read from the text; nothing was executed.
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. Review the folder before installing.
Ag2 Telemetry is published under the Apache-2.0 licence (declared in SKILL.md). It allows redistribution, so the full SKILL.md is shown on this page.
About 1.9k tokens (SKILL.md is roughly 7.4k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full.
Skills that share tags, products or a category with Ag2 Telemetry: Observability Architecture (majiayu000/litellm-rs, 117 stars), Frontmcp Observability (agentfront/frontmcp, 146 stars), Monitoring Observability (ahmedasmar/devops-claude-skills, 203 stars) and App Observability (grafana/skills, 279 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
ag2ai (a GitHub organization) maintains it in ag2ai/build-with-ag2, which has 252 GitHub stars. The repository holds 16 skills in this directory. The repository was last updated on September 6, 2026.
Source: ag2ai/build-with-ag2 on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.