Langfuse Observability
jeremylongshore/tons-of-skills-marketplace
Set up comprehensive observability for Langfuse with metrics, dashboards, and alerts.
Monitoring and observability patterns for Prometheus metrics, Grafana dashboards, Langfuse v4 LLM tracing (astype, scorecurrentspan, shouldexportspan, LangfuseMedia), and drift detection.
$ npx skills add yonatangross/orchestkit --skill monitoring-observability -a claude-codeProject install by default; add -g for ~/.claude/skills/.
$ gh skill install yonatangross/orchestkit monitoring-observability --agent claude-codeProject scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).
$ git clone --depth 1 https://github.com/yonatangross/orchestkit.git skills-src && mkdir -p .claude/skills && cp -r skills-src/src/skills/monitoring-observability .claude/skills/monitoring-observability && rm -rf skills-srcUse ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.
Claude Code skills documentation · loads skills from .claude/skills/
Install the "monitoring-observability" agent skill from https://github.com/yonatangross/orchestkit/tree/main/src/skills/monitoring-observability into .claude/skills/monitoring-observability/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "monitoring-observability", then confirm the skill loads.Claude Code copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$skill-installer install https://github.com/yonatangross/orchestkit/tree/main/src/skills/monitoring-observabilityType this inside Codex. $skill-installer <name> installs a curated skill from openai/skills. The installer writes to $CODEX_HOME/skills (default ~/.codex/skills). Restart Codex if the skill does not show up.
$ npx skills add yonatangross/orchestkit --skill monitoring-observability -a codexProject install goes to .agents/skills/; add -g for ~/.codex/skills/.
$ gh skill install yonatangross/orchestkit monitoring-observability --agent codexProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/yonatangross/orchestkit.git skills-src && mkdir -p .agents/skills && cp -r skills-src/src/skills/monitoring-observability .agents/skills/monitoring-observability && rm -rf skills-srcUse ~/.agents/skills/ instead of .agents/skills for a personal install.
Codex skills documentation · loads skills from .agents/skills/
Install the "monitoring-observability" agent skill from https://github.com/yonatangross/orchestkit/tree/main/src/skills/monitoring-observability into .agents/skills/monitoring-observability/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "monitoring-observability", then confirm the skill loads.Codex copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add yonatangross/orchestkit --skill monitoring-observability -a cursorProject install goes to .agents/skills/; add -g for ~/.cursor/skills/.
$ gh skill install yonatangross/orchestkit monitoring-observability --agent cursorProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/yonatangross/orchestkit.git skills-src && mkdir -p .cursor/skills && cp -r skills-src/src/skills/monitoring-observability .cursor/skills/monitoring-observability && rm -rf skills-srcUse ~/.cursor/skills/ instead of .cursor/skills for a personal install.
Cursor skills documentation · loads skills from .cursor/skills/, .agents/skills/, .claude/skills/, .codex/skills/
Install the "monitoring-observability" agent skill from https://github.com/yonatangross/orchestkit/tree/main/src/skills/monitoring-observability into .cursor/skills/monitoring-observability/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "monitoring-observability", then confirm the skill loads.Cursor copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gemini skills install https://github.com/yonatangross/orchestkit.git --path src/skills/monitoring-observability--scope user (default) or --scope workspace; --path is the subfolder of the repo that holds the skill; --consent skips the security confirmation prompt.
$ npx skills add yonatangross/orchestkit --skill monitoring-observability -a gemini-cliProject install goes to .agents/skills/; add -g for ~/.gemini/skills/.
$ gh skill install yonatangross/orchestkit monitoring-observability --agent gemini-cliProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/yonatangross/orchestkit.git skills-src && mkdir -p .gemini/skills && cp -r skills-src/src/skills/monitoring-observability .gemini/skills/monitoring-observability && rm -rf skills-srcUse ~/.gemini/skills/ instead of .gemini/skills for a personal install, then run /skills reload.
Gemini CLI skills documentation · loads skills from .gemini/skills/, .agents/skills/
Install the "monitoring-observability" agent skill from https://github.com/yonatangross/orchestkit/tree/main/src/skills/monitoring-observability into .gemini/skills/monitoring-observability/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "monitoring-observability", then confirm the skill loads.Gemini CLI copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ gh skill install yonatangross/orchestkit monitoring-observabilityInstalls for Copilot at project scope by default; add --scope user for a personal install. Preview a skill first with gh skill preview. Needs GitHub CLI 2.90.0 or later (public preview).
$ npx skills add yonatangross/orchestkit --skill monitoring-observability -a github-copilotProject install goes to .agents/skills/; add -g for ~/.copilot/skills/.
$ git clone --depth 1 https://github.com/yonatangross/orchestkit.git skills-src && mkdir -p .github/skills && cp -r skills-src/src/skills/monitoring-observability .github/skills/monitoring-observability && rm -rf skills-srcUse ~/.copilot/skills/ instead of .github/skills for a personal install. Commit .github/skills so cloud agent and code review can use it.
GitHub Copilot skills documentation · loads skills from .github/skills/, .claude/skills/, .agents/skills/
Install the "monitoring-observability" agent skill from https://github.com/yonatangross/orchestkit/tree/main/src/skills/monitoring-observability into .github/skills/monitoring-observability/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "monitoring-observability", then confirm the skill loads.GitHub Copilot copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
$ npx skills add yonatangross/orchestkit --skill monitoring-observability -a opencodeOpenCode documents no install command of its own. Project install goes to .agents/skills/; add -g for ~/.config/opencode/skills/.
$ gh skill install yonatangross/orchestkit monitoring-observability --agent opencodeProject scope by default (.agents/skills/); add --scope user for a personal install.
$ git clone --depth 1 https://github.com/yonatangross/orchestkit.git skills-src && mkdir -p .opencode/skills && cp -r skills-src/src/skills/monitoring-observability .opencode/skills/monitoring-observability && rm -rf skills-srcUse ~/.config/opencode/skills/ instead of .opencode/skills for a personal install.
OpenCode skills documentation · loads skills from .opencode/skills/, .claude/skills/, .agents/skills/
Install the "monitoring-observability" agent skill from https://github.com/yonatangross/orchestkit/tree/main/src/skills/monitoring-observability into .opencode/skills/monitoring-observability/ in this project. Copy the whole folder (SKILL.md and every file beside it), keep the folder name "monitoring-observability", then confirm the skill loads.OpenCode copies the folder itself, the same result as the manual copy. Check what it changed before you commit it.
monitoring-observabilityMonitoring and observability patterns for Prometheus metrics, Grafana dashboards, Langfuse v4 LLM tracing (astype, scorecurrentspan, shouldexportspan, LangfuseMedia), and drift detection.
Monitoring Observability is an agent skill from yonatangross/orchestkit. Monitoring and observability patterns for Prometheus metrics, Grafana dashboards, Langfuse v4 LLM tracing (astype, scorecurrentspan, shouldexportspan, LangfuseMedia), and drift detection. Use when adding logging, metrics, distributed tracing, LLM cost tracking, or quality drift monitoring.
Its SKILL.md is about 2.2k tokens, which your agent loads only when the skill is triggered. The skill folder holds 30 other files, including scripts and reference files (for example `examples/orchestkit-monitoring-dashboard.md`, `metadata.json` and `references/dashboards.md`). Compatibility notes: Claude Code 2.1.277+.
It sits in DevOps & Cloud, covering Monitoring and alerting, Observability and LLM observability. It works with Prometheus, Langfuse and Grafana. The repository describes itself as: The Complete AI Development Toolkit for Claude Code. 106 skills, 36 agents, 171 hooks. Install ork for stable (v9.x), or ork-alpha for the v10 line, which ships daily. The licence is MIT.
Read from SKILL.md and the folder at commit 0ef71d2. It shows what the files ask for, not the result of running them.
Pre-approves these tools, so the agent can use them without asking each time:
ReadGlobGrepWebFetchWebSearchFrom allowed-tools in the SKILL.md frontmatter.
Ships 1 file in scripts/, which the agent can run.
From the folder's file list and the shell code blocks in SKILL.md.
Links to these hosts (documentation or services it may open):
langfuse.comprometheus.iografana.comopentelemetry.ioevidentlyai.comitl.nist.govstructlog.orgFrom URLs in SKILL.md, links to its own repository left out.
Names no API keys, tokens, secrets or passwords.
From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.
Claude Code 2.1.277+.
From compatibility in the SKILL.md frontmatter.
Monitoring Observability loads about 2.2k tokens when it runs, and up to ~16k if it reads all its reference files. Until then it costs about 80 tokens; SKILL.md has 678 words of instructions outside code blocks.
Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.
The automated check found no risky patterns in SKILL.md.
Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); the scripts in this folder are not scanned.
The full file from yonatangross/orchestkit at commit 0ef71d2, republished under its MIT licence (© yonatangross). 678 words, ~2,222 tokens.
.claude/skills/monitoring-observability/SKILL.md (or your agent's skills folder). This skill also uses 27 other files; get the full folder from GitHub.A wrap around Prometheus, Grafana, OpenTelemetry and Langfuse, not a re-teaching of them. This
skill carries OrchestKit's delta (version floors, house decisions, scars) and points at the
vendor for everything else. Start at references/ork-delta.md.
These topics are fully covered first-party. Read the source, do not add a local copy.
| Topic | First-party source |
|---|---|
| Prometheus metric types, RED method, cardinality, PromQL | https://prometheus.io/docs/practices/ |
| Alertmanager grouping, inhibition, escalation, runbooks | https://prometheus.io/docs/alerting/latest/configuration/ |
| Grafana dashboards, Loki and LogQL, Promtail | https://grafana.com/docs/ |
| OpenTelemetry spans, sampling, context propagation | https://opentelemetry.io/docs/ |
Langfuse Python SDK (@observe, as_type, score_current_span, should_export_span, LangfuseMedia) | https://langfuse.com/docs/sdk/python |
| Langfuse v2 to v4 Python and v3 to v5 JS migration paths | https://langfuse.com/docs/sdk/python/v4-migration |
| Langfuse self-hosting (ClickHouse, Redis, S3, Helm) | https://langfuse.com/docs/deployment/self-host |
| Langfuse cost tracking, model pricing, Metrics API v2 | https://langfuse.com/docs/model-usage-and-cost |
| Langfuse scores, online evaluators, annotation queues, prompt management | https://langfuse.com/docs/scores/overview |
| Langfuse framework integrations (LangChain, LangGraph, CrewAI, Pydantic AI, Bedrock, LiveKit) | https://langfuse.com/docs/integrations |
| Agent Graphs, observation types, rendered tool calls | https://langfuse.com/docs/tracing-features/agent-graphs |
| PSI, KS test, KL and JS divergence, Wasserstein, embedding drift | https://www.evidentlyai.com/blog/data-drift-detection-large-datasets |
| EWMA control charts | https://www.itl.nist.gov/div898/handbook/pmc/section3/pmc324.htm |
| structlog, Winston, correlation IDs, log sampling | https://www.structlog.org/en/stable/ |
| Category | Rules | Impact | When to Use |
|---|---|---|---|
| Infrastructure Monitoring | 1 | CRITICAL | Grafana dashboards, Golden Signals, SLO/SLI |
| LLM Observability | 1 | HIGH | Langfuse tracing, observation types, agent graphs |
| Silent Failures | 3 | HIGH | Tool skipping, quality degradation, loop/token spike alerting |
Total: 5 rules across 3 categories. Drift detection, cost tracking, eval scoring, Prometheus instrumentation and alert-rule authoring moved to the upstream sources listed above.
# Langfuse v4 LLM tracing: semantic as_type plus inline scoring
from langfuse import observe, get_client
@observe(as_type="generation", name="analyze_content")
async def analyze_content(content: str):
get_client().update_current_trace(
user_id="user_123", session_id="session_abc",
tags=["production", "orchestkit"],
)
result = await llm.generate(content)
get_client().score_current_span(name="response_quality", value=0.85)
return result# Prometheus RED method, wired the way this repo expects (bounded labels only)
from prometheus_client import Counter, Histogram
http_requests = Counter('http_requests_total', 'Total requests', ['method', 'endpoint', 'status'])
http_duration = Histogram('http_request_duration_seconds', 'Request latency',
buckets=[0.01, 0.05, 0.1, 0.5, 1, 2, 5])Dashboard and health-check patterns. Metric instrumentation and alert-rule syntax are upstream.
| Rule | File | Key Pattern |
|---|---|---|
| Grafana Dashboards | rules/monitoring-grafana.md | Golden Signals, SLO/SLI, health checks |
CC 2.1.161 — OTEL resource attributes as metric labels:
OTEL_RESOURCE_ATTRIBUTESvalues are now attached as labels on metric datapoints, so usage metrics can be sliced by custom dimensions (team, repo, environment). Add label selectors to dashboards for multi-tenant / per-team cost and usage tracking.
Langfuse-based tracing for LLM applications. Cost tracking, scoring and drift statistics are upstream; what stays here is how this repo wires traces.
| Rule | File | Key Pattern |
|---|---|---|
| Langfuse Traces | rules/llm-langfuse-traces.md | @observe decorator, OTEL spans, agent graphs |
Detection and alerting for silent failures in LLM agents.
| Rule | File | Key Pattern |
|---|---|---|
| Tool Skipping | rules/silent-tool-skipping.md | Expected vs actual tool calls, Langfuse traces |
| Quality Degradation | rules/silent-degraded-quality.md | Heuristics + LLM-as-judge, z-score baselines |
| Silent Alerting | rules/silent-alerting.md | Loop detection, token spikes, escalation workflow |
CC 2.1.169 — OTEL client-cert paths require trust: untrusted project settings can no longer set OTEL client-certificate paths without a trust confirmation. If your OTEL exporter uses client certs configured in project
.claude/settings.json, expect a one-time trust prompt on first use in an untrusted project — telemetry silently not flowing after 2.1.169 is usually this gate, not the collector.
| Decision | Recommendation | Rationale |
|---|---|---|
| Metric methodology | RED method (Rate, Errors, Duration) | Industry standard, covers essential service health |
| Log format | Structured JSON | Machine-parseable, supports log aggregation |
| Tracing | OpenTelemetry | Vendor-neutral, auto-instrumentation, broad ecosystem |
| LLM observability | Langfuse (not LangSmith) | Open-source, self-hosted, built-in prompt management |
| LLM tracing API | @observe(as_type=...) + score_current_span() | v4: semantic types, inline scoring, span filtering |
| Langfuse APIs | Observations API v2 + Metrics API v2 | v4 (Mar 2026): faster querying, aggregations at scale |
| Hook telemetry transport | JSONL under ~/.claude/analytics/, never an SDK in-process | Hooks are per-event processes; SDK init would be paid on every spawn (references/ork-delta.md) |
| Resource | Description |
|---|---|
references/ork-delta.md | Start here. Floors, house decisions and scars that upstream docs do not carry |
references/langfuse-js-v5.md | JS/TS SDK v5 delta from Python 4.x: package map, phantom packages, SpanProcessor vs exporter. Read before writing any JS Langfuse code |
references/experiments-api.md | Langfuse experiments and dataset runs as this repo uses them |
references/evaluation-scores.md | Score shapes and scoring pipeline wiring |
references/session-tracking.md | Session and user grouping across multi-step workflows |
references/metrics-collection.md | Claude Code OTEL metric inventory and collector-side joins |
references/dashboards.md | Dashboard layout conventions |
references/structured-logging.md | Structured log field conventions |
references/dev-agent-lens.md | LiteLLM proxy layer for API-boundary observability |
examples/orchestkit-monitoring-dashboard.md | Worked monitoring dashboard example |
scripts/ | Templates: Prometheus, OpenTelemetry, health checks, Langfuse |
defense-in-depth - Layer 8 observability as part of security architecturedevops-deployment - Observability integration with CI/CD and Kubernetesresilience-patterns - Monitoring circuit breakers and failure scenariosllm-evaluation - Evaluation patterns that integrate with Langfuse scoringcaching - Caching strategies that reduce costs tracked by Langfuse© yonatangross, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file
SKILL.md and 27 other files (scripts, references) in src/skills/monitoring-observability of yonatangross/orchestkit.
Open the folder on GitHubat commit 0ef71d2
Monitoring Observability next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.
| Skill | Stars | Used in | Tokens | Auto-check | Licence | Repo updated |
|---|---|---|---|---|---|---|
| Monitoring Observability this skillyonatangross/orchestkit | 289 | — | ~2.2k | Automated safety check: Pass | MIT | |
| Langfuse Observabilityjeremylongshore/tons-of-skills-marketplace | 2.8k | — | ~2.2k | Automated safety check: Pass | MIT | |
| Ag2 Telemetryag2ai/build-with-ag2 | 252 | — | ~1.9k | Automated safety check: Pass | Apache-2.0 | |
| Cost Exportruvnet/ruflo | 74k | 1 repos | ~687 | Automated safety check: Notes | MIT | |
| Archestra Dev Observabilityarchestra-ai/archestra | 4.4k | — | ~1.2k | Automated safety check: Pass | Custom licence | |
| Frontmcp Observabilityagentfront/frontmcp | 146 | — | ~4.6k | Automated safety check: Pass | Apache-2.0 |
jeremylongshore/tons-of-skills-marketplace
Set up comprehensive observability for Langfuse with metrics, dashboards, and alerts.
ag2ai/build-with-ag2
Add OpenTelemetry traces to an AG2 beta Agent via TelemetryMiddleware (autogen.beta.middleware.builtin).
ruvnet/ruflo
Export cost-tracking telemetry in Prometheus textfile or webhook JSON formats — for external observability (Grafana, Datadog, custom dashboards)
archestra-ai/archestra
A skill your agent uses when changing Archestra tracing, metrics, OpenTelemetry, Tempo, Grafana, Prometheus, LLM/MCP spans, observability labels, or local observability setup.
agentfront/frontmcp
A skill your agent uses when adding tracing, structured logging, metrics, or monitoring to a FrontMCP server.
ahmedasmar/devops-claude-skills
Monitoring and observability strategy, implementation, and troubleshooting.
yonatangross/orchestkit
API contract design for REST and GraphQL, covering resource shape, URL and header versioning with deprecation windows, RFC 9457 Problem Details error handling, and OpenAPI specs.
yonatangross/orchestkit
ADR templates in the Nygard format with context, decision, consequences, and alternatives.
yonatangross/orchestkit
Single-pass codebase analysis leveraging a 1M-token context window for comprehensive security scanning, architecture review, and dependency auditing.
yonatangross/orchestkit
Structured review processes, conventional comments, language-specific checklists, and feedback templates.
yonatangross/orchestkit
Creates GitHub pull requests with pre-flight validation, conventional title formatting, and structured summary generation.
yonatangross/orchestkit
Multi-angle codebase exploration spawning 3-5 parallel agents for code structure, data flow, architecture patterns, and health assessment.
Works with
Categories
Monitoring and observability patterns for Prometheus metrics, Grafana dashboards, Langfuse v4 LLM tracing (astype, scorecurrentspan, shouldexportspan, LangfuseMedia), and drift detection. Monitoring Observability is an agent skill from yonatangross/orchestkit. Monitoring and observability patterns for Prometheus metrics, Grafana dashboards, Langfuse v4 LLM tracing (astype, scorecurrentspan, shouldexportspan, LangfuseMedia), and drift detection.
Monitoring Observability fits situations like: distributed tracing; LLM cost tracking; quality drift monitoring.
Run `npx skills add yonatangross/orchestkit --skill monitoring-observability -a claude-code`. Or copy the skill folder (src/skills/monitoring-observability in yonatangross/orchestkit) into .claude/skills/monitoring-observability in your project. Claude Code loads it when a task matches its description.
Run `npx skills add yonatangross/orchestkit --skill monitoring-observability -a codex`. Or copy the skill folder (src/skills/monitoring-observability in yonatangross/orchestkit) into .agents/skills/monitoring-observability in your project. Codex loads it when a task matches its description.
Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add yonatangross/orchestkit --skill monitoring-observability -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/monitoring-observability, .gemini/skills/monitoring-observability, .github/skills/monitoring-observability and .opencode/skills/monitoring-observability in your project.
SKILL.md names no scripts, command-line tools or credentials: Monitoring Observability is instructions for the agent only. Our summary lists: Python 3. Its frontmatter pre-approves these tools: Read, Glob, Grep, WebFetch, WebSearch. Compatibility (from SKILL.md): Claude Code 2.1.277+..
SKILL.md names 7 domains. As links in the text: langfuse.com, prometheus.io, grafana.com, opentelemetry.io, evidentlyai.com, itl.nist.gov and structlog.org. This is read from the text; nothing was executed.
Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. The check reads SKILL.md only: the scripts in the folder are not scanned, so read them before running anything.
Monitoring Observability is published under the MIT licence (declared in SKILL.md). It allows redistribution, so the full SKILL.md is shown on this page.
About 2.2k tokens (SKILL.md is roughly 8.9k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full. Its references folder adds about 13k tokens, read only when the agent opens those files.
Skills that share tags, products or a category with Monitoring Observability: Langfuse Observability (jeremylongshore/tons-of-skills-marketplace, 2.8k stars), Ag2 Telemetry (ag2ai/build-with-ag2, 252 stars), Cost Export (ruvnet/ruflo, 74k stars) and Archestra Dev Observability (archestra-ai/archestra, 4.4k stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.
yonatangross (a GitHub user) maintains it in yonatangross/orchestkit, which has 289 GitHub stars. The repository holds 108 skills in this directory. The repository was last updated on October 7, 2026.
Source: yonatangross/orchestkit on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.