New: 98,000+ skills indexed from GitHub repos with 100+ stars · every skill safety-checkedBrowse top skillsTop skills
Esc to closeBrowse all skills
Submit a skillBrowse skills
Submit a skillBrowse skills

Agent skill

Langsmith Trace Analyzer

by soba-labs in soba-labs/langchain-agent-skills

Fetch, organize, and analyze LangSmith traces for debugging and evaluation.

MITAuto-check passedAI & LLM Engineering

Install Langsmith Trace Analyzer

skills CLI
$ npx skills add soba-labs/langchain-agent-skills --skill langsmith-trace-analyzer -a claude-code

Project install by default; add -g for ~/.claude/skills/.

GitHub CLI
$ gh skill install soba-labs/langchain-agent-skills langsmith-trace-analyzer --agent claude-code

Project scope by default; add --scope user for a personal install. Needs GitHub CLI 2.90.0 or later (public preview).

Manual copy
$ git clone --depth 1 https://github.com/soba-labs/langchain-agent-skills.git skills-src && mkdir -p .claude/skills && cp -r skills-src/skills/langsmith-trace-analyzer .claude/skills/langsmith-trace-analyzer && rm -rf skills-src

Use ~/.claude/skills/ instead of .claude/skills for a personal install. The folder must contain SKILL.md.

Claude Code skills documentation · loads skills from .claude/skills/

Facts

Skill name
langsmith-trace-analyzer
GitHub stars
107
Token cost
~1.4k tokens
SKILL.md length
365 words
Files
7 (incl. scripts, references)
Skills in repo
9
Repo updated
First seen
Licence
MIT

At a glance

Fetch, organize, and analyze LangSmith traces for debugging and evaluation.

  • Works in 3 steps: Download and organize traces → Analyze downloaded traces → Query traces correctly (SDK)
  • You need to: query traces/runs by project
  • SKILL.md covers Quick Start, Decision Guide, Core Workflows and Accuracy and Schema Notes, plus 3 more sections
  • Runs Python and TypeScript scripts from its folder; calls uv; needs LANGSMITH_API_KEY

What it does

Langsmith Trace Analyzer is an agent skill from soba-labs/langchain-agent-skills. Fetch, organize, and analyze LangSmith traces for debugging and evaluation. Use when you need to: query traces/runs by project, metadata, status, or time window; download traces to JSON; organize outcomes into passed/failed/error buckets; analyze token/message/tool-call patterns; compare passed vs failed behavior; or investigate benchmark and production failures.

Its SKILL.md is about 1.4k tokens, which your agent loads only when the skill is triggered. The skill folder holds 8 other files, including scripts and reference files (for example `references/analysis-patterns.md`, `references/benchmark-analysis.md` and `references/filtering-querying.md`).

It sits in AI & LLM Engineering, covering LLM observability. It works with LangSmith. The repository describes itself as: A collection of agent-optimized LangChain, LangGraph and LangSmith skills for AI coding assistants. The licence is MIT.

When your agent uses it

  • You need to: query traces/runs by project
  • Download traces to JSON
  • Organize outcomes into passed/failed/error buckets
  • Analyze token/message/tool-call patterns

Example prompts

  • “/langsmith-trace-analyzer”

Requirements

  • Python 3
  • Node.js
  • A credential in LANGSMITH_API_KEY

Workflow steps

3 steps, taken from the step headings in SKILL.md.

  1. Download and organize traces
  2. Analyze downloaded traces
  3. Query traces correctly (SDK)

What it can do on your machine

Read from SKILL.md and the folder at commit a2d4a10. It shows what the files ask for, not the result of running them.

  • Tool permissions

    Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

    From allowed-tools in the SKILL.md frontmatter.

  • Runs code

    Ships 3 files in scripts/ (Python and TypeScript), which the agent can run.

    Shell commands in SKILL.md call:

    • uv

    From the folder's file list and the shell code blocks in SKILL.md.

  • Network

    No URLs in SKILL.md. Its commands use uv, which can reach the network depending on how they are called.

    From URLs in SKILL.md, links to its own repository left out.

  • Credentials

    Names these keys or tokens, usually read from environment variables:

    • LANGSMITH_API_KEY

    From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Langsmith Trace Analyzer loads about 1.4k tokens when it runs, and up to ~9.9k if it reads all its reference files. Until then it costs about 98 tokens; SKILL.md has 365 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it
~98
When it runs · the whole SKILL.md, loaded when a task matches
~1.4k
With references · SKILL.md plus every file in references/, read only if the agent opens them
~9.9k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Auto-check passed

The automated check found no risky patterns in SKILL.md.

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); the scripts in this folder are not scanned.

SKILL.md

The full file from soba-labs/langchain-agent-skills at commit a2d4a10, republished under its MIT licence (© soba-labs). 365 words, ~1,401 tokens.

Download SKILL.mdSave it as .claude/skills/langsmith-trace-analyzer/SKILL.md (or your agent's skills folder). This skill also uses 6 other files; get the full folder from GitHub.
name
langsmith-trace-analyzer
description
Fetch, organize, and analyze LangSmith traces for debugging and evaluation. Use when you need to: query traces/runs by project, metadata, status, or time window; download traces to JSON; organize outcomes into passed/failed/error buckets; analyze token/message/tool-call patterns; compare passed vs failed behavior; or investigate benchmark and production failures.

LangSmith Trace Analyzer

Use this skill to move from raw LangSmith traces to actionable debugging/evaluation insights.

Quick Start

bash
# Install dependencies
uv pip install langsmith langsmith-fetch

# Auth
export LANGSMITH_API_KEY=<your_langsmith_api_key>
Fast workflow
  1. Download traces with scripts/download_traces.py (or scripts/download_traces.ts).
  2. Analyze downloaded JSON with scripts/analyze_traces.py.
  3. Load targeted references only when needed:
    • references/filtering-querying.md for query/filter syntax
    • references/analysis-patterns.md for deeper diagnostics
    • references/benchmark-analysis.md for benchmark-specific workflows

Decision Guide

  1. Known trace IDs
    Use langsmith-fetch trace <id> directly, or --trace-ids in downloader scripts.

  2. Need to discover traces first
    Use LangSmith SDK list_runs/listRuns with filters, then download selected trace IDs.

  3. Need aggregate insights
    Run analyze_traces.py for summary stats, patterns, and passed-vs-failed comparisons.

Core Workflows

1) Download and organize traces

Python:

bash
uv run skills/langsmith-trace-analyzer/scripts/download_traces.py \
  --project "my-project" \
  --filter "job_id=abc123" \
  --last-hours 24 \
  --limit 100 \
  --output ./traces \
  --organize

TypeScript:

bash
ts-node skills/langsmith-trace-analyzer/scripts/download_traces.ts \
  --project "my-project" \
  --filter "job_id=abc123" \
  --last-hours 24 \
  --limit 100 \
  --output ./traces

Output layout:

text
traces/
├── manifest.json
└── by-outcome/
    ├── passed/
    ├── failed/
    └── error/
        ├── GraphRecursionError/
        ├── TimeoutError/
        └── DaytonaError/

Notes:

  • Python script supports --organize/--no-organize.
  • Both scripts use SDK filtering plus langsmith-fetch for full trace payload export.
2) Analyze downloaded traces
bash
# Markdown report
uv run skills/langsmith-trace-analyzer/scripts/analyze_traces.py ./traces --output report.md

# JSON output
uv run skills/langsmith-trace-analyzer/scripts/analyze_traces.py ./traces --json

# Compare passed vs failed (expects by-outcome folders)
uv run skills/langsmith-trace-analyzer/scripts/analyze_traces.py ./traces --compare --output comparison.md

The analyzer reports:

  • message/tool-call/token/duration summaries
  • top tool usage
  • anomaly patterns (high message count, repeated tools, quick failures)
  • passed-vs-failed metric deltas when comparison is enabled
3) Query traces correctly (SDK)

Use official LangSmith run filter syntax via filter and/or start_time:

python
from datetime import datetime, timedelta, timezone
from langsmith import Client

client = Client()

start = datetime.now(timezone.utc) - timedelta(hours=24)
filter_query = 'and(eq(metadata_key, "job_id"), eq(metadata_value, "abc123"))'

runs = client.list_runs(
    project_name="my-project",
    is_root=True,
    start_time=start,
    filter=filter_query,
)

For TypeScript:

ts
import { Client } from "langsmith";

const client = new Client();
for await (const run of client.listRuns({
  projectName: "my-project",
  isRoot: true,
  filter: 'and(eq(metadata_key, "job_id"), eq(metadata_value, "abc123"))',
})) {
  console.log(run.id, run.status);
}

Accuracy and Schema Notes

  • LangSmith run fields are commonly top-level (status, error, total_tokens, start_time, end_time).
  • Some exported traces also include nested metadata (metadata or extra.metadata) and/or messages.
  • analyze_traces.py is resilient to multiple payload shapes, including raw array payloads.
  • For full conversation content, prefer downloaded trace payloads over bare list_runs results.
Show full SKILL.md (144 more words)Show less

Troubleshooting

IssueLikely CauseAction
LANGSMITH_API_KEY missingAuth not configuredexport LANGSMITH_API_KEY=<your_langsmith_api_key>
No runs returnedWrong project/filter/time rangeVerify project name and filter syntax
Empty/partial message arraysRun schema differs or incomplete dataUse downloaded trace JSON and inspect status/error fields
JSON parse error on downloaded filesBad/incomplete exportRe-download trace; use --format raw paths in scripts
Re-downloading same traces repeatedlyExisting files in nested foldersUse current scripts (they check existing files across output tree)

Safety for Open Source

  • Do not commit downloaded trace artifacts (manifest.json, trace JSON dumps) unless sanitized.
  • Trace payloads can contain user prompts, outputs, metadata, and other sensitive runtime data.
  • Keep this skill repository focused on scripts/templates, not production trace exports.

Resources

scripts/
  • scripts/download_traces.py: Python downloader + organizer
  • scripts/download_traces.ts: TypeScript downloader + organizer
  • scripts/analyze_traces.py: Offline analysis and reporting
references/
  • references/filtering-querying.md: LangSmith query/filter examples
  • references/analysis-patterns.md: Diagnostic patterns and heuristics
  • references/benchmark-analysis.md: Benchmark-oriented analysis

© soba-labs, MIT. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 6 other files (scripts, references) in skills/langsmith-trace-analyzer of soba-labs/langchain-agent-skills.

  • SKILL.md
  • references/analysis-patterns.md
  • references/benchmark-analysis.md
  • references/filtering-querying.md
  • scripts/analyze_traces.py
  • scripts/download_traces.py
  • scripts/download_traces.ts

Open the folder on GitHubat commit a2d4a10

Compare with similar skills

Langsmith Trace Analyzer next to the 5 skills that share the most tags, products or categories with it. Stars are the repository's; “used in” counts other GitHub owners with a copy.

Langsmith Trace Analyzer compared with similar skills
SkillStarsUsed inTokensAuto-checkLicenceRepo updated
Langsmith Trace Analyzer this skillsoba-labs/langchain-agent-skills107—~1.4kAutomated safety check: PassMIT
Langsmith Online Eval Engineeringlangchain-ai/langsmith-skills159—~1.4kAutomated safety check: PassMIT
Langsmith ObservabilityOrchestra-Research/AI-Research-SKILLs13k3 repos~2.4kAutomated safety check: PassMIT
LangSmith Trace DebuggingComposioHQ/awesome-claude-skills77k9 repos~2.7kAutomated safety check: PassNone
Migrate To Langfuselangfuse/skills299—~1.7kAutomated safety check: NotesMIT
Langchain Dependencieslangchain-ai/langchain-skills1.3k1 repos~3.6kAutomated safety check: PassMIT

Similar skills

  • Langsmith Online Eval Engineering

    langchain-ai/langsmith-skills

    Official

    Design, test, create, and attach LangSmith online evaluators for production traces or conversation threads.

    159 GitHub stars~1.4k tokensUpdated 5 days ago
    AI & LLM EngineeringAuto-check passed

  • Langsmith Observability

    Orchestra-Research/AI-Research-SKILLs

    LLM observability platform for tracing, evaluation, and monitoring.

    13k GitHub starsUsed in 3 repos~2.4k tokens
    AI & LLM EngineeringAuto-check passed

  • LangSmith Trace Debugging

    ComposioHQ/awesome-claude-skills

    Debugs LangChain and LangGraph agents by pulling recent execution traces with the langsmith-fetch CLI and reporting errors, tool calls, timings and token use.

    77k GitHub starsUsed in 9 repos~2.7k tokens
    AI & LLM EngineeringAuto-check passed

  • Migrate To Langfuse

    langfuse/skills

    Migrate to Langfuse from another LLM observability/evals platform (LangSmith, Arize AX, Phoenix, Braintrust, Helicone, Promptfoo, ...).

    299 GitHub stars~1.7k tokensUpdated 7 days ago
    AI & LLM EngineeringAuto-check: notes

  • Langchain Dependencies

    langchain-ai/langchain-skills

    Official

    INVOKE THIS SKILL when setting up a new project or when asked about package versions, installation, or dependency management for LangChain, LangGraph, LangSmith, or Deep Agents.

    1.3k GitHub starsUsed in 1 repo~3.6k tokens
    AI & LLM EngineeringAuto-check passed

  • Langsmith Custom Apps

    langchain-ai/langsmith-skills

    Official

    INVOKE THIS SKILL when building, iterating on, copying, or sharing a LangSmith Custom App — a React/TypeScript UI that runs inside LangSmith and reads the LangSmith API.

    159 GitHub stars~3.2k tokensUpdated 5 days ago
    AI & LLM EngineeringAuto-check passed

More from soba-labs/langchain-agent-skills

All 9 skills in this repo

  • Deepagents Planning Todos

    soba-labs/langchain-agent-skills

    Use the writetodos tool effectively for task planning and decomposition in Deep Agents.

    107 GitHub stars~2.3k tokensUpdated 1 mo ago
    Auto-check passed

  • Deepagents Setup Configuration

    soba-labs/langchain-agent-skills

    Initialize, validate, and troubleshoot Deep Agents projects in Python or JavaScript using the deepagents package.

    107 GitHub stars~1.9k tokensUpdated 1 mo ago
    Auto-check passed

  • Langgraph Agent Patterns

    soba-labs/langchain-agent-skills

    Implement multi-agent coordination patterns (supervisor-subagent, router, orchestrator-worker, handoffs) for LangGraph applications.

    107 GitHub stars~3.6k tokensUpdated 1 mo ago
    Auto-check passed

  • Langgraph Error Handling

    soba-labs/langchain-agent-skills

    Implement LangGraph error handling with current v1 patterns.

    107 GitHub stars~1.5k tokensUpdated 1 mo ago
    Auto-check passed

  • Langgraph Project Setup

    soba-labs/langchain-agent-skills

    Initialize and configure LangGraph projects with proper structure, langgraph.json configuration, environment variables, and dependency management.

    107 GitHub stars~2.4k tokensUpdated 1 mo ago
    Auto-check: notes

  • Langgraph State Management

    soba-labs/langchain-agent-skills

    Design state schemas, implement reducers, configure persistence, and debug state issues for LangGraph applications.

    107 GitHub stars~3.4k tokensUpdated 1 mo ago
    Auto-check passed

Works with

Categories

Questions about Langsmith Trace Analyzer

What does Langsmith Trace Analyzer do?

Fetch, organize, and analyze LangSmith traces for debugging and evaluation. Langsmith Trace Analyzer is an agent skill from soba-labs/langchain-agent-skills. Fetch, organize, and analyze LangSmith traces for debugging and evaluation.

When should I use Langsmith Trace Analyzer?

Langsmith Trace Analyzer fits situations like: you need to: query traces/runs by project; download traces to JSON; organize outcomes into passed/failed/error buckets; analyze token/message/tool-call patterns.

How do I install Langsmith Trace Analyzer in Claude Code?

Run `npx skills add soba-labs/langchain-agent-skills --skill langsmith-trace-analyzer -a claude-code`. Or copy the skill folder (skills/langsmith-trace-analyzer in soba-labs/langchain-agent-skills) into .claude/skills/langsmith-trace-analyzer in your project. Claude Code loads it when a task matches its description.

How do I install Langsmith Trace Analyzer in Codex?

Run `npx skills add soba-labs/langchain-agent-skills --skill langsmith-trace-analyzer -a codex`. Or copy the skill folder (skills/langsmith-trace-analyzer in soba-labs/langchain-agent-skills) into .agents/skills/langsmith-trace-analyzer in your project. Codex loads it when a task matches its description.

Can I use Langsmith Trace Analyzer in Cursor, Gemini CLI or GitHub Copilot?

Cursor, Gemini CLI, GitHub Copilot and OpenCode also load SKILL.md folders. With the skills CLI, run `npx skills add soba-labs/langchain-agent-skills --skill langsmith-trace-analyzer -a cursor` (or -a gemini-cli, github-copilot or opencode for the others). To copy it by hand, put the folder in .cursor/skills/langsmith-trace-analyzer, .gemini/skills/langsmith-trace-analyzer, .github/skills/langsmith-trace-analyzer and .opencode/skills/langsmith-trace-analyzer in your project.

What does Langsmith Trace Analyzer need to run?

Going by SKILL.md and its folder, Langsmith Trace Analyzer needs Python and TypeScript for the scripts in its folder, the command-line tools its instructions call (uv) and credentials named LANGSMITH_API_KEY. Our summary lists: Python 3; Node.js; A credential in LANGSMITH_API_KEY.

Does Langsmith Trace Analyzer access the network?

SKILL.md contains no URLs. Its commands use uv, which can reach the network depending on how they are called. This is read from the text; nothing was executed.

Is Langsmith Trace Analyzer safe to install?

Our automated static check of SKILL.md found no risky patterns, such as piping downloads into a shell, reading credential files or hidden Unicode. It is not a guarantee. The check reads SKILL.md only: the scripts in the folder are not scanned, so read them before running anything.

What licence does Langsmith Trace Analyzer use?

Langsmith Trace Analyzer is published under the MIT licence (the repository's licence). It allows redistribution, so the full SKILL.md is shown on this page.

How many tokens does Langsmith Trace Analyzer use?

About 1.4k tokens (SKILL.md is roughly 5.6k characters). Agents keep only the skill's name and description in context until a task matches; then they load SKILL.md in full. Its references folder adds about 8.5k tokens, read only when the agent opens those files.

What are the alternatives to Langsmith Trace Analyzer?

Skills that share tags, products or a category with Langsmith Trace Analyzer: Langsmith Online Eval Engineering (langchain-ai/langsmith-skills, 159 stars), Langsmith Observability (Orchestra-Research/AI-Research-SKILLs, 13k stars), LangSmith Trace Debugging (ComposioHQ/awesome-claude-skills, 77k stars) and Migrate To Langfuse (langfuse/skills, 299 stars). The comparison table on this page puts their stars, adoption, token cost, safety result and licence side by side.

Who maintains Langsmith Trace Analyzer?

soba-labs (a GitHub organization) maintains it in soba-labs/langchain-agent-skills, which has 107 GitHub stars. The repository holds 9 skills in this directory. The repository was last updated on August 17, 2026.

Source: soba-labs/langchain-agent-skills on GitHub. Facts on this page come from the repository at the commit we read; the author's words are quoted as theirs.