Phoenix Error Analysis

Error Analysis

Two phases against sampled traces, spans, or sessions. Open coding reads each sampled entity and writes a short, specific note on what went wrong. Axial coding groups those notes into a few narrow annotations, each judging one dimension of the entity with a small label set, and counts the labels to feed eval design and fix prioritization. Open coding always comes first: dimensions and labels that are not grounded in notes are invented top-down, which is the failure this workflow exists to avoid. Axial coding can also start from any existing set of open-ended observations.

Recorded notes are the deliverable. Open coding ends with one note per problematic entity written to the server and mirrored in the local sidecar, not with a summary in chat. Reading traces and reporting what you noticed without recording anything is a read-only diagnosis, the top-down shortcut this workflow exists to replace. A project-wide question such as "find any issues", "what's going wrong", or "any errors here" is a request for phase 1: pick the identifier, pick the unit, record a note for each problem as you find it, then summarize. A project with no annotation names yet is the normal starting state, not a reason to hold off writing.

Choosing the unit of analysis

Pick trace, span, or session deliberately before recording — the choice determines what every note and annotation below targets, and a wrong default is expensive to undo mid-run. The unit is where the failure modes you're investigating actually live:

• Trace — one input → one output. Classifiers, single-shot summarizers, stateless tool-using agents, single-query RAG.
• Span — one operation inside a trace. In-isolation mechanical failures (exception, tool error response, malformed output) or failures attributable on sight to one component.
• Session — a sequence of traces sharing a session.id. Multi-turn agents where the failure is a trajectory — context loss across turns, goal drift, forgotten preferences — that exists only across traces.

Three signals to read:

1. User framing. "Conversation", "agent forgot", "drift", "across turns" → session. "This trace", "wrong output" → trace. "Exception", "malformed", "the retrieval failed" → span.

2. Session wiring. The session id lives at the root span's session.id attribute (not a top-level trace field); "" means absent. List ~200 recent traces and compute how many carry a non-empty session id, how many distinct session ids appear, and the median traces per session. No session ids or median 1 → trace or span level; median 2+ → session level is plausibly right.

3. Turn structure. Open one recent trace's root-span input.value (it may be large — filter rather than reading wholesale). A single user message → one shot; a message array ([{role: user}, {role: assistant}, ...]) → a turn of a dialogue that lives at the session level.

State the chosen unit explicitly before recording any note. It can shift if the data demands it — trace-level notes that keep surfacing "the agent never remembers earlier turns" should pivot the next batch to session. The unit is a starting hypothesis, not a contract.

Coding annotation identifier (pick this first)

Every artifact this workflow produces — open-coding notes, axial-coding labels, and the local sidecar files — is tagged with one coding annotation identifier so the run is queryable and revertible as a unit. Pick a descriptive, unique value before recording, e.g. coding-run:chatbot-context-loss-2026-05-06. (The coding-run: prefix is a naming convention; the value is not a Phoenix session id.)

Pass the identifier explicitly on every server write. A shell variable is fine for readability, but do not rely on shell inheritance — many agent harnesses spawn each command in a fresh subshell.

The local sidecars live at .px/coding/<sanitized-identifier>.jsonl (open-coding notes) and .px/coding/<sanitized-identifier>-axial.jsonl (axial labels), CWD-relative; sanitization replaces every character outside [a-zA-Z0-9_-] with -. Uniqueness is a local file check, not a server query:

CODING_ANNOTATION_IDENTIFIER="coding-run:chatbot-context-loss-2026-05-06"
SLUG=$(echo -n "$CODING_ANNOTATION_IDENTIFIER" | sed 's/[^a-zA-Z0-9_-]/-/g')
NOTES_SIDECAR=".px/coding/${SLUG}.jsonl"
AXIAL_SIDECAR=".px/coding/${SLUG}-axial.jsonl"
test ! -f "$NOTES_SIDECAR" || { echo "Sidecar already exists at $NOTES_SIDECAR — pick a new identifier or delete the file"; exit 1; }
mkdir -p .px/coding

If $NOTES_SIDECAR already exists, append a disambiguator (-v2, -dustin, etc.) and re-check.

Resuming a run — the two phases may run in independent invocations. When axial coding starts in a fresh shell, set the identifier to the same value chosen during open coding (recoverable from the wrap-up UI URL or by listing .px/coding/*.jsonl), skip the uniqueness check, and re-derive the sidecar paths from it. Never mint a new identifier mid-run.

Where the writes go

The steps below name operations, not commands. Two surfaces implement them; use whichever the harness gives you. The phoenix-cli skill documents the px flags, and the phoenix-graphql skill's references (project-spans-traces.md, sessions.md, annotations.md) document the GraphQL fields and mutation inputs. This skill ships no reference files of its own.

Note and annotation writes are synchronous and upsert on (entity, name, identifier). Where the harness gates mutations behind an approval step, describe the write each command makes and keep every mutation in its own call.

Phase 1: Open coding

Free-form note-writing. Write what you saw, not the category you think it belongs to — categorization is phase 2.

Process

1. Pick a coding annotation identifier and verify the sidecar does not yet exist
2. Pick the unit — trace, span, or session
3. Inspect — fetch one entity at the chosen unit and read its input, output, exceptions, tool calls, retrieved context, and (at session level) the trajectory across child traces
4. Note — write one specific sentence describing what went wrong, or skip if correct
5. Record — write the note to the server and append a sidecar row
6. Iterate until the sample is exhausted or saturation hits

Inspection

• Trace unit — read one trace's input → tool calls → retrieved context → output as one story.
• Span unit — read one operation's input/output and surrounding spans for context.
• Session unit — read the sequence of traces in order; the trajectory across traces is the data, not any single trace.

Don't sample by span status ERROR. OTel's status_code only flips to ERROR when an instrumentor catches a raised exception. Hallucinations, wrong tone, retrieval misses, and bad tool selection all complete cleanly as OK or UNSET — filtering to error status excludes the population this workflow exists to surface.

What to look for

A checklist, not a taxonomy — categories come later. Note the first thing that goes wrong; a downstream symptom gets its own note only if it has an independent cause.

• Explicit errors — exceptions or error messages in tool, LLM, or retriever spans
• Cost and latency — unusually high token counts or slow spans
• Retrieval quality — irrelevant, missing, or low-scoring chunks
• Response quality — hallucination, factual errors, wrong tone, inappropriate refusals
• Tool use — wrong tool, malformed call, mishandled result
• Trajectory — loops, detours, unfinished tasks; across a session, lost context or goal drift
• Instrumentation gaps — missing spans or attributes; a blind spot is a finding

Treat existing evals and annotations as one input among many. Read content, not status: a success status can hide an error in the attributes, and an exception can be expected behavior.

Whatever the tooling, the fetches are: sample recent traces (trace id, root span name, status, root-span input.value / output.value); expand one trace into its spans ordered by start time; drill into a single span by id when the unit is the span; and check existing notes on entities you are about to review — notes are stored server-side as annotations with the reserved name note. As always, be aware that the data may be verbose, so take care not to blow up the context.

Recording notes

For each session, trace, or span you inspect, submit a note to the server (see Where the writes go) and also save a local copy. Every write should carry the note text and an explicit identifier set to the coding annotation identifier. Record as you go, entity by entity — do not batch the notes into a closing summary.

After every successful note write to the server, append one JSONL line to $NOTES_SIDECAR — the sidecar is what axial coding reads, with no server round-trip:

{"entity_kind":"trace","entity_id":"<trace-id>","note":"<text>","identifier":"<original identifier value, unsanitized>","ts":"<ISO-8601 UTC>"}

What makes a good note

Saturation

Stop when observations stop being new: the last 10–15 entities repeat failures you've already seen, you catch yourself paraphrasing earlier notes, or skips outnumber notes. Resist grouping into categories while still collecting. You do not need to annotate every trace — annotating correct ones dilutes signal.

Between phases: proceed or offer

At saturation, decide whether to continue into axial coding:

• If the user asked for categories, a taxonomy, counts, eval targets, or fix priorities — or framed the task as error analysis end to end — continue directly into phase 2.
• Otherwise stop here: summarize what the notes surfaced, then offer to continue with axial coding and wait for the user's answer rather than starting it. If the run ends here, follow Wrapping up.

Phase 2: Axial coding

Turn the open-coding notes into a small set of annotations, each judging one dimension of the entity: one question about its behavior, answered with a few labels. An entity carries one annotation per dimension its note touches, so a trace can end up with two or three. The output is several narrow annotations with two to four labels each — not one wide <app>_failure_mode annotation whose labels span unrelated concerns. A wide annotation cannot be aggregated per dimension, cannot be reused by an eval that judges one thing, and forces one label onto entities that failed in two ways.

Process

1. Set the coding annotation identifier to the open-coding value and derive the sidecar paths (see Resuming a run)
2. Gather — read this run's notes from $NOTES_SIDECAR. An absent file means open coding hasn't run for this identifier in this CWD — stop and run phase 1 first. The newest ts per entity wins; fix or drop a malformed line without touching its neighbors. To include notes from other reviewers or earlier runs, fetch them from the server (annotations with the reserved name note).
3. Group — cluster notes that answer the same question about the entity ("did it pick the right tool?", "did the answer stay grounded in what was retrieved?", "did it finish the task it was asked to do?"). Each cluster is a candidate dimension. Work bottom-up from the note text; do not start from a list of dimensions.
4. Define — for each dimension, name the annotation for the question it asks (tool_selection, answer_faithfulness) and choose its labels: the failure outcomes the notes describe, plus one passing value so an eval can later apply the same annotation to entities that pass. Labels within a dimension are mutually exclusive; if two labels could both be true of one entity, they belong to different dimensions.
5. Attribute — decide what level each dimension lives at (see Choosing the level)
6. Configure — register one categorical annotation config per dimension (see Annotation configs)
7. Record — for each noted entity, write one annotation per dimension its note touches, and append one sidecar row per annotation
8. Quantify — count labels per dimension from $AXIAL_SIDECAR: group the current rows by (annotation_name, axial_label), newest ts per (entity, annotation_name) wins. No server query — the sidecar holds exactly the labels this run wrote.