Ax Agent Observability

AxAgent Observability Rules (@ax-llm/ax)

Use this skill when an agent needs runtime visibility, progress reporting, tracing, usage accounting, or chat-log access. For ordinary agent setup use ax-agent. For RLM runtime policy use ax-agent-rlm. For memories and dynamic skill loading use ax-agent-memory-skills.

Choose The Smallest Hook

• Need a quick prompt/runtime trace during development -> start with debug: true.
• Need structured per-turn code, raw runtime result, formatted output, provider thoughts, or actor stage -> use actorTurnCallback.
• Need context-pressure and compaction telemetry -> use onContextEvent.
• Need real-time task progress emitted by actor code -> use agentStatusCallback.
• Need every runtime function call before execution -> use onFunctionCall.
• Need model prompts/responses after a run -> use getChatLog().
• Need centralized chat/embed usage across APIs, users, agents, and services -> use axGlobals.onUsage plus usageContext.
• Need token usage by actor/responder -> use getUsage() and resetUsage().
• Need usage split by context and task stages -> use getStagedUsage().
• Need Ax program traces -> use getTraces().
• Do not add multiple hooks unless the user clearly needs each output stream.

Global Runtime Defaults

OpenTelemetry and debug defaults come from the shared Ax runtime surface:

import { axGlobals, axCreateDefaultColorLogger } from '@ax-llm/ax';
import { metrics, trace } from '@opentelemetry/api';

axGlobals.rateLimiter = async (next, info) => next();
axGlobals.tracer = trace.getTracer('agent-app');
axGlobals.meter = metrics.getMeter('agent-app');
axGlobals.debug = true;
axGlobals.logger = axCreateDefaultColorLogger();
axGlobals.onUsage = (event) => usageQueue.enqueue(event);

Each agent run snapshots these globals. A forward-scoped rateLimiter, tracer, or meter overrides agent defaults, child-generator defaults, service hooks, and globals for that invocation. Ax carries it through the distiller, executor, responder, repeated actor turns, citation repair, built-in llmQuery, context-map work, checkpoint/tombstone summaries, and other direct internal model calls without mutating child programs or leaking across concurrent runs.

The limiter wraps model execution and its failures propagate. Tracer, meter, and usage-observer failures are fail-open. Runtime-hook spans contain metadata only—never prompts, outputs, tool arguments, or tool results—and preserve agent → internal AxGen → provider/tool parentage. External meter instruments do not replace or derive from balancer-local getMetrics() snapshots. Use AxAgent callbacks below when the caller needs structured agent-turn events rather than spans or metrics.

Centralized Usage Observer

Use the process-wide usage observer for application accounting across many agents, API routes, tenants, and users. Keep getUsage() for inspecting one agent instance after a run.

import { axGlobals } from '@ax-llm/ax';

axGlobals.onUsage = (event) => {
  usageQueue.enqueue(event); // Must return immediately.
};

await supportAgent.forward(
  llm,
  { query: request.body.query },
  {
    usageContext: {
      tenantId: auth.tenantId,
      userId: auth.userId,
      requestId: request.id,
      runId: crypto.randomUUID(),
      feature: 'support-chat',
      attributes: { environment: 'production' },
    },
  }
);

Rules:

• The observer receives one immutable normalized event for each completed chat or embedding call that reports provider usage. A fully consumed stream emits once; an unconsumed or cancelled stream may not emit.
• Events include operation, ai, model, normalized tokens, streaming, optional context, and available session or remote request IDs.
• Put stable attribution such as application or environment in AI service-level usageContext. Put tenant, user, request, run, and feature attribution in per-call or per-forward usageContext.
• Per-call context overrides service defaults. attributes are shallow-merged.
• The observer is best-effort and fail-open. Ax does not await it and ignores observer failures, so synchronously enqueue and persist or aggregate out of band.
• The registration is process-wide. A new assignment replaces the previous observer; set axGlobals.onUsage = undefined during test teardown or shutdown when appropriate.
• In multi-process or serverless deployments, send events to a shared durable pipeline. Do not treat an in-memory total as application-wide accounting.
• Keep identifiers opaque and attributes low-cardinality. Avoid prompts, responses, secrets, and other sensitive payloads.
• Token events do not estimate currency cost. Apply a versioned provider/model pricing table downstream.

For direct AI calls and the complete event shape, also read ax-ai.

Actor Turn Callback

Use actorTurnCallback when the caller needs structured telemetry for each actor turn.

What it gives you:

• code: the normalized JavaScript code the actor produced
• stage: which actor produced the turn (distiller or executor)
• result: the raw untruncated runtime return value from executing that code
• output: the formatted action-log output string after Ax normalizes and truncates it for prompt replay
• thought: the actor model's thought field when showThoughts is enabled and the provider returns one
• executorResult: the full actor payload returned by the current actor stage, kept under this historical field name for compatibility
• isError: whether the execution path for that turn was treated as an error
• usage: token usage for this actor turn only
• model: model used for this turn when explicitly set through executorModelPolicy
• chatLogMessages: raw ChatML conversation for this turn, populated only when an actor turn callback is set

Use it for:

• debug UIs that want to show code plus raw runtime results
• tracing and analytics
• capturing thought for internal diagnostics when supported by the provider
• storing per-turn execution artifacts without scraping the prompt/action log

Important:

• output is not raw stdout; it is the formatted replay string used in the action log.
• result is the raw runtime result before Ax applies type-aware serialization and budget-proportional truncation.
• thought is optional and only appears when the underlying AxGen call had showThoughts enabled and the provider actually returned a thought field.
• actionLogEntryCount and guidanceLogEntryCount reflect the live log sizes after the turn is processed, including resumed runs.
• actorTurnCallback fires for the configured agent instance. Child agents passed through functions: [...] should define their own callback if you need their internal actor turns; use onFunctionCall on the parent to observe the parent-side child-agent invocation.

Good pattern:

const supportAgent = agent('query:string -> answer:string', {
  contextFields: ['query'],
  runtime,
  actorTurnCallback: ({
    stage,
    turn,
    actionLogEntryCount,
    guidanceLogEntryCount,
    code,
    result,
    output,
    thought,
    isError,
    usage,
    model,
  }) => {
    console.log({
      turn,
      stage,
      model,
      actionLogEntryCount,
      guidanceLogEntryCount,
      isError,
      code,
      rawResult: result,
      replayOutput: output,
      thought,
      usage,
    });
  },
  executorOptions: {
    model: 'gpt-5.4-mini',
    showThoughts: true,
  },
});

Callback type:

actorTurnCallback?: (turn: {
  stage: 'distiller' | 'executor';
  turn: number;
  actionLogEntryCount: number;
  guidanceLogEntryCount: number;
  executorResult: Record<string, unknown>;
  code: string;
  result: unknown;
  output: string;
  isError: boolean;
  thought?: string;
  usage?: AxProgramUsage[];
  model?: string;
  chatLogMessages?: ReadonlyArray<{ role: string; content: string }>;
}) => void | Promise<void>;

actorTurnCallback?: (turn: {
  stage: 'distiller' | 'executor';
  turn: number;
  actionLogEntryCount: number;
  guidanceLogEntryCount: number;
  executorResult: Record<string, unknown>;
  code: string;
  result: unknown;
  output: string;
  isError: boolean;
  thought?: string;
  usage?: AxProgramUsage[];
  model?: string;
  chatLogMessages?: ReadonlyArray<{ role: string; content: string }>;
}) => void | Promise<void>; // deprecated alias

Context Event Observability

Use onContextEvent when the caller needs structured telemetry about prompt pressure and compaction. It does not change model behavior directly; it is for logs, evals, and dashboards.

Events:

• budget_check: character-based prompt pressure before an actor turn, with detailed metrics kept out of the actor prompt
• checkpoint_created / checkpoint_cleared: checkpoint lifecycle events with covered turns and reason
• tombstone_created: compact resolved-error summary creation
• relevance_ranking: emitted once per ranked domain per forward when relevanceRanking is enabled; carries domain ('modules' | 'skills' | 'memories'), the shortlist ({ id, score }[], most relevant first), and suppressed (true when the low-confidence guard emitted no hint)
• field_auto_promoted: emitted once per field per run when autoUpgrade keeps an oversized undeclared input value runtime-only; carries fieldName, originalChars, and promptPreviewChars (undefined when no inline preview was kept)

To measure whether the advisory hint helps, join per forward: relevance_ranking.shortlist ids against what the actor then loaded — for modules the internal discover calls (onFunctionCall with kind: 'internal', name: 'discover', args.request) plus the module part of external qualifiedNames; for skills onLoadedSkills / used(id); for memories onLoadedMemories / used(id).

Rules:

• contextPressure in the actor prompt is intentionally compact (ok, watch, critical plus one short instruction).
• Budget metrics are character-based for provider neutrality and are exposed through onContextEvent, not the actor prompt.
• Callback errors are swallowed so telemetry cannot break the agent run.
• Do not scrape actor prompts for pressure metrics.

const supportAgent = agent('query:string -> answer:string', {
  contextFields: ['query'],
  runtime,
  contextPolicy: { preset: 'checkpointed', budget: 'balanced' },
  onContextEvent: (event) => {
    if (event.kind === 'budget_check') {
      console.log(event.pressure, event.mutablePromptChars);
    }
  },
});

Type:

onContextEvent?: (event: AxAgentContextEvent) => void | Promise<void>;