Logfire Evals

Evaluate AI code with Logfire

How This Works

Python's pydantic_evals and Node.js's logfire/evals run a task against cases, apply evaluators, and return a report. An active Logfire or OpenTelemetry provider may export inputs and outputs even if this skill did not configure it. Intentional upload needs logfire.configure() in Python or a configured Node.js exporter.

Span-based evaluators inspect the task's OpenTelemetry span tree. Without working Logfire instrumentation, Python reports "No span tree available"; Node.js HasMatchingSpan can produce no evaluator result at all. Treat either signal as a setup failure, not evidence about the agent.

Step 1: Check for an Existing Braintrust Suite First

Cheap check, before anything else: does this repo already have an existing Braintrust suite — actual Eval(...) calls or from braintrust import Eval in source, not just a braintrust dependency listed without any real usage? This path needs no CLI auth at all — don't run Step 2 for it. The compatibility endpoint is documented only for Logfire Cloud US and EU; for any other supplied origin, use the native evaluation path instead.

Keep the existing Eval() code (Python braintrust>=0.30.1 / TypeScript braintrust>=3.24.0 — verified versions) and redirect its next run to Logfire by changing environment variables only, no pydantic_evals involved:

export BRAINTRUST_APP_URL="https://logfire-us.pydantic.dev/v1/braintrust"  # EU: logfire-eu.pydantic.dev
export BRAINTRUST_API_KEY="<logfire-project-api-key>"                     # Settings -> API Keys
unset BRAINTRUST_API_URL BRAINTRUST_PROXY_URL  # these override the endpoint above if set — the #1 "it still hit Braintrust" cause

The destination project's API key needs project:write_otlp and project:read_datasets: the SDK writes the run, then reads experiment metadata for its summary. An ingest-only write token fails with 403.

This compatibility preview covers inline/callable data, local tasks and scorers, multiple scores, one label per name, and normal summary finalization. It excludes Braintrust-hosted resources, BTQL, the model proxy, server-side scoring, and post-finalization feedback. Rust, summarize_scores=False, and manual flush() without comparison do not request the summary needed for projection. See the full coverage and concept mapping.

Skip straight to Step 5 (Verify) — the SDK's own printed result URL also opens directly in Logfire, and nothing else here (auth, dataset definition) applies to this path.

No existing Braintrust suite? Continue to Step 2 now, before the more detailed identification in Step 3 — nothing past this point requires knowing the function/agent or dataset shape yet.

Step 2: Authenticate When the Run Needs Logfire

For an explicitly local-only run without span evaluators, use a fresh process that neither preloads nor imports application telemetry; omit Python's logfire.configure() and any Node.js exporter bootstrap. If the task configures an exporter and has no documented disable switch, stop rather than claiming local-only. Uploading, hosted datasets, and span evaluators require Logfire authentication to the exact project.

For a Logfire-backed run, use Authenticate and Select the Exact Project to derive the CLI target from the supplied Logfire URL and run its target-aware whoami check with a verified CLI path — for JS/TS projects without uv, use the external-prefix npm fallback instead of plain npx, which can execute a repository-local binary. Skip to Step 3 if that already reports the right project and resolved --region or --base-url target; otherwise, continue through the full authentication and project-selection sequence there. This CLI flow is for logfire.configure(); Step 3's hosted-dataset operations use a separate API key with different scopes.

Step 3: Detect What to Evaluate

Identify the real task and any dataset. Follow repository package, test, and dependency conventions. For a first eval, prefer 3-5 cases from tests, schemas, examples, or synthetic fixtures, with deterministic checks for defined behavior. Do not copy the example unless it fits, replace an eval framework, or refactor unrelated code. Without a runnable task or safe expected behavior, ask one focused question instead of inventing either.

• In-code dataset: a Python module using pydantic_evals, or a Node.js module using logfire/evals. This is the default for an agent-driven workflow.
• Hosted/managed dataset: cases live in the Logfire UI, edited by non-engineers, pulled/pushed via a separate LogfireAPIClient (from logfire.experimental.api_client import LogfireAPIClient). Hosted inputs and outputs must be JSON objects; scalar roots accepted by code-defined datasets are rejected. client.get_dataset(name) with no type arguments returns a raw dict, not something push_dataset or .evaluate_sync() can take — pass the input/output (and metadata, if used) types to get back a real pydantic_evals.Dataset: client.get_dataset(name, MyInputType, MyOutputType). If the stored dataset contains custom evaluators, also pass their classes with custom_evaluator_types=[MyEvaluator] (and custom report evaluators with custom_report_evaluator_types=[...]) so they can be deserialized. Push with client.push_dataset(dataset). This needs its own API key from Settings → API Keys (scoped project:read_datasets/project:write_datasets), not Step 2's CLI auth flow. Only relevant if the user specifically wants case editing outside code.

Step 4: Define the Dataset and Run It

Use the repository's existing package manager and lockfile. Install only the missing integration for its language.

Python

Add pydantic-evals[logfire] with the detected Python manager: uv add, poetry add, or pdm add. For a pip/requirements project, update its declared requirements and install from that file; do not introduce a second manager or lockfile.

import logfire
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import EqualsExpected, IsInstance

logfire.configure()  # omit only in the isolated local-only process described above

def classify_sentiment(text: str) -> str:
    return 'positive' if 'love' in text else 'negative'


dataset = Dataset[str, str, None](
    name='sentiment-eval',
    cases=[
        Case(name='positive', inputs='I love this', expected_output='positive'),
        Case(name='negative', inputs='This is terrible', expected_output='negative'),
    ],
    evaluators=[EqualsExpected(), IsInstance(type_name='str')],
)

report = dataset.evaluate_sync(classify_sentiment)  # or `await dataset.evaluate(...)`
report.print(include_input=True, include_output=True)

Use logfire[datasets] instead only when the task specifically needs the hosted-dataset API from Step 3.

JavaScript or TypeScript on Node.js

Do not apply this section to Deno, Bun, browsers, or workers; their exporter setup is not validated by this skill.

Add logfire and @pydantic/logfire-node with the manager selected by the existing lockfile: pnpm add, yarn add, bun add, or npm install. Do not introduce a second lockfile.

Configure Logfire before loading the task. Reuse an existing instrumentation entry point rather than configuring it twice.

import * as logfire from '@pydantic/logfire-node'
import { Case, Dataset, EqualsExpected, renderReport } from 'logfire/evals'

logfire.configure()

const classifySentiment = (text: string) => (text.includes('love') ? 'positive' : 'negative')
const dataset = new Dataset<string, string>({
  name: 'sentiment-eval',
  cases: [new Case({ name: 'positive', inputs: 'I love this', expectedOutput: 'positive' })],
  evaluators: [new EqualsExpected()],
})

await dataset.evaluate(classifySentiment).then((report) => {
  console.log(renderReport(report, { includeInput: true, includeOutput: true }))
}).finally(() => logfire.shutdown({ timeoutMillis: 5000 }))

Other built-ins include Equals, Contains, IsInstance, MaxDuration, HasMatchingSpan, and LLMJudge. Node.js custom evaluators extend Evaluator, and LLMJudge needs a judge callback. Use @pydantic/logfire-node/datasets only for hosted datasets.