Agent Observability Online Experiment

Set up an online experiment on an LLM application

An online experiment splits live traffic between a control and a treatment version of an LLM application. A Datadog feature flag assigns each subject to a variant, and an Agent Observability evaluation score (plus cost and token usage) measures the outcome. This skill walks through the whole setup, from the flag to starting the experiment.

Online experiments for Agent Observability may require the feature to be enabled for the user's organization. If any step reports the feature is unavailable, tell the user to contact their Datadog representative rather than retrying.

When to use this skill

• The user wants to test a change (prompt, model, retrieval strategy, tool set) on real traffic rather than a fixed dataset.
• The user has a change on a branch and wants to know whether it is better, cheaper, or both in production.
• The user wants an experiment whose metrics come from Agent Observability evaluations, token counts, or cost.

Do not use it for offline experiments over a dataset (use the LLM Observability experiment tooling), for analyzing an experiment that already ran (agent-observability-experiment-analyzer or the Datadog experiments tooling), or for promoting an existing offline experiment.

Ground rules

• Use one setup approval checkpoint. Before any write, finish the read-only discovery in Phase 0, make sensible recommendations, and present the complete proposed setup in one confirmation request. Include the flag, variants, metric definitions or reused metric IDs, desired-change directions, environment, subject, traffic split, code change, and every organization write you intend to make. Once the user approves that proposal, do not ask again before each flag, draft, link, metric, enablement, or allocation write that was clearly listed.
• Make production risk explicit in that same checkpoint. If the proposal enables a production flag or replaces production allocations, say that the change takes effect immediately. A clear approval of that production proposal is sufficient; do not add a second confirmation that repeats the same information. Ask again only if read-back reveals a materially different or destructive replacement that was not in the approved proposal.
• Starting is separate unless already explicit. Start only if the user explicitly asked to start the experiment. If the original request included starting, include it in the single setup proposal; otherwise finish with a draft and ask no start question until the user requests it.
• Go one step at a time and read back. After each write, read the object back and report what the server actually stored, not what you sent.
• Never attach metrics in parallel or alongside other metric edits. The attach operation sends the experiment's complete metric set to the server, so concurrent edits can overwrite each other.
• Report failures plainly. If a step is refused (permissions, production restrictions), say so and give the user the manual alternative. Do not look for a workaround.
• Do not guess identifiers. Environment IDs, flag IDs, metric IDs, and variant keys all come from the server.

Phase 0 — Discover, recommend, and confirm once

Do not interrupt the investigation to ask for one input at a time. First perform all available read-only discovery:

• Inspect the active tool list and verify that the session has the llmobs, experiments, and feature-flags capabilities needed by this workflow: environment discovery; flag lookup, creation, enablement, and allocation management; experiment lookup, creation, linking, and start; and experiment-metric lookup, creation, and attachment. The skill's toolset configuration controls its visibility with OR semantics; it does not guarantee that all three toolsets are enabled. If capabilities are missing, report every missing group and the toolset to enable, then stop before any write instead of repeatedly retrying individual operations.
• Inspect the application and its deployment configuration. Confirm it reports to Agent Observability, find where it submits evaluations (if it does), find the code path the treatment changes, and infer its ml_app, DD_ENV, and stable subject identifier.
• List environments, look for an existing flag and experiment for the proposed change, and search existing experiment metrics before proposing new objects.

Then establish the complete proposal. Infer from context and use the defaults below wherever reasonable:

When metric-name search is needed, try naming variants before concluding that no metric exists: spaces, hyphens, underscores, and a meaningful substring. Inspect duplicate definitions rather than choosing by name alone. Prefer a definition already used by active experiments only when its source, columns, filters, aggregation, and desired-change direction match the proposal. Do not infer ML-app scope from a metric's name. If the user's requested existing metric lacks an ML-app filter or otherwise differs from the recommended definition, disclose that in the proposal and recommend whether to reuse or replace it.

Present one compact confirmation request containing:

• application and ml_app;
• control, treatment, and hypothesis;
• proposed flag key and Boolean variants;
• subject identifier and code location where the flag will be evaluated;
• target environment and 50/50 or proposed split;
• primary evaluation metric, stored definition or proposed definition, and desired-change direction;
• token and cost secondary metrics;
• whether existing objects will be reused or new ones created;
• the complete list of writes, including production enablement/allocation when applicable;
• whether starting is included (only when the user already requested it).

End with one question: Confirm this complete setup, or tell me what to change. Use follow-up questions only for a genuinely blocking ambiguity that cannot be resolved from code, server state, or a safe default.

Phase 1 — Create the feature flag

Create a Boolean flag with two variants: control = false and treatment = true. Make control the default variant.

• Do not add allocations, targeting rules, or a percentage rollout when creating the flag. Linking it to an experiment does not create an allocation by itself: create or update one experiment-linked FEATURE_GATE allocation in Phase 5 after linking. Extra rules interfere with the split.
• Check first whether a flag with that key already exists, and reuse it if the user says so.
• If the organization has no feature flags at all, the server may point to a guided onboarding flow for first-time setup. Direct creation is fine here when the user has asked for a specific flag; mention the onboarding flow only if the user seems new to Datadog Feature Flags.
• Record the flag's ID and the IDs of its two variants.

Listing environments gives the available ones (typically Production, Staging, Development). Note which are production: some actions on production are not available through the server and must be done in the Datadog UI.

Phase 2 — Wire the application

The application must do three things for every subject: evaluate the flag, run the matching behavior, and report the score and spans under the same subject identifier.

Choose the subject identifier. It must be stable across the whole interaction being measured.

• Application with signed-in users: a stable user ID.
• No users, but sessions: the session ID.
• Autonomous workflows: generate one UUID at the start of a run and reuse it for the whole run.

Use the same value in three places: the feature flag evaluation context's targetingKey, the subject_identifier tag on the evaluation score (the join key the Datadog online experiments guide requires), and the subject_identifier tag on the root span of each trace, which is what the cost and token metrics read. If these differ, Datadog cannot join exposure to outcome and the experiment shows missing metric data.

Evaluate the flag. Follow the Feature Flags SDK for the application's language. For Python this means a recent ddtrace plus the OpenFeature SDK, registering the Datadog provider once, and evaluating a Boolean flag with targeting_key set to the subject identifier. The SDK needs the Datadog API key, site, and DD_ENV set. Default the evaluation to false (control) and catch failures so a flag outage degrades to control instead of breaking the app.

Feature-flag management guidance may point to a React integration resource regardless of the application's language. Treat that as generic tool guidance, not as an instruction to add React code. Follow the SDK for the application's actual language and prefer an existing working provider/evaluation pattern in the repository.

Choose where the variant is applied. If the behavior is baked into state that persists (for example a system prompt stored in a conversation object), evaluate the flag once when that state is created, not on every call, so a subject cannot change variants mid-interaction. Derive the treatment from the control in code where possible so the two cannot drift.

Report the score. Submit the score evaluation with the Agent Observability SDK and add subject_identifier to its tags. Only evaluations with a numeric score metric type can be experiment metrics; Boolean, categorical, and other types cannot.

Tag the spans and the evaluations. Add the subject_identifier tag (for example subject_identifier:73b2efe1-03e3-40c1-ab6a-bd2a6cfbc865) to the Agent Observability spans and to the evaluation scores in the application's own code. Datadog converts the tag into the event's @usr.id field during ingestion, and that field is what the experiment joins to the flag exposure. Set the tag, not @usr.id: the SDK does not set that field directly, and the conversion happens in Datadog's backend.

Cost and token metrics read each completed root trace's rollup, so the tag must be on the trace's root span: the outermost workflow or agent span. Annotate that span through the SDK's span annotation call. For a span-scoped managed evaluator, also add the same tag to every span that evaluator scores because its outcome derives the subject from the evaluated span's own tags. A trace-scoped managed evaluator can propagate the root span's subject to its outcomes. Verify the evaluator's scope before reusing its metric. Tagging evaluated child spans does not double count cost or tokens because those metrics still measure only root traces.

The code must be deployed for the experiment to collect anything. Tell the user plainly that the flag, allocation, and experiment can exist before the deploy but record nothing until the application evaluates the flag in the linked environment, and that the application's DD_ENV must match that environment.

Phase 3 — Create the experiment and link the flag

1. Create a draft experiment with a clear name and the hypothesis. The primary metric can be attached afterwards, so it is fine to create the draft before the metrics exist.
2. Link the flag from Phase 1 to the experiment.
3. Read the experiment back. A draft may report an incomplete set-up state until it is started; that alone is not an error. The start step reports concrete blockers if there are any.
4. Resolve the draft's subject type and verify that its Product Analytics attribute is @usr.id, matching the field produced from Phase 2's subject_identifier tag. Starting requires a subject type, but do not accept an arbitrary server-selected default: subject types mapped to another field cannot join these Agent Observability outcomes. Explicitly select an @usr.id-compatible subject type before proceeding. If the available capabilities cannot inspect or configure the mapping, direct the user to the Datadog UI and stop until it is confirmed.

An alternative to creating the flag first is a single operation that creates the flag together with an experiment-linked allocation (the create-experiment-feature-flag tool). It needs the experiment and the environment to exist up front. The step-by-step path above keeps each write small and easy to read back, which is why this skill uses it.

Report the experiment's name and ID. Construct its direct Product Analytics URL as https://<app-host>/product-analytics/experiments/<experiment-id>, using the ID returned by the server and the application host for the organization's Datadog site. For US1 use app.datadoghq.com; for EU use app.datadoghq.eu; regional sites such as us5.datadoghq.com use that host directly; and datad0g.com uses dd.datad0g.com. Include this link in the final handoff after completing the setup instructions, whether the experiment remains a draft or has been started.

Phase 4 — Create and attach metrics

Reuse before creating

List the existing experiment metrics first. If a metric already measures the user's score for this ML app, reuse it. If any existing metric is built on Agent Observability data, read its stored definition and copy its column names instead of guessing them. The same applies to cost and token metrics.