Creating Experiments

Creating experiments

This skill walks through creating a new A/B test experiment: read the project, then the 3-step flow, then report the draft.

Core principle: draft first, iterate on details

Create the experiment as a draft quickly, then iterate on metrics and configuration. The user gets a tangible draft immediately and can refine it. Choose settings from the project's facts rather than asking, and say which choices are guesses. Bucketing is the exception: the facts cannot decide it, so ask before you create the experiment.

Step 0: Read the project

The right bucketing, metric and running time depend on the project: who sees the page, which SDKs evaluate the flag, how often the metric event happens. Read that before configuring anything.

1. From the request, infer the target (the event that marks someone reaching the change, usually $pageview for a page) and a candidate metric event. Confirm both exist with read-data-schema. Don't ask the user for event names you can find. For a page, read event_property_values for $host and $pathname on the target event as well. The response samples the values rather than listing them all, so read it for the shape the project records (a trailing slash, a www. prefix, the casing) rather than as proof that a value is absent. An exact filter has to carry that shape: /pricing matches nothing where every pageview says /pricing/, and example.com matches nothing where the host is www.example.com.

2. If the experiment-setup-context tool is available, call it once with target_event and metric_event. For a web surface, add target_properties with an exact $host. Add an exact $pathname as well when the surface is one page. target_url_contains is a substring match on $current_url. A bare domain matches any host that contains it, notexample.com included, and a homepage path matches every page under it. Both overstate the traffic and the exposure rate. For a surface that spans several pages, keep the exact $host and put only the path fragment in target_url_contains. Add metric_properties in the same call when the candidate metric counts only some occurrences of its event. Pass previous_experiments_limit: 5. Five experiments are enough to read a precedent, and the default of 10 roughly doubles the response for no more signal. If the tool is not available, continue without it and treat every choice below as a best guess. Never call a tool you can't see.

   Each filter needs a type of event or person, a key, an operator and a value. The call rejects the flag_evaluates_to operator with a 400 that names it. For the homepage of one domain:

   {
     "target_event": "$pageview",
     "target_properties": [
       { "key": "$host", "type": "event", "operator": "exact", "value": ["www.example.com"] },
       { "key": "$pathname", "type": "event", "operator": "exact", "value": ["/"] }
     ],
     "metric_event": "your_conversion_event",
     "previous_experiments_limit": 5
   }

3. Apply references/setup-decisions.md to the result. It maps each fact to a choice (bucketing, where the flag is evaluated, exposure, primary metric, running time, precedent) and to a tier for the summary.

4. Carry those choices into steps 1 to 3.

Ask the user only when a choice is "not decided" and the user is in the conversation. If the user said they are away, or asked for a draft without questions, decide with the defaults and list the open points in the report.

The 3-step creation flow

Step 1: What are we testing?

Gather these before calling experiment-create:

• Experiment name — descriptive, inferred from context when possible
• Hypothesis — what you expect to happen (goes in description)
• Feature flag key — kebab-case. Infer a new key unless the user names an existing flag. The flag is auto-created — do NOT create one separately. The one exception is device-id bucketing, which experiment-create cannot set (see the recipe in configuring-experiment-rollout).
• Type — leave empty (will internally default to "product". The "web" value is reserved for no-code experiments configured visually with the PostHog toolbar in a browser; it cannot be meaningfully driven via MCP. If a user asks for a no-code/toolbar experiment, point them to the PostHog UI instead of creating one here.)

If the user gives enough context to infer these, don't ask — just proceed.

Step 2: Who sees what variant?

This is about rollout configuration.

Before asking any rollout question, load configuring-experiment-rollout. The disambiguation wording, recommendations, and post-answer branches live there — do not formulate rollout questions yourself, and do not assume an example you remember covers the user's path.

Key decision points (covered in detail by configuring-experiment-rollout):

• Variant split (how many variants, what percentage each)
• Overall rollout percentage (what % of all users enter the experiment)
• Bucketing and whether to persist the flag across authentication steps — the user chooses, because the project's facts cannot decide it

If the user doesn't mention rollout specifics, use defaults: 50/50 control/test, 100% rollout.