Measure Instrumentation Spec

<!-- PM-Skills | https://github.com/product-on-purpose/pm-skills | Apache 2.0 -->

Instrumentation Spec

An instrumentation spec defines what analytics events to track, when to fire them, and what properties to include. It serves as a contract between product and engineering, ensuring consistent data collection that enables accurate measurement. Good instrumentation specs prevent the "we can't answer that question because we didn't track it" problem.

When to Use

• Before engineering implements a new feature
• When defining analytics requirements for experiments
• When auditing existing tracking for gaps or inconsistencies
• When onboarding a new analytics tool
• Before launch to ensure measurement is in place

When NOT to Use

• You are specifying the dashboard built on top of the events -> use measure-dashboard-requirements
• You need experiment-specific metrics and variants, not product-wide tracking -> use measure-experiment-design
• The feature itself is not yet specified (no flows to instrument) -> use deliver-prd first
• You are analyzing data you already collect -> use measure-experiment-results or measure-survey-analysis

Instructions

When asked to create an instrumentation spec, follow these steps:

1. Define Analytics Goals Start with the questions you need to answer. What will you measure? What decisions will this data inform? This prevents over-instrumentation while ensuring nothing important is missed.

2. Identify Events to Track List each user action or system event that should be tracked. Follow consistent naming conventions (typically noun_verb or verb_noun in snake_case). Each event should represent a distinct, meaningful action.

3. Specify Event Triggers For each event, describe exactly when it fires. Be precise: "When user clicks Submit button" vs. "When form is submitted successfully." These are different events with different meanings.

4. Define Event Properties List the properties (attributes) attached to each event. Include property name, data type, description, and example values. Properties provide context that makes events useful.

5. Document User Properties Identify persistent user-level attributes that should be associated with all events (e.g., subscription tier, account creation date). These enable segmentation in analysis.

6. Address PII and Privacy Flag any properties that contain personally identifiable information. Document how PII should be handled - hashing, encryption, or exclusion.

   When any model input, output, retrieval context, or tool call is captured, extend this section to cover the trace as well. The trigger is capture, not authorship: it fires whether the sensitive text was typed by a user, retrieved from a tenant's documents, carried in a system prompt, passed to or returned from a tool, read out of an uploaded file or a batch job, or generated by the model itself. A feature with no direct user input can still write a customer's contract text into a trace store.

   A trace is not an event: an event carries properties you chose in advance, while a trace carries free text that can contain anything, including data no property schema anticipated. Decide and record:

   • Data classes captured. Name them (user text, retrieved documents, system prompt, tool arguments and results, file contents, model output). "The trace" is not an answer.
   • Minimization, at both boundaries. What is dropped or redacted before the trace leaves the process, and separately what is dropped before it is stored. A redactor that runs only at the storage layer has already sent the raw text over the wire.
   • Access. Who can read traces, and whether reads are logged. Trace stores are routinely the least-governed copy of the most sensitive data a feature handles.
   • Retention and deletion. How long, what deletes them, and how a deletion request reaches a trace that was copied into an evaluation set.
   • Consent and opt-out. Whether the subject agreed, and what the feature does when they decline.
   • Sampling. What fraction is captured and how that sample is chosen. A uniform sample is the wrong instrument for finding rare failures; if traces exist to diagnose bad output, oversample the flagged cases and say so.

7. Create Testing Checklist Define how QA should verify that tracking is implemented correctly. Include steps to validate events fire at the right times with correct properties.