Distribution Profiler

Skill: Distribution Profiler

Purpose

Take any numeric data column and produce a complete analytical playbook: identify the distribution, compute the right summary statistics, recommend the correct statistical tests, flag common traps, and give specific A/B testing guidance.

This skill exists because the #1 mistake in product analytics is assuming data is normal when it's not — leading to wrong tests, false positives, and misleading dashboards. The profiler catches this automatically.

When to Use

• Before any analysis involving a numeric metric
• When the user asks "what distribution is this?" or "what test should I use?"
• When checking assumptions for an A/B test
• When a user says "profile" or "understand" a metric
• Proactively when you notice an analysis is about to use a t-test or OLS on data that hasn't been checked

Invocation

/distribution-profiler — profile a data column's distribution

Instructions

Step 0: Identify the Target

Figure out what column/metric the user wants profiled. This could be:

• A specific column name (e.g., "total_amount from orders")
• A derived metric (e.g., "revenue per user", "sessions per user per month")
• A SQL query result

If unclear, ask. If the user hasn't specified, look at what they're analyzing and suggest the most relevant metric to profile.

Step 1: Extract the Data

Write and execute a Python script to extract the target column from the active dataset. Use helpers/data/data_helpers.py to resolve the data source:

from helpers.data.data_helpers import detect_active_source, check_connection
source = detect_active_source()

Query through ConnectionManager (helpers/data/connection_manager.py) whatever the source; it resolves local vs. remote and auto-logs the query for provenance.

For per-user metrics (revenue per user, sessions per user), aggregate first — the unit of analysis matters. Profile the metric at the level it will be used in the analysis (per-user, per-session, per-day, etc.).

Step 2: Run the Diagnostic Pipeline

Write and execute a single Python script that computes all diagnostics. The script should print results as structured output that you'll use to build the report.

Core diagnostics to compute:

import numpy as np
from scipy import stats

# --- Data basics ---
n = len(x)
n_zeros = (x == 0).sum()
pct_zeros = n_zeros / n
n_unique = len(np.unique(x))
is_integer = np.all(x == np.floor(x))
x_min, x_max = x.min(), x.max()

# --- Central tendency ---
mean = np.mean(x)
median = np.median(x)
mean_median_ratio = mean / median if median != 0 else float('inf')

# --- Spread ---
sd = np.std(x, ddof=1)
cv = sd / mean if mean != 0 else float('inf')
iqr = stats.iqr(x)
mad = stats.median_abs_deviation(x)

# --- Shape ---
skewness = stats.skew(x)
excess_kurtosis = stats.kurtosis(x, fisher=True)

# --- Distribution-specific diagnostics ---
vmr = np.var(x, ddof=1) / mean if mean != 0 else float('inf')  # for counts

# --- Formal tests ---
# Normality (on raw data)
if n <= 5000:
    shapiro_stat, shapiro_p = stats.shapiro(x)
anderson_result = stats.anderson(x, dist='norm')

# Normality of log-transformed data (if all positive)
if x_min > 0:
    log_x = np.log(x)
    shapiro_log_stat, shapiro_log_p = stats.shapiro(log_x)

# Bimodality
bimodality_coeff = (skewness**2 + 1) / (excess_kurtosis + 3)

Key decision points the diagnostics must answer:

1. Is the data discrete (counts) or continuous? → determines count vs continuous branch
2. Is there zero-inflation? → compare observed zero % to expected under fitted distribution
3. Is it bounded (0,1)? → beta distribution
4. For counts: VMR near 1, >> 1, or < 1? → Poisson vs NB vs Binomial
5. For continuous positive: is log(x) normal? → log-normal check
6. Is it symmetric? → normal check
7. Is it bimodal? → dip test + bimodality coefficient
8. Is the tail extremely heavy? → power law check (mean/median ratio, top 1% share)

Step 3: Identify the Distribution

Use the diagnostic results and the decision flowchart from the reference guide at .knowledge/references/statistical-distributions-guide.md (Section 15: Distribution Identification Flowchart) to identify the most likely distribution.

Read the relevant section of the reference guide for the identified distribution to get the full details on summary stats, tests, transformations, and A/B implications.

Report your confidence level:

• High confidence: Formal tests agree, visual shape matches, domain makes sense
• Moderate confidence: Some tests agree, shape is consistent but not perfect
• Low confidence: Tests disagree or data doesn't fit cleanly into one family. In this case, report the top 2-3 candidates and what would distinguish them.