Cja Dimension Analysis

CJA Dimension Analysis

Analyze one or more CJA dimensions to understand their cardinality, distribution, trends, anomalies, data quality issues, and forecasts. Produces an actionable report that helps teams understand what's inside their dimensions and where to focus attention.

Workflow

Execute phases in order. Each phase is selectable — the user can ask for a subset (e.g., "just cardinality and errors") or the full analysis. Default is all phases.

Phase 0 — Setup

1. Call findDataViews to list available data views. If the user hasn't specified one, ask which data view to analyze. Set it with setDefaultSessionDataViewId.
2. Ask which dimensions to analyze. Options:
   • Named dimensions: "Analyze Page Name and Browser Type"
   • By ID: user provides dimension IDs directly
   • All dimensions: warn that this may be slow; ask for a limit (default: top 50 by name)
3. Ask which analyses to run (or confirm "all" as the default):
   • Cardinality, Distribution/Skew, Trends, Anomalies, Data Quality, Comparisons, Forecasting
4. Ask for the date range. If the user hasn't specified one, test a few ranges to find data:
   • Try last 30 days, last 90 days, last 6 months, last year — use the first that returns rows.
5. Ask for the primary metric to use for distribution/skew (default: occurrences or visits).
6. Confirm the plan with the user before proceeding.

Phase 1 — Cardinality

For each dimension:

1. Call searchDimensionItems(dimensionId, limit: 50000) to estimate unique value count, or runReport with the dimension as rows and a count metric to get row count.
2. Classify cardinality:

   Level	Threshold
   LOW	< 100 unique values
   MEDIUM	100 – 1,000
   HIGH	1,000 – 10,000
   VERY HIGH	> 10,000

3. Track cardinality over time (optional): runReport with dimension + date breakdown; count unique dimension values per day/week to see cardinality growth trend.
4. Flag HIGH and VERY HIGH dimensions with performance recommendations.

Store: {dimensionId, name, uniqueValueCount, cardinalityLevel, cardinalityTrend}

Phase 2 — Distribution & Skew

For each dimension:

1. runReport with dimension as rows + primary metric (e.g., occurrences/visits). Request at least 50 rows to capture the distribution shape.
2. Compute top-N % share (top 1, 5, 10), Gini coefficient, and cumulative distribution.
3. Classify skew:

   Label	Condition
   Extreme skew	Top 1 value > 50% of total
   High skew	Top 1 value > 30% of total
   Moderate	Top 5 values < 70% of total
   Long tail	Top 10 values < 50% of total

4. Note: per-value breakdown with percentage and cumulative %.

Store: {dimensionId, distribution: [{value, metric, pct, cumulative}], gini, skewLabel, top1Pct, top5Pct, top10Pct}

Phase 3 — Trends

For each dimension:

1. runReport with dimension + date granularity (day or week depending on range). Compare two periods: first half vs second half of the selected date range.
2. Identify:
   • New values: appeared in period 2 but not period 1
   • Disappeared values: present in period 1, absent in period 2
   • Growth: metric in period 2 > metric in period 1 by > 10%
   • Decline: metric in period 2 < metric in period 1 by > 10%
   • Stable: < 10% change between periods
3. Assign trend badges per value: 🟢 Growing | 🔴 Declining | 🟡 Stable | 🆕 New | ⬜ Disappeared

Store: {dimensionId, periodComparison: {period1, period2, changes: [{value, p1Metric, p2Metric, pctChange, badge}]}, newValues: [], disappearedValues: []}

Phase 4 — Anomalies

For each dimension:

1. From the Phase 3 time-series, compute rolling mean and stddev per dimension value.
2. Z-score detection: flag (value, date) pairs where the z-score exceeds the threshold (default: 2.0; sensitive: 1.5; conservative: 3.0).
3. Threshold alerts:
   • Any single value holding > 50% of total metric on a given day
   • Value count that is > 2× the rolling average for that value
4. New/disappeared alerts: flag values that appear or disappear mid-period (from Phase 3).
5. Collect: anomaly type (spike, drop, new, disappeared, threshold), dimension value, date, magnitude.

Store: {dimensionId, anomalies: [{value, date, type, magnitude, zScore}]}

Phase 5 — Data Quality / Errors

For each dimension:

1. Search for known bad values using searchDimensionItems:
   • "Unspecified", "None", "(empty)", "", "null", "undefined", "N/A", "unknown"
2. Count occurrences with runReport filtering to each known bad value.
3. Compute: missing data % = (sum of bad value occurrences) / total occurrences.
4. Flag: dimensions where missing data > 5% (warning), > 20% (critical).
5. If the dimension has an expected format (URL, email, date), note it — but don't auto-validate patterns unless the user asks.

Store: {dimensionId, errorPatterns: [{pattern, count, pct}], missingDataPct, missingDataSeverity}

Phase 6 — Comparisons (multi-dimension or time-period)

This phase runs when the user is analyzing 2+ dimensions OR requests period comparison.

Side-by-side (2–3 dimensions):

1. For each dimension pair, compare cardinality level, skew, top-5 values, error rate.
2. Produce a comparison table: dimension A vs B vs C on each metric.

Time-period comparison (single dimension):

1. Compare two custom date ranges provided by the user (or auto-detect: first half vs second half).
2. For each value: metric in period 1, metric in period 2, delta, % change.
3. Surface the biggest movers (top 5 growing, top 5 declining).

Store: {comparisons: [{type, dimensions or periods, table}]}

Phase 7 — Forecasting

For each dimension with sufficient time-series data (>= 7 data points):

1. For the top 5–10 values by metric, fit a linear regression to the time series.
2. Project 7 periods forward.
3. Report:
   • Trend direction: Upward / Downward / Flat (based on slope)
   • Confidence: High (R² > 0.7), Medium (0.4–0.7), Low (< 0.4)
   • Projected value at end of forecast window
4. Flag values with strong upward trend (might become dominant) or strong downward trend (might disappear soon).

Store: {dimensionId, forecasts: [{value, slope, r2, direction, confidence, projectedValues: []}]}

Phase 8 — Report Generation

After all analysis phases complete:

1. Save all collected data to a JSON file: dimension_analysis_results_YYYY-MM-DD_HH-MM.json (in a temp output directory, e.g. /tmp/cja-dimension-analysis/, or a path the user specifies)

2. Run the Python report generator:

   python3 scripts/cja_dimension_analysis.py \
     <analysis_json> \
     "<data_view_name>" \
     "<data_view_id>" \
     [output_directory] \
     [--format=html|markdown] \
     [--keep-analyses=N]

   Options:

   • --format=html (default): Interactive HTML dashboard with Chart.js visualizations
   • --format=markdown: Comprehensive text-based report with tables
   • --keep-analyses=N (default: 0 = keep all): Auto-cleanup of old analysis files

3. The script generates a second output file: the report (HTML or markdown).

4. Open with open <output_directory>/dimension_analysis_report_*.html

5. Present the report path to the user and summarize key findings:

   • Dimensions with HIGH/VERY HIGH cardinality
   • Dimensions with extreme or high skew
   • Any anomalies found
   • Data quality issues above warning threshold
   • Forecast trends worth watching