---
name: pytrendy
description: Use when working on pytrendy code, tests, or data. Covers the 5-stage pipeline, module map, install/verify, datasets, PyTrendyResults API, and docs tooling.
---

# pytrendy package architecture

## 5-stage pipeline

`pytrendy/detect_trends.py` — the only entrypoint that matters. `detect_trends(df, date_col, value_col, plot=True, method_params=None, debug=False)` runs:

```text
df → process_signals → get_segments → refine_segments → analyse_segments → [plot_pytrendy] → PyTrendyResults
```

Stages mutate a copy of the input DataFrame and a list of segment dicts. The final `PyTrendyResults` wraps the segment list.

## Module map

```text
pytrendy/
├── __init__.py            # public exports: detect_trends, load_data, plot_pytrendy, dtw
├── detect_trends.py       # pipeline orchestrator (85 lines — read it first)
├── process_signals.py     # Savitzky-Golay smoothing + rolling stats → flat/noise/trend flags
├── simpledtw.py           # DTW for gradual-vs-abrupt classification (standalone, no deps on rest)
├── io/
│   ├── data_loader.py     # load_data('series_synthetic' | 'classes_signals')
│   ├── plot_pytrendy.py   # annotated matplotlib viz; called by detect_trends when plot=True
│   ├── results_pytrendy.py # PyTrendyResults: .print_summary(), filtering, ranking, tabular access
│   └── data/              # series_synthetic.csv, classes_signals.csv (packaged CSVs)
└── post_processing/
    ├── segments_get.py        # extract contiguous segments from trend_flag; min-length filter
    ├── segments_refine/       # package — the heaviest stage
    │   ├── __init__.py        # refine_segments() orchestrator
    │   ├── gradual_expand_contract.py  # boundary adjust via local extrema (±7d window)
    │   ├── trend_classify.py          # DTW-based gradual/abrupt labeling
    │   ├── abrupt_shaving.py          # z-score outlier changepoint detection on abrupt segs
    │   ├── segment_grouping.py        # merge short consecutive same-direction segments
    │   ├── artifact_cleanup.py        # remove invalid segments, fill gaps with flats
    │   └── update_neighbours.py       # boundary cascade when a neighbor changes
    └── segments_analyse.py    # per-segment metrics: change, duration, SNR, change_rank
```

## Datasets

`load_data(name)` reads CSVs shipped in `pytrendy/io/data/`:

- `'series_synthetic'` — synthetic daily time series with embedded up/down/flat regions. The README quickstart and core plot tests use this.
- `'classes_signals'` — reference signals used internally by `trend_classify` for DTW alignment (gradual vs abrupt templates).

Adding a dataset = drop a CSV in `pytrendy/io/data/` + extend `data_loader.py`. It's a plain `pd.read_csv` wrapper, not a registry.

## PyTrendyResults (`io/results_pytrendy.py`)

Returned by `detect_trends()`. Constructed from the segment list and auto-populates `best`, `df`, `df_summary`, `summary` in `__init__`.

### Attributes

| Attr | Type | What it is |
|---|---|---|
| `.segments` | `list[dict]` | Raw segment dicts (all directions including Flat/Noise). |
| `.trend_segments` | `list[dict]` | Subset with a `trend_class` key (Up/Down only — Flats/Noise excluded). |
| `.best` | `dict \| None` | Segment with lowest `change_rank` among `trend_segments`; `None` if no trends. Picked on `total_change` magnitude (steepness × length), not raw `change`. |
| `.df` | `pd.DataFrame` | Full segment table, indexed by `time_index`. All columns: `direction, start, end, trend_class, change, pct_change, days, total_change, SNR, change_rank` (+ `padded` when `abrupt_padding>0`). |
| `.df_summary` | `pd.DataFrame` | Slimmer view: `direction, start, end, days, total_change, change_rank, trend_class`. What `.print_summary()` prints. |
| `.summary` | `dict` | `direction_counts` (Counter→dict over Up/Down/Flat/Noise), `trend_class_counts` (gradual/abrupt), `highest_total_change`. |

### Methods

- `.print_summary()` — prints direction counts, best trend (direction + date range), and the `df_summary` table. **Use for high-level overview**; cheap to paste into an LLM context when developing.
- `.filter_segments(direction='Any', sort_by='time_index', format='df')` — the workhorse for narrowing focus.
  - `direction`: `'Any'` | `'Up/Down'` | `'Up'` | `'Down'` | `'Flat'` | `'Noise'`.
  - `sort_by`: `'time_index'` (ascending) or `'change_rank'` (descending by `abs(total_change)`).
  - `format`: `'df'` (default) or `'dict'`.
  - Returns DataFrame/list; slice with `[:3]` for top-N. Invalid `direction`/`sort_by`/`format` print a hint and fall through.

### Usage patterns (from the example notebooks)

- **High-level overview** → `results.print_summary()`. One-glance counts + best trend.
- **Debugging one direction** → `results.filter_segments(direction='Up')` to isolate uptrends only (the abrupt notebook does this to compare uptrends against each other).
- **Rank the strongest trends** → `results.filter_segments(direction='Up/Down', sort_by='change_rank')`. Rank 1 = steepest+longest by `total_change` magnitude.
- **Top-N** → `results.filter_segments(direction='Up', sort_by='change_rank')[:3]`.
- **Full per-segment metrics** → `results.df` (includes `change`, `pct_change`, `SNR` that `df_summary` omits).
- **Raw dicts for downstream code** → `results.filter_segments(format='dict')` or `results.segments` / `results.trend_segments`.
- **Best single segment** → `results.best` (dict).

### Metric columns (what they mean)

- `change` / `total_change` — cumulative sum of day-over-day differences across the segment.
- `pct_change` — relative change; e.g. `3.67` = +367%. Useful for comparing trends of different baselines.
- `days` — segment length.
- `SNR` — signal-to-noise ratio. **Lower SNR ≈ noisier segment.** Noise segments typically have SNR < ~7; clean gradual trends sit around 17-22. Borderline trends (e.g. SNR ~5.9) show up with high `change_rank` numbers — still detected but flagged as weak.
- `change_rank` — 1 = strongest trend by `abs(total_change)`. Lower is stronger. Artifact trends (short, low-magnitude) get high rank numbers and are effectively de-prioritized.
- `padded` — `True` only when `abrupt_padding>0` extended an abrupt segment's end date; `NaN` for gradual/flat/noise.
- `trend_class` — `gradual` / `abrupt` / `NaN` (NaN for Flat/Noise).

### Use-case notes (from the notebooks)

- **Gradual trends** — the default case; `trend_class='gradual'` for all Up/Down. Compare via `filter_segments(direction='Up', sort_by='change_rank')` to rank competing uptrends.
- **Abrupt trends** — short, sharp shifts (changepoint-style). Pass `method_params=dict(abrupt_padding=N)` to extend each abrupt segment N days forward (for Interrupted Time Series / Pre-Post analysis). Padding stops early if the signal reverses; segments gain a `padded=True` column. Combine with gradual in one run — DTW classifies each segment independently, so a single `detect_trends()` call returns both classes.
- **Noise** — `direction='Noise'` segments are flagged by a rolling SNR threshold, not by magnitude. Use `filter_segments(direction='Noise')` to inspect what got rejected. As noise std rises, trends shorten and more Noise/Flat segments appear; borderline trends keep high `change_rank` so they're easy to filter out.
- **Noise control via `avoid_noise`** — `method_params=dict(avoid_noise=False)` opts out of noise detection entirely. Spikes and noisy regions are ignored and trend detection proceeds straight through them. **Use case**: new-market launches or quasi-experiments where the signal is zero before and after an activation window — with `avoid_noise=True` (default), the step-change boundaries produce Noise artifacts; with `avoid_noise=False`, you get clean Up/Down segments. See `whats-new.md` v1.2.0 entry / PR #110.

Tests exercise this heavily in `tests/test_io_results.py` (all `@pytest.mark.core`) — any API change there breaks the core suite immediately. Read `results_pytrendy.py` before extending the class.

## Segment classification

- **direction** ∈ {`Up`, `Down`, `Flat`, `Noise`} — assigned during `process_signals`/`segments_get`.
- **trend_class** ∈ {`gradual`, `abrupt`, `NaN`} — assigned by `segments_refine/trend_classify.py` via DTW against `classes_signals` references. Only Up/Down segments get a class; Flat/Noise stay NaN.
- **Min segment length**: Up/Down ≥3 days, Flat/Noise ≥1 day (enforced in `segments_get.py`).

## `method_params` (the only tuning surface)

```python
{'abrupt_padding': 0, 'avoid_noise': True}  # defaults; see maintenance skill for deprecation note
```

`detect_trends` reconstructs this dict from an allowlist — unknown keys are dropped. Adding a tunable = update the allowlist in `detect_trends.py:72` + the docstring + add a test.

## Install & verify

During development you can import `pytrendy` directly — no rebuild step needed.
The commands below are primarily for end users or when setting up a fresh environment.

```bash
pip install -e ".[dev]"           # dev: pytest, pytest-cov, pytest-mpl, pytest-timeout
pip install -e ".[dev,docs]"      # add mkdocs material + mkdocstrings for docs work
```

**Verify before push** — CI runs core first, then non-core with `--cov-append`:

```bash
pytest tests/ -m core                       # essential, must pass
pytest tests/ -m "not core" --cov-append    # the rest
# or just: pytest  (pyproject addopts already set --mpl --cov=pytrendy)
```

- 15s per-test timeout (`pytest-timeout`).
- Plot tests use **pytest-mpl** with baselines pinned to **matplotlib==3.10.8** (hard pin in `pyproject.toml`). Regenerating baselines on another version = false failures. See the `test` skill.
- `coverage.xml` + `.coverage` are gitignored artifacts; don't commit.

## Docs

- MkDocs Material + mkdocstrings (Google-style). New page → register under `nav:` in `mkdocs.yml`.
- `mkdocs serve` for local preview. PRs touching `docs/`, `mkdocs.yml`, or `pytrendy/` auto-deploy a preview at `russellsb.github.io/pytrendy/pr-<N>/` (bot comments the URL; removed on PR close; fork PRs skipped).
- JupyterLite notebooks in `docs/examples/` — run `scripts/normalize_notebooks.sh` before building docs (works around a JupyterLite text-output rendering bug).