Assess Covariance Method

Assess a new covariance/correlation methodology

A protocol for turning "here's a covariance idea" into a defensible verdict. Work the steps in order; stop early only if a step fails. Install: pip install precise[research] (the research/ scripts use scikit-learn and randomcov).

0. Classify the method first

• Estimator (produces a matrix) vs assessor (scores a matrix)? Different paths below.
• Online (updatable per observation) or batch? precise is an online library; a batch method can still be wrapped, but say so.
• Does it target the covariance, correlation, or precision? It must expose, or be convertible to, covariance_.
• Does evaluating it need the ground-truth covariance (simulation only) or work on real data?

1. Implement it to the contract

Subclass BaseOnlineCovariance. The cheapest correct route: copy the simplest existing estimator and modify it — read precise/empirical.py (and precise/base.py for the exact hook signatures, typically _init_state / _update_state / _state_to_cov / _state_to_mean). The base class owns partial_fit, fit, and the derived attributes; the subclass only supplies the state update and the map to a covariance. Register it in precise/registry.py so all_estimators() includes it.

A scoring rule instead? Implement an Assessor (see precise/assessment/), set needs_data / needs_truth, and follow the higher = better convention.

2. Conformance — non-negotiable

Run the conformance suite (tests/test_estimators.py parametrizes over all_estimators()), or check the invariants directly:

• covariance_ is symmetric and PSD (eigenvalues ≥ 0);
• correlation_ has unit diagonal; precision_ @ covariance_ ≈ I when well-conditioned;
• streaming rows via partial_fit equals fit(X) for non-windowed estimators;
• set_state(get_state()) round-trips;
• it runs on the numpy-only install (no hidden heavy deps).

If it fails any of these, fix the implementation before any benchmarking — numbers from a non-conformant estimator are meaningless.

3. Pick the right judge BEFORE looking at results

This is where most covariance evaluations go wrong. In high dimensions (p comparable to n), do not rank estimators by the held-out Gaussian likelihood — it is dominated by unidentifiable small eigenvalues and ranks estimators below chance. Use the assessor panel and choose by regime (see the score-covariance-estimate skill): BlockPseudoLikelihood / SchurLikelihood / VariogramScore / GMVVariance in high-d; LogLikelihood only when low-d and well-conditioned; a QLIKE/Bregman-consistent loss (not RMSE) when the target is a noisy variance proxy. To choose defensibly, measure the statistical power of candidate judges — the probability they reproduce a known quality ordering — with research/metric_power.py.

4. Benchmark against the registry

research/bakeoff.py runs every estimator over discriminating scenarios and scores them with all_assessors(). Add the new estimator and compare. Report relative error vs a naive / shrinkage baseline within each scenario, then averaged (RMSE is scale-sensitive and can be won by doing well only in high-volatility regimes). Always include a 0/historical-mean baseline for returns-like targets.

5. Sweep the data-generating process — results are ensemble-sensitive

A win on one generator need not transfer. Generate ground truth across several ensembles (LKJ, Wishart, factor/spiked, Toeplitz/AR, equicorrelation) with randomcov plus plain numpy, sample from each, and report per-ensemble, not just pooled. A single ensemble can manufacture or hide any effect.