Regression Consistency Checker

What it does

When your agent uses it

• Comparing two versions of code to detect regressions
• Verify refactoring safety
• Validate bug fixes dont break existing functionality
• Ensure backward compatibility

Example prompts

• “Use the regression-consistency-checker skill to check whether a new version of a repository preserves the behavior observed by tests on the old…”
• “/regression-consistency-checker”

Requirements

• Python 3

Workflow steps

8 steps, taken from the step headings in SKILL.md.

1. 1Prepare Versions
2. 2Run Tests on Old Version
3. 3Run Tests on New Version
4. 4Compare Results
5. 5Analyze Regressions
6. 6Investigate Root Causes
7. 7Document Findings
8. 8Fix or Accept Changes

What it can do on your machine

Read from SKILL.md and the folder at commit 4f38503. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves nothing: there is no allowed-tools line, so your agent's usual permission prompts apply.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  Ships 1 file in scripts/ (Python), which the agent can run.

  Shell commands in SKILL.md call:

  • git
  • python
  • pytest
  • npm

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  No URLs in SKILL.md. Its commands use git and npm, which can reach the network depending on how they are called.

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Regression Consistency Checker loads about 2.1k tokens when it runs, and up to ~5k if it reads all its reference files. Until then it costs about 191 tokens; SKILL.md has 451 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~191

When it runs · the whole SKILL.md, loaded when a task matches

~2.1k

With references · SKILL.md plus every file in references/, read only if the agent opens them

~5k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); the scripts in this folder are not scanned.

SKILL.md

The full file from ArabelaTso/Skills-4-SE at commit 4f38503, republished under its Apache-2.0 licence (© ArabelaTso). 451 words, ~2,092 tokens.

name

regression-consistency-checker

description

Checks whether a new version of a repository preserves the behavior observed by tests on the old version. Use this skill when comparing two versions of code to detect regressions, verify refactoring safety, validate bug fixes don't break existing functionality, or ensure backward compatibility. Detects differences in function outputs, exceptions, observable states, and performance between versions. Generates reports highlighting potential regressions (critical, high, medium, low severity), improvements, and areas requiring verification. Triggers when users ask to check for regressions between versions, compare test behavior across versions, verify behavior preservation, or validate that changes don't break existing tests.

# Tag or note the old version
git tag old-version

# Or checkout specific commit
git checkout <old-commit-hash>

# Tag the new version
git tag new-version

# Or checkout new commit
git checkout <new-commit-hash>

# Python (pytest with JSON report)
git checkout old-version
pytest --json-report --json-report-file=old_results.json

# JavaScript (Jest with JSON report)
git checkout old-version
npm test -- --json --outputFile=old_results.json

# Run multiple times to check stability
pytest --json-report --json-report-file=old_results_1.json
pytest --json-report --json-report-file=old_results_2.json
# Compare to ensure deterministic

# Python
git checkout new-version
pytest --json-report --json-report-file=new_results.json

# JavaScript
git checkout new-version
npm test -- --json --outputFile=new_results.json

python scripts/compare_results.py old_results.json new_results.json

# With custom tolerance for floats
python scripts/compare_results.py old_results.json new_results.json --tolerance 0.001

# Save detailed report
python scripts/compare_results.py old_results.json new_results.json --output regression_report.json

# Find commits that caused regression
git bisect start
git bisect bad new-version
git bisect good old-version
# Test each commit
git bisect run pytest path/to/failing_test.py

REGRESSION ANALYSIS REPORT
==========================

Version Comparison: v1.0.0 → v1.1.0
Date: 2024-01-15
Tests Run: 156

SUMMARY
-------
Critical Regressions: 2
High Severity: 5
Medium Severity: 3
Low Severity: 8
Improvements: 4
Unchanged: 134

CRITICAL REGRESSIONS
--------------------
1. test_user_authentication
   - Status: PASS → FAIL
   - Error: KeyError: 'user_id'
   - Root Cause: Removed field from response
   - Action: Restore field or update API contract

2. test_payment_processing
   - Status: PASS → FAIL
   - Error: AssertionError: expected 100.00, got 100.01
   - Root Cause: Rounding change in calculation
   - Action: Fix rounding logic

HIGH SEVERITY REGRESSIONS
--------------------------
1. test_data_export
   - Output changed: CSV format → JSON format
   - Impact: Breaking change for consumers
   - Action: Maintain backward compatibility

[... continue for all regressions ...]

EXPECTED CHANGES
----------------
1. test_error_messages
   - Error messages now include more context
   - Intentional improvement
   - Action: Update baseline

RECOMMENDATIONS
---------------
1. Fix critical regressions before release
2. Review high severity changes with team
3. Document breaking changes in changelog
4. Update tests for intentional changes

# Fix the code
git checkout new-version
# Make fixes
git commit -m "Fix: regression in user authentication"

# Re-run tests
pytest --json-report --json-report-file=fixed_results.json

# Verify fix
python scripts/compare_results.py old_results.json fixed_results.json

# Update baseline
cp new_results.json baseline_results.json

# Document in changelog
echo "- Changed: CSV export now returns JSON" >> CHANGELOG.md

old_output == new_output

abs(old_output - new_output) < tolerance

# Ignore timestamps, IDs
compare_ignoring_fields(old, new, ['timestamp', 'id'])

# Compare as sets
set(old_list) == set(new_list)

# Basic comparison
python scripts/compare_results.py old_results.json new_results.json

# Custom float tolerance
python scripts/compare_results.py old_results.json new_results.json --tolerance 0.001

# Save detailed report
python scripts/compare_results.py old_results.json new_results.json --output report.json

© ArabelaTso, Apache-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 2 other files (scripts, references) in skills/regression-consistency-checker of ArabelaTso/Skills-4-SE.

• SKILL.md
• references/detection_strategies.md
• scripts/compare_results.py

Open the folder on GitHubat commit 4f38503