QA

Slop-Guard QA Audit

Black-box QA of slop-guard through its public interfaces only. Do not read source files or tests to find bugs — discover issues by using the product as an agent or user would.

Launch parallel subagents per testing angle. File genuine issues with gh issue create. If $ARGUMENTS specifies a focus area (mcp, cli, fit, docs, workflows, or effectiveness), run only that angle. Otherwise run all 6 in parallel.

Setup

1. Read README.md for documented behavior.
2. If mcp__slop_guard__* tools are available, fetch their schemas via tool search.
3. Fetch open issues to avoid duplicates:

   gh issue list --repo eric-tramel/slop-guard --limit 100 --state open --json number,title,body,labels

4. Prefer local QA fixtures if present. Otherwise create temp files under /tmp/slop-guard-qa. Do not write throwaway files into the repo.

Pass the open issues list into every subagent prompt.

Testing Angles

1. MCP Agent Interface (mcp)

Is the MCP surface clear, consistent, and useful for an agent?

• Tool descriptions, parameter clarity, distinction between check_slop and check_slop_file
• JSON output structure: score, band, violations, counts, advice, file
• Edge cases: short text, empty strings, unicode, markdown, code fences, long inputs
• Missing/unreadable file paths
• Response size and parseability
• Cross-check MCP vs CLI JSON output on the same input

Issue prefix: mcp:

2. CLI UX (cli)

Does uv run sg behave predictably?

• All flags from sg --help: --json, --verbose, --quiet, --threshold, --score-only, --counts, --config, --version
• Input modes: file paths, inline text, stdin (-), multiple inputs
• Exit codes: 0 (pass), 1 (threshold fail), 2 (error)
• Stdout/stderr separation for scripting
• Path-vs-inline-text ambiguity
• Missing files, bad config paths

Issue prefix: cli:

3. Fit Workflow (fit)

Can a user fit a custom rule config from docs alone?

• Legacy shorthand: sg-fit TARGET_CORPUS OUTPUT
• Multi-input mode with --output
• All input formats: .jsonl, .txt, .md
• --negative-dataset, --no-calibration, --init
• Error cases: malformed JSONL, missing text field, invalid label, missing files, unsupported suffixes
• Round-trip: is the fitted JSONL immediately usable with sg -c?

Issue prefix: fit:

4. Docs and Onboarding (docs)

Do install flows and docs match actual behavior?

• Install paths: uvx slop-guard, uv tool install slop-guard, uv run sg
• MCP setup snippets for Claude and Codex
• Custom config examples
• Whether documented commands, flags, and tool names match reality

Issue prefix: docs:

5. End-to-End Workflows (workflows)

Does slop-guard compose well for real agent work?

1. Lint prose via MCP or CLI, assess whether advice is actionable enough to drive a rewrite
2. Lint a real file via check_slop_file or sg README.md
3. CI gating with sg -t 60 ... — is the output machine-parseable?
4. Compare MCP and CLI results on the same input

Focus on cross-surface consistency, scoring coherence, and whether the product actually improves agent writing workflows.

Issue prefix: workflow:

6. Writing Effectiveness (effectiveness)

Does slop-guard's feedback actually make an agent produce better writing? This angle tests the quality of the feedback loop — not whether the tool runs correctly, but whether following its guidance leads to measurably improved prose.

Method: Act as an agent that writes, scores, interprets advice, rewrites, and rescores. Evaluate every step for friction, ambiguity, and actual improvement.

6a. Advice Actionability

For each test, write a paragraph in a specific register (technical docs, blog post, marketing copy, academic summary, casual explanation), score it, then attempt to follow every piece of advice literally.

• Is each advice string specific enough to know what to change? ("Replace 'crucial' — what specifically do you mean?" is good; a vague suggestion would be bad)
• Does the advice tell you how to fix it, or only that something is wrong?
• When multiple violations overlap in the same sentence, does the combined advice make sense or contradict itself?
• Are there violations with no corresponding advice entry? (The advice array should cover every fixable issue)
• Does any advice lead you toward a different slop pattern? (e.g., replacing a slop word with another slop word)

6b. Feedback Loop Convergence

Run iterative rewrite cycles and track whether the score converges upward:

1. Write 3-5 paragraphs of deliberately sloppy AI-style prose (heavy on slop words, uniform rhythm, bold-bullet structures, contrast pairs)
2. Score via check_slop or sg --json
3. Rewrite following only the advice array — no independent judgment
4. Rescore the rewrite
5. Repeat until score stabilizes or advice is empty

Evaluate:

• Does the score improve monotonically with each cycle? If not, what caused a regression?
• How many cycles to reach clean band? (Should be 1-2 for light, 2-3 for moderate)
• Does the advice array shrink each cycle, or do new violations appear as old ones are fixed?
• Is there a score plateau where advice remains but following it doesn't move the score? What's blocking?
• At convergence, does the text actually read well — or has it been "optimized" into something sterile/awkward?

6c. Score Sensitivity and Proportionality

Test whether scores reflect actual writing quality differences:

• Write two versions of the same content: one natural, one AI-sloppy. Do scores separate cleanly?
• Take clean prose (human-written, score >80) and introduce one slop word. How much does the score drop? Is the penalty proportional?
• Take a short text (<50 words) vs a long text (~500 words) with the same density of violations. Are scores comparable?
• Does the concentration penalty feel right? Write text with 1 contrast pair (should be mild) vs 5 contrast pairs (should be harsh). Check that scoring reflects this.