Audit Filters

Audit Filters

When to Use

• Analyzing negative feedback (downvotes, deletions) on AI comments to find recurring patterns
• Suggesting new filter.yaml exception rules for a language based on feedback themes
• Opening a PR with proposed filter additions after user confirmation

Overview

This is a multi-phase workflow:

1. Collect — Pull feedback and memories for the specified language and time period
2. Analyze — Categorize feedback by reason, theme, and IsGeneric status; identify recurring bad-comment patterns
3. Recommend — Propose new numbered DO NOT ... lines for metadata/{lang}/filter.yaml
4. Confirm — Present recommendations and ask the user whether to proceed
5. PR — Create a branch, apply changes, commit, push, and open a pull request

Defaults

Unless the user says otherwise, always apply these defaults:

• Environment: production
• Feedback types: Focus on bad and delete feedback (exclude good)
• Format: JSON (redirect to file)

Language Resolution

Map the user's language name to the metadata directory name:

Date Resolution

The user will typically specify a calendar month by name (e.g. "March", "January 2025"). Resolve to the full month date range:

When only a month name is given without a year, use the current year. Be careful with month lengths (28/29/30/31 days).

Phase 1: Collect Data

Run both commands sequentially in the same foreground terminal. Use a 120-second timeout for each.

Step 1a: Pull feedback

New-Item -ItemType Directory -Path output -Force | Out-Null; if (Test-Path output/feedback_output.json) { Remove-Item output/feedback_output.json }; python cli.py report feedback -s <start_date> -e <end_date> -l <language> --exclude good --include-implicit | Out-File -Encoding UTF8 output/feedback_output.json

Step 1b: Pull memories

if (Test-Path output/memory_output.json) { Remove-Item output/memory_output.json }; python cli.py report memory -s <start_date> -e <end_date> -l <language> | Out-File -Encoding UTF8 output/memory_output.json

After both commands complete, read both output files with read_file.

Phase 2: Analyze

Read the current filter file at metadata/{lang}/filter.yaml so you know what rules already exist.

Then analyze the collected feedback and memories. Produce a summary organized as follows:

Analysis Structure

By Feedback Reason — Group comments by their Feedback[].Reasons values (e.g. AcceptedRenderingChoice, FactuallyIncorrect, RenderingBug, NotRelevant, TooNitpicky, Other). For each reason, count occurrences and list representative CommentText excerpts.

By Theme — Identify recurring themes across the bad comments. A theme is a pattern you can describe in one sentence (e.g. "commenting on interface method implementations", "suggesting consolidating overloads"). Include the count of comments matching each theme.

By IsGeneric Status — Report how many bad comments had IsGeneric: true vs false. Generic comments are not tied to a specific guideline and are more likely candidates for filter rules.

By Submitter — Note which users (Feedback[].SubmittedBy) provided the most feedback. The most significant contributor will be used as the PR assignee.

Cross-reference with Memories — Check if any memories (especially those with is_exception: true) suggest filter rules that are not yet in filter.yaml.

Present this analysis to the user in a clear summary table or grouped list.

Phase 3: Recommend Filter Additions

Based on the analysis, propose specific new lines to add to metadata/{lang}/filter.yaml. Each recommendation must:

1. Follow the existing format: N. DO NOT <description>
2. Be numbered sequentially after the last existing rule
3. Not duplicate an existing rule — Before proposing a rule, compare it against every existing rule in the current filter.yaml. If an existing rule already covers the same behavior (even with different wording), do NOT propose it again. Explain in the analysis that the theme was already covered and cite the existing rule number.
4. Be phrased as a clear, actionable instruction the LLM can follow

Signal Strength

When presenting recommendations, clearly label each with its signal strength:

• Strong signal: 2+ explicit feedback items (downvotes with reasons) or 1 memory with is_exception: true
• Low signal: Only 1 explicit feedback item, or only implicit bad comments (no explicit downvote/reason)

Do NOT automatically exclude low-signal items. Present ALL actionable patterns to the user with their signal strength clearly marked, and let the user (or reviewer) decide whether to include them in the PR.

Present the recommendations in a numbered list, each with:

• The proposed rule text
• The evidence (feedback count, representative comment texts, memory references)
• Whether the pattern was IsGeneric or guideline-linked

Example recommendation format:

Proposed rule 8: DO NOT comment on explicit interface implementations for serialization (IJsonModel, IPersistableModel)

• Evidence: 4 bad comments with reason FactuallyIncorrect, all IsGeneric: true
• Example: "Interface method implementation for AzureAISearchIndex is unexpected here"