Experimental Code Coverage Cq Debugger

State Management

Throughout the triage process, maintain a state file at scratch/triage_state.json. In Step 1, you MUST load the existing state file (initialized by the orchestrator) and initialize ALL variables defined in templates/triage_state_template.json in scratch/triage_state.json initially, even if many of them will initially be null. Update these variables in scratch/triage_state.json as they are resolved in subsequent steps.

Variables to Initialize:

When initializing variables in Step 1, reference templates/triage_state_template.json for the required variables and data contract. All variables from this schema MUST be loaded into scratch/triage_state.json at the start of Phase 1.

---

Commit Queue (CQ) Coverage Debugging Playbook

This skill defines the end-to-end triaging and resolution process for Gerrit Commit Queue (CQ) underreported coverage bugs. It guides the agent through Phase 1 (Information Gathering), Phase 2 (Audit Configuration), Phase 3 (Prepare Experiment), and Phase 4 (Validation).

[!IMPORTANT] Sequential Execution & Status Updates:

1. Sequential Execution: You must execute each step in the workflow sequentially. Do NOT skip any steps or proceed to a subsequent step until the current step has fully completed, returned its outputs, and validated its constraints.
2. Status Updates: You MUST update the status field in scratch/triage_state.json whenever transferring to another step (i.e., at the end of each step) to track progress. The status value should reflect the step name or number just completed (e.g., "CQ Triage: Step 5 Completed"). The final terminal status values must be set to either "FIXED" or "ESCALATED".

---

Phase 1: Information Gathering

Step 1: Initialize State File

• Action: Load scratch/triage_state.json and initialize/append ALL variables defined in templates/triage_state_template.json in scratch/triage_state.json (setting unresolved variables to null or empty dictionaries). Ensure all existing variables already defined in scratch/triage_state.json (such as bug_id, status, bug_details, etc., initialized by the orchestrator) are preserved.
• Required Outputs: All schema variables appended and initialized in scratch/triage_state.json.

Step 2: Scrape Gerrit CL Details & Verify Repository

• Action: Extract the CL URL from bug_details in scratch/triage_state.json and store it in the original_cl variable in scratch/triage_state.json. Then, verify the repository and retrieve modified files for original_cl. (Note: You can use tools/code_coverage/parse_gerrit_url.py to extract host, project, change, and patchset details from any CL URL when needed).
• Skill: Use the gerrit-cli skill.
• Required Outputs: original_cl (the CL URL) and the list of modified_files. Update these variables in scratch/triage_state.json.
• Verify Repository Contract: Confirm that the target CL belongs strictly to the chromium/src repository. If the CL targets recipe or infrastructure repositories (e.g. infra/build), stop immediately and inform the user that recipe triage is out of scope for this playbook.
• State: Update original_cl and modified_files in scratch/triage_state.json.
• Note: If the CL URL cannot be extracted from bug_details or is not provided, prompt the user to get it.

Step 3: Validate Against Bug Description & Filter Target Files

• Action: Cross-reference modified_files (retrieved in Step 2) with the developer's bug report (found in bug_details in scratch/triage_state.json).
• Filter Files & Metrics:
  • Files: Identify which specific application source files (.cc, .cpp, .java, etc.) the developer is complaining about in bug_details. Set target_files to only contain those specific files matching the bug report. If the bug report does not specify files, keep all modified application source files from modified_files.
  • Metrics: Identify which specific coverage metrics the developer is complaining about in bug_details. Set metrics_of_concern to only contain matching metrics from the standard options ("absolute_coverage", "incremental_coverage", "absolute_unit_tests_coverage", "incremental_unit_tests_coverage"). If the bug report does not specify, keep all four standard coverage metrics.
• Required Outputs: The resolved target_files list and metrics_of_concern list under investigation. Update both target_files and metrics_of_concern in scratch/triage_state.json.

Step 4: Read Coverage Metrics

• Action: Fetch code coverage metrics and unexecuted line details strictly for target_files using a sub-agent.
• Invocation: Launch a metrics_reader sub-agent equipped with the experimental-read-code-coverage-metrics skill. You MUST explicitly pass cl_url (read original_cl from scratch/triage_state.json), artifact_path: "scratch/original_metrics.json", and target_files (read from scratch/triage_state.json) in the subagent prompt.
• Gate: Wait for the metrics_reader sub-agent to complete before proceeding.
• Required Outputs: The subagent records coverage metrics and unexecuted code lines strictly for target_files in scratch/original_metrics.json.

Step 5: Map Source Files to Test Suites

• Action: Identify which test suites (e.g. unit_tests, browser_tests) compile the modified target files by calling test_suite_mapper.py directly.
• Command:

  vpython3 tools/code_coverage/test_suite_mapper.py <target_file>

  (For additional flags such as --build-dir or historical --revision mapping, see vpython3 tools/code_coverage/test_suite_mapper.py --help).
• Required Outputs: The list of mapped test_suites for the target files. Update test_suites in scratch/triage_state.json.

Step 6: Identify Builders of Concern

• Action: Identify the initial list of builders_of_concern based on the file language:
  • Java Files: Use only the Android try builders:
    • android-arm64-rel, android-x86-rel, android-12-x64-rel, android-desktop-x64-rel (android-internal-desktop-x64-rel for internal developers), android-cronet-x64-dbg-14-tests
  • JavaScript / TypeScript (WebUI) Files: Use only the JS coverage try builders:
    • linux-js-coverage-rel, chromeos-js-coverage-rel
  • C++ Files: Use all standard platform builders (none are ruled out):
    • Linux: linux-rel
    • Mac/iOS: mac-rel, ios-simulator, ios-simulator-full-configs
    • Windows: win-rel
    • ChromeOS: linux-chromeos-rel
    • Android: android-arm64-rel, android-x86-rel, android-12-x64-rel, android-desktop-x64-rel (android-internal-desktop-x64-rel for internal developers), android-cronet-x64-dbg-14-tests
• Note on Android Desktop Configurations:
  • The public android-desktop-x64-rel CQ bot and its internal equivalent android-internal-desktop-x64-rel both inherit their base build configuration from infra/config/subprojects/chromium/try/tryserver.chromium.android.desktop.star (via the equivalent_builder property).
  • For specific test suite configurations on internal Clank bots, check the *.pyl files inside https://chrome-internal.googlesource.com/clank/internal/apps/+/HEAD/build/bot.
• Required Outputs: The initial list of builders_of_concern. Update builders_of_concern in scratch/triage_state.json.

---

Phase 2: Audit Configuration Patching

Step 7: Check & Patch Test Suite Execution on Builders

• Action: For each builder in builders_of_concern:
  • Verify if the mapped test suites are configured to run on it.
  • If a mapped test suite is NOT configured to run on the builder, evaluate whether the test suite would be compatible to run with it. At your discretion, if compatible:
    1. Create a new branch from main (e.g., verify-coverage-fix) if fix_branch_name is None, and update fix_branch_name in scratch/triage_state.json. Check out fix_branch_name.
    2. Add the test suite configuration to the builder in Starlark.
    3. Regenerate the configuration files:

       lucicfg generate infra/config/main.star

    If fundamentally incompatible, stop or exclude the builder.

• Required Outputs: Confirmation that test suites are configured (or added) on all builders_of_concern, and update fix_branch_name in scratch/triage_state.json (if created).

Step 8: Audit Builder Configurations (Parallel Audit)

• Action: Audit the builder configurations for all builders_of_concern in parallel by invoking a specialized sub-agent for each builder.
• Invocations: Launch a builder_auditor sub-agent for each builder in builders_of_concern. Equip each sub-agent with the experimental-code-coverage-config-validator skill and the experimental-ci-only-validator skill. When launching each sub-agent, instruct it to:
  1. Pass builders_of_concern and language_context ("cpp", "objc", "rust", "java", "js", or "ts", inferred from target_files) to experimental-code-coverage-config-validator to validate GN arguments.
  2. Run experimental-ci-only-validator to verify if any of the mapped test_suites (from scratch/triage_state.json) are barred by ci_only = True on that builder (checking all mapped test suites, not just one).
• Gate: Wait for all builder_auditor sub-agents to return their results before proceeding to Phase 3.
• Resolution: If any sub-agent reports GN argument validation failures or that any test suite is barred by ci_only = True:
  1. Apply all suggested GN arg fixes to the Starlark files.
  2. Remove the ci_only = True flag for any barred test suite on that builder in Starlark.
  3. Commit all configuration fixes together in a single commit to the fix_branch_name branch (creating the branch from main if fix_branch_name is None).
  4. Regenerate the configuration files:

     lucicfg generate infra/config/main.star

• Required Outputs: Confirmation that GN arguments and ci_only settings are valid across all builders_of_concern and all mapped test_suites. If configuration fixes were applied, update fix_branch_name in scratch/triage_state.json (if created).

Step 9: Compile Configuration Fixes Summary

• Action: Compile a summary of all configuration audits and fixes performed in Step 7 and Step 8 across all builders_of_concern.
• Required Outputs: Create a markdown artifact at scratch/config_summary.md documenting whether any test suites were added or patched, any GN arguments were corrected, or any ci_only flags were removed (or confirming that existing builder configurations were valid without changes).

---

Phase 3: Prepare Experiment

Step 10: Analyze Required CLs & Branches

• Action: Determine CL strategy based on fix_branch_name:
  • fix_branch_name is NOT None: We need two branches:
    1. An Experimental Branch (already exists as fix_branch_name).
    2. A Baseline Branch (create a new branch from main, e.g. verify-coverage-baseline, and set control_branch_name to this branch).
  • fix_branch_name is None: We only need one branch:
    1. A Baseline Branch (create a new branch from main, e.g. verify-coverage-baseline, and set control_branch_name to this branch).
• Required Outputs: The branch name control_branch_name and, if it wasn't already created, fix_branch_name. Update control_branch_name and fix_branch_name in scratch/triage_state.json.

Step 11: Generate CLs (Parallel CL Generation)

• Action: Generate the baseline and, if necessary, experimental CLs in parallel by invoking a specialized sub-agent for each CL.
• Invocations: Launch up to two cl_generator sub-agents in parallel (one for the baseline branch and one for the experimental fix branch if created). Equip both sub-agents with the experimental-code-coverage-verification-prep skill to inject dummy comments, optimize Starlark configs, and upload CLs to Gerrit. When launching each cl_generator sub-agent, instruct it to read its primary inputs (target_files, builders_of_concern, test_suites, bug_id) directly from scratch/triage_state.json. You MUST pass role ("Control" or "Fix"). For "Fix" runs, also pass in a summary of the configuration and recipe fixes applied (e.g. from scratch/config_summary.md, scratch/swarming_summary.md, or scratch/local_debugging.md) so the sub-agent can generate a descriptive CL description.
• Gate: Wait for all cl_generator sub-agents to return their results before proceeding to Step 12.
• Required Outputs: The Gerrit CL URLs for control_cl and fix_cl (if applicable). Update control_cl and fix_cl in scratch/triage_state.json.
• Note: If control_cl is already instantiated with a non-null value in scratch/triage_state.json (such as during subsequent iterations or if pre-populated), skip generating control_cl and do NOT launch a cl_generator sub-agent for the control run. Only launch the sub-agent for the experimental fix branch (role: "Fix").

Step 12: Trigger Try Jobs (Parallel Execution)

• Action: Launch try jobs on the target builders for the prepared CLs in parallel using sub-agents.
• Invocations: Launch up to two build_invoker sub-agents in parallel (one for the control CL and one for the fix CL if created). Equip both sub-agents with the experimental-code-coverage-build-invoker skill. When launching each sub-agent, instruct it to:
  1. For control_cl: pass builders_of_concern and the designated control_cl URL (read from scratch/triage_state.json).
  2. For fix_cl (if created): pass builders_of_concern and the designated fix_cl URL (read from scratch/triage_state.json).
• Gate: Wait for all build_invoker sub-agents to return their results before proceeding to Phase 4.
• Required Outputs: The Buildbucket build links/URLs for all triggered try jobs, mapped to their builder name inside control_builds (for control CL) or fix_builds (for fix CL). Update control_builds and fix_builds in scratch/triage_state.json.
• Note: If control_builds is already instantiated with non-empty values in scratch/triage_state.json (such as during subsequent iterations or if pre-populated), skip triggering try jobs for control_cl and do NOT launch a build_invoker sub-agent for the control run. Only trigger try jobs for fix_cl.

---