Fix Broken Datadog Provider Tests

What it does

When your agent uses it

• Fixing a failing TestAcc integration test in terraform-provider-datadog
• Checking whether a reported test failure still reproduces in recent CI runs
• Turning a Jira ticket about broken tests into a draft pull request
• Triaging a flaky provider test and recording why it fails

Example prompts

• “Fix the failing TestAccDatadogDashboard test and open a draft PR.”
• “Our integration run fails with a bad request error on monitor creation, so find the test behind it and reproduce it locally.”
• “Look at the Jira ticket for the broken synthetics tests and work out which tests are affected.”

Requirements

• GitHub CLI (gh) for reading CI runs
• The Jira CLI (acli) for ticket lookups
• A local checkout of the Datadog Terraform provider repository
• Pre-approved tools (allowed-tools): Read, Write, Edit, Glob, Grep, Bash, Agent, Skill, TaskCreate, TaskUpdate, TaskGet, AskUserQuestion

Workflow steps

8 steps, taken from the step headings in SKILL.md.

1. 1Identify Failing Tests
2. 2Validate in Recent CI
3. 3Reproduce Locally
4. 4Diagnose and Plan Fix
5. 5Execute the Fix
6. 6Commit and Open Draft PR
7. 7Monitor Integration Tests
8. 8Report Results

What it can do on your machine

Read from SKILL.md and the folder at commit ca0801d. It shows what the files ask for, not the result of running them.

• Tool permissions

  Pre-approves these tools, so the agent can use them without asking each time:

  • Read
  • Write
  • Edit
  • Glob
  • Grep
  • Bash
  • Agent
  • Skill
  • TaskCreate
  • TaskUpdate

  …and 2 more on the same allowed-tools line.

  From allowed-tools in the SKILL.md frontmatter.

• Runs code

  Shell commands in SKILL.md call:

  • gh
  • git
  • make

  From the folder's file list and the shell code blocks in SKILL.md.

• Network

  No URLs in SKILL.md. Its commands use gh and git, which can reach the network depending on how they are called.

  From URLs in SKILL.md, links to its own repository left out.

• Credentials

  Names no API keys, tokens, secrets or passwords.

  From names ending in _API_KEY, _TOKEN, _SECRET, _KEY or _PASSWORD in SKILL.md.

Context cost

Fix Broken Datadog Provider Tests loads about 2.4k tokens when it runs. Until then it costs about 107 tokens; SKILL.md has 836 words of instructions outside code blocks.

Always · name and description, kept in context so the agent knows when to use it

~107

When it runs · the whole SKILL.md, loaded when a task matches

~2.4k

Estimates: characters ÷ 4, the usual rule of thumb; real counts depend on the model's tokenizer. Scripts and assets cost tokens only if the agent reads them.

Safety

• NotePre-approves every shell command (allowed-tools: Bash)SKILL.md

  allowed-tools: Read, Write, Edit, Glob, Grep, Bash, Agent, Skill, TaskCreate, TaskUpdate, TaskGet, AskUserQuestion

Automated static check — not a guarantee. Review scripts before installing. It scans the text of SKILL.md for risky patterns (piping downloads into a shell, reading credential files, hidden Unicode, destructive commands); files beside SKILL.md are not scanned.

SKILL.md

The full file from DataDog/terraform-provider-datadog at commit ca0801d, republished under its MPL-2.0 licence (© DataDog). 836 words, ~2,356 tokens.

name

fix-broken-integration-tests

description

End-to-end workflow to diagnose, reproduce, fix, and validate a failing Datadog Terraform provider integration test. Takes any input pointing at specific tests: test function names, error messages, resource names, or any other description of what's failing. Runs autonomously through 8 phases — identify → validate in CI → reproduce locally → plan fix → execute → open draft PR → monitor → report.

allowed-tools

Read, Write, Edit, Glob, Grep, Bash, Agent, Skill, TaskCreate, TaskUpdate, TaskGet, AskUserQuestion

user-invocable

true

argument-hint

<test name(s), error description, or other context about the failing test>

REPO_ROOT=$(git rev-parse --show-toplevel)

acli jira workitem view <TICKET-ID> --fields 'summary,status,description' 2>&1

Grep pattern "<keyword>" path "datadog/tests/" glob "*_test.go"

grep -A4 "<TestName>" flaky_tests.yaml

TaskCreate: "Fix integration tests: <list of tests>"

gh run list --workflow test_integration.yml \
  --repo DataDog/terraform-provider-datadog \
  --branch master --limit 5 --json databaseId,conclusion,createdAt

gh run view <RUN_ID> --log-failed \
  --repo DataDog/terraform-provider-datadog 2>&1 \
  | grep -E "FAIL.*<TestName>" | head -20

Skill: "dd-tf-provider-test-runner"
Args: "Test pattern: <TestName1>|<TestName2>  Record mode: none  Working directory: <value of REPO_ROOT>"

Read: .claude/skills/fix-broken-integration-tests/fix-patterns.md

git checkout -b fix/<resource>-integration-test

Skill: "dd-tf-provider-test-runner"
Args: "Test pattern: <TestName>  Record mode: true  Working directory: <REPO_ROOT>"

Skill: "dd-tf-provider-test-runner"
Args: "Test pattern: <TestName>  Record mode: false  Working directory: <REPO_ROOT>"

make fmtcheck
make test

git add <changed files>
git commit -m "[datadog_<resource>] Fix integration test — <brief root cause>

<one paragraph explaining what was failing and why, and how it is fixed>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>"

git push -u origin <branch-name>

gh pr create --draft \
  --title "[datadog_<resource>] Fix integration test — <brief description>" \
  --label "ci/integrations" \
  --label "<changelog/bugfix or changelog/no-changelog>" \
  --body "$(cat <<'EOF'
## Summary

- **Failing tests:** `<TestName1>`, `<TestName2>`
- **Root cause:** <one-line root cause>
- **Fix:** <one-line fix description>

## Details

<paragraph about what was failing and the error observed in CI>

<paragraph about the fix approach and what changed>

## Test plan

- [ ] Tests pass with `RECORD=none` locally
- [ ] Tests pass in CI integration run (triggered by `ci/integrations` label)
EOF
)"

# Wait for the run to be created (retry up to 10 minutes)
gh run list --repo DataDog/terraform-provider-datadog \
  --branch <branch-name> --workflow test_integration.yml \
  --limit 3 --json databaseId,status,conclusion,createdAt

# Check run status
gh run view <RUN_ID> --repo DataDog/terraform-provider-datadog

gh run view <RUN_ID> --log-failed --repo DataDog/terraform-provider-datadog 2>&1 \
  | grep -E "FAIL.*<TestName>|Error:" | head -40

© DataDog, MPL-2.0. Rendered from Markdown: HTML in the file is shown as text, images as links, and headings moved down two levels. Raw file

Files

SKILL.md and 1 other file in .claude/skills/fix-broken-integration-tests of DataDog/terraform-provider-datadog.

• SKILL.md
• fix-patterns.md

Open the folder on GitHubat commit ca0801d