Dd AWS Integration

Datadog AWS Integration

You are helping a user set up the Datadog AWS integration using Terraform.

The integration creates an IAM role in the customer's AWS account that Datadog assumes via cross-account role delegation. Datadog's AWS account is granted sts:AssumeRole with an external ID for security. No long-lived credentials are stored - Datadog assumes the role on demand.

This is a hands-on setup: run the commands yourself as part of the conversation rather than handing the user a list, keep them in the loop, and pause for confirmation before terraform apply.

Phase 0: Preflight

Terraform or OpenTofu. Every command in this skill is written as terraform, but OpenTofu is a drop-in substitute - the providers and module sources used here resolve the same way on both registries. Check which binary the user actually has before Phase 1:

command -v terraform tofu

If only tofu is on the PATH, read every terraform <subcommand> below as tofu <subcommand>. If both are present, ask which one the user wants rather than guessing.

Datadog credentials. Load DD_SITE / DD_API_KEY / DD_APP_KEY from the environment (falling back to .env.local / .env) and validate both keys - they fail independently:

for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
: "${DD_SITE:=datadoghq.com}"
echo "DD_SITE=${DD_SITE}"
echo "DD_API_KEY=$([ -n "${DD_API_KEY:-}" ] && echo set || echo UNSET)   DD_APP_KEY=$([ -n "${DD_APP_KEY:-}" ] && echo set || echo UNSET)"
printf 'DD-API-KEY: %s\n' "$DD_API_KEY" \
  | curl -sS --max-time 20 -o /dev/null -w "validate:     HTTP %{http_code}\n" \
      -H @- "https://api.${DD_SITE}/api/v1/validate"
printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" \
  | curl -sS --max-time 20 -o /dev/null -w "current_user: HTTP %{http_code}\n" \
      -H @- "https://api.${DD_SITE}/api/v2/current_user"

App URL. The Datadog app host is not app.${DD_SITE} for every site. It is https://app.datadoghq.com (US1), https://app.datadoghq.eu (EU1), https://app.ddog-gov.com (Gov), and for every other site it is https://${DD_SITE} itself - https://us3.datadoghq.com, https://us5.datadoghq.com, https://ap1.datadoghq.com, https://ap2.datadoghq.com, https://uk1.datadoghq.com. Resolve it once and substitute it wherever <APP_BASE> appears below. Full list: https://docs.datadoghq.com/getting_started/site/

Remember the resolved DD_SITE. Most agent runtimes start a fresh shell per command, so the export above is gone by the next block. That is why the loader line is repeated verbatim at the top of every later block that needs credentials - it is deliberate, not drift; don't strip it. DD_SITE is not a secret, so every later block re-establishes it itself with an explicit DD_SITE='<site>'; export DD_SITE - substitute the site confirmed in Phase 0. It is a plain assignment rather than : "${DD_SITE:=...}" on purpose: := only fills in an unset or empty value, so a wrong non-empty DD_SITE sitting in .env would survive it and every call would go to the wrong region. The keys are guarded with :? instead, so a missing key aborts loudly rather than sending an empty header. Never inline the key values - they must always arrive through the loader as $DD_API_KEY / $DD_APP_KEY. Two consequences follow, and both are deliberate:

• Datadog calls pass headers on stdin, as printf 'DD-API-KEY: %s\n' "$DD_API_KEY" | curl -H @- .... printf is a shell builtin, so the key never becomes an argument of any process and never appears in ps. Writing -H "DD-API-KEY: $DD_API_KEY" instead would put it in curl's argv. (-H @- needs curl 7.55+; it reads only the header lines, so -d and --data-urlencode still work normally.)
• Terraform never receives the keys as values at all for AWS, Azure, and GCP: the Datadog provider reads DD_API_KEY / DD_APP_KEY from the environment, so there are no root variables, no -var= arguments, and nothing for Terraform to record in state or a saved plan. (OCI is the exception - its module needs them as inputs, so there they travel as TF_VAR_*.)

Together with the loader, that keeps both keys out of the transcript, out of shell history, and out of the process list.

Tools. terraform is required; the aws CLI is optional (it only looks up the account ID):

command -v terraform || echo "MISSING terraform - https://developer.hashicorp.com/terraform/install"
command -v aws >/dev/null 2>&1 && echo "aws: available" || echo "aws: not installed"

The snippets here are POSIX shell. Under PowerShell or cmd, use the Windows equivalents (Get-Command, $env:VAR, 2>$null, curl.exe) - same calls, same order.

Phase 1: Determine Scope

Ask the user for:

• Their AWS account ID (12-digit number).
• Which AWS regions they want to monitor. Default to all regions if they have no preference.

If they don't know their account ID:

If the aws CLI is available:

aws sts get-caller-identity --query "Account" --output text

Otherwise, offer to install it - it's the fastest way to look this up automatically:

I can read your AWS account ID directly if you install the AWS CLI. On macOS: brew install awscli. Otherwise: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html. After install, run aws configure (or set AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY env vars). Want to install it now, or paste your account ID yourself?

If the user prefers to find it manually: AWS Console top-right account menu, or My Account at https://console.aws.amazon.com/billing/home#/account.

Also determine the correct Datadog trusted AWS account ID. It depends on two things - the Datadog site and the AWS partition the monitored account lives in - and getting it wrong writes an sts:AssumeRole trust for the wrong account, so do not guess.

For a commercial AWS account (partition aws):

us2.ddog-gov.com (US2-FED) is deliberately not in that table, and neither is any GovCloud or China AWS account. Datadog publishes a different id for a GovCloud-partition account than for a commercial account on the same government site, and this skill does not carry those values. In any of those cases, stop and read the id from Datadog's AWS manual setup guide with the DATADOG SITE selector on that page set to the user's site - the page renders the id per site, and distinguishes the commercial value from the GovCloud one. Never fall back to the commercial default for a site or partition that isn't listed above.

Phase 2: Generate and Apply Terraform

Check what already exists - local Terraform first, then Datadog

Do this before generating or applying anything, and in this order. Local state first, because it decides whether an existing integration is something you can update or something you must not touch:

find . -maxdepth 1 -type f \( -name '*.tf' -o -name 'terraform.tfstate' \) -print
# A project is "present" if it has configuration - .terraform/ may simply not exist yet on a fresh clone
# with a remote backend, and terraform.tfstate does not exist at all when state is remote.
if [ -n "$(find . -maxdepth 1 -type f \( -name '*.tf' -o -name '*.tf.json' \) -print -quit)" ]; then
  terraform init -input=false >/dev/null || { echo "terraform init failed - resolve that before concluding anything about existing state"; exit 1; }
  out=$(terraform state list 2>&1); rc=$?
  if [ "$rc" -ne 0 ]; then
    case $out in
      *'No state file'*|*'no state'*|*'Backend initialization required'*)
        echo "project is initialized but has no state yet - treat as a clean install" ;;
      *)
        printf '%s\n' "$out"
        echo "could not read state (backend or credentials problem) - do NOT treat this as 'nothing exists'"; exit 1 ;;
    esac
  elif [ -z "$out" ]; then
    echo "state is empty - treat as a clean install"
  else
    printf '%s\n' "$out" | grep -F 'datadog_integration_aws_account' || echo "state exists but holds no datadog_integration_aws_account resource"
  fi
else
  echo "no Terraform configuration here yet - clean install"
fi

Match the exact resource address datadog_integration_aws_account, not a loose grep -i datadog: unrelated Datadog resources, or cloud IAM left behind by a partial apply, would otherwise read as a managed integration.

Then ask Datadog what it already has:

for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
# Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
: "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
resp=$(printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" \
  | curl -sS -w '\n%{http_code}' -X GET -H @- "https://api.${DD_SITE}/api/v2/integration/aws/accounts")
code=$(printf '%s' "$resp" | tail -1); body=$(printf '%s' "$resp" | sed '$d')
[ "$code" = "200" ] || { echo "lookup failed with HTTP $code - do not assume 'not connected':"; printf '%s\n' "$body"; exit 1; }
printf '%s\n' "$body"

Now reconcile the two answers before doing anything:

• Not in Datadog, nothing in local state - a clean install. Continue.

• In Datadog and present in local state - this is the update/repair case, not a duplicate. Continue into the Terraform below as a change to the existing resources, and let the plan show what it will alter.

• In Datadog but absent from local state (the AWS account ID from Phase 1 already appears) - stop. Applying would either create a duplicate or fight with whatever manages it. Say so plainly and offer the options: import the existing object into this project (terraform import datadog_integration_aws_account.datadog_integration "<config-id>", where the config id is not the AWS account id - get it from the List all AWS integrations endpoint, querying by AWS account id), manage it where it is already managed, or delete it in Datadog first. Only continue if the user picks one and confirms. Two things about importing, in this order. Generate the configuration first (the Terraform below, adapted to the identity that already exists - same role/app/service-account name), because terraform import binds an existing object to a configured resource address and fails without one. And importing the Datadog registration alone is not enough: the cloud-side identity (the IAM role, the app registration, the service account) is still outside state, so either import those too or reference them with data sources, or the next apply will try to create them again and collide.

  # 1. get the Datadog-side config id (NOT the AWS account id)
  for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
  DD_SITE='datadoghq.com'; export DD_SITE   # <- the site confirmed in Phase 0
  : "${DD_API_KEY:?}"; : "${DD_APP_KEY:?}"
  printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" \
    | curl -sS -H @- "https://api.${DD_SITE}/api/v2/integration/aws/accounts"
  # 2. import it, after the configuration below exists
  terraform import datadog_integration_aws_account.datadog_integration "<config-id-from-above>"

• Present in local state but absent from Datadog - a partial or rolled-back install. The cloud-side resources may exist while the registration does not. Do not start from scratch: run the plan and let it show what is missing, and expect it to re-create only the registration.

Check for Existing Terraform

Before generating a new Terraform configuration, check if the user already has a Terraform project in the current directory or nearby:

find . -maxdepth 1 -type f \( -name '*.tf' -o -name 'terraform.tfstate' \) -print

If existing .tf files are found:

• Read them to understand what providers and resources are already configured.
• If a datadog provider already exists, reuse its configuration - do not create a duplicate.
• If an aws provider already exists, reuse it.
• Only add the new resources needed (IAM role, policies, datadog_integration_aws_account) to the existing project. Do not regenerate providers, variables, or terraform blocks that already exist.
• If the user has a modular layout (e.g., separate files per concern), create a new file like datadog-aws-integration.tf for the Datadog resources.

If no existing Terraform is found, generate a standalone configuration.

The full HCL template - providers, the trust policy, the dynamically-fetched permission set with its 6144-character chunking, the role, and the datadog_integration_aws_account registration - is in references/terraform.md. Read it now and emit it with the placeholders filled in.

The template leaves logs_config.lambda_forwarder empty (metrics only). Forwarding AWS logs to Datadog is a separate follow-on that deploys the Datadog Forwarder Lambda and registers its ARN here; point the user at https://docs.datadoghq.com/logs/guide/forwarder/ once the integration is live. Don't set up the forwarder inline.