Dd GCP Integration

Datadog GCP Integration

You are helping a user set up the Datadog GCP integration using Terraform.

The integration creates a GCP service account in the customer's project and grants Datadog's delegate principal the ability to impersonate it via roles/iam.serviceAccountTokenCreator. This avoids long-lived keys entirely.

This is a hands-on setup: run the commands yourself as part of the conversation rather than handing the user a list, keep them in the loop, and pause for confirmation before terraform apply.

Phase 0: Preflight

Terraform or OpenTofu. Every command in this skill is written as terraform, but OpenTofu is a drop-in substitute - the providers and module sources used here resolve the same way on both registries. Check which binary the user actually has before Phase 1:

command -v terraform tofu

If only tofu is on the PATH, read every terraform <subcommand> below as tofu <subcommand>. If both are present, ask which one the user wants rather than guessing.

Datadog credentials. Load DD_SITE / DD_API_KEY / DD_APP_KEY from the environment (falling back to .env.local / .env) and validate both keys - they fail independently:

for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
: "${DD_SITE:=datadoghq.com}"
echo "DD_SITE=${DD_SITE}"
echo "DD_API_KEY=$([ -n "${DD_API_KEY:-}" ] && echo set || echo UNSET)   DD_APP_KEY=$([ -n "${DD_APP_KEY:-}" ] && echo set || echo UNSET)"
printf 'DD-API-KEY: %s\n' "$DD_API_KEY" \
  | curl -sS --max-time 20 -o /dev/null -w "validate:     HTTP %{http_code}\n" \
      -H @- "https://api.${DD_SITE}/api/v1/validate"
printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" \
  | curl -sS --max-time 20 -o /dev/null -w "current_user: HTTP %{http_code}\n" \
      -H @- "https://api.${DD_SITE}/api/v2/current_user"

App URL. The Datadog app host is not app.${DD_SITE} for every site. It is https://app.datadoghq.com (US1), https://app.datadoghq.eu (EU1), https://app.ddog-gov.com (Gov), and for every other site it is https://${DD_SITE} itself - https://us3.datadoghq.com, https://us5.datadoghq.com, https://ap1.datadoghq.com, https://ap2.datadoghq.com, https://uk1.datadoghq.com. Resolve it once and substitute it wherever <APP_BASE> appears below. Full list: https://docs.datadoghq.com/getting_started/site/

Remember the resolved DD_SITE. Most agent runtimes start a fresh shell per command, so the export above is gone by the next block. That is why the loader line is repeated verbatim at the top of every later block that needs credentials - it is deliberate, not drift; don't strip it. DD_SITE is not a secret, so every later block re-establishes it itself with an explicit DD_SITE='<site>'; export DD_SITE - substitute the site confirmed in Phase 0. It is a plain assignment rather than : "${DD_SITE:=...}" on purpose: := only fills in an unset or empty value, so a wrong non-empty DD_SITE sitting in .env would survive it and every call would go to the wrong region. The keys are guarded with :? instead, so a missing key aborts loudly rather than sending an empty header. Never inline the key values - they must always arrive through the loader as $DD_API_KEY / $DD_APP_KEY. Two consequences follow, and both are deliberate:

• Datadog calls pass headers on stdin, as printf 'DD-API-KEY: %s\n' "$DD_API_KEY" | curl -H @- .... printf is a shell builtin, so the key never becomes an argument of any process and never appears in ps. Writing -H "DD-API-KEY: $DD_API_KEY" instead would put it in curl's argv. (-H @- needs curl 7.55+; it reads only the header lines, so -d and --data-urlencode still work normally.)
• Terraform never receives the keys as values at all for AWS, Azure, and GCP: the Datadog provider reads DD_API_KEY / DD_APP_KEY from the environment, so there are no root variables, no -var= arguments, and nothing for Terraform to record in state or a saved plan. (OCI is the exception - its module needs them as inputs, so there they travel as TF_VAR_*.)

Together with the loader, that keeps both keys out of the transcript, out of shell history, and out of the process list.

Tools. terraform is required, plus jq for Phase 1. The gcloud CLI is optional for discovery, but the google provider needs ambient GCP credentials either way (gcloud auth application-default login, or GOOGLE_APPLICATION_CREDENTIALS):

command -v terraform || echo "MISSING terraform - https://developer.hashicorp.com/terraform/install"
command -v jq || echo "MISSING jq - needed to read the delegate email in Phase 1"
command -v gcloud >/dev/null 2>&1 && echo "gcloud: available" || echo "gcloud: not installed"

If jq isn't available, read data.attributes.delegate_account_email out of the raw JSON response in Phase 1 yourself instead of piping through jq.

The snippets here are POSIX shell. Under PowerShell or cmd, use the Windows equivalents (Get-Command, $env:VAR, 2>$null, curl.exe) - same calls, same order.

Phase 1: Gather Datadog Delegate Principal

Datadog impersonates the customer's service account through a per-org delegate service account. Get its email - GET first, and only POST if the org has no delegate yet, so an existing one is reused rather than re-created:

for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
# Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
: "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
resp=$(printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" \
  | curl -sS -w '\n%{http_code}' -X GET \
      -H @- "https://api.${DD_SITE}/api/v2/integration/gcp/sts_delegate")
code=$(printf '%s' "$resp" | tail -1); body=$(printf '%s' "$resp" | sed '$d')
if [ "$code" = "404" ]; then
  echo "no delegate exists yet (HTTP 404) - creating one"
  resp=$(printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" \
    | curl -sS -w '\n%{http_code}' -X POST -H "Content-Type: application/json" -d '{}' \
        -H @- "https://api.${DD_SITE}/api/v2/integration/gcp/sts_delegate")
  code=$(printf '%s' "$resp" | tail -1); body=$(printf '%s' "$resp" | sed '$d')
elif [ "$code" != "200" ]; then
  echo "GET sts_delegate returned HTTP $code - not creating anything. Response:"
  printf '%s\n' "$body"
  echo "403 means the app key lacks permission, 429 is rate limiting, 000 is a transport failure, 5xx is"
  echo "server-side. None of those mean 'no delegate exists', so a POST here could create one you did not"
  echo "intend. Resolve the error, then re-run. If the body says no delegate exists, POST explicitly."
  exit 1
fi
[ "$code" = "200" ] || { echo "sts_delegate failed with HTTP $code:"; printf '%s\n' "$body"; exit 1; }
email=$(printf '%s' "$body" | jq -r '.data.attributes.delegate_account_email // empty')
case "$email" in
  *@*.iam.gserviceaccount.com) echo "DATADOG_PRINCIPAL_ID=$email" ;;
  *) echo "did not get a delegate service-account email; response was:"; printf '%s\n' "$body"; exit 1 ;;
esac

Remember the printed email as DATADOG_PRINCIPAL_ID - the Terraform below needs it. It is org-specific and cannot be hardcoded.

Never continue on an empty or null email. The guards above exist because curl -s | jq -r on an error response prints null, which would silently become member = "serviceAccount:null" in the Terraform and produce an integration that cannot authenticate.

If jq is unavailable, run the block exactly as written except the email=$(... | jq -r ...) line - do not remove the printf ... | curl -H @- pipeline, which is what supplies authentication. Read data.attributes.delegate_account_email out of $body yourself and apply the same rule: refuse to proceed unless it looks like a ...@....iam.gserviceaccount.com address.

Phase 2: Determine Scope

Ask the user if they already know which GCP project IDs and/or folder IDs they want Datadog to monitor.

If they do, collect:

• Which GCP project should host the Datadog service account (the "host project")?
• The list of project IDs and/or folder IDs to monitor.

If they don't know or want help figuring it out:

If the gcloud CLI is available, offer to discover their GCP environment. Explain that you will use gcloud to list their organizations, folders, and projects so they can pick which ones to monitor. This is best-effort - run each command independently and work with whatever succeeds:

Organizations:

gcloud organizations list --format="table(displayName, name)"

Folders (uses the REST API to search all active folders across the org; also requires gcloud for the access token):

printf 'Authorization: Bearer %s\n' "$(gcloud auth print-access-token)" \
  | curl -sS -H @- -H "Content-Type: application/json" \
      -d '{"query": "lifecycleState=ACTIVE"}' \
      "https://cloudresourcemanager.googleapis.com/v2/folders:search"

If the response contains a nextPageToken, paginate by adding "pageToken": "<token>" to the request body until all folders are retrieved.

Expand nested folders yourself. The search above returns every active folder in the org, so use it to build the full descendant set for whatever the user picks: for each chosen folder, collect every folder whose parent chain leads back to it, then list the projects of all of them:

# replace these with the chosen folder and every folder nested beneath it
set -- 123456789012 987654321098
for folder do
  gcloud projects list --filter="parent.id=${folder} AND lifecycleState=ACTIVE AND NOT projectId:sys*" \
    --format="value(projectId)"
done | sort -u

Put that expanded list into the Terraform's project_ids in addition to keeping the chosen folders in folder_ids. The reason is in the template: folder-level IAM is inherited by every descendant project, but API enablement is not, and the google_projects data source matches on immediate parent only. Folders in folder_ids cover IAM (including projects created later); the expanded project_ids is what actually enables the required APIs in each existing project. Skipping the expansion produces an integration that looks configured but silently collects nothing from projects in sub-folders.

Projects (active only, excluding system projects):

gcloud projects list --filter="lifecycleState=ACTIVE AND NOT projectId:sys*" --format="table(projectId, name, parent.id)"

If any individual command fails (e.g., the user lacks permission to list organizations or folders), inform the user which command failed and why, but continue with whatever information was successfully retrieved. For example, if they can list projects but not folders, proceed with project-level setup.

Otherwise (gcloud not installed or not authenticated), ask the user to gather the IDs from the GCP Console:

• Project IDs: console.cloud.google.com → project picker (top bar) → each project's ID is shown next to its name.
• Folder IDs (optional): console.cloud.google.com → IAM & Admin → Manage Resources → expand the org tree; folder IDs are visible in the resource manager.
• Every project inside those folders, including nested sub-folders - expand the whole subtree in Manage Resources and collect each project ID. This is not optional busywork: the Terraform enables the required APIs per project, so any project you don't list gets folder-inherited IAM and no API enablement, and collects nothing. If the user won't enumerate them, tell them folder-scoped setup needs gcloud (which can expand the subtree automatically) and offer project-scoped setup instead.
• Host project: ask which project the Datadog service account should live in.

Present whatever results were gathered in a readable format and let the user choose:

• Specific projects: list of project IDs
• Folders: list of folder IDs (Datadog will discover all active projects within them)
• Both: a combination of explicit projects and folders

Then ask which project should host the Datadog service account.

Phase 3: Generate and Apply Terraform

Check what already exists - local Terraform first, then Datadog

Do this before generating or applying anything, and in this order. Local state first, because it decides whether an existing integration is something you can update or something you must not touch:

find . -maxdepth 1 -type f \( -name '*.tf' -o -name 'terraform.tfstate' \) -print
# A project is "present" if it has configuration - .terraform/ may simply not exist yet on a fresh clone
# with a remote backend, and terraform.tfstate does not exist at all when state is remote.
if [ -n "$(find . -maxdepth 1 -type f \( -name '*.tf' -o -name '*.tf.json' \) -print -quit)" ]; then
  terraform init -input=false >/dev/null || { echo "terraform init failed - resolve that before concluding anything about existing state"; exit 1; }
  out=$(terraform state list 2>&1); rc=$?
  if [ "$rc" -ne 0 ]; then
    case $out in
      *'No state file'*|*'no state'*|*'Backend initialization required'*)
        echo "project is initialized but has no state yet - treat as a clean install" ;;
      *)
        printf '%s\n' "$out"
        echo "could not read state (backend or credentials problem) - do NOT treat this as 'nothing exists'"; exit 1 ;;
    esac
  elif [ -z "$out" ]; then
    echo "state is empty - treat as a clean install"
  else
    printf '%s\n' "$out" | grep -F 'datadog_integration_gcp_sts' || echo "state exists but holds no datadog_integration_gcp_sts resource"
  fi
else
  echo "no Terraform configuration here yet - clean install"
fi

Match the exact resource address datadog_integration_gcp_sts, not a loose grep -i datadog: unrelated Datadog resources, or cloud IAM left behind by a partial apply, would otherwise read as a managed integration.

Then ask Datadog what it already has:

for f in .env.local .env; do [ -f "$f" ] || continue; for k in DD_SITE DD_API_KEY DD_APP_KEY; do eval "[ -n \"\${$k:-}\" ]" && continue; v=$(grep -E "^$k=" "$f" | head -1 | cut -d= -f2- | sed 's/^["'\'']//;s/["'\'']$//'); [ -n "$v" ] && export "$k=$v"; done; done
DD_SITE='datadoghq.com'; export DD_SITE   # <- replace with the site confirmed in Phase 0.
# Explicit assignment, not ':=': a wrong non-empty DD_SITE in .env would otherwise survive.
: "${DD_API_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"; : "${DD_APP_KEY:?not set - run dd-account-setup (commercial sites) or supply it directly (government sites)}"
resp=$(printf 'DD-API-KEY: %s\nDD-APPLICATION-KEY: %s\n' "$DD_API_KEY" "$DD_APP_KEY" \
  | curl -sS -w '\n%{http_code}' -X GET -H @- "https://api.${DD_SITE}/api/v2/integration/gcp/accounts")
code=$(printf '%s' "$resp" | tail -1); body=$(printf '%s' "$resp" | sed '$d')
[ "$code" = "200" ] || { echo "lookup failed with HTTP $code - do not assume 'not connected':"; printf '%s\n' "$body"; exit 1; }
printf '%s\n' "$body"

Now reconcile the two answers before doing anything:

• Not in Datadog, nothing in local state - a clean install. Continue.

• In Datadog and present in local state - this is the update/repair case, not a duplicate. Continue into the Terraform below as a change to the existing resources, and let the plan show what it will alter.

• In Datadog but absent from local state (a service account for the host project from Phase 2 is already registered) - stop. Applying would either create a duplicate or fight with whatever manages it. Say so plainly and offer the options: import the existing object into this project (terraform import datadog_integration_gcp_sts.datadog_integration "<integration-uuid>" - the uuid comes from GET /api/v2/integration/gcp/accounts, not the project id or service-account email), manage it where it is already managed, or delete it in Datadog first. Only continue if the user picks one and confirms. Two things about importing, in this order. Generate the configuration first (the Terraform below, adapted to the identity that already exists - same role/app/service-account name), because terraform import binds an existing object to a configured resource address and fails without one. And importing the Datadog registration alone is not enough: the cloud-side identity (the IAM role, the app registration, the service account) is still outside state, so either import those too or reference them with data sources, or the next apply will try to create them again and collide.

  # the uuid comes from the accounts listing, not the project id or service-account email
  terraform import datadog_integration_gcp_sts.datadog_integration "<integration-uuid>"

• Present in local state but absent from Datadog - a partial or rolled-back install. The cloud-side resources may exist while the registration does not. Do not start from scratch: run the plan and let it show what is missing, and expect it to re-create only the registration.

Check for Existing Terraform

Before generating a new Terraform configuration, check if the user already has a Terraform project in the current directory or nearby:

find . -maxdepth 1 -type f \( -name '*.tf' -o -name 'terraform.tfstate' \) -print

If existing .tf files are found:

• Read them to understand what providers and resources are already configured.
• If a datadog provider already exists, reuse its configuration - do not create a duplicate.
• If a google provider already exists, reuse it.
• Only add the new resources needed (service account, IAM bindings, datadog_integration_gcp_sts) to the existing project. Do not regenerate providers, variables, or terraform blocks that already exist.
• If the user has a modular layout, create a new file like datadog-gcp-integration.tf for the Datadog resources.

If no existing Terraform is found, generate a standalone configuration.

The full HCL template - providers, API enablement, the service account, the token-creator binding for Datadog's delegate, the project and folder role bindings, and the datadog_integration_gcp_sts registration - is in references/terraform.md, along with the projects-only and folders-only variants. Read it now and emit it with the placeholders filled in.