Agentcore Observability Setup

AgentCore Observability Setup & Validation

Validate and bootstrap observability for Amazon Bedrock AgentCore workloads, aligned with the AgentCore observability documentation. Produce a readiness report that states, per check, what is configured vs. what should be, and generate the exact remediation for every gap.

Operating Principle

Verify where reachable; prescribe everywhere else. Every check is exactly one of:

• VERIFY — confirmable through a read-only API (log group exists, data arriving, spans flowing, Transaction Search enabled, tracing toggle, delivery configured).
• PRESCRIBE — cannot be read directly (code-level OTEL instrumentation in the agent's source). Validate the effect (are spans arriving?) and emit the exact steps. Do not claim to read source code.

This skill is read-only. It validates and generates configuration/commands. The customer decides whether to apply them. Never mutate IAM, resources, or configuration.

When to Use

Activate when the user asks to:

• Check / validate / audit whether AgentCore observability is configured correctly
• Diagnose missing traces, spans, metrics, or logs for a Bedrock agent
• Set up or bootstrap observability (tracing, metrics, logging) for an agent
• Run an AgentCore observability readiness review
• Understand why agent telemetry isn't appearing in CloudWatch GenAI Observability

Step 1: Detect Permission Tier

This skill degrades gracefully based on which read-only permissions the DevOps Agent role holds. Detect the tier by attempting calls and noting failures; run the checks the tier allows and report what to add to unlock the rest. Never attempt to modify IAM.

If Tier 2/3 permissions are absent, tell the user the check is prescriptive-only here and give the exact scoped inline policy to attach to the DevOps Agent role (see references/iam-tiers.md) so a re-run can verify it. The shared managed policy is cross-tenant and must not be modified per-skill.

Step 2: Determine Scope (Host & Surface)

Host detection:

• Runtime — auto-detect with bedrock-agentcore:GetAgentRuntime / bedrock-agentcore:ListAgentRuntimes (Tier 2). If absent, ask the user.
• Non-runtime — the agent cannot reliably auto-detect the host. Ask: "Is the agent on Lambda, ECS, EKS, or on-prem/another cloud?"

Surfaces to assess (ask which apply, or discover via Tier 2 List*): Runtime agents · Memory resources · Gateway resources · Built-in tools (code interpreter, browser) · non-runtime host.

Outcome matrix:

Step 3: Run Checks

Assign each finding a severity (CRITICAL / HIGH / MEDIUM / LOW / INFO) and a type (VERIFY / PRESCRIBE). The complete check catalog with APIs, log-group patterns, and pass/fail logic is in references/checks-catalog.md. Summary below.

3.1 Account prerequisite — CloudWatch Transaction Search (all hosts)

• VERIFY (Tier 2): xray:GetTraceSegmentDestination → destination must be CloudWatchLogs and status ACTIVE. If XRay/inactive, Transaction Search is not enabled → CRITICAL (spans will not be delivered to CloudWatch Logs; this is the single most common silent root cause).
• VERIFY (Tier 1 fallback): if the destination API is unavailable, infer from whether span log streams are receiving data (below). Report as inferred, not confirmed.
• PRESCRIBE: enable Transaction Search and ingest spans as structured logs (see references/remediation-runtime.md).

3.2 Runtime agent path (start here — primary surface)

1. Agent log group exists (VERIFY, Tier 1): logs:DescribeLogGroups prefix /aws/bedrock-agentcore/runtimes/. Missing → HIGH.
2. Logs arriving (VERIFY, Tier 1): logs:FilterLogEvents on the runtime log stream in the last 24h. No recent events after invocations → HIGH.
3. Spans flowing (VERIFY, Tier 1): check the spans log stream in the agent's log group (unified destination) or the shared aws/spans log group. No spans despite invocations → CRITICAL (points to Transaction Search disabled or missing instrumentation).
4. Session metrics emitting (VERIFY, Tier 1): cloudwatch:ListMetrics namespace bedrock-agentcore. Absent → MEDIUM.
5. Runtime tracing / span destination (VERIFY, Tier 2): bedrock-agentcore:GetAgentRuntime → inspect env for UNIFIED_TRACES_DESTINATION_ENABLED and DISABLE_ADOT_OBSERVABILITY. Starting 2026-07-20, newly created agents in supported AWS Regions default to the unified span destination (the agent's own log group) — UNIFIED_TRACES_DESTINATION_ENABLED=false is now the opt-out. Agents created before that date remain on shared aws/spans unless opted in. If ADOT observability disabled unintentionally → HIGH.
6. X-Ray resource policy on log group (VERIFY, Tier 2): logs:DescribeResourcePolicies — must allow xray.amazonaws.com to logs:PutLogEvents on the agent's log group. Applies by default for agents created after 2026-07-20 in supported Regions, since unified span destination is the new default (per the AgentCore release notes). Mark N/A only when the agent was created before 2026-07-20 and still delivers to shared aws/spans, or the customer has explicitly opted out via UNIFIED_TRACES_DESTINATION_ENABLED=false. Missing → HIGH.
7. Code-level instrumentation (PRESCRIBE): cannot read source. If spans are absent, prescribe: aws-opentelemetry-distro>=0.10.0 (≥0.18.0 for unified span destination) + boto3 in requirements.txt; launch with opentelemetry-instrument python main.py (container CMD ["opentelemetry-instrument","python","main.py"]); framework tracing enabled (e.g. Strands tracer, opentelemetry-instrumentation-langchain); session id via X-Amzn-Bedrock-AgentCore-Runtime-Session-Id.