Agents Debug

debug

Diagnose why your AgentCore agent or environment isn't working correctly.

When to use

• Your agent is returning wrong answers or errors
• Tool calls are failing or timing out
• Agent works locally but fails after deploying
• Logs aren't showing up in CloudWatch
• The AgentCore CLI isn't working or environment seems broken
• agentcore command not found or prerequisites are missing

Do NOT use for:

• Deploy failures (CDK errors, IAM during deploy) → use agents-deploy
• Scaffolding a new project → use agents-get-started
• Measuring quality or setting up monitoring → use agents-optimize

Input

$ARGUMENTS is optional:

/agents-debug                      # interactive — describe what's wrong
/agents-debug traces               # read and explain recent traces
/agents-debug logs                 # search recent logs for errors
/agents-debug memory               # diagnose memory recall issues specifically
/agents-debug doctor               # check environment prerequisites

Process

Step 0: Determine problem type

If the developer's issue is about the CLI itself (command not found, prerequisites, environment setup), load references/doctor.md and follow its diagnostic checklist.

If the issue is about agent behavior (wrong answers, errors, timeouts, tool failures), continue with Step 1 below.

Step 1: Verify CLI version

Run agentcore --version. This skill requires v0.9.0 or later. If the version is older, tell the developer to run agentcore update before proceeding.

Step 2: Understand the symptom

Ask (or infer from context):

"What's happening?

1. The agent returns an error message
2. The agent returns a wrong or unhelpful answer
3. A specific tool call is failing
4. Memory isn't working (agent doesn't remember things)
5. The agent is slow or timing out
6. I want to understand what the agent did in a specific session"

Step 3: Read traces and logs automatically

Don't ask the developer to paste logs — read them directly.

# List recent traces
agentcore traces list --runtime <AgentName> --since 1h

# Get the most recent trace ID
agentcore traces list --runtime <AgentName> --since 1h --limit 1

# Download and read the trace
agentcore traces get <traceId> --runtime <AgentName>

# Search logs for errors
agentcore logs --runtime <AgentName> --since 1h --level error

# Search logs for a specific pattern
agentcore logs --runtime <AgentName> --since 2h --query "timeout"
agentcore logs --runtime <AgentName> --since 2h --query "model access"

Important: CloudWatch put-to-get latency is ~10 seconds end-to-end — that's the delay from when a span is emitted to when it's readable by agentcore traces get or agentcore run eval. There is no separate "trace ingested but eval not ready yet" window; the same ingestion step unlocks both paths. Older skills and docs said 30–60s for traces and 2–5 minutes for evals — both are stale. If you just invoked the agent, wait ~15 seconds and both trace reads and evals will work.

Read agentcore/agentcore.json to get the agent name if not provided.

Step 4: Diagnose by symptom

Symptom: "model access denied" or model error

Most common cause: The model isn't enabled in the Bedrock console for your region.

Fix:

1. Go to AWS Console → Amazon Bedrock → Model access
2. Enable the model your agent uses
3. Wait 1–2 minutes for access to propagate

Second cause: The execution role is missing bedrock:InvokeModel.

Check:

aws iam simulate-principal-policy \
  --policy-source-arn $(agentcore status --json | jq -r '.runtimes[0].executionRoleArn') \
  --action-names bedrock:InvokeModel \
  --resource-arns "arn:aws:bedrock:*::foundation-model/*"

Third cause: Cross-region inference profile requires model access in all regions.

Model IDs starting with a geographic prefix are cross-region inference profiles that route requests within that geography:

The AgentCore CLI scaffolds global. by default (e.g., global.anthropic.claude-sonnet-4-5-20250929-v1:0). All prefixes require model access enabled in every destination region the profile covers. For us. profiles, enable in all US regions; for eu., all EU regions; for global., all supported regions. Not all models support all prefixes — global. is currently available for select models only. Use global. for maximum throughput when available, or a geographic prefix when data residency requirements constrain where inference can run. Check the Bedrock inference profiles docs for current model × prefix availability.

Symptom: Tool call failing

Step 1: Find the failing tool call in the trace:

agentcore traces get <traceId> --runtime <AgentName>

Look for tool call entries with error status.

Step 2: Check the gateway status:

agentcore status --type gateway
agentcore fetch access --name <AgentName> --type agent

Step 3: Common tool call failures:

Gateway URL not set (local dev): The AGENTCORE_GATEWAY_*_URL env var is only set after deploy. In agentcore dev, gateway tools aren't available. This is expected — the agent should handle this gracefully.

Auth failure on tool call:

agentcore logs --runtime <AgentName> --since 1h --query "auth"

Check that the credential is configured correctly: agentcore status --type credential

Lambda function error: The Lambda itself is failing. Check Lambda logs directly:

aws logs tail /aws/lambda/<function-name> --since 1h

Policy denial: If a policy engine is attached, check policy decision logs:

agentcore logs --runtime <AgentName> --since 1h --query "policy"
agentcore status --type policy-engine

Symptom: Wrong or unhelpful answers

Step 1: Read the trace to see the agent's reasoning:

agentcore traces get <traceId> --runtime <AgentName>

The trace shows the model's reasoning steps, tool calls made, and the final response. Look for:

• Did the agent use the right tools?
• Did the tool calls return the expected data?
• Is the system prompt providing the right context?

Step 2: Check if memory is involved: If the agent should be using memory context but isn't, see the "Symptom: Memory not persisting" section later in this skill, or load references/doctor.md if this is an environment issue.

Step 3: Common causes:

• System prompt is too vague or missing key context
• Agent isn't calling the right tools (tool descriptions need improvement)
• Tool is returning unexpected data format
• Model ID is wrong for the task (e.g., using a smaller model for complex reasoning)

Symptom: Memory not working

Memory not persisting across sessions (LTM):

1. Verify LTM strategies are configured (SEMANTIC or USER_PREFERENCE):

agentcore status --type memory --json | jq '.memories[].strategies'

1. Wait 5–30 seconds after a session ends — LTM extraction is async. The agent must finish its session before facts are extracted.

2. Use UUIDs (v4) for session IDs — the platform requires a minimum of 33 characters. Short IDs like "session-1" cause LTM to fail silently. agentcore invoke generates compliant IDs by default.

3. Verify the memory resource is ACTIVE:

agentcore status --type memory

Memory not loading at session start:

1. Check the MEMORY_*_ID env var is set:

agentcore status --type memory --json | jq '.memories[].id'

1. Verify the actor_id is consistent across sessions — memory is scoped per actor.

2. Check the namespace paths in your retrieval config match the namespaces used when writing.

Symptom: Agent timeout

Step 1: Check the trace for where time is being spent:

agentcore traces get <traceId> --runtime <AgentName>

Look for long-running steps — model calls, tool calls, memory operations.

Step 2: Common timeout causes:

Slow agent initialization: If the first invocation after an idle period is slow but subsequent requests are fast, the agent is spending too much time initializing. Check for heavy imports at module level, database connections in global scope, or MCP client initialization during startup. Move expensive setup into the request handler or use lazy initialization. See the agents-harden skill for optimization guidance.

Model call timeout: The model is taking too long. Consider using a faster model for time-sensitive operations (e.g., Haiku instead of Sonnet for simple tasks).

Tool call timeout: The Lambda or external API is slow. Check the tool's own logs.

Memory retrieval timeout: Semantic search can be slow for large memory stores. Consider reducing top_k in your retrieval config.

VPC connectivity issue: If the agent is in a VPC, check security group rules and route tables. See agents-build (loads references/vpc.md) for VPC-specific debugging.

Symptom: ServiceQuotaExceededException: maxVms limit exceeded (despite low observed concurrency)

Your CloudWatch "concurrent sessions" metric shows modest numbers (maybe 30–50) but InvokeAgentRuntime calls return ServiceQuotaExceededException: maxVms limit exceeded.

What's actually happening: CloudWatch's concurrent-sessions metric is not the same as live microVM count. The maxVms quota counts all environments your account has active — including ones that finished their invocation but haven't been reclaimed yet. Idle-but-not-yet-reclaimed environments count against the quota until idleRuntimeSessionTimeout expires (default 900 seconds / 15 minutes) or you explicitly stop them.

If your code uses a new session ID per request and doesn't call StopRuntimeSession, every request leaves an environment sitting idle for 15 minutes counting against the quota.

Fix order (try in this order before requesting a quota increase):

1. Call StopRuntimeSession after each logical request completes. If you're not going to send more requests on this session, stop it explicitly.

   client.stop_runtime_session(
       agentRuntimeArn=runtime_arn,
       runtimeSessionId=session_id,
   )

2. Reuse session IDs across related requests. If a user interaction produces multiple backend calls, route them to the same session instead of generating a new session ID per call.

3. Lower idleRuntimeSessionTimeout. If your sessions are short-lived and you can't add StopRuntimeSession everywhere, lower the timeout by editing the runtime's lifecycleConfiguration in agentcore/agentcore.json and running agentcore deploy.

4. Only after the above, request a quota increase. See agents-harden (loads references/limits.md) — request it through the Service Quotas console (Amazon Bedrock AgentCore), not by filing a support ticket directly.

See agents-harden Session lifecycle management section for the full pattern.

Symptom: 424 Failed Dependency on invoke

This usually means the agent container failed to start or crashed during initialization.

Step 1: Check the agent logs for startup errors:

agentcore logs --runtime <AgentName> --since 30m --level error

Step 2: Common causes:

Missing Python dependency: The agent code imports a package not in pyproject.toml. The container starts but crashes on first request. Fix: add the dependency and redeploy.

Entrypoint crash: The main.py throws an exception during import or app.run(). Check logs for the traceback.

Container image pull failure: If using Container build, the ECR image may not exist or the execution role lacks ecr:BatchGetImage. Check:

agentcore status --runtime <AgentName> --json

Memory resource not ACTIVE: If the agent code assumes memory is available but the memory resource is still in CREATING state, the entrypoint may fail. Check:

agentcore status --type memory

Initialization timeout: The agent takes too long to be ready for its first request — heavy imports at module level, synchronous database connections, or MCP client initialization during startup can exceed the service's health-check window. The symptom looks like a 424 on the first invoke but healthy on subsequent ones. Fix: move expensive setup out of module level, use lazy initialization, or warm the agent before production traffic. See agents-harden Initialization time section for patterns.

Symptom: Local invocations fail with connection-refused / exit code 7

Usually not an agent bug — the dev server is on a different port than you expect.

Default ports agentcore dev binds:

When the default is occupied (second dev session, a lingering process from a previous run, another service on 8080), the CLI auto-increments silently: 8080 → 8081 → 8082. A test harness or curl script hardcoded to 8080 will get Connection refused (curl exit code 7) while the agent is running fine on 8082.

Diagnose in this order:

1. Read the CLI banner that agentcore dev prints — it shows the actual bound port and URL. This is always the source of truth.

2. If the banner is gone (terminal cleared, running in background), check the log file:

   tail -20 agentcore/.cli/logs/dev/*.log

3. Or find the process directly:

   # macOS / Linux
   ps aux | grep -E 'agentcore dev|uvicorn' | grep -v grep
   lsof -iTCP -sTCP:LISTEN -n -P | grep -E '8080|8081|8082|8000|9000'

Fix options:

• Pin the port explicitly: agentcore dev --port 8080
• Kill the process squatting on the default: lsof -tiTCP:8080 -sTCP:LISTEN | xargs kill
• Update the hardcoded port in your test harness to read from the CLI output or from an env var

This is also a common source of "works locally one day, fails the next" reports — the port shifted between runs.

Symptom: Gateway tool calls failing with auth errors

Step 1: Verify the auth type matches the target type. This is the most common gateway error — using the wrong outbound auth for the target:

Step 2: Check for expired OAuth tokens. If the gateway target uses OAuth, the access token may have expired. Look for auth-related errors:

agentcore logs --runtime <AgentName> --since 1h --query "auth"
agentcore logs --runtime <AgentName> --since 1h --query "401"
agentcore logs --runtime <AgentName> --since 1h --query "403"

If tokens are expiring, verify the OAuth credential provider's token endpoint is reachable and the client credentials are still valid. For MCP server targets with OAuth, the gateway handles token refresh automatically — if it's failing, the credential provider config may be wrong.

Step 3: Check the credential is configured:

agentcore status --type credential
agentcore status --type gateway --json

Symptom: No traces appearing

Wait ~15 seconds — there's a short delay (typically ~10s) between invocation and trace availability.

If still no traces after ~30 seconds:

1. Verify observability was enabled when the agent was deployed
2. Check the agent was actually invoked: agentcore logs --runtime <AgentName> --since 1h
3. Check CloudWatch permissions on the execution role