Bedrock Operation Review

Amazon Bedrock Operational Review

Conduct a comprehensive operational review of Amazon Bedrock workloads aligned with the AWS Well-Architected Framework and Amazon Bedrock best practices.

This skill uses the AWS Bedrock, Bedrock Agent, CloudWatch, Service Quotas, and EC2 APIs only — all data is collected through native AWS control-plane APIs and CloudWatch metrics. It performs no data-plane model invocations and reads no prompt or response content.

When to Use

Activate this skill when the user asks to:

• Review, audit, or assess an Amazon Bedrock workload
• Check Bedrock best-practices compliance
• Evaluate Bedrock security, performance, cost, quotas, or resilience
• Perform a Bedrock operational readiness review (ORR)
• Investigate Bedrock configuration drift, throttling, or cost drivers

Step 1: Identify Target Scope

Ask the user which accounts and regions to review. Accept:

• Specific account IDs and regions
• "all regions" for a given account
• A specific pillar or set of checks (e.g. "just cost optimization")

If no scope is given, default to all configured account regions and all pillars. Default the analysis window to the last 7 days unless the user specifies a range (historical windows older than ~2 weeks may have reduced CloudWatch resolution).

Step 2: Discover Bedrock Resources

Per account/region, begin by checking whether the account actually uses Bedrock in this region. Do not use bedrock.ListFoundationModels as an activity signal — it returns the regional model catalog available to the account and is generally non-empty in every supported region regardless of usage, so it is treated as catalog data only (see ListFoundationModels).

Instead, determine activity from account-owned resources and AWS/Bedrock metrics:

• Account-owned resources: bedrockagent.ListAgents, bedrockagent.ListKnowledgeBases, bedrock.ListGuardrails, bedrock.ListCustomModels, bedrock.ListProvisionedModelThroughputs, bedrock.ListInferenceProfiles (application-scoped), and bedrock.ListModelCustomizationJobs.
• Invocation activity: cloudwatch.ListMetrics for the AWS/Bedrock namespace.

If no account-owned Bedrock resources exist AND CloudWatch shows no AWS/Bedrock metrics for the region, record "No Bedrock activity detected — skipping pillar analysis" for that region and move on. Do not generate empty findings tables for inactive regions.

For regions where Bedrock is active, collect the full resource inventory:

bedrock.ListFoundationModels                   # catalog reference only: model
                                               # availability + lifecycle status
                                               # (NOT an account-activity signal)
bedrock.ListGuardrails / GetGuardrail          # configured guardrails
bedrock.GetModelInvocationLoggingConfiguration
bedrock.ListInferenceProfiles / GetInferenceProfile
bedrock.ListPromptRouters / GetPromptRouter
bedrock.ListProvisionedModelThroughputs / GetProvisionedModelThroughput
bedrock.ListCustomModels / GetCustomModel
bedrock.ListModelCustomizationJobs / GetModelCustomizationJob
bedrockagent.ListAgents / GetAgent             # agents (draft version)
bedrockagent.ListAgentVersions / GetAgentVersion  # deployed versions — read only the
                                               # per-version model ID, orchestration
                                               # type, and whether a prompt override is
                                               # present. Do NOT read the override
                                               # base-prompt template text.
bedrockagent.ListAgentAliases
bedrockagent.ListKnowledgeBases / GetKnowledgeBase
bedrockagent.ListDataSources / GetDataSource
bedrockagent.ListPrompts                       # Prompt Management: presence/metadata
                                               # only (identifiers, versions, counts).
                                               # Do NOT call GetPrompt — it returns
                                               # prompt template/variant content, which
                                               # this skill does not read.

Capture per resource: identifiers, ARNs, status, creation/update timestamps, encryption configuration (AWS-managed vs customer-managed KMS key), and any VPC configuration.

Step 3: Collect CloudWatch Metrics (namespace AWS/Bedrock)

Before querying metrics, load the authoritative thresholds reference:

read_skill_resource(skill_id="bedrock-operation-review", path="references/metrics-thresholds.md")

Use the thresholds from that file when classifying metric values as Normal, Warning, or Critical throughout this step and Step 4.

Discover which models are actually invoked with cloudwatch.ListMetrics (dimension ModelId), then pull metric data with cloudwatch.GetMetricData. Use the user's selected window; default to 7 days.

Key model-level metrics (dimension ModelId):

GPU metrics for self-managed workloads live in the CWAgent namespace (nvidia_smi_utilization_gpu, nvidia_smi_memory_util).

cloudwatch.GetMetricData allows up to 500 metric-data queries per call — batch requests and paginate when a region has many invoked models.

Step 4: Analyze Against Best Practices

Before evaluating findings, load the best-practices checklist:

read_skill_resource(skill_id="bedrock-operation-review", path="references/best-practices-checklist.md")

Use the checklist as the canonical list of items to evaluate for each pillar. Mark each item as ✅ Pass, ⚠️ Warning, ❌ Fail, or ➖ Not Applicable, and generate a finding for every Fail or Warning.

Evaluate all collected data across the pillars below and assign a severity to every finding: CRITICAL, HIGH, MEDIUM, LOW, or INFO. The pillars mirror the Bedrock operational review structure: Security, Performance, Service Quotas, Cost Optimization, Resilience.

4.1 Security

Ref: Security in Amazon Bedrock

• Guardrails: no guardrail configured on high-volume workloads → HIGH. Use guardrails to filter hate, insult, sexual, violence, and PII content and to block denied topics relevant to the use case. Guardrails
• Guardrail signals: high InvocationClientErrors (>5% of Invocations) indicates problematic content reaching models — filter before invocation → MEDIUM. InvocationsIntervened <5% of invocations on an active guardrail suggests under-configuration; >15% suggests policy over-tuning → review.
• Model invocation logging: for workloads handling sensitive content (PII, PCI, HIPAA), enabling invocation logging persists prompt/output on the account and may conflict with data-handling requirements. Flag logging configuration mismatches → MEDIUM. Model invocation logging
• Knowledge Base configuration: chunking (200–1000 tokens), embedding strategy, encryption, access controls, and VPC endpoints → MEDIUM where missing. Knowledge bases
• Knowledge Base data source encryption: data sources without a customer-managed KMS key when handling sensitive content → MEDIUM. Encryption of knowledge base resources
• VPC configuration for model customization jobs: customization jobs without a configured VPC expose training data to the internet → HIGH. (us-east-1, us-west-2) Configure a VPC for Bedrock
• IAM fine-grained access control: agent/alias IAM policies using resource * without justification → MEDIUM. Follow least privilege for inference endpoints. A comprehensive least-privilege audit of all IAM policies is out of scope for this skill — flag obvious *-resource findings only, at INFO severity. IAM for Amazon Bedrock
• Knowledge Base logging: KB ingestion log delivery not configured → LOW. CloudTrail and CloudWatch not enabled for anomaly detection → MEDIUM.
• Prompt injection: this skill does not read prompt template content, so it does not assess whether individual templates are hardened. Instead, treat a guardrail configured with prompt-attack filtering as the control signal — workloads with no guardrail (or a guardrail lacking a prompt-attack content filter) are exposed to prompt injection → MEDIUM. Provide general hardening guidance by reference only. Prompt engineering best practices
• Model access: Amazon Bedrock foundation models are enabled by default (no explicit access request needed) — model access should instead be controlled via IAM and SCP policies scoped to the essential models for the use case (least privilege). A comprehensive least-privilege audit is out of scope for this skill — report gaps at INFO severity only. Model access

4.2 Performance

Ref: Monitoring Amazon Bedrock

• Latency and throttling: InvocationLatency p95, TimeToFirstToken p95, throttle rate (InvocationThrottles / Invocations), and server error rate. High throttle rate → HIGH (add retries with exponential backoff + jitter, request quota increase, spread load, use CRIS). Prefer smaller models and streaming for latency-sensitive apps. Improve Bedrock performance
• Agent performance: agents can use a latency-optimized flow when they have a single knowledge base, no enabled action groups, don't ask follow-ups, and use the default orchestration template → flag agents that miss these conditions. Optimize agent performance
• Agent version configuration drift: compare deployed alias versions (from GetAgentVersion) against the draft. Aliases pinned to outdated versions with different model IDs or orchestration types than the current draft → MEDIUM (may miss performance or capability improvements).
• Invoked model versions: models in Legacy or End-of-Life state still receiving invocations → MEDIUM (plan upgrade; check the model card for its EOL date rather than assuming a fixed notice period — Legacy notice periods are either 6 months or 45 days depending on the model, with most models using 6 months). Model lifecycle
• Data automation: success rate <95%, error rate >5%, or throttle rate >1% for production workloads → MEDIUM. Bedrock Data Automation
• Bedrock service tier: verify tier selection (default, flex, priority, reserved) matches workload criticality. Latency-sensitive apps on Flex, or batch workloads on Priority, are misaligned → MEDIUM. Service tiers