Agent Platform Alert Configuration

Agent Platform Alert Configuration

Critical Steps

1. Safety & Confirmation Tiers (CRITICAL)

Before executing any commands or writing configurations on behalf of the user, you MUST adhere to the following safety tiers based on the action requested:

1. Tier R: Read-only (check_telemetry.py / gather_agent_info.py)
   • Rule: No confirmation needed. You may execute these scripts immediately to inspect telemetry status or gather agent configuration details.
2. Tier B: Billing & Resource Creation (create_online_monitor.py / provisioning)
   • Rule: Explicit User Confirmation Required. These actions incur additional billing charges and create cloud resources. The agent MUST ALWAYS warn the user explicitly about the potential extra billing costs of BOTH the Online Monitor (specifically mentioning LLM evaluations) and Telemetry (specifically mentioning Cloud Trace/Cloud Logging export). You MUST STOP and ask for explicit approval before proceeding with provisioning or providing setup commands.

2. Prerequisites & Dependencies

Agent Telemetry

• Disclaimer: For Reliability, Cost, Safety, and Security alerts to function, the underlying agent MUST be instrumented to emit OpenTelemetry (OTel) metrics. If the agent does not emit these metrics, the alerting policies will have no data stream to evaluate.

Python Environment

Before executing any python script in this skill you MUST install the required dependencies in your environment. Run this command first:

pip install -r scripts/requirements.txt

3. Input Assumptions

• Explicit Project Adherence: You must ONLY configure alerts, query telemetry, or interact with the Google Cloud Project(s) explicitly provided by the user in the prompt. Do NOT assume or use other projects from your environment or history unless the user explicitly directs you to do so.
• Sequential File Transformations: If the user explicitly asks to copy a file and then modify it, you MUST perform these actions sequentially (copy first, then modify) rather than writing the final content directly.

4. Execution Steps

1. Mandatory Prerequisite Execution Protocol (SEQUENTIAL): Before generating or writing ANY configuration, you MUST execute these steps in order:

   1. Step 1: Streamlined Discovery (Mandatory): Run gather_agent_info.py to automatically identify agent runtime, verify telemetry, metric scopes, linked datasets, and more. This script covers most of the manual verifications listed in subsequent steps.
      • Command: python3 scripts/gather_agent_info.py --project-id {project_id} --agent-name {agent_name}
      • Note: If this script fails, returns partial data, or doesn't produce everything you need, you MUST satisfy requirements by running the manual fallback steps listed in Step 2 and then perform Step 3 below. If Step 1 succeeds and provides all info, SKIP to Step 3 (Pre-existing Policies Verification).
   2. Step 2: Metric Scope Verification (Fallback): Run this ONLY if Step 1 failed to determine the metric scope.
      • Action A (CLI): Run gcloud beta monitoring metrics-scopes list projects/{project_id}. If a scoping project is returned, you MUST deploy policies there.
      • Action B (Code Scan): Search Terraform configurations for google_monitoring_monitored_project resources to extract the scoping project.
      • Action C (Fallback): If ambiguous, ASK the user: "Are you using a multi-project Cloud Monitoring Metric Scope? If so, what is the scoping project ID?"
   3. Step 3: Pre-existing Policies Verification: Avoid duplicates.
      • Action: Scan the target directory to see if aggregated policies already exist targeting the same metrics (grouped by reasoning_engine_id or gen_ai_agent_name). Use scan_duplicates.py to verify.

2. Alert Policy Type Resource Files: You MUST list and read files under references/ with names ending in _alert_policies.md to learn how to configure alert policies based on type. By default you MUST configure all of the following alert types UNLESS the user requests to generate explicit alert policies and/or types. Follow their tables of content to help you find the reference sections you need to read:

   Alert Type	Reference File
   Reliability	reliability_alert_policies.md
   Quality	quality_alert_policies.md
   Cost	cost_alert_policies.md
   Safety	safety_alert_policies.md
   Security	security_alert_policies.md

5. Outputs & Formats

• Always configure the supported alerting policies for the target agent:

  • For Reliability Monitoring: You MUST configure exactly five alerting policies:
    1. Latency (anomaly monitoring)
    2. Error Rate - Fast Burn SLO (1-Hour Window)
    3. Error Rate - Slow Burn SLO (3-Day Window)
    4. Model Call Error Rate (SQL-based Observability Analytics Alerting)
    5. Tool Call Error Rate (SQL-based Observability Analytics Alerting)
  • For Quality Monitoring: You MUST configure exactly three alerting policies (Requires Vertex AI Online Monitors):
    1. Final Response Quality
    2. Tool Use Quality
    3. Hallucination
  • For Cost Monitoring: You MUST configure exactly one cost alerting policy:
    1. Rapid Token Burn Rate (anomaly monitoring)
  • For Safety Monitoring: You MUST configure exactly one safety alerting policy:
    1. High Model Armor Safety Policy Trigger Rate (SQL-based Observability Analytics Alerting)
  • For Security Monitoring: You MUST configure exactly one security alerting policy:
    1. High IAM Permission Denied Trigger Rate (SQL-based Observability Analytics Alerting)

• Terraform Only: Write the generated observability configuration ONLY as Terraform (.tf) files (such as alerts.tf, variables.tf).

  • You ONLY need to install Terraform if you're asked to deploy the alerts AND there is no valid Terraform install. SQL-based alerting using condition_sql requires the provider version >= 6.0.0 (or late 5.x versions supporting the feature).
  • If you are NOT asked to deploy the alerts you do not need to install terraform.

• Dynamic Multi-Resource Alerting (No Single-Resource Pinning): You MUST NOT hardcode specific agent IDs or resource name filters (for example, {gen_ai_agent_name="{agent_name}"} or metric.labels.agent_resource_name="{agent_name}") in alerting conditions unless explicitly requested (for example, "ONLY for this agent"). Merely mentioning a specific agent name or ID in the request does NOT constitute an explicit request to pin/filter; you MUST still default to dynamic grouping to cover all agents. To cover all active agents in the project dynamically:

  Good Example (PromQL Grouping):

  sum(rate(workload_googleapis_com:gen_ai_invoke_agent_duration_count{monitored_resource="generic_node"}[5m])) by (gen_ai_agent_name)

  Bad Example (PromQL Hardcoded Filter):

  sum(rate(workload_googleapis_com:gen_ai_invoke_agent_duration_count{monitored_resource="generic_node", gen_ai_agent_name="support-bot"}[5m]))

  • For Reliability Metrics using PromQL: ALWAYS use grouping aggregations. Group by gen_ai_agent_name (for example, by (gen_ai_agent_name)). Avoid filtering to a single ID/Name unless requested.
  • For Quality Metrics using Standard Threshold Filters: Omit the agent_resource_name filter entirely. Configure the condition filter to only target the monitored resource type (aiplatform.googleapis.com/OnlineEvaluator) and metric type (aiplatform.googleapis.com/online_evaluator/scores) globally for the project.

  Good Example (SQL Grouping):

  SELECT
    JSON_VALUE(resource.attributes, '$."cloud.resource_id"') as agent_id,
    ...
  FROM ...
  GROUP BY agent_id

  Bad Example (SQL Hardcoded Filter):

  SELECT ...
  FROM ...
  WHERE JSON_VALUE(resource.attributes, '$."cloud.resource_id"') = 'support-bot'

  • For Downstream Calls using SQL: Omit the ENDS_WITH filter targeting a specific agent name. Instead, extract the agent identifier (for example, JSON_VALUE(resource.attributes, '$."cloud.resource_id"')) and add it to the GROUP BY clause alongside the model or tool name.

• Directory Inference: Prefer the path explicitly provided by the user (if any). Otherwise, deploy configuration files to target Terraform or SRE folders (such as monitoring/, ops/, sre/). Use tools to locate where alert policies or state pointers exist in the project, rather than blindly writing to the root.

• Notification Channels: By default, never configure any notification channels without user input. If the user explicitly provides a notification channel in their prompt, configure the alerts to use it. If no notification channel is provided, you MUST explicitly ask the user in your final response if they would like to configure notification channels. This is a mandatory question and you MUST NOT omit it from your response. IMPORTANT Do NOT make assumptions about notification channels. If you search the codebase for a notification channel you must ALWAYS confirm with the user before using it.

• Plain English Response: You MUST include a plain English explanation for what the alerts do in your response. This must explain in plain English what the alert measures, how the algorithm works, and what a trigger indicates.