Cxas Composite Voice Agent Optimizer

CXAS Composite Voice Agent Optimizer

This skill audits, optimizes, and remediates Google Cloud CX Agent Studio (CXAS) and Customer Engagement Suite (CES) agent configurations for Gemini Composite V1 voice naturalness, persona stability, and multi-language parity.

All core operations execute purely on local workspace files (app.json, global_instruction.txt, agents/*/instruction.txt, tools/). The cxas CLI can be used to manage local agent workspaces (cxas pull, cxas lint, cxas push).

When to Use This Skill

Activate this skill when:

• Adapting to Composite Models: Evaluating or migrating an existing or new CXAS agent configuration to Gemini Composite V1.
• Generating Readiness Reports: Generating a prioritized (P0/P1/P2) HTML/Markdown assessment report of required voice adaptations when upgrading a CXAS agent to Gemini Composite V1.
• Optimizing for Composite Models: Optimizing a CXAS agent to apply best practices for Gemini Composite V1.

When NOT to use this skill:

• Standard text-only chat agents without voice/audio synthesis.
• Non-composite standard TTS/STT pipelines.
• Generic non-voice dialog flow refactoring.

⚠️ Fundamental Rule: Non-Destructive Preservation of Existing Instructions

CRITICAL — DO NOT REMOVE EXISTING INSTRUCTIONS: When optimizing an agent for Gemini Composite V1, you MUST NOT delete, remove, or strip existing business logic, domain instructions, taskflows, steps, or operational rules from global_instruction.txt, agents/*/instruction.txt, or tool docstrings.

All adaptations must be strictly additive and non-destructive:

1. Preserve Full Instruction Sets: Retain all existing domain instructions, guardrails, step transitions, and business logic verbatim.
2. Relocate Static Voice & Speech Guidance to Director's Notes: Identify static voice, speech delivery, accent, vocal tone, and acoustic pacing presets inside global_instruction.txt and agents/*/instruction.txt, migrate/consolidate them into the global Director's Note in app.json (synthesizeSpeechConfigs), and remove them from agent text prompts to prevent context waste and instruction dilution. (Note: Static voice personas and accent profiles belong exclusively in Director's Notes in app.json. However, dynamic turn-by-turn emotion recognition, the 7 affective registers, and inline empirical audio tags MUST remain defined in global_instruction.txt under <Affective Delivery and Voice Guidelines> per references/natural_speech_patterns.md).
3. Rephrase Prohibited Tags Without Deleting Logic: When resolving prohibited platform XML tags (strictly restricted to the explicit prohibited list: <state_update>, <context>, <reasoning>, <thought>, <internal>, <call_tool>, <parameter_update>, <variable_update>, <voice_lock>, <voice_output>, <state>, <transition>, <transitions>, and banned legacy CamelCase tags like <Agent>, <Role>, <Persona>), rephrase the tag references into plain natural language descriptions (e.g., "system context", "state update") rather than deleting the surrounding rules or instructions. Do NOT flag or check all general/standard XML tags.
4. Augment Tool Docstrings Additively (Only if Insufficient): In tools/*/python_function/python_code.py, preserve all existing descriptions, parameter documentation, and implementation details. Only append or integrate explicit When to Call: and When NOT to Call: execution boundaries if the existing docstring or description is missing, ambiguous, or insufficient.
5. Harmonize Conflicting Directives Collaboratively: When resolving contradictory instructions across scopes, ask the user which behavior to preserve and adjust the wording to eliminate the contradiction without deleting core domain logic.
6. Enrich Spoken Cues Incrementally: Add natural voice cues (ellipses ..., brief bridge words like "umm...") into response instructions without altering the core messaging or domain content.

Checklist & Inspection Gates

The optimizer evaluates agent configurations against a prioritized checklist:

🔴 Priority P0: Critical Synthesis Blockers, Contradictions & Instruction Clean Up (Must Fix First)

1. Audio Profile & Director's Note Configuration: Add Audio Profile & Director's Note to app.json under synthesizeSpeechConfigs. Preserve the complete Director's Note with mandatory trailing ## Transcript:\n hook to prevent style prompt leakage into spoken audio.
2. Relocate Voice, Accent & Speaking Instructions to Director's Notes: Relocate all static voice, accent, pronunciation, delivery style, vocal tone, pitch, speaking pace, and speech-related directives from global_instruction.txt and agents/*/instruction.txt into app.json under audioProcessingConfig.synthesizeSpeechConfigs (Director's Note and Audio Profile). Placing static voice/speech directives in agent instructions is ineffective, wastes reasoning context tokens, and causes model confusion.
3. Natural Language Accent Strings: Set Accent using Natural Language (e.g., Accent: American English, Accent: Contemporary Irish English, Accent: Australian English, Accent: British English, Accent: Latin American Spanish) rather than locale codes (en-US).
4. Eliminate Prohibited Platform Tags: Eliminate prohibited platform tags (strictly restricted to the explicit list: <state_update>, <context>, <reasoning>, <thought>, <internal>, <call_tool>, <parameter_update>, <variable_update>, <voice_lock>, <voice_output>, <state>, <transition>, <transitions>, and banned legacy CamelCase tags like <Agent>, <Role>, <Persona>) which trigger thought-leakage regex safety filters. Do NOT check or flag general XML tags outside this explicit list.
5. Global Voice Guidelines & Emotion Attunement: If global instructions already exist (whether in global_instruction.txt, inline within app.json, or directly in the agent instructions), append and integrate the canonical <Affective Delivery and Voice Guidelines> block (from references/natural_speech_patterns.md §1.5) into that existing location. If a new global_instruction.txt is created, ensure to add and reference it in app.json (under "globalInstruction": "global_instruction.txt"). This block instructs the model to: (1) dynamically evaluate caller emotion on every turn into one of 7 core affective registers (angry, sad, anxious, frustrated, confused, positive, neutral); (2) enforce acute emotion precedence (angry > sad > anxious > frustrated > confused > positive > neutral); (3) enforce the Negative-Emotion Latch (never reverting to a cheerful tone when an upset caller gives terse/neutral replies until explicit relief); (4) weave verified inline empirical audio tags ([prosody rate="65%"], [slow], [seriousness], [sigh], [short pause], [uhm], [whispers], [positive], [neutral]); (5) apply voice texture rules (1–3 tags/turn, no adjacent stacked tags, ellipses ... micro-pauses, localized bridge words, digit clustering, no exclamation marks, max 1 apology); and (6) execute sub-agent transitions and escalation/wrap-up tools silently. Additionally, conversational sub-agents' instruction.txt must explicitly reference and enforce <Affective Delivery and Voice Guidelines> in their <response_protocol> or persona.
6. Cross-Scope Contradictory Instruction Resolution: Detect and resolve all mutually conflicting instructions across global_instruction.txt, agents/*/instruction.txt, tool descriptions/docstrings, and callbacks. Systematically scan and resolve:
   • Spoken Transfer Announcements vs. Silent Callbacks/Tools: When after_model_callbacks or handoff tools (escalation_call, call_wrap_up) handle transfer announcements and farewell audio, agent prompt directives commanding the agent to speak transfer messages (e.g., "say 'I am transferring you to a live agent now.'") MUST be harmonized to silent handoffs to prevent double-speaking.
   • Invalid Tool Invocation Markup: Eliminate tool markup containing embedded arguments (e.g., {@TOOL: tool_name(arg="val")}) and convert them to valid {@TOOL: tool_name} references with parameters described in natural language text.
   • Pre-Call Tool Pacing vs. Output Grounding: Harmonize prompts that forbid speaking before tool returns with tool docstrings that require pre-call conversational pacing phrases before latency-sensitive lookups.
   • "Say Exactly" vs. Anti-Repetition Deadlocks: Remove blanket repetition bans (e.g., "Never repeat any prompt or confirmation question verbatim across retry attempts") that clash with verbatim compliance, authentication, or legal disclosure requirements (e.g., confirming a 6-character booking reference). Allow explicit multi-turn retry ladders (Strike 1, Strike 2) to govern rephrasing instead of a global ban.
7. Strip Negative Conversational Micro-Bans (Filler & Bridge Prohibitions): Remove prohibitions against natural conversational connective words ("Sure", "Okay", "One moment", "Let's see", "I understand"). Negative bans on conversational fillers contradict voice warmth and prevent the agent from delivering natural conversational pacing phrases before long-running tool executions.
8. Decouple Global Loop Prohibitions from Local Escalation Ladders: Remove blunt turn-count repetition caps (e.g., "Strictly prohibited from repeating the same question > 2 times") that prematurely abort structured multi-turn escalation flows. Rely on explicit, stateful retry ladders to govern escalation deterministically.
9. Prune Monolithic Few-Shot <examples> Debt: Remove large legacy dialog transcripts from prompt text. Legacy examples accumulate format drift, violate newer negative operational rules, and bloat reasoning context. Maintain vetted golden test cases in evaluation suites instead.
10. Eliminate Text-Based Variable Setting Antipatterns: Avoid text-based variable setting (e.g., "Set booking_verified = true" or "Set user_language = es"); state mutations cannot occur via raw text output in CES. Use structured tool invocations (e.g., update_booking_status) instead.
11. Remove Redundant Platform Formatting Rules: Remove instructions mandating plaintext, forbidding markdown headers, or instructing digit spacing; these are handled automatically by the platform baseline.
12. Eliminate Reflexive Turn Closings: Eliminate reflexive turn closings (avoid ending every turn with "Is there anything else?").
13. Eliminate Deprecated Language-Switching Tools: Remove dynamic language-switching tools (language_switcher, en_to_es); session language is established at IVR/session initialization and dynamic switching tools add latency and risk hallucination.
14. Model Settings Configuration: Set modelSettings.model to "gemini-composite-v1" and modelSettings.temperature to 1.0 (prevents acoustic repetition loops).
15. Incorporate Natural Speech Cues: Incorporate natural speech cues (ellipses ... and brief bridge words like "um", "hmm", "let's see") in LLM response instructions.
16. Preserve Domain Logic & Taskflows (MANDATORY Non-Destructive Rule): Never delete, wipe, or strip existing business logic, validation rules, or negative operational constraints. Optimization must be strictly additive and restorative.

🟡 Priority P1: Multi-Language Parity, Session Stability & Call Flow

1. Multi-Language Session Variable (user_language): When the application is multi-lingual (declares languageSettings.supportedLanguageCodes with multiple locales), ensure user_language or app_language is declared in app.json.variableDeclarations to track active caller language and prevent language drift. (Single-language/unilingual apps skip this check).
2. Multi-Language Voice Parity: Every configured locale in languageSettings.supportedLanguageCodes has a matching entry in synthesizeSpeechConfigs with localized Director's Notes, appropriate voice IDs, and native bridge words.
3. Verify Long-Call Stability (5+ Minutes): Verify long-call stability (5+ minutes) without speaker drift, voice fry, or turn exhaustion.
4. Minimize Proactive Unnecessary Call Transfers: Transfer only on explicit customer escalation or hold the line and be rigorous on conversational design. Sub-agent handoffs execute silently via tool calls without speaking internal transition jargon.
5. Employ Validated Physical Acoustic Tags: Employ validated physical acoustic tags (e.g., [whispers], [sigh], [chuckles], [slow], [seriousness]).

🟢 Priority P2: Lowest Priority — Tool Conversational Pacing & Docstring Hygiene (User-Approved Only)

1. Tool-Level Conversational Pacing Directives (Lowest Priority — User-Approved Only): Spoken conversational pacing phrases ("Before calling this tool, speak a brief, natural conversational pacing phrase...") are the lowest priority (P2). They MUST NOT be automatically applied across all tools. Pacing directives should be added ONLY for specific latency-sensitive tools that are explicitly selected and approved by the person executing the skill.
2. Tool Docstring Sufficiency (Only if Insufficient): Inspect declared active tools. If the existing tool docstring or description is incomplete or ambiguous, refine it with clear positive/negative execution boundaries (When to Call: and When NOT to Call:) upon user approval.

Modes of Execution

                       ┌──────────────────────────────────────────────┐
                       │    CXAS Composite Voice Agent Optimizer      │
                       └──────────────────────┬───────────────────────┘
                                              │
                     ┌────────────────────────┴────────────────────────┐
                     ▼                                                 ▼
     ┌───────────────────────────────┐                 ┌───────────────────────────────┐
     │  Mode 1: Report Generation    │                 │       Mode 2: Fix Mode        │
     │  (Readiness Assessment)       │                 │(Audio Patch & Guided Refactor)│
     └───────────────┬───────────────┘                 └───────────────┬───────────────┘
                     │                                                 │
     1. Discover workspace configuration               1. Execute `--remediate` for `app.json`
     2. Run multi-pass acoustic & tool audit           2. Declare `user_language` (if multi-lingual)
     3. Generate prioritized Markdown report           3. Contextually refactor XML, prompts & tags
     4. Review prioritized P0/P1/P2 plan               4. Refactor Python tool docstrings & pacing
                                                       5. Run verification audit & `cxas lint`

Mode 1: Report Generation Mode (Comprehensive Readiness Assessment)

Assesses an existing CXAS agent workspace, checks all checklist inspection gates across both static configurations and semantic prompt policies, and generates a unified prioritized report (P0/P1/P2) detailing required voice adaptations for Gemini Composite V1.

Workflow Steps:

1. Workspace Discovery: Locate app.json, global_instruction.txt, sub-agent instructions (agents/*/instruction.txt), and tool definitions (tools/) in the workspace.

2. Pass 1 — Execute Automated Structural & Audio Audit: Run the local auditor CLI to evaluate deterministic configuration rules:

   # Generate baseline structural Markdown report
   python3 .agents/skills/cxas-composite-voice-agent-optimizer/scripts/audit_agent.py \
     --workspace=. --report

   (What this covers: app.json synthesizeSpeechConfigs (A007), Director's Notes headers (A007), trailing ## Transcript:\n hooks (A007), natural language accents (A008), multi-language audio profile parity (A009), model settings and sampling temperature (A010), prohibited platform XML tags (I015), unregistered template variables (V104), and conversational tool pacing (T014 - P2)).

3. Pass 2 — Semantic Voice & Policy Review (LLM Checklist Evaluation): Actively evaluate the workspace instructions against qualitative checklist gates not covered by static scripts:

   • Audit Global Instructions for Voice Guidelines & Emotion Tags (P0 Critical): Verify whether global instructions exist (in global_instruction.txt, inline within app.json, or directly in the agent instructions) and define the canonical <Affective Delivery and Voice Guidelines> block with all 7 affective registers (angry, sad, anxious, frustrated, confused, positive, neutral), acute emotion precedence, the Negative-Emotion Latch, and inline empirical audio tags ([prosody rate="65%"], [slow], [seriousness], [sigh], [short pause], [uhm], [whispers], [positive], [neutral]). Flag as 🔴 P0 if <Affective Delivery and Voice Guidelines> is missing from the global instructions. If a new global_instruction.txt will be created, verify that app.json will be updated to link it ("globalInstruction": "global_instruction.txt").
   • Relocate Static Voice & Speech Directives to Director's Notes (P0): Review global_instruction.txt and agents/*/instruction.txt for static voice styling, accent directives, vocal tone, speech pace, pronunciation rules, or <voice_lock>/<voice_output> blocks that belong in Director's Notes rather than reasoning prompts. With Gemini Composite V1, Director's Notes configured in app.json are the only way to provide static speech presets to the TTS model.
   • Cross-Scope Contradictory Instructions Audit (P0 Critical): Review global_instruction.txt, agents/*/instruction.txt, tool docstrings, and callbacks to detect conflicting directives. Specifically scan for:
     • Spoken Transfer Announcements vs. Silent Callbacks/Tools: Prompt instructions commanding the agent to speak transfer messages (e.g., "I am transferring you to a live agent now") when callbacks (e.g. after_model_callbacks) or tools (escalation_call, call_wrap_up) inject transfer messaging, causing double-speaking.
     • Invalid Tool Invocation Markup: Prompt instructions containing tool calls with embedded arguments (e.g., {@TOOL: tool_name(arg="val")}).
     • Tool Pacing vs. Output Grounding: Prompts mandating complete silence before tool return vs. tool docstrings requiring pre-call conversational pacing phrases.
     • "Say Exactly" vs. Anti-Repetition Deadlocks: Conflicts between exact verbatim compliance/verification phrases and blanket anti-repetition rules.
   • Strip Negative Conversational Micro-Bans (P0): Identify and remove blanket prohibitions against natural conversational connective words ("Sure", "Okay", "One moment", "Let's see", "I understand") that impede natural speech and tool pacing.
   • Decouple Global Loop Prohibitions from Local Escalation Ladders (P0): Replace blunt turn-count repetition caps with deterministic multi-strike retry and escalation ladders.
   • Prune Monolithic Few-Shot <examples> Debt (P0): Strip large legacy dialog transcripts from prompt text to prevent format drift, rule contradiction, and reasoning token bloat.
   • Eliminate Text-Based Variable Setting Antipatterns (P0): Inspect instruction files for raw text variable mutation statements (e.g., Set user_language = es, Set booking_verified = true). In CES, state mutations cannot occur via raw output text; verify that state changes are mediated through tool calls (e.g., update_booking_status) instead.
   • Remove Redundant Platform Formatting Rules (P0): Remove instructions mandating plaintext, forbidding markdown headers, or instructing digit spacing, as these are natively handled by the platform.
   • Eliminate Reflexive Turn Closings (P0): Eliminate reflexive turn closings (avoid ending every turn with "Is there anything else?").
   • Eliminate Deprecated Language-Switching Tools (P0): Deprecate dynamic language switching tools (language_switcher, en_to_es); session language is established at IVR/session initialization.
   • Speech Texture & Natural Hesitation Directives (P0): Inspect instructions for micro-pause ellipses (...), natural hesitation bridge words, and digit clustering rules.
   • Long-Call Stability & Empathy Capping (P1): Verify that empathetic fillers and apologies are capped to a maximum of 1 occurrence per call.
   • Tool Conversational Pacing & Docstring Sufficiency (P2 — Lowest Priority / User-Approved Only): Spoken conversational pacing phrases are lowest priority (P2) and MUST NOT be assumed or applied across all tools. In the readiness assessment, list tools missing conversational pacing as Priority P2 with an explicit Open Question asking the person executing the skill whether they approve adding pacing to that specific tool. Only recommend adding docstring contracts (When to Call: and When NOT to Call:) if the existing docstring is ambiguous or insufficient.

4. Synthesize & Present Unified Prioritized Report: Combine findings from both Pass 1 (Static) and Pass 2 (Semantic) into a single structured assessment. Always generate a comprehensive markdown table of all P0, P1, and P2 issues with the following structure:

   • Executive Summary: Overall readiness status (PASSED / FAILED) and issue count breakdown across P0, P1, and P2.
   • Prioritized Assessment Table: A unified table with the following columns:
     • Priority: 🔴 P0 (Critical Voice & Synthesis Blockers), 🟡 P1 (High Impact Multi-Language & Stability), or 🟢 P2 (Lowest Priority: Tool Pacing, Hygiene & Texture).
     • Issue Description: Issue identifier/code, concise explanation of the problem, and clickable markdown links to affected files with exact line numbers.
     • Possible Resolution: Concrete, actionable, non-destructive remediation steps complying with the non-destructive guidelines.
     • Open Questions: Add open questions ONLY when there is a genuine conflict or business-logic ambiguity that cannot be resolved without clarification from the user (e.g., asking which specific latency-sensitive tools are approved for conversational pacing phrases, cross-scope contradictory instructions between prompt silence and tool pacing, or conflicting brand routing rules). For deterministic/standard technical fixes (e.g., missing Director's Note header, standard app.json schema patches), omit questions or state *(None - deterministic fix)*.

   Table Schema Example:

   | Priority | Issue Description | Possible Resolution | Open Questions |
   | :--- | :--- | :--- | :--- |
   | 🔴 **P0** | **`MISPLACED_VOICE_INSTRUCTIONS`**<br>`instruction.txt:L8` contains voice tone and accent directives. | Relocate voice/accent instructions into `app.json` Director's Note and remove from agent prompt. | *(None - deterministic fix)* |
   | 🔴 **P0** | **`CONTRADICTORY_INSTRUCTIONS`**<br>`instruction.txt:L20` forbids speech before tool calls, but `tools/search.py` requires pacing. | Harmonize prompt to permit conversational pacing before backend lookup. | Does the business require total silence during tool execution or is conversational pacing preferred? |
   | 🔴 **P0** | **`MISSING_DIRECTORS_NOTE`**<br>`app.json:L6` lacks Director's Note and `## Transcript:\n` hook. | Inject standardized Director's Note with Audio Profile and trailing hook. | *(None - deterministic fix)* |
   | 🟢 **P2** | **`MISSING_TOOL_CONVERSATIONAL_PACING`**<br>`tools/search_flights/python_code.py` lacks spoken pacing phrase. | Add varied pacing phrase to docstring if approved by user. | Do you approve adding a pre-call conversational pacing phrase to `search_flights`? |