Ambient Healthcare Agent With Nemotron Voice Agent

Ambient Healthcare Agent with Nemotron Voice Agent

Purpose

Customize src/examples/generic/ in a user-selected NVIDIA Nemotron Voice Agent (NVA) checkout. Use this skill for appointment making, five-field patient intake, or a developer-defined healthcare workflow that fits one Pipecat pipeline with an LLM prompt, OpenAI-style tool schemas, and Python handlers.

This skill is owned by Healthcare TME. It is not maintained or endorsed by the NVA team, and it does not live in the NVA repository. For ordinary NVA deployment or non-healthcare voice work, follow the public NVA documentation instead.

Read the user-experience flowchart when a visual overview of the gated workflow or bundled scenario state machines would help.

Requirements

• A compatible checkout of https://github.com/NVIDIA-AI-Blueprints/nemotron-voice-agent
• Python 3.10+ with PyYAML, Git, and Docker Compose
• Network access and credentials for the user-selected LLM, ASR, and TTS services
• Permission to write only to the NVA checkout path the user explicitly supplies
• Human review before sending fictional healthcare-style histories to any model endpoint

Treat the skill loader's installed directory as SKILL_DIR. Resolve bundled scripts/ and references/ from that directory, not from the current working directory.

Required Welcome

Begin a new positive workflow with this phrase verbatim:

Welcome to the NVIDIA Nemotron Voice Agent (NVA for short). We will customize NVA for creating ambient healthcare agents.
Now I will make a fresh clone of the Nemotron Voice Agent repository, and this will be the directory we work out of. Where would you like me to clone the repo to? Please provide a path.
If you already have a clone of the repository somewhere, please point me to the path.

Then stop. Do not inspect files, search for clones, reuse a path from earlier context, run tools, or choose a default. Set NVA_ROOT only from a path the user provides or confirms after this welcome.

Instructions

1. Resolve and validate the NVA checkout

If the user requests a fresh clone, clone only to their exact destination and only with network and write permission:

git clone https://github.com/NVIDIA-AI-Blueprints/nemotron-voice-agent.git "$NVA_ROOT"

If cloning fails, report the observed reason and ask the user to either grant the session the required access via /permissions and request a retry, or manually clone the NVA repository and provide its path. Do not retry until the user grants access or supplies a checkout path. For either a fresh or existing checkout, require these markers:

test -f "$NVA_ROOT/docker-compose.yml" && \
test -f "$NVA_ROOT/examples_registry.yaml" && \
test -d "$NVA_ROOT/src/examples/generic"

An invalid path is a hard stop. Do not modify any repository before this check passes.

After the markers pass, create $NVA_ROOT/.env by copying $NVA_ROOT/.env.example when the target does not already exist. Preserve an existing .env and never print its contents. If .env is absent and the template is missing, report that setup failure and stop. Do this immediately after validating a fresh clone or an existing checkout, before asking the user to choose hosted services.

2. Inspect compatibility and service defaults

Run:

python3 "$SKILL_DIR/scripts/inspect_nva_generic_defaults.py" \
  --nva-root "$NVA_ROOT" --check-compatibility

Stop on compatibility errors. Report the inspected prompt plus the LLM, ASR, and TTS key, display name, model/server, base URL when present, and catalog source. Never substitute release-specific defaults from memory.

The preflight checks Python structure and required capabilities rather than an NVA version number or one exact source string. It also discovers deployment skills from both skills/*/SKILL.md and .agents/skills/*/SKILL.md. A passing check is not permission to guess through an unknown layout: stop when syntax or a required semantic hook is ambiguous.

3. Select the runtime and verify setup access

Explain that the default is the compatible public NVIDIA AI Endpoint entries in the NVA cloud catalog. Before any inference, let the user choose public NVIDIA endpoints, NVA-managed local NIMs, existing NIM endpoints, or a mixed layout. Never silently fall back to public endpoints after opt-out. In the same message as these choices, give the user the actual absolute path to $NVA_ROOT/.env and tell them to fill in its NVIDIA_API_KEY= entry if they choose public NVIDIA AI Endpoints. Explain that the key authenticates access to those endpoints. Direct the user to the public NVA deployment documentation for credential setup; never ask them to paste a secret into chat or display the file contents. State:

The NVIDIA_API_KEY is required to utilize public NVIDIA AI Endpoints. With this key configured, I will be running live tests while customizing and standing up a Nemotron Voice Agent application.

Before applying a healthcare overlay:

1. Run python3 "$SKILL_DIR/scripts/verify_docker_compose_access.py" and require all checks to pass. If Docker Compose access is blocked by session permissions, report the observed failure, ask the user to grant the required access via /permissions, and stop until the user requests a retry.
2. Use the public NVA deployment instructions to identify one recipe and the selected runtime's credential and endpoint requirements. Do not start the unmodified Generic recipe or run inference at this stage.

If repository compatibility or Docker Compose access fails, report the exact failed gate and stop before applying an overlay. Missing credentials or unavailable endpoints do not prevent overlay and static validation, but they block the later authenticated service checks, live validation, and handoff. Do not change runtime modes silently or claim the app is ready.

4. Obtain informed scenario selection

After checkout compatibility and Docker Compose access pass, restate the resolved runtime and services, then ask exactly:

What type of voice agent application would you like to create? We have two example default use cases, appointment making and patient intake, or you could tell me your own use case.

Offer appointment-making example, patient-intake example, and customize your own use case. In the same message, replace the destination placeholder below with the actual resolved LLM display name and base URL or host:

If you choose either example, I will apply its customization and send its bundled fictional conversation histories to <resolved destination> for live validation during setup and before handoff. The appointment example includes the fictional patient Jordan Patel, date of birth 1979-09-24, and appointment details. The patient-intake example includes the fictional patient Maya Chen, date of birth 1988-04-12, symptoms, current medications, and pharmacy details.

Wait for selection. Selecting a preset after this disclosure authorizes only the disclosed fixture and destination. Ask again if either changes.

5. Apply a preset

For a selected preset, apply the bundled overlay without additional design questions:

python3 "$SKILL_DIR/scripts/apply_generic_agent_template.py" \
  --nva-root "$NVA_ROOT" \
  --scenario-dir "$SKILL_DIR/references/appointment-making"

Use references/patient-intake for patient intake. The applier must preserve current LLM/ASR/TTS defaults, set the scenario prompt as the Generic default, patch only supported insertion points, and remain idempotent.

The applier reruns compatibility with the selected scenario before writing. To inspect that gate separately, pass --scenario appointment-making, --scenario patient-intake, or --scenario custom together with --check-compatibility. Scenario checks must cover every scenario-specific hook, including deterministic session startup for both examples.

Appointment making installs SQLite support, initializes data/appointment-making/appointment_schedule.sqlite, and creates or merges docker-compose.override.yml; it must not edit the base Compose file. It also queues the exact fixed opening greeting once at session start and suppresses the model-generated intro. Patient intake collects name, date of birth, symptoms, current medications, and preferred pharmacy, and installs deterministic turn/speech guards plus generated unit tests. Preserve its shared conversation state, earliest-missing-field question, exactly-once direct tool responses, one-time welcome, silent empty tool transitions, interruption forwarding, and interruptible welcome. These are code-backed safety invariants, not prompt-only suggestions. Read references/example-design-choices.md when implementation detail is needed.