Nim Operator Uninstall

NVIDIA NIM Operator Uninstall

Use this skill to remove the NVIDIA NIM Operator Helm release from a Kubernetes cluster. This skill is intentionally separate from install because uninstall is destructive and needs stronger confirmation.

By default, uninstall only removes the NIM Operator Helm release. Do not delete NIM custom resources, CRDs, namespaces, persistent volumes, secrets, GPU Operator, cert-manager, KServe, or Dynamo dependencies unless the user explicitly approves that specific action.

Workspace Root

Assume commands run from the root of the k8s-nim-operator repository unless the user gives another working directory. Before using repo-relative paths such as .agents/skills/..., verify the current directory:

pwd
test -f .agents/skills/nim-operator-uninstall/SKILL.md

If this check fails, ask for the correct repository root or cd to it before continuing.

Safety Contract

Run read-only inventory before proposing any destructive command. Before running each destructive step, print the exact command, summarize what will be removed, and ask for confirmation.

Read-only examples: kubectl get, kubectl describe, helm list, helm status, helm get values.

Destructive examples: helm uninstall, kubectl delete, namespace deletion, CRD deletion.

Defaults

• NIM Operator release: nim-operator
• NIM Operator namespace: nim-operator
• Keep CRDs by default
• Keep custom resources by default
• Keep namespace by default
• Keep GPU Operator by default
• Keep cert-manager and KServe by default

References

• For validation levels, inventory checks, and evidence format, read references/validation.md.
• For a read-only validation helper from the repository root, run .agents/skills/nim-operator-uninstall/scripts/validate-nim-operator-uninstall.sh.

How To Ask For This Skill

End users do not need to know the internal file layout. They should ask the agent for the cleanup outcome they want. Recognize and support these prompt patterns:

Inventory only:

Use the NIM Operator uninstall skill to inventory the current installation. Do not delete anything.

Safe default uninstall:

Use the NIM Operator uninstall skill to uninstall the NIM Operator Helm release. Preserve CRDs, custom resources, namespace, GPU Operator, cert-manager, and KServe unless I explicitly approve deleting them.

Uninstall a specific release or namespace:

Use the NIM Operator uninstall skill to remove release <release> from namespace <namespace>. Inventory resources first and ask before uninstalling.

Full API cleanup:

Use the NIM Operator uninstall skill to remove the Helm release and then ask me whether to delete NIM Operator CRDs. Show existing custom resources before deleting any CRDs.

Validate after uninstall:

Use the NIM Operator uninstall skill to validate that the operator release and controller pods are gone. Tell me which CRDs and custom resources remain.

Remote cluster through SSH:

Use the NIM Operator uninstall skill against my remote Kubernetes host <user>@<host>. Run commands over SSH, show every command before running it, and do not delete anything until I approve.

Manual CLI Usage

This section is for humans, CI jobs, and reviewers who want to run the same workflow without an agent. Run local commands from the repository root and ensure kubectl points at the target cluster before running helm uninstall.

Inventory only:

.agents/skills/nim-operator-uninstall/scripts/validate-nim-operator-uninstall.sh

Inventory with overrides:

NIM_OPERATOR_RELEASE=nim-operator \
NIM_OPERATOR_NAMESPACE=nim-operator \
.agents/skills/nim-operator-uninstall/scripts/validate-nim-operator-uninstall.sh

Safe default uninstall. This removes only the Helm release and preserves CRDs, custom resources, namespace, GPU Operator, cert-manager, KServe, and Dynamo dependencies:

.agents/skills/nim-operator-uninstall/scripts/validate-nim-operator-uninstall.sh
helm uninstall nim-operator -n nim-operator
.agents/skills/nim-operator-uninstall/scripts/validate-nim-operator-uninstall.sh
helm list -n nim-operator
kubectl get pods -n nim-operator

Remote SSH usage if the skill folder has been copied to the remote host:

ssh <user>@<host> '~/.agents/skills/nim-operator-uninstall/scripts/validate-nim-operator-uninstall.sh'
ssh <user>@<host> 'helm uninstall nim-operator -n nim-operator'
ssh <user>@<host> '~/.agents/skills/nim-operator-uninstall/scripts/validate-nim-operator-uninstall.sh'
ssh <user>@<host> 'helm list -n nim-operator'
ssh <user>@<host> 'kubectl get pods -n nim-operator'

Initial Questions

Ask only for missing choices that materially affect removal:

1. Which release and namespace should be uninstalled?
2. Should existing NIM and NeMo custom resources be deleted before the Helm release is uninstalled, or preserved? Custom resources must be deleted while the operator is still running (see "Optional Custom Resource Cleanup").
3. Should NIM Operator CRDs be deleted after the Helm release is uninstalled, or preserved?
4. Should the namespace be deleted after cleanup, or preserved?

If the user wants a quick default uninstall, uninstall only the nim-operator Helm release from namespace nim-operator and preserve CRDs, custom resources, namespace, GPU Operator, cert-manager, and KServe.

Inventory Checks

Start inventory by calling the bundled validation helper:

.agents/skills/nim-operator-uninstall/scripts/validate-nim-operator-uninstall.sh

This is the canonical pre-uninstall call site for the skill. It checks client tools, cluster access, the Helm release, operator namespace resources, NIM Operator CRDs, and any NIM/NeMo custom resources.

If the helper is unavailable or a narrower manual check is needed, run these read-only checks before proposing uninstall commands:

kubectl config current-context
kubectl cluster-info
helm list -A | grep nim-operator
helm status nim-operator -n nim-operator
helm get values nim-operator -n nim-operator
kubectl get pods -n nim-operator
kubectl get deployment -n nim-operator -l app.kubernetes.io/instance=nim-operator,app.kubernetes.io/name=k8s-nim-operator
kubectl get crd | grep -E 'apps.nvidia.com'

Inventory NIM and NeMo custom resources across all namespaces:

kubectl get nimservices.apps.nvidia.com -A
kubectl get nimcaches.apps.nvidia.com -A
kubectl get nimpipelines.apps.nvidia.com -A
kubectl get nimbuilds.apps.nvidia.com -A
kubectl get nemodatastores.apps.nvidia.com -A
kubectl get nemoentitystores.apps.nvidia.com -A
kubectl get nemocustomizers.apps.nvidia.com -A
kubectl get nemoevaluators.apps.nvidia.com -A
kubectl get nemoguardrails.apps.nvidia.com -A

If any custom resources exist, warn that deleting CRDs will delete or orphan API access to those resources. Ask whether the user wants to delete custom resources first.

Optional Custom Resource Cleanup

Do this step BEFORE uninstalling the Helm release.

NIM and NeMo custom resources each carry an operator-managed finalizer (for example finalizer.nimcache.apps.nvidia.com, finalizer.nimservice.apps.nvidia.com). Only the running operator removes these finalizers during deletion. If the Helm release is uninstalled first, the controller is gone, so any later kubectl delete of a custom resource blocks forever: the object keeps its deletionTimestamp and its finalizer, which in turn blocks CRD deletion and wedges the namespace in Terminating. If you have already hit this, see "Recovery From Stuck Finalizers".

Only if the user explicitly approves deleting NIM and NeMo custom resources, show and run targeted deletes while the operator is still running. Prefer deleting specific resources the user selected. If the user approves deleting all NIM Operator custom resources, use:

kubectl delete nimservices.apps.nvidia.com --all -A
kubectl delete nimcaches.apps.nvidia.com --all -A
kubectl delete nimpipelines.apps.nvidia.com --all -A
kubectl delete nimbuilds.apps.nvidia.com --all -A
kubectl delete nemodatastores.apps.nvidia.com --all -A
kubectl delete nemoentitystores.apps.nvidia.com --all -A
kubectl delete nemocustomizers.apps.nvidia.com --all -A
kubectl delete nemoevaluators.apps.nvidia.com --all -A
kubectl delete nemoguardrails.apps.nvidia.com --all -A

Warn that this may remove model-serving workloads, caches, jobs, and service state owned by those custom resources.

Verification gate: before moving on to the Helm uninstall, confirm every custom resource is actually gone (not just marked for deletion). Re-run the inventory and ensure each command returns no resources:

kubectl get nimservices.apps.nvidia.com -A
kubectl get nimcaches.apps.nvidia.com -A
kubectl get nimpipelines.apps.nvidia.com -A
kubectl get nimbuilds.apps.nvidia.com -A
kubectl get nemodatastores.apps.nvidia.com -A
kubectl get nemoentitystores.apps.nvidia.com -A
kubectl get nemocustomizers.apps.nvidia.com -A
kubectl get nemoevaluators.apps.nvidia.com -A
kubectl get nemoguardrails.apps.nvidia.com -A

If any resource is still present with a deletionTimestamp and a lingering finalizer, do not proceed to helm uninstall. The operator must stay running to drain the finalizer; wait for it to clear before continuing, or see "Recovery From Stuck Finalizers".