Paidf Orchestration Setup

PAIDF Orchestration Environment Setup

Prepare a Kubernetes GPU environment for PAIDF Orchestration without assuming a cloud provider. Treat Kubernetes—not the host vendor—as the integration contract.

Safety boundary

Start with a read-only audit. Before any prepare, Helm install, Airflow connection change, or remote mutation, summarize the exact changes and obtain user approval. Never print kubeconfig, NGC keys, Hugging Face tokens, AWS secrets, or Kubernetes Secret bodies.

Select two independent axes

1. Select controller placement: use an existing controller, or deploy Airflow into the current Kubernetes cluster.
2. Select model-service placement separately: use external VLM/LLM/image-edit endpoints, or deploy those services in-cluster. Never infer one choice from the other.
3. Prefer external endpoints on a one-GPU H100 node. The controller and augmentation worker may still run in that node's cluster.
4. Reject a Docker-only or SSH-only host until a supported Kubernetes distribution and NVIDIA device plugin expose nvidia.com/gpu.

Read topologies.md before changing infrastructure.

Audit the compute cluster

The cluster is reached only through a kubeconfig the user supplies. It carries a cluster address and admin credentials, so it is never part of the repository. Resolve it in this order:

1. Use $KUBECONFIG if it is already set in the environment.
2. Otherwise ask the user for the path and export it.

echo "${KUBECONFIG:-unset}"   # ask the user for a path when this is unset
export KUBECONFIG=/path/the/user/gave

Never guess a path, assume a repository-relative location, or fall back to ~/.kube/config. If the path the user names does not exist, say so and ask again.

Run locally when the agent already has the kubeconfig (remote_k8s.py audit has no --kubeconfig flag; pass it via the env var):

python scripts/remote_k8s.py audit --service-mode external --json

Alternatively, pass it inline through --kubectl-command:

python scripts/remote_k8s.py audit \
  --kubectl-command "kubectl --kubeconfig $KUBECONFIG" \
  --service-mode external --json

Run through SSH when Kubernetes tooling exists only on the remote host:

python scripts/remote_k8s.py audit \
  --ssh-target ubuntu@host \
  --kubectl-command "kubectl" \
  --service-mode external --json

remote_k8s.py is bundled with this skill — run it from the skill directory, not the repository scripts/ directory. Use --kubectl-command "k3s kubectl" when appropriate. Do not pass SSH passwords or private-key contents in the prompt; use SSH configuration or an agent.

Before a Helm install the audit exits non-zero with ready: false and blocker NGC image-pull secret is missing. That is the expected first-install state, because the chart creates that secret itself — read the facts block and continue. Do not resolve it with --create-registry-secret, which makes the subsequent install fail on ownership metadata.

Interpret capacity conservatively:

• Image Attribute Augmentation: External endpoints deploy no in-cluster inference services, and the checked-in augmentation and attribute-search tasks use CPU profiles. Internal mode deploys VLM, LLM, and image-edit services, each claiming one GPU; require at least three allocatable GPUs for one replica of each, plus one for each additional replica.
• Event Video Generation: External endpoints deploy no in-cluster inference services, but detection/tracking, captioning, and visual-QA auto-labeling task pods each claim one GPU while active. Internal mode additionally needs one GPU per VLM and LLM replica plus two per image-to-video replica — at least four allocatable GPUs for one replica of each service.
• A single-GPU node (for example, one H100) can use external model endpoints, subject to Event Video Generation's GPU auto-labeling capacity.

The compute cluster is shared. Other users' DAG runs may be active in the same namespace. Always report GPUs as free-versus-total (check running pods for GPU requests, not just node allocatable), and never issue broad destructive commands (delete pods --all) against the compute namespace without first checking pod ownership via dag_id and run_id labels.

Deploy a controller on the current cluster

When the user requests setup (not audit-only), confirm which steps to run before executing anything. Present the exact commands you plan to run and obtain explicit approval:

"I will run the following commands in order:

1. make setup — validates secrets from secrets.env and generates the Helm values for install
2. make install sdg-controller — packages Airflow runtime dependencies, uploads DAGs and plugins to S3, and installs/upgrades the Helm release

Proceed?"

There are exactly two install-related targets: make setup and make install sdg-controller. There is no bare make install and no make install nfs unless NFS storage is also needed.

Always run make setup first on any deploy, install, or redeploy request — even if a previous run already generated the Helm values. Secrets rotate; make setup is cheap and safe. Only skip it mid-session when the agent itself just ran it moments earlier.

Missing namespace — if kubectl get ns sdg-workflow returns NotFound, this is a normal first-install condition, not an error to diagnose. Route directly to make install sdg-controller.

Read deploy-controller.md for the full make setup / make install sdg-controller walkthrough: required secrets.env variables, the sandbox DNS failure signature, post-install cluster-state verification, capacity pre-flight, and storage requirements — before running either command.