Helm Dev Environment

Helm Dev Environment

Set up, run, and tear down the local Kubernetes development environment for OpenShell. The stack is: k3d (Docker-backed k3s) for the cluster, Skaffold for image builds and Helm deploys, and the OpenShell Helm chart (deploy/helm/openshell/).

Prerequisites

• Docker Desktop (macOS) or Docker Engine (Linux) running
• mise install completed (provides k3d, kubectl, skaffold, helm)

Startup

1. Create the cluster

mise run helm:k3s:create

Creates a k3d cluster and merges its kubeconfig into the worktree-local kubeconfig file. When the named cluster already exists, the task starts any stopped containers and refreshes same-named kubeconfig entries so a recreated load balancer's current API port takes effect. Also applies the upstream agent-sandbox CRDs/controller (pinned via AGENT_SANDBOX_VERSION in tasks/scripts/helm-k3s-local.sh, fetched from github.com/kubernetes-sigs/agent-sandbox releases), enables its OTLP tracing on v0.5 and later, installs an OTLP trace collector and UI in the observability namespace, and preloads the default sandbox image into k3d so the first sandbox create does not wait on a large registry pull. Traefik is disabled at cluster creation time.

Multi-worktree support: the cluster name is derived from the last component of the current git branch (e.g. branch chore/1234-local-dev/octocat → cluster openshell-dev-octocat). Each worktree therefore gets its own isolated cluster and its own kubeconfig file. Override with HELM_K3S_CLUSTER_NAME to force a specific name or share one cluster across worktrees.

Port mappings created at cluster time (cannot be changed without recreating):

Override with env vars before running helm:k3s:create:

• HELM_K3S_LB_HOST_PORT (default: 8080)
• HELM_K3S_PRELOAD_SANDBOX_IMAGE (default: nvcr.io/nvidia/base/ubuntu:24.04; set to an empty value to skip)
• HELM_K3S_COLLECTOR_IMAGE (default: mcr.microsoft.com/dotnet/aspire-dashboard:latest)
• HELM_K3S_COLLECTOR_HEALTH_TIMEOUT (default: 120 seconds)

2. Deploy OpenShell

Iterative dev (rebuilds on file changes, recommended during active development):

mise run helm:skaffold:dev

One-shot deploy (build once and leave running):

mise run helm:skaffold:run

Resource admission defaults to enabled and caller driver config to disabled. Driver-config scenarios need an explicit allowDriverConfig opt-in; external attachments also need administrator-controlled approval labels in the target namespace. GPU attachments and operator-selected image-pull Secrets are exempt from labels. Managed workspace image-pull Secrets are copied from the configured source in the gateway namespace; do not grant approval to the gateway database PVC or disable admission to make tests pass.

The Skaffold flow builds distinct gateway, sandbox, and supervisor images and deploys the OpenShell Helm chart. The Kubernetes driver creates a capability-free workload Pod and a directly managed capability-free supervisor Pod. One namespace-wide NetworkPolicy denies direct egress from every OpenShell workload Pod. The pkiInitJob hook (a pre-install Job that runs openshell-gateway generate-certs) generates gateway and CLI TLS secrets on first install. Supervisor Pods project only ca.crt and authenticate gateway RPCs with sandbox bearer tokens. User client certificates and private keys remain outside supervisor and workload Pods. The default Skaffold values export gateway and Kubernetes-driver traces to the collector service installed by helm:k3s:create. Envoy Gateway is opt-in; see the Optional Add-ons section.

The gateway Service uses ClusterIP. Access is via Envoy Gateway (port 8080) or the unified local forwarding task:

mise run helm:k3s:forward

The task forwards OTLP/gRPC to http://127.0.0.1:4317 and the trace UI to http://127.0.0.1:18888. When Skaffold has deployed a Kubernetes gateway, it also forwards the gateway to http://127.0.0.1:8090; otherwise it continues with the collector ports only. A successful plaintext helm:skaffold:run registers the gateway under the worktree-specific k3d cluster name and selects it as the active gateway. Keep the forwarding task running while using those endpoints.

Viewing local traces

The gateway exports OTLP/gRPC to http://openshell-collector.observability.svc.cluster.local:4317 through the default Skaffold values. Forward OTLP/gRPC and the trace UI to the host:

mise run helm:k3s:forward

Open http://127.0.0.1:18888 and exercise the gateway to inspect gateway and Kubernetes compute-driver spans under their distinct service names, along with Agent Sandbox controller reconciliation spans linked through the Sandbox trace-context annotation. The same command exposes OTLP/gRPC on http://127.0.0.1:4317 and, when deployed, the Kubernetes gateway on http://127.0.0.1:8090. The local gateway:docker, gateway:podman, and gateway:vm tasks detect the collector listener at startup and enable trace export only while it is reachable.

The Skaffold profile for HA reverse-proxy development is available from deploy/helm/openshell/:

# Two gateway replicas + external PostgreSQL Secret + Envoy Gateway + Gateway API route.
KUBECONFIG=../../../kubeconfig skaffold run -p high-availability

The high-availability profile expects a Secret named openshell-ha-pg in the openshell namespace with a uri key. For local manual testing, either create your own PostgreSQL Secret or use the e2e PostgreSQL fixture manifest in e2e/kubernetes/postgres-fixture.yaml.

For the high-availability profile, return to the repository root and apply the GatewayClass and BackendTrafficPolicy manifest after Skaffold has installed Envoy Gateway:

KUBECONFIG=kubeconfig mise run helm:gateway:apply

The BackendTrafficPolicy disables Envoy request and stream-duration timeouts for OpenShell's GRPCRoute. Keep that policy in deploy/kube/manifests/envoy-gateway-openshell.yaml, not in the Helm chart; it is required for long-lived gRPC create/watch/exec/relay streams during gateway rollouts and scale events.

TLS behaviour

ci/values-skaffold.yaml sets server.disableTls: true, so Skaffold-based deploys run plaintext by default. Override server.disableTls=false to exercise TLS/mTLS.

Connecting through the forwarding task

Port 8080 is already bound by the k3d load balancer when Envoy Gateway is active, so the forwarding task uses local port 8090 for the gateway. In a second terminal, confirm that the gateway is registered and active:

openshell gateway list

Plaintext (default Skaffold deploy):

openshell sandbox list

With mTLS enabled — extract the client cert the PKI hook wrote to the cluster, then place it where the CLI expects it. Run once after each fresh install:

mkdir -p ~/.config/openshell/gateways/openshell/mtls
KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \
  -o jsonpath='{.data.ca\.crt}'  | base64 -d > ~/.config/openshell/gateways/openshell/mtls/ca.crt
KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \
  -o jsonpath='{.data.tls\.crt}' | base64 -d > ~/.config/openshell/gateways/openshell/mtls/tls.crt
KUBECONFIG=kubeconfig kubectl get secret openshell-client-tls -n openshell \
  -o jsonpath='{.data.tls\.key}' | base64 -d > ~/.config/openshell/gateways/openshell/mtls/tls.key

The server cert SANs include localhost and 127.0.0.1, so hostname verification passes over a port-forward without any extra flags:

openshell sandbox list --gateway-endpoint https://localhost:8090

Teardown

Remove the Helm releases (keep cluster)

mise run helm:skaffold:delete

Delete the cluster entirely

mise run helm:k3s:delete

This removes the k3d cluster and all resources. Kubeconfig context is left behind but will point to a deleted cluster — safe to ignore or clean up manually.

Optional Add-ons

Some add-ons can be enabled by uncommenting values in skaffold.yaml, but prefer the dedicated Skaffold profiles when they exist. Profiles avoid leaving local manual edits in the worktree.

Envoy Gateway (Gateway API / GRPCRoute)

Use the high-availability Skaffold profile for HA reverse-proxy testing. The profile intentionally includes Envoy Gateway so multi-replica behavior is exercised through the same Gateway API path used by reverse-proxy deployments:

cd deploy/helm/openshell
KUBECONFIG=../../../kubeconfig skaffold run -p high-availability
cd ../../..
KUBECONFIG=kubeconfig mise run helm:gateway:apply

values-gateway.yaml creates a Gateway (listener on port 80, class eg) and GRPCRoute in the openshell namespace. The high-availability profile installs the Envoy Gateway Helm chart and layers both values-high-availability.yaml and values-gateway.yaml onto the OpenShell release.

deploy/kube/manifests/envoy-gateway-openshell.yaml creates:

• GatewayClass/eg
• BackendTrafficPolicy/openshell-grpc-timeouts

The Envoy Gateway proxy Service is usually exposed through the k3d load balancer at http://127.0.0.1:8080. If the cluster was created with a different HELM_K3S_LB_HOST_PORT, use that host port instead.

For manual tests against an existing cluster, prefer forwarding the Envoy proxy Service rather than svc/openshell. That keeps client traffic on the same path as a real reverse proxy while gateway pods rotate behind it:

KUBECONFIG=kubeconfig kubectl get svc -A \
  -l gateway.envoyproxy.io/owning-gateway-name=openshell
KUBECONFIG=kubeconfig kubectl -n <envoy-service-namespace> port-forward \
  svc/<envoy-service-name> 8080:80
openshell gateway add http://127.0.0.1:8080 --name openshell --local

When running e2e tests manually through Envoy, register gateway metadata (as above) instead of relying only on OPENSHELL_GATEWAY_ENDPOINT; some tests call openshell gateway info and expect metadata for the active gateway.