Openhands Enterprise Troubleshooting

OpenHands Enterprise Troubleshooting

This skill helps diagnose and resolve common issues on OpenHands Enterprise (OHE) self-hosted installations using Replicated. It covers triage, guided recovery, support bundle generation, and escalation handoffs.

Diagnostic Workflow

When a user reports an OHE issue:

1. Collect symptoms - Ask user to describe what they see, error messages, when it started
2. Identify failure mode - Match symptoms to one of the common issues below
3. Run targeted diagnostics - Use commands in references/diagnostics.md
4. Guide recovery - Follow resolution steps for the identified issue, one at a time
5. Verify fix - Confirm the original symptom is gone, not just that the last command succeeded
6. Generate handoff - If unresolved, produce a clear summary for the OpenHands team

Take recovery one step at a time. Before each step, state what it will change and what you expect to see afterwards; after it, run the check that confirms it before moving on. If the check fails or shows something unexpected, stop and re-diagnose — a step applied on top of a failed one buries the evidence, and several applied blind can leave the install worse than the fault you started with. Destructive steps (restarts, rollbacks, config changes) need the user's agreement first, and are worth recording as you go so the handoff can say exactly what was changed.

Work from whatever the user has. A described symptom starts at step 1; a support bundle goes to Analyzing the Support Bundle. If they paste raw log output, the triage method in references/support-bundle-analysis.md applies to pasted text as well as to files, and notes what a hand-picked excerpt can hide.

Common Failure Modes

1. Sandbox Fails to Start / Timeout

Symptoms:

• Conversation hangs then times out
• "Sandbox failed to start" error
• A timeout in the logs. Read the actual value rather than assuming one: the timeouts in runtime-api are configurable and differ between the Kubernetes client and the app, so quoting a fixed number back to a customer is how you end up chasing the wrong one.

Diagnosis: Check sandbox service status, the sysbox-runc RuntimeClass and its containerd runtime, resource availability

Reference: See references/diagnostics.md - Section "Sandbox Startup"

2. Git Provider Auth Broken

Symptoms:

• "Authentication failed" for the configured provider
• Can't clone or push repos
• The provider shows as disconnected

Diagnosis: Check the provider's secret in Kubernetes and that the provider is enabled. GitHub, GitLab, Bitbucket Data Center, and Azure DevOps are each configured separately — confirm which one the user is actually on before diagnosing

Reference: See references/diagnostics.md - Section "Git Provider Auth"

3. Certificate Errors

Symptoms:

• "certificate expired" or "self-signed certificate" errors
• TLS handshake failures
• Browser shows insecure connection warning

Diagnosis: Check cert expiry, certificate chain, ingress configuration

Reference: See references/diagnostics.md - Section "Certificate Issues"

4. LLM Connectivity Failures

Symptoms:

• "LLM endpoint unreachable"
• "Authentication failed" for LLM API
• Conversations fail to start

Diagnosis: Check LLM endpoint URL, API key secrets, network policies

Reference: See references/diagnostics.md - Section "LLM Connectivity"

5. Keycloak Login Issues

Symptoms:

• Can't access admin console
• Login loop or "invalid credentials"
• Keycloak pod showing errors

Diagnosis: Check Keycloak pod status, database connectivity, realm configuration

Reference: See references/diagnostics.md - Section "Keycloak"

6. Replicated Admin Console Unreachable

Symptoms:

• Can't access admin console URL
• Connection refused or timeout
• Browser shows "site cannot be reached"

Diagnosis: Check Replicated operator pod, ingress, service endpoints

Reference: See references/diagnostics.md - Section "Replicated Admin Console"

7. Upgrade Stuck or Failed

Symptoms:

• Replicated shows upgrade as "failed"
• Pods in crash loop after upgrade
• Migration jobs failing

Diagnosis: Check failed job logs, resource availability, pre-flight failures

Reference: See references/diagnostics.md - Section "Upgrade Issues"

8. OOM / Resource Exhaustion

Symptoms:

• Pods being OOMKilled
• "Too many open files" errors
• Services becoming unresponsive

Diagnosis: Check node resources (memory, disk, file descriptors)

Reference: See references/diagnostics.md - Section "Resource Exhaustion"

Diagnostic Commands Quick Reference

Access the VM and run these common commands.

An empty result never means "healthy". kubectl get pods -l <selector> prints No resources found and exits 0 both when a component is down and when the selector is wrong, so the two are indistinguishable. When you need to know whether something is running, ask its Deployment or StatefulSet for a READY count instead — that object exists either way, and 0/1 means down while a NotFound error means you had the name wrong.

# Is it up? READY answers this; an empty pod list does not.
kubectl get deploy,statefulset -n openhands

# Check overall pod status
kubectl get pods -n openhands

# View pod logs (replace POD_NAME)
kubectl logs -n openhands POD_NAME
kubectl logs -n openhands POD_NAME --previous

# Describe a pod for events
kubectl describe pod -n openhands POD_NAME

# Check resource usage
kubectl top nodes
kubectl top pods -n openhands

# Check certificate expiry
echo | openssl s_client -connect HOST:443 2>/dev/null | openssl x509 -noout -dates

# Check the Replicated components. On Embedded Cluster these are the Admin
# Console in `kotsadm`, the operator in `embedded-cluster`, and the Replicated
# SDK in `openhands` under app.kubernetes.io/name=replicated. A `replicated`
# namespace belongs to the older kURL topology and is absent here.
kubectl get pods -n kotsadm
kubectl get pods -n embedded-cluster
kubectl get pods -n openhands -l app.kubernetes.io/name=replicated

Support Bundle Generation

When the issue requires deeper investigation — or before escalating — generate a support bundle. It captures both host- and cluster-level state in one archive.

Generating the Support Bundle

SSH to the VM, then from the directory containing the installer binary:

sudo ./openhands support-bundle

This uses the default Embedded Cluster spec to collect cluster- and host-level information, and automatically includes the OpenHands application-specific collectors. Run it on a controller node — on a non-controller node it cannot capture cluster-wide information.

For Embedded Cluster versions earlier than 1.17.0, use the support-bundle plugin from within the cluster shell instead:

sudo ./openhands shell
kubectl support-bundle --load-cluster-specs /var/lib/embedded-cluster/support/host-support-bundle.yaml

The bundle is written to the working directory as support-bundle-<UTC timestamp>.tar.gz. Share it with the OpenHands team, or analyze it directly with the steps below.

Support bundles carry potentially sensitive data. Replicated's redactor masks common secret patterns as ***HIDDEN***, but it is not a guarantee — hostnames, user and installation identifiers, and config values routinely survive it. Treat a bundle as confidential, send it only through the channel the OpenHands team gives you, and avoid pasting raw excerpts into public issues or chats.

Analyzing the Support Bundle

Full guide: references/support-bundle-analysis.md. Read it before drawing conclusions — the bundle's layout is not what you would guess from kubectl, and several of its gaps produce convincing false negatives.

Fast path — the bundled triage script reconstructs the standard first pass (cluster meta, analyzer results, pod table, OOM and restart scan, top equivalent, allocatable headroom, events) in one command. It reports; the ranking and the diagnosis are yours to make:

tar -xzf support-bundle-2026-07-28T06_54_18.tar.gz
python3 scripts/bundle_triage.py support-bundle-2026-07-28T06_54_18

You are reading this bundle because something is broken, so treat a clean run as "not here" rather than "nothing wrong" — the script sees pod objects, analyzer verdicts, node conditions and resource totals, and reads no application logs at all. references/support-bundle-analysis.md has a section on where to look next when the objects come back clean.

Then the four things that most often answer the question outright:

Four traps worth knowing before you start:

• Never use file mtimes for timing. They record when you extracted the archive. Take the capture time from the bundle directory name, which is UTC.
• Log filenames are container names, not pod names. Init-container failures (migrate-db, wait-for-db) are invisible to kubectl logs <pod> and are the easiest real failure to miss.
• ***HIDDEN*** means "redacted", not "unset". The redactor over-redacts, including non-secrets.
• Check a log's format before filtering it. Most bundle logs are plain text, not JSON, and jq aborts on the first non-JSON line — so a severity filter can print nothing on a file full of errors. Prefix with grep '^{', and read the non-JSON lines separately.

Once triage points at a failure mode, use references/diagnostics.md for that mode's specific commands and error patterns.