Paperclip Dev Workspace Run Verify Fix

Paperclip Dev Workspace Run / Verify / Fix

This skill is for Paperclip-specific development workspaces whose service is started through project execution workspace runtime services, typically a worktree service such as paperclip-dev.

Success means all of these are true:

• the service was started through the normal managed runtime path, not a detached workaround
• the worktree database is a full bootstrapped isolated clone of the primary instance database
• /api/health returns status: ok and bootstrapStatus: ready
• the root page returns 200 and does not show the first-admin setup gate
• the board user can log in with their normal dev credentials
• the app shows populated cloned data, not only a manually copied auth user
• the main control plane shows the service as running / healthy with the expected URL
• the served workspace app also knows about that service and shows it as running / healthy
• the process that owns the service port really belongs to the target workspace: readlink /proc/<pid>/cwd resolves inside the target worktree, not a sibling workspace

If any item fails, keep fixing. Do not mark the issue done because one probe passed.

Hard rules

• The master checkout /srv/paperclip/home/paperclipai/paperclip (the primary repo with the real instance .env) is never a workspace repair target. Never point a worktree runtime service at it, never git worktree add into it or under <master>/.paperclip/worktrees/, and never edit its .env while repairing a workspace. Managed workspaces run in their own folders, for example ~/.paperclip-worktrees/instances/<slug>/….
• Never trust a running / healthy runtime row or a passing /api/health alone. The service:paperclip-dev port is pinned in service config, so a sibling workspace's process can own the port and still answer health checks. Confirm the port owner's real cwd (readlink /proc/<pid>/cwd) before and after every repair; the runtime row's recorded cwd and pid can be fabricated by port adoption.
• Read doc/DEVELOPING.md before running Paperclip CLI, dev server, worktree, database, build, or test commands.
• Use the Paperclip CLI and API as the source of truth for worktree and database operations. Do not use psql, raw embedded-Postgres commands, or ad hoc row copying for the normal fix path.
• Do not manually copy only auth rows as the final fix. That proves login, but it does not prove the isolated workspace has the full bootstrapped database.
• Prefer managed runtime start/stop routes over pnpm dev or detached shell processes when the task is about a reusable workspace service.
• Avoid destructive git or database actions. Preserve user changes in the worktree.
• Run the smallest verification that proves the repair. Add focused tests only when code changed.
• Leave an issue comment with root cause, exact fix, verification, and any commit link. Set a clear final status.

Inputs to collect

Use environment variables when available. Do not print API keys or passwords.

• PAPERCLIP_API_URL: main control-plane API URL
• PAPERCLIP_API_KEY: agent API key
• PAPERCLIP_RUN_ID: current run id for runtime-service mutations
• PAPERCLIP_TASK_ID: current issue id
• PAPERCLIP_COMPANY_ID: company id
• PAPERCLIP_AGENT_ID: current agent id
• execution workspace id for the worktree service
• runtime workspace command id, usually service:paperclip-dev
• expected service URL, for example http://paperclip-dev:40631
• dev credential owner, if the user supplied one; never post the password

If the execution workspace id or service command id is missing, read the issue, project, or execution-workspace API records first. Do not guess and start an unmanaged server on a random port.

Normal run sequence

Use this when the user says to start the workspace, start it again, or fix a workspace that should be freshly ready.

1. Confirm the latest issue comment and restate the success condition in your own words.
2. Inspect current runtime state from the main control plane.
3. Check the target port for conflicts: identify the current port owner and its real cwd, and check whether any sibling workspace's runtime row also claims the same port (see "Port conflicts and workspace identity").
4. If another live workspace or agent run owns the port, stop that owner through its managed control path and verify the port stays free. Do this before restarting the target; otherwise the owner can respawn and reclaim the port.
5. Stop any managed instance of the target runtime service.
6. Reseed the worktree with a full clone from the primary instance when there is any doubt about database completeness.
7. Start the runtime service through the managed runtime API.
8. Verify health, bootstrap state, login readiness, populated data, runtime visibility, and port-owner identity from both the main control plane and served workspace app.
9. If a verification item fails, diagnose that exact failure and loop back to the narrowest repair step.
10. Comment on the issue and set the final disposition.

Managed start / stop

Use the runtime-service endpoints on the main control plane. Include X-Paperclip-Run-Id so the mutation is associated with the current heartbeat.

curl -sS -X POST \
  "$PAPERCLIP_API_URL/api/execution-workspaces/$EXECUTION_WORKSPACE_ID/runtime-services/stop" \
  -H "Authorization: Bearer $PAPERCLIP_API_KEY" \
  -H "X-Paperclip-Run-Id: $PAPERCLIP_RUN_ID" \
  -H "Content-Type: application/json" \
  --data-binary '{"workspaceCommandId":"service:paperclip-dev"}'

curl -sS -X POST \
  "$PAPERCLIP_API_URL/api/execution-workspaces/$EXECUTION_WORKSPACE_ID/runtime-services/start" \
  -H "Authorization: Bearer $PAPERCLIP_API_KEY" \
  -H "X-Paperclip-Run-Id: $PAPERCLIP_RUN_ID" \
  -H "Content-Type: application/json" \
  --data-binary '{"workspaceCommandId":"service:paperclip-dev"}'

If the API returns an existing service, treat that as a candidate only. Verify its real /api/health and its port-owner identity before trusting it.

Port conflicts and workspace identity

service:paperclip-dev pins an explicit port in service config, so every workspace that realizes that service claims the same port. On start, service adoption will adopt whatever process already owns the port when its command line looks similar — and it records the requested cwd, not the owner's real cwd. Two consequences:

• multiple workspaces can hold running rows for the same port while only one process exists; whichever workspace URL you open, you get that one process
• the runtime row's cwd, pid, and health can all look correct while the URL actually serves a sibling worktree's app

Identity check — run this before trusting any existing service and again after every start or restart:

pid=$(lsof -nP -iTCP:"$SERVICE_PORT" -sTCP:LISTEN -t | head -1)
readlink "/proc/$pid/cwd"

The resolved cwd must be inside the target worktree (the dev server usually runs in <worktree>/server). Only /proc/<pid>/cwd is authoritative. Do not accept the runtime row's cwd field, a root 200, or a healthy /api/health as proof of identity — a squatting sibling passes all three.

If the port owner is a sibling workspace's process (port squat):

1. Decide which workspace should own the pinned port right now — normally the workspace named in the issue.
2. Identify the current owner's workspace and whether a live agent run or managed runtime service is responsible for keeping it alive. Use the real /proc/<pid>/cwd, the sibling execution-workspace records, and the owning issue/run state; do not infer ownership from the target runtime row.
3. Stop the owner first through its own managed control path: stop the sibling workspace's runtime service, or stop/cancel the live run that supervises and respawns it. Do not kill the sibling process directly while its supervisor is still active, because it will simply come back and reclaim the port.
4. Verify the port remains unbound for a stability wait. If it is immediately rebound, find and stop the still-live supervisor instead of restarting the target in a loop.
5. Only after the conflicting owner is stopped, start or restart the target service and re-run the identity check:

curl -sS -X POST \
  "$PAPERCLIP_API_URL/api/execution-workspaces/$EXECUTION_WORKSPACE_ID/runtime-services/restart" \
  -H "Authorization: Bearer $PAPERCLIP_API_KEY" \
  -H "X-Paperclip-Run-Id: $PAPERCLIP_RUN_ID" \
  -H "Content-Type: application/json" \
  --data-binary '{"workspaceCommandId":"service:paperclip-dev"}'

6. A restart only holds if the conflicting owner stays stopped. If the port is reclaimed, stop and diagnose the owner/supervisor again; do not loop target restarts against it.
7. If two workspaces genuinely need to run at the same time, they cannot share a pinned port. Stop the one that is not needed (through its own workspace's managed stop, or by ending the run that keeps respawning it), say in your issue comment which workspace now owns the port, and escalate the pinned-port collision as a product issue instead of looping restarts.

Before declaring success, also fetch the sibling workspaces you suspect and compare their runtimeServices[].port values: more than one running row claiming the target port across workspaces is always a conflict to resolve, not a state to work around.

Full database reseed

Use a full reseed when the app says setup is incomplete, login works but data is missing, the cloned app does not have the expected companies/issues/agents, or the user explicitly asks for the normal isolated-workspace database.

npx paperclipai worktree reseed --from-instance default --seed-mode full --yes

After reseed, restart through the managed runtime path. A reseed can copy runtime-service rows whose ids no longer match the local process registry, so runtime adoption and reconciliation must be verified after the start.

Do not consider the reseed complete until the served app has both auth and populated product data. A one-off inserted user/account row is a diagnostic clue, not the final state.

Core verification probes

Set SERVICE_URL to the service URL returned by the runtime API.

curl -sS "$SERVICE_URL/api/health" | jq
curl -sS -I "$SERVICE_URL/" | head

Expected health:

• status: "ok"
• bootstrapStatus: "ready"
• bootstrapInviteActive: false

Failures to reject:

• bootstrap_pending: the instance will show the first-admin setup gate
• database_unreachable: a web process is listening but its database is dead
• a root 200 with unhealthy /api/health: stale process adoption bug or a dead embedded database behind a live Node process

Check the port owner when a process is already listening:

lsof -nP -iTCP:"$SERVICE_PORT" -sTCP:LISTEN || true

Use this only to identify and remove a stale matching Paperclip dev-runner process after managed stop fails. Do not kill unrelated processes.

Verify main control-plane runtime state

Read the execution workspace from the main API and inspect the runtime service record.

curl -sS \
  "$PAPERCLIP_API_URL/api/execution-workspaces/$EXECUTION_WORKSPACE_ID" \
  -H "Authorization: Bearer $PAPERCLIP_API_KEY" | jq

The target service should show:

• matching workspace command id, usually service:paperclip-dev
• status: "running"
• healthy health fields, if present
• the same URL you are probing
• a local provider reference that maps to the running port owner

If the main app says running but /api/health is bad, stop and replace the stale process through the managed runtime.

Verify served workspace runtime state

The cloned Paperclip app must also know about the service. Query the same execution workspace through the served app when agent auth is available there:

curl -sS \
  "$SERVICE_URL/api/execution-workspaces/$EXECUTION_WORKSPACE_ID" \
  -H "Authorization: Bearer $PAPERCLIP_API_KEY" | jq

The served app should agree that the service is running / healthy at the same URL. If the main control plane and served app disagree after a reseed, the cloned database may contain copied runtime-service ids that do not match the local process registry. Use the normal start/adoption path again and verify both sides. If code changed in this area, add a focused regression test.