Triage Sync Fix Repair

Triage a sync/fix repair

What the report means

app/sync/fix/index.js runs five checks in series against one blog's Redis state and the blog's local folder. If any check returns rows it bumps the blog's cacheID and returns the rows as its report. Fix() no longer emails on its own (the standalone "sync/fix repaired" email, SYNC_FIX_REPAIRED, is gone); callers report it:

• the hourly sweep digest, e.g. "Dropbox sync issue" (app/helper/email/admin/DROPBOX_SYNC_ISSUE.txt; Google Drive's, sent at minute 30 by app/clients/google-drive/validate.js, is GOOGLE_DRIVE_SYNC_ISSUE.txt): a blog's "Fix() repaired:" lines, one per check with the first 10 rows JSON.stringifyd. Formatting lives in app/clients/util/syncReport.js.
• the dashboard resync's "Resync found changes" email (RESYNC_FOUND_CHANGES.txt), when Fix() repaired something after the resync.
• iCloud: the hourly "iCloud sync issue" digest (ICLOUD_SYNC_ISSUE.txt, sent at minute 45 by app/clients/icloud/init.js) and the "iCloud resync found changes" email (ICLOUD_RESYNC_ISSUE.txt) that a macserver-requested resync sends from app/clients/icloud/routes/site/status.js. Both run the walk and Fix() from app/clients/icloud/sync/validateBlog.js, and the resync email carries the macserver's reason for requesting it.

Other callers (dashboard rebuild, local setup, template folders and the startup resync) only log Fix repaired <blog> check=N. Fix() doesn't hold the folder lock and repairs as it goes, so the report says what was wrong; it's already been changed by the time you read it.

Fix() is called from:

• Dropbox hourly validation, minute 0, green (app/clients/dropbox/init.js, blogs with last_sync in the past hour, after the resync)
• iCloud hourly validation, minute 45 (app/clients/icloud/init.js, blogs with lastSync in the past hour, after a clean walk), and the resync the macserver requests after a failed push (app/clients/icloud/routes/site/status.js), both through app/clients/icloud/sync/validateBlog.js
• Google Drive hourly validation, minute 30 (app/clients/google-drive/validate.js, blogs with lastSync in the past hour, after a clean walk)
• Local client setup (app/clients/local/setup.js)
• Dashboard "rebuild"/fix of a site (app/dashboard/site/client.js)
• Template folder installs (app/templates/folders/index.js)

The digest's client (its subject) tells you which hourly job probably ran it. Each check logs Fix: <blogID> <check> duration=Nms, so the email time gives you the run: grep 'Fix: <blogID>' to find it.

Fix() repairs often have benign, by-design causes. Work out which of those applies before you look for a bug.

Reading the report: check by check

Row shapes come straight from each check's report.push.

entry-ghosts (["MISSING", {id, path}], ["CASE", {oldPath, path}])

Walks every entry (Entries.each, the all list) and checks its file exists on disk (localPath), with case-insensitive fallbacks.

• MISSING: the entry isn't deleted but no file backs it, so it's Entry.dropped. That means a sync missed a delete/rename, or the local folder was lost or reset. Usually a real sync bug. Check whether the path was renamed or moved recently (grep the path in green's logs). Known cause (fixed): a whole local folder removed by a sync walk. The Google Drive, iCloud and Dropbox reset-to-blot walks fs.remove a folder that's gone remotely, and used to call update() on the folder path only. drop on a folder path drops nothing beneath it, so every entry inside stayed live until Fix() caught it. The walks now update each path inside (clients/util/localDescendants). If this recurs, look for Removing <parent folder> which does not exist remotely in the logs.
• CASE: the file exists with different casing. Entry path rewritten. Common after case-only renames on Dropbox/macOS. Mostly benign, but if you see it repeatedly for one blog, case handling is broken.

list-ghosts ([list, "MISSING", id], [list, "MISMATCH", id])

• MISSING rows come from Entries.pruneMissing (app/models/entries/index.js): a member of one of the lists all, created, entries, drafts, scheduled, pages, deleted, entries:lex has no blog:<id>:entry:<id> key. Known benign cause: expired deleted entries. Entry.drop doesn't delete an entry. It re-saves it with deleted: true, and models/entry/set.js puts a 24h TTL on that key, while _assign.js keeps the id in all (and adds it to deleted). Once the key expires, the id lingers in all/deleted until the next Fix() prunes it. Rows only in all and deleted for files the user deleted (or renamed away) more than 24h ago are this housekeeping, not corruption. Expect roughly one all row plus one deleted row per deleted file, so the count is about 2× the number of files. Only the first 10 rows are quoted, and all is listed first, so deleted rows are often cut off. Since Entries.sweepExpiredDeleted landed, every entry save on the blog and the start of list-ghosts silently clear these ids using the deleted scores, so expired deleted entries should no longer appear in the email. If MISSING rows in all/deleted turn up after that deploy, a deleted entry's key disappeared before its 24h was up. That needs investigating. It becomes suspicious if a MISSING id is in entries, drafts, pages, scheduled or created, because live lists should never point at an absent key.
• MISMATCH: the key exists but the JSON's id ≠ the list member (an entry stored under the wrong key). It's re-saved under its real id and the stale member is removed. Real corruption. Find out how the entry got there (rename handling, path normalization).

tag-ghosts (["EMPTY TAG", tag], ["MISSING", id], ["MISMATCH", staleId, realId])

• EMPTY TAG: tag set with no members, deleted. Benign leftover.
• MISSING: a tag points at an entry key that doesn't exist. This is usually the same TTL expiry as above, when the deleted entry's tags weren't cleared. Check it against the list-ghosts rows.
• MISMATCH: same as list-ghosts MISMATCH. Real corruption.

menu-ghosts (["Delete", item], ["Delete duplicate", item], ["Changed label/metadata/URL of", item])

Syncs blog.menu page links with their entries. Delete = page entry now deleted; Changed … = page title/URL changed. These are normal side-effects of editing pages, unless the same item flips back and forth every run. The row holds a reference to the menu item, which is mutated after the push, so every row shows the post-repair values (a "Changed label of" row already carries the new label and metadata). Known benign cause: demo template folders (client: empty, handles from app/templates/folders/config.js). On every app boot setupBlogs.js resets the blog's menu to the bare config menu (no metadata, config labels), then folders/index.js calls Fix(), which copies titles/metadata back from the page entries. This sent the email after every deploy until setupBlogs started keeping the entry-derived label/url/metadata from the existing menu; if it recurs, check that merge.

entries-path-index (["MISMATCH", {entries, pathIndex}], ["BACKFILLED", n])

entries:lex cardinality ≠ entries, rebuilt. A small diff can be a race with a sync writing entries mid-check. A persistent or large diff means a code path updates entries without pathIndex.

Method

1. Read the incident log below first. If you've seen the pattern before, say so and keep it short.
2. Classify from the email alone where you can. Group the rows by check and reason. Look at the file names and list names. Deleted-looking drafts or renamed duplicates (… 1.md, Untitled.md) that only appear in all point to TTL housekeeping.
3. Confirm on production if the email isn't conclusive, or to settle a first-of-its-kind case. Ask the user before every production command, even read-only ones, and stick to read-only commands. Host blot, containers blot-container-{blue,green,yellow}, helpers in remote ~/.bashrc (see investigate-production-container-restarts). Use docker logs blot-container-green --timestamps 2>&1 | … so you can window by time. Logs disappear when a deploy recreates the container.
   • Find the Fix() run: grep 'Fix: <blogID>' (per-check duration= lines).
   • Find what happened to a reported path earlier: grep '<blog12>' | grep -F '<path>'. models/entry/set.js logs <blog12> delete <path> and <blog12> update <path>. A delete >24h before the Fix() run confirms TTL expiry. If a reported path doesn't appear at all, that still points to TTL expiry when the logs cover the 24h or so before the run. Sync lines (sync_…, Removing, Downloading) show renames and moves.
   • Read-only Redis check, if needed (ask first): TTL/EXISTS blog:<id>:entry:<path>, ZSCORE blog:<id>:<list> <path> to see which lists still hold an id.
4. Classify as one of:
   • Expected housekeeping: TTL-expired deleted entries, empty tags, menu label or URL updates after page edits. No bug. Consider whether the email should be filtered (see follow-ups).
   • Race with a live edit: the user changed files while Fix() ran. Converges on the next run.
   • Sync bug: entry-ghosts MISSING/CASE repeatedly, or a missed delete or rename. Cross-check with investigate-dropbox-sync-issue for Dropbox blogs.
   • Data-model bug: any MISMATCH, live-list MISSING, or persistent path-index drift. Find the write path that produced it.
5. Report to the user in chat (blog and path details are fine here): the checks, the cause, the evidence and whether code needs to change. Don't change code unless asked. If a fix is warranted, add a compact line to the root TODO or note it for a PR.
6. Append an entry to the Incident log (newest last), following the privacy rules. Keep it to about 5 lines. If you learned a better query or a new benign cause, update the check notes above as well.