Memory Load Check

Memory Load Check

Use this skill when a PR or diff could load unbounded data into a Node/Bun process, especially in cron routes, background tasks, API routes, workflow execution, file parsing, cleanup jobs, migrations, import/export flows, and external API integrations.

Review Goal

Prove each changed path has explicit bounds for:

• rows held in memory
• bytes held in memory
• concurrent promises, DB queries, HTTP calls, storage operations, and jobs
• number of pages, batches, chunks, retries, and retained intermediate objects

If any bound depends only on current production size or "probably small" data, treat it as a finding.

References

Read these when doing a deeper pass:

• Node.js streams/backpressure: https://nodejs.org/learn/modules/backpressuring-in-streams
• Node.js stream usage: https://nodejs.org/en/learn/modules/how-to-use-streams
• Keyset/cursor pagination over offset scans: https://blog.sequinstream.com/keyset-cursors-not-offsets-for-postgres-pagination/
• Postgres pagination tradeoffs: https://www.citusdata.com/blog/2016/03/30/five-ways-to-paginate/

Sim Helpers To Prefer

• apps/sim/lib/cleanup/batch-delete.ts
  • chunkedBatchDelete: bounded SELECT -> optional side effect -> DELETE loop.
  • batchDeleteByWorkspaceAndTimestamp: common workspace/timestamp cleanup wrapper.
  • selectRowsByIdChunks: chunks large ID sets and enforces an overall row cap.
• chunkArray from @sim/utils/helpers: use only after the input set itself is already bounded.
• apps/sim/lib/core/utils/stream-limits.ts
  • PayloadSizeLimitError
  • assertKnownSizeWithinLimit
  • assertContentLengthWithinLimit
  • readStreamToBufferWithLimit
  • readNodeStreamToBufferWithLimit
  • readResponseToBufferWithLimit
  • readResponseTextWithLimit
• Cleanup dispatcher pattern in apps/sim/lib/billing/cleanup-dispatcher.ts
  • page active workspaces with WHERE id > afterId ORDER BY id LIMIT N
  • dispatch concrete chunks (workspaceIds, retention, label) instead of one giant scope
  • prefer Trigger.dev queue/concurrency keys when available
  • execute inline fallback chunks sequentially, not with unbounded Promise.all
• File parse pattern in apps/sim/lib/internal/file/parser.ts and apps/sim/lib/uploads/utils/fetch-external-url.server.ts
  • cap downloads and parsed output separately
  • preserve partial results when a later item exceeds the cap
  • never read untrusted response bodies without a byte cap
• KB connector file downloads in apps/sim/connectors/utils.ts
  • CONNECTOR_MAX_FILE_BYTES: shared per-file cap (aligned with the manual KB upload limit)
  • readBodyWithLimit: stream a download body to a Buffer with a hard byte cap (null on overflow)
  • stubOrSkipBySize: listing-time skip when the reported size exceeds the cap
  • markSkipped / sizeLimitSkipReason: surface oversized files as failed (skipped) KB rows
  • ConnectorFileTooLargeError: thrown mid-download when the listing under-reported size
• Large workflow value payloads
  • prefer durable references/manifests over inlining large arrays or files
  • materialize refs only behind an explicit byte budget

KB Connector File Size Handling

The connector size pattern in apps/sim/connectors/utils.ts (CONNECTOR_MAX_FILE_BYTES + readBodyWithLimit + stubOrSkipBySize/markSkipped) exists for one risk: a knowledge-base connector downloading arbitrary, user-controlled file bytes that the source does not hard-cap. Apply it by that risk, not by the connector's name.

Use the pattern when the connector downloads file content via a stream/download_url where the user controls the size:

• file-storage connectors: Dropbox, OneDrive, SharePoint, Google Drive, S3, GitHub, GitLab, Azure DevOps
• any connector that fetches a file via a download URL even if it is not a "storage" service (e.g. the Zoom transcript .vtt)

For those, require all three:

• stream the body with readBodyWithLimit(resp, CONNECTOR_MAX_FILE_BYTES) — never raw response.text()/response.arrayBuffer()
• skip oversize at listing (stubOrSkipBySize with the reported size) and again at fetch time (overflow -> markSkipped), since the listing size can be missing or under-reported
• never drop/truncate silently — oversized files become content-less failed rows carrying skippedReason, so they stay visible in the KB UI instead of vanishing from the index

Skip the pattern when the source already bounds the payload:

• pure API/structured-data connectors (Jira, Linear, Sentry, Slack, Zendesk, Gmail, ...) — paginated JSON/text; apply normal pagination + concurrency bounds instead of a per-file byte cap
• native-document connectors whose platform caps each document — a 100 MB cap can never fire there

Some connectors also budget the response body (google-docs MAX_DOCS_RESPONSE_BYTES, google-sheets MAX_CONTENT_BYTES, a remaining-bytes budget in notion); Confluence attachments use the full file pattern. Follow the connector's existing approach rather than adding a cap to every response.json().

Litmus test: "Can a user make this one fetch arbitrarily large, with nothing upstream stopping it?" Yes -> use the pattern. No (platform hard-cap, or already paginated) -> a per-file byte cap adds noise, not safety. Borderline: a user-configured/self-hosted endpoint with no platform cap (e.g. Obsidian) — bound it only if the content is genuinely unbounded.