Fd Audit

Resource leak audit — descriptors and memory

OpenAlgo runs production as a single long-lived Gunicorn worker (--worker-class eventlet -w 1). It never restarts between deploys, so anything leaked once per request accumulates until the process dies. There is no second worker to absorb the failure and no natural recycling point.

Two failure modes, same root cause — unbounded growth in a process that never restarts:

Audit the change you just made, not the whole repo.

Step 1 — scope

If the change touches none of these, the audit is done. Say so and move on.

Descriptor-holding: SQLAlchemy engines/sessions · DuckDB connections · HTTP clients · WebSockets · ZeroMQ sockets · subprocesses · files · raw sockets · threads and executors · inotify/selectors

Memory-holding: module-level dicts, lists and sets · caches · event-bus subscriptions · SocketIO rooms · registries keyed by symbol/user/strategy · retained DataFrames · closures capturing large objects

Step 2 — descriptor conventions

SQLite engines. Only via database.engine_factory.create_db_engine(), which applies NullPool. Never create_engine() directly, never StaticPool — a shared connection has its cursor state corrupted under concurrency, producing "bad parameter or other API misuse" and "cannot commit - SQL statements in progress".

DuckDB is separate. database/historify_db.py calls duckdb.connect() directly — it does not go through engine_factory and NullPool does not apply. Each connection is an FD plus a memory arena. Use a context manager or guarantee .close(); a DuckDB connection left open also holds its buffer pool.

Sessions. Every scoped_session is either registered in the app.py teardown_appcontext handler or used as with db_session() as session:. A scoped_session created in a module and never .remove()d holds a connection per green thread forever. Existing cleanup layers to match: app.py teardown, traffic_logger.py logs_session.remove() in a finally, security_middleware.py for the banned-IP WSGI path, and teardown handlers in blueprints/traffic.py and blueprints/security.py.

HTTP. Use the shared utils/httpx_client.get_httpx_client(). A per-call httpx.Client() opens a fresh connection pool and leaks it unless closed; the shared client is what keeps HTTP/2 keep-alive to broker APIs working. Always pass an explicit timeout= — a hung request holds its socket indefinitely, which is a slow leak that looks like a hang.

WebSocket adapters. Close before reconnect. A reconnect path that opens a new socket without closing the old one leaks one descriptor per retry — and retries run unbounded during a broker outage, which is exactly when you cannot afford it.

ZeroMQ. Sockets closed on shutdown and adapter teardown; cleanup_zmq() in disconnect(). Never create a context per call. Read the SUB-binds/PUBs-connect invariant in CLAUDE.md before changing any bind/connect.

Subprocesses. Write to a log file, not PIPE, and .wait()-reap. An unreaped child leaves a zombie plus its pipe FDs; undrained PIPE output deadlocks the child once the buffer fills. Note telegram_bot_service's kaleido renderer spawns a real OS thread and an image-export subprocess — both must be joined/reaped on every path.

Threads and executors. Shared module-level singletons. Never a ThreadPoolExecutor per call or per request — each holds threads plus an internal control pipe until shut down. Under eventlet, threading.local() maps to green threads, so per-green-thread state accumulates with connection count, not with CPU count.

Files. with blocks. Temp files cleaned up via tempfile context managers.