fix(gateway): refuse to boot when the data home cannot persist state

[rubencu]

Problem

A gateway whose environment breaks basic file syscalls still binds its port and serves traffic — but every save silently fails. Chat-slot history writes, cron-history appends, and session-PID tracking all require tempfile.mkstemp (via atomic_write) plus an advisory lock (platform_compat.file_lock) in the data home.

Observed in the wild (Linux dev host, 2026-08-08): a gateway spawned from inside a sandboxed agent session inherits the sandbox's seccomp filter — seccomp survives fork/exec, nohup/disown included — so fcntl.flock and mkstemp raise OSError(ENOSYS) while writes to already-open fds keep working. The result over one 5-minute lifetime:

• every chat-slot save at shutdown failed (Failed to save slot ... to history × N — silent chat-history loss),
• cron-history appends crashed with [Errno 38] Function not implemented,
• a second gateway started from the same environment died with a raw traceback in cleanup_orphaned_sessions instead of anything actionable.

The process never exits on its own — it limps for its whole lifetime dropping writes.

Fix

• platform_compat.probe_file_persistence(directory): creates a probe file with mkstemp and locks it with file_lock — the exact two primitives the persistence paths use. Returns None on success or a human-readable failure description; when errno == ENOSYS, the message names the inherited-seccomp cause and the remedy (start from a regular shell or the system service). Probe file removed in all outcomes.
• GatewayOrchestrator.run() runs the probe against data_home() before the first lock consumer (cleanup_orphaned_sessions, where the raw-traceback crash happened). On failure: CRITICAL log + one-line stderr message + SystemExit(1).

Fail-loud over fail-silent, matching the repo's existing lock philosophy (file_lock already "FAILS CLOSED" on Windows rather than proceeding unserialized) and the pidfd ENOSYS probe precedent in cli.py.

Tests

test/test_persistence_probe.py (11 tests): healthy dir passes / leaves no litter / creates missing dirs; ENOSYS on create and on lock produce step-specific messages with the seccomp hint; non-ENOSYS errors get no misleading hint; probe file removed even when locking fails; orchestrator wiring — run() exits 1 on probe failure and does so before cleanup_orphaned_sessions runs.

Verified: new module green, test_no_config_dir_in_async ratchet green (probe uses data_home(), issue #1057), isort/flake8/mypy clean (mypy 1.14.1, CI-parity venv, 847 files). Full suite run: 38,395 passed; 69 failures reproduced identically on clean origin/main @ 6590a1c on the same host (git/network-dependent modules timing out in the sandboxed environment) — environmental, not from this diff.

[github-actions]

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed 5e0c688d51b07e88749a0c254534f4da4529b320 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 5e0c688

[github-actions]

Design Review (Fable 5, fork) — ✅ PASS

Advisory design-level review of 5e0c688d51b07e88749a0c254534f4da4529b320 via the fork AI-review pipeline — updated in place on each push; does not block merge.

Design assessment complete. The change is a boot-time persistence preflight (probe_file_persistence in platform_compat, wired into GatewayOrchestrator.run() before the first lock consumer) that converts a documented silent-data-loss failure mode into a loud, actionable refusal to start. The probe exercises exactly the primitives the real save paths use (mkstemp, write, fsync, file_lock, replace_with_retry, unlink), lives in the module that owns the lock shim, runs off-loop, and is trivially reversible with no contract or data migration surface. Tests pin both the probe semantics and the wiring order. I found no design-level concerns.

Design-Verdict: PASS

Real observed harm, guarded with the exact production primitives, at the right layer, before the first consumer — loud, scoped, and trivially reversible.

[DESIGN-REVIEWED] 5e0c688

[rubencu]

Disposition — GPT round 1 (SHA 3a359f7 → 03c431a):

• Event-loop blocking probe (BLOCKING) — ✅ fixed as suggested. The whole probe, including the data_home() resolution, now runs off-loop via await asyncio.to_thread(...) in GatewayOrchestrator.run(), matching the existing warm_backend off-loop pattern a few lines below. A stalled filesystem can no longer wedge the event loop during startup. Comment updated to state the off-loop rationale. All 11 probe tests, isort/flake8/mypy, and the brand gate pass locally.

[github-actions]

Opus 5 Review (fork) — ✅ no blocking findings

Reviewed 5e0c688d51b07e88749a0c254534f4da4529b320 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 5e0c688

[rubencu]

Disposition — GPT round 2 (SHA 03c431a → b7860fc):

• Thread-exhaustion RuntimeError from asyncio.to_thread (BLOCKING) — ✅ fixed as suggested. The await asyncio.to_thread(...) call is now wrapped: RuntimeError (executor exhausted/shut down) is converted into a preflight-failure message and routed through the existing clean CRITICAL-log + SystemExit(1) exit instead of a raw traceback. Rationale in-code: a process that cannot spawn one worker thread at boot cannot run session pools either. Added test_thread_exhaustion_routes_through_clean_exit proving RuntimeError produces the same clean exit code 1. 11 probe tests + isort/flake8/mypy + brand gate green locally.

[bolichen97]

Approved for rc.3: green CI, targeted fix, reviewed for insider release cut.

[rubencu]

Disposition — GPT round 4 (SHA 375ec8f → 872a57c):

• Probe never tests writing or atomic replacement (BLOCKING) — ✅ fixed as suggested, and generalized. The probe now exercises the complete persistence contract in production order: mkstemp → os.write (a byte-quota/ENOSPC environment that allows empty files now fails) → file_lock → os.replace onto a sibling path (the exact commit primitive atomic_write uses; inlined rather than imported because atomic_write imports platform_compat) → os.unlink. Each step reports its own failure message. Added 3 tests (write-ENOSPC, replace-EPERM, replace-failure cleanup) for 15 total. isort/flake8/mypy + brand gate green locally.

This closes the enumeration: the probe now covers create, write, lock, atomic-replace, and delete — every filesystem primitive the persistence layer performs.

[rubencu]

Disposition — GPT round 5 (SHA 872a57c → 70b6694):

• Raw os.replace rejects healthy Windows data homes (BLOCKING) — ✅ fixed as suggested. The probe now commits through atomic_write.replace_with_retry — the exact production primitive — which absorbs the bounded Windows AV/indexer sharing-violation window and is a plain os.replace on POSIX. Imported lazily inside the probe because atomic_write imports platform_compat at module top (cycle avoidance, noted in-code). The probe runs off-loop via asyncio.to_thread, so the helper's retry path (gated on no running loop in the thread) is active exactly as it is for production writers. Added test_probe_uses_the_production_replace_primitive (spy asserts the probe routes through the helper) and repointed the two replace-failure tests at the helper. 17 tests + isort/flake8/mypy + brand gate green locally.

[rubencu]

Disposition — GPT round 6 (SHA 70b6694 → 5e0c688):

• Probe omits fsync (BLOCKING) — ✅ fixed as suggested. os.fsync(fd) now runs after the probe write (its own flush files in step), covering environments where buffered writes succeed but the flush fails with EIO/ENOSPC — matching chat history's atomic_write(..., fsync=True). Added test_fsync_failure_reports_flush_step. 17 tests + isort/flake8/mypy + brand gate green locally.

With this the probe covers create → write → fsync → lock → atomic-replace (via replace_with_retry) → delete — the complete set of I/O primitives any persistence path performs. There is no further filesystem operation in atomic_write or the lock helpers left unprobed.