feat: stagger gateway startup after a recent loopstall crash dump

[rubencu]

Cautious boot after a recent loop-stall crash

Problem

When a Kiro Crew gateway wedges under host memory pressure, the loop-stall watchdog dumps thread stacks and hard-exits so the service manager can restart it. The NEW instance already detects that dump at boot — it logs ⚠️ Prior loop-stall crash dump found: ... (0.0 hours ago) and notifies the user — and then launches its entire startup battery at once anyway: MCP gateway sidecar, cron scheduler (which immediately fires every overdue job), app backends, MCP server probes, and session restores, each of which spawns subprocesses.

On a host that is still under the same memory pressure that killed the previous instance, that simultaneous burst is exactly what re-wedges the new one. This was observed in practice: a gateway died under memory pressure, the restarted instance detected the minutes-old dump, launched everything at once, its own watchdog wedged within seconds, and the host froze shortly after. The gateway had the signal and did not act on it.

What changed

A new dashboard/cautious_boot.py module turns the existing detection into behavior:

• One decision, made once, off the event loop (initialize(), evaluated via asyncio.to_thread early in gateway startup): is there a prior loop-stall dump with real stack content younger than 30 minutes?
• Posture-scaled delay: the current resource posture (resource_status.probe) picks the pause length —
  • recent dump + tight/critical host → 10s between battery groups (maximum caution)
  • recent dump + ample host → 2s between groups (mild stagger; an unknown posture reading deliberately does not escalate)
• Five pause points turn the burst into a sequence of small groups: MCP gateway sidecar → cron scheduler (this also defers the post-restart cron catch-up burst out of the app/MCP launch window) → app backends → MCP server probes → session restores. Each pause is a plain await asyncio.sleep(...), so the event loop stays responsive and the loop-stall watchdog keeps getting its heartbeat throughout.
• Fail-open everywhere: an unreadable dump store, a config error, a failed posture probe, or worker-thread exhaustion all mean a normal, un-staggered boot. pause_before() without a prior initialize() (tests, embedded starts) is a no-op.
• Loud when active: 🐢 Cautious boot ACTIVE: ... is logged at WARNING with the dump name, its age, the posture, and the chosen delay, so an operator reading the journal after an incident sees the gateway both noticed the crash and changed its behavior.

One new config key, following the existing loader conventions (typed accessor with default, non-bool values fall back to the default):

• dashboard.cautious_boot (bool, default on) — the recency threshold (30 min) and the two delays are deliberately module constants, keeping the config surface to a single switch.

What is deliberately NOT here

• No scheduling machinery: no priority queues, no dependency graph, no adaptive back-off. The stagger is sequential groups with short sleeps — the minimal change that spreads the burst.
• No admission control: nothing is skipped or refused; every service still starts, just not simultaneously. Posture-gated admission of new work is a separate concern.
• No configurable thresholds/delays: one bool. If field experience shows the constants need tuning, that is a follow-up with evidence.
• No change to the watchdog or the crash-dump store: detection and dump lifecycle are untouched; this only consumes the existing signal.

Testing

New test/test_cautious_boot.py (19 tests) covering the decision matrix (recent dump × ample/tight/critical/unknown posture, old dump, no dump, header-only dump, config off, missing config attribute), fail-open paths (unreadable store, probe exception, worker-thread exhaustion), and the async plumbing (decision caching, loud activation log, pause_before no-op when uninitialized/inactive, correct sleep when active).

Also run green: crash-dump store and resource-status neighbors (65 tests), config schema/loader suites (275 tests), and the dashboard/API server startup suites (96 tests). isort, flake8, and mypy pass on all touched files.

[github-actions]

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed 611c6cea2a2937912536b625acb13841dd661376 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 611c6ce

[github-actions]

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed 611c6cea2a2937912536b625acb13841dd661376 via the fork AI-review pipeline; updated in place on each push.

Review details

No semantic defects found. The change is well-guarded: _evaluate fails open on every exception path, runs blocking I/O off-loop via asyncio.to_thread, pause_before uses asyncio.sleep (not a blocking syscall) and is a documented no-op when _decision is None (embedded/test callers), the config key defaults on and routes through _safe_bool, and the tests stay within tmp_path/monkeypatch with no host side-effects. Boot ordering (pauses before app backends, MCP probe, session restore, MCP gateway, cron) matches the documented intent, and the loop stays responsive throughout.

No findings.

[OPUS-REVIEWED] 611c6ce

[github-actions]

Design Review (Fable 5, fork) — ✅ PASS

Advisory design-level review of 611c6cea2a2937912536b625acb13841dd661376 via the fork AI-review pipeline — updated in place on each push; does not block merge.

Design-Verdict: PASS

A real, observed re-wedge loop mitigated with the minimal proportionate shape: one reversible bool, fail-open everywhere, no new scheduling machinery.

Suggestions

• The pauses only space each group's launch; fire-and-forget tails (cron catch-up sessions, _bg_mcp_probe handshakes) still overlap later groups — if field evidence shows the stagger under-delivers, awaiting the cron catch-up window before app backends is the cheapest next lever, not longer sleeps.

[DESIGN-REVIEWED] 611c6ce