fix: surface the wedged thread's stack in doctor and boot replay - #2490
Conversation
… idle workers faulthandler.dump_traceback_later writes threads newest-first, putting the main thread (where the gateway's asyncio loop wedged) LAST in a loop-stall dump, behind dozens of idle thread-pool workers parked in Queue.get. Both consumers read the dump top-down: - kirocrew doctor printed the FIRST thread under 'MainThread stuck at:' -- always an idle worker, actively misleading whoever diagnoses the stall. - The boot-time journal replay (120-line/8KB caps) truncated before ever reaching the wedged stack on real dumps. Fix: parse per-thread blocks, pick the wedged thread (explicit 'Current thread' marker if present, else the last block), surface it first in both paths. Falls back to old top-down behavior for unrecognizable content.
9cf4da5 to
5c4c72d
Compare
Design Review (Fable 5, fork) — ✅ PASSAdvisory design-level review of The patch is small and self-contained; I've verified the base file context (parsing lives in Design-Verdict: PASS Real diagnostic harm, fixed at the right layer: one shared parser in Suggestions
[DESIGN-REVIEWED] 5c4c72d |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed Review detailsThis is a diagnostic-display change: reorders/selects which thread's stack The header skip ( No findings. [OPUS-REVIEWED] 5c4c72d |
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
… idle workers (kirodotdev#2490) faulthandler.dump_traceback_later writes threads newest-first, putting the main thread (where the gateway's asyncio loop wedged) LAST in a loop-stall dump, behind dozens of idle thread-pool workers parked in Queue.get. Both consumers read the dump top-down: - kirocrew doctor printed the FIRST thread under 'MainThread stuck at:' -- always an idle worker, actively misleading whoever diagnoses the stall. - The boot-time journal replay (120-line/8KB caps) truncated before ever reaching the wedged stack on real dumps. Fix: parse per-thread blocks, pick the wedged thread (explicit 'Current thread' marker if present, else the last block), surface it first in both paths. Falls back to old top-down behavior for unrecognizable content.
Problem
A loop-stall crash dump exists to answer one question: where did the event loop wedge? Both consumers of the dump currently answer it wrong.
faulthandler.dump_traceback_later(armed by the loop watchdog) writes threads newest-first, so the MAIN thread — where the gateway's asyncio loop runs — lands last in the dump, behind dozens of idle thread-pool workers parked inQueue.get. Both readers scan top-down:kirocrew doctorprints the first thread in the file under the labelMainThread stuck at:— on every real dump that is an idle worker (concurrent/futures/thread.py:90 in _worker), actively misleading whoever is diagnosing the stall.Replaying prior crash dump stacks, capped at 120 lines / 8KB) truncates before ever reaching the wedged stack. A gateway with a saturated default executor produces 200+ lines of idle workers first.Evidence (from a real stall, sanitized)
A gateway stalled ≥25s and hard-exited. Doctor reported:
The journal replay of the same dump ended in
[truncated — full dump at above path]while still printing idle workers. The actual wedged stack — sitting unread at the bottom of the dump — was:Fix
In
crash_dump_store.py, parse the dump into per-thread blocks and pick the wedged thread: a block explicitly markedCurrent threadwhen present (synchronous Python-initiated dumps mark the dumping thread), otherwise the last block (CPython dumps threads newest-first; the main thread is created first, so it is written last — anddump_traceback_laterfires from an internal C thread, so no marker exists in watchdog dumps).dump_first_stack_lines(doctor) now returns the preamble (Timeout (…)!) + the wedged thread's top frames. Doctor shows 8 lines so the first project frame past the asyncio plumbing is visible.dump_replay_lines(boot journal replay) now replays the wedged thread first, then the remaining threads until the caps — the one stack that explains the stall always survives truncation.Tests
Current threadmarker preference, fallback on unrecognizable content, and replay-truncation survival (40 idle workers, 12-line cap — wedged frames still present).test_crash_dump_store.py: 46 passed.test_loop_watchdog.py: 13 passed. Doctor tests: 3 passed.