close
Skip to content

fix(terminal): stop scanning PTY output on the way to the browser - #2152

Merged
buluoray merged 1 commit into
mainfrom
fix/terminal-output-redaction
Aug 10, 2026
Merged

fix(terminal): stop scanning PTY output on the way to the browser#2152
buluoray merged 1 commit into
mainfrom
fix/terminal-output-redaction

Conversation

@buluoray

@buluoray buluoray commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Problem

The web terminal scanned every 4096-byte PTY read for credentials on its way to
the browser. That scan protected nothing, corrupted output, and hid secrets the
user had deliberately printed.

It protected nothing. The panel renders into the authenticated operator's own
browser, showing bytes their own shell just wrote a moment earlier. Any threat
that can read it can read the terminal application sitting beside it — and the
scrollback ring buffer is replayed only to that same browser. The one path by
which terminal output reaches a model is the selection hand-off, and
POST /api/terminal/redact already scans that.

It corrupted output. Scanning required decoding, and a PTY read ends wherever
the kernel had bytes, so a multi-byte character is routinely split across two
reads. Decoding each read independently turned both halves into U+FFFD,
permanently destroying CJK, emoji and a TUI's box-drawing glyphs — the client
cannot recover the original bytes from a replacement character.

It hid secrets printed on purpose. Observed false positives: the output of
gh auth token, an entire device-code login URL, a presigned S3 URL, and npm
integrity sha512-… lines.

Why it matters

A terminal that silently rewrites what your shell printed is not a terminal. The
failure is worst exactly where the user needs the output most — copying a token
they just minted, following a device-code URL, reading a TUI in a non-Latin
locale — and it is invisible: [REDACTED: …] looks deliberate, and a U+FFFD
looks like a font problem rather than data loss.

Fix

Forward each read byte-for-byte and delete the scan.

  • The read loop sends data unchanged. Reassembly moves to xterm.js, which runs
    its own incremental decoder, so no read boundary can corrupt a character and
    the server holds no decoder state that could desynchronize from the client's.
  • Reconnect replay is the same byte copy (bytes(sess.scrollback)). A truncated
    ring buffer can begin mid-character; the client renders that head as it finds
    it, the lead byte being genuinely gone.
  • POST /api/terminal/redact is unchanged and remains unconditional. It is now
    documented as the whole credential boundary for the web terminal, which it
    already was in practice — and a contiguous selection is the only input the
    regex redactors can be accurate on, since a secret split across two reads is
    invisible to a per-chunk scan by construction.
  • Separately: the read loop now captures the WebSocket into a local and
    revalidates it after taking the send lock. sess.ws is set to None by the WS
    handler on disconnect, and AttributeError is not caught by the loop's
    except OSError, so dereferencing it after a suspension point killed the
    reader task and stopped PTY draining and scrollback capture for a session the
    client could still reconnect to.

README.md's claim that Kiro Crew "redacts credential patterns from output
before it reaches a chat surface" stays accurate: that is the selection hand-off,
which is untouched.

Tests

Each test was verified to fail with the per-chunk scan restored (a temporary
patch reinstating main's _redact_terminal call in the read loop), then to
pass once removed.

  • test_multibyte_output_survives_read_boundaries — drives the real PTY loop
    with an unbroken 9 KB run of 3-byte characters and asserts no U+FFFD arrives.
    Falsification: with the scan restored it fails on
    a read boundary corrupted a character. The payload is deliberately one line
    with no newline in it: an earlier version of this test used many short lines
    and passed against the corrupting code, because a shell that writes line by
    line hands the reader whole lines and every read then lands on a character
    boundary by accident. A single run of 3-byte characters longer than one
    4096-byte read cannot be split cleanly, since 4096 is not a multiple of 3.
  • test_stream_and_replay_forward_output_verbatim — an AKIA key printed by the
    shell arrives intact, and so does the scrollback replayed on reconnect. The
    literal is split in the typed line ('AKIAIOSFODNN7''EXAMPLE') so the shell's
    echo of the command cannot satisfy the assertion; only the printf's real output
    joins the halves. Falsification: with the scan restored the needle never
    arrives and the assertion shows the redacted echo.
  • test_redactors_are_called_only_by_the_selection_handler — source guard: each
    redactor has exactly one call site, inside api_terminal_redact's _scan.
    Falsification: with the scan restored, assert 2 == 1.
  • test_no_send_through_the_session_field_after_an_await — source guard for the
    capture-and-revalidate fix; sess.ws.send must not appear in the module.
  • test_session_reservation_has_no_await — source guard pinning the
    max-sessions-check → placeholder-assignment region await-free, so two
    concurrent opens for one session id cannot both spawn a PTY.

test_terminal_handler.py: 217 passed, 1 skipped. Full backend suite: 40679
passed; the 93 failures are pre-existing on this host (120 s timeouts in
test_worktree_create, test_cli_manifest_signature, the ops-mission-control
git suites, and sandbox-unavailable failures in test_terminal_commands's probe
tier and auto_improvement) — none of the 13 affected files import
dashboard.handlers.terminal. flake8, isort, mypy (866 files) and
scripts/docs-lint.sh are clean.

Note for reviewers who saw the earlier revision

This PR previously kept the scan on by default and added a
dashboard.terminal.redact_output opt-out. That was the wrong shape: the switch
was undiscoverable (there is no Settings → Terminal surface, and
kirocrew config set cannot even create a key that does not already exist), and
it required latching the mode per session because the two modes disagree about
who holds a partial character — complexity that existed only to preserve a scan
with no threat model behind it. Removing the scan removes the configuration, the
latch, and that entire class of bug.

@buluoray
buluoray requested a review from a team as a code owner August 8, 2026 00:47
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Aug 8, 2026
@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Advisory design-level review of a5c91a9695915af8a1f7ea2cfaefa786580b70f4 — updated in place on each push; does not block merge.

Design-Verdict: PASS

Removes a scan with no threat behind it at the root cause, and correctly relocates the credential boundary to the one path output reaches a model.

The corruption (per-read decode of split multi-byte characters) and false positives are inherent to per-chunk scanning — the description's own point that a straddled secret is "invisible to a per-chunk scan by construction" means the deleted layer was never a reliable backstop, so consolidating on the unconditional selection hand-off loses nothing real. The rejected earlier shape (opt-out config + per-session mode latch) was the right thing to kill; this version is simpler, reversible, and one-way-door-free. The bundled sess.ws capture-and-revalidate fix touches the same send site and is disclosed, so it belongs here. Docs and spec move in the same commit as required.

[DESIGN-REVIEWED] a5c91a9

@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of a5c91a9695915af8a1f7ea2cfaefa786580b70f4 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] a5c91a9

False positive or not applicable? A repository writer can comment:
/ai-review override gpt a5c91a9695915af8a1f7ea2cfaefa786580b70f4: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Aug 8, 2026
@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed a5c91a9695915af8a1f7ea2cfaefa786580b70f4 — this comment is updated in place on each push.

Review details

I've verified the diff, the base-branch backend-security-controls rule (blocking: true, matches src/kiro_crew/**/*.py), and the candidate's evidence directly.

Falsifying Candidate 1:

  • The deleted helper and its former call sites match the diff (evidence grounded).
  • (a) input: a credential/exfil URL written to the PTY — occurs in practice.
  • (b) path: read_ptylive.send_bytes(data) and reconnect → ws.send_bytes(bytes(existing.scrollback)), both unscanned — holds.
  • (c) observable wrong outcome: this is where it dies. The per-session WebSocket reaches only the one authenticated operator's own browser (not a broadcast, not a multi-party surface), rendering bytes that operator's own shell just wrote — bytes they already see in their real terminal. Under the repo's stated single-user, same-UID trust boundary there is no privilege crossing and thus no re-derivable wrong outcome. The rule is anchored to "Never trust LLM output"; terminal output is the operator's own shell, and the one path where that output actually crosses into a model (/api/terminal/redact) retains both redact_credentials() and redact_exfiltration_urls() — a compensating guard present in the same diff and pinned by a source test. The candidate's own confidence is "low" and concedes the removal is defensible under this trust model.

No observable wrong outcome survives; the model-facing boundary is still scanned. The candidate does not reach 80.

No findings.

[OPUS-REVIEWED] a5c91a9

Verdict parsed from the review's SHA-scoped output markers for commit a5c91a9695915af8a1f7ea2cfaefa786580b70f4.

False positive or not applicable? A repository writer can comment:
/ai-review override fable a5c91a9695915af8a1f7ea2cfaefa786580b70f4: <one-sentence reason>

@buluoray
buluoray force-pushed the fix/terminal-output-redaction branch from 6185303 to fd52982 Compare August 8, 2026 00:59
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Aug 8, 2026
@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

UX Review (Fable 5) — ⏭️ skipped

Revision a5c91a9695915af8a1f7ea2cfaefa786580b70f4 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@buluoray
buluoray force-pushed the fix/terminal-output-redaction branch from fd52982 to 312d5b3 Compare August 8, 2026 01:09
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Aug 8, 2026
@buluoray
buluoray force-pushed the fix/terminal-output-redaction branch from 312d5b3 to f49dd0a Compare August 8, 2026 01:30
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Aug 8, 2026
@buluoray
buluoray force-pushed the fix/terminal-output-redaction branch from f49dd0a to c59fc92 Compare August 8, 2026 01:52
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Aug 8, 2026
@buluoray
buluoray force-pushed the fix/terminal-output-redaction branch from c59fc92 to a09175f Compare August 8, 2026 02:39
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Aug 8, 2026
CrysisDeu pushed a commit that referenced this pull request Aug 9, 2026
The Opus lane had a 2.7% blocking rate against GPT's 18% across 73 PRs, and
63 of 73 posted comments contained literally "No findings." A controlled
experiment on this repo found the cause is not the model and not the
architecture -- it is three clauses in the prompt, each independently
sufficient to silence a defect the same model reports 3/3 times without them:
the closed-list reading of the residual defect classes, the certainty
threshold, and "drop the finding if the fix touches untouched code".

Splitting the call while keeping those clauses in the first half did NOT help:
the discovery pass produced zero candidates, so the filter had nothing to
keep. Precision enforcement has to sit downstream of discovery, which is also
what Anthropic's own code-review plugin does (parallel discovery agents, then
a per-candidate validation agent with a confidence floor).

What changes:

* Stage 1 (discovery) generates candidates with generous recall and no
  precision gates. Its output is never posted and gates nothing.
* Stage 2 (validation) is an independent call that re-derives input / call
  path / observable outcome for every candidate from code it opens itself,
  keeps only those it scores >= 80, and only then applies the closed blocking
  list. It keeps `id: review`, so the existing transcript capture, comment
  upsert and fail-closed gate are unchanged.
* The prompts move out of the YAML into .github/review-prompts/ and are shared
  by the same-repo and fork lanes, which previously carried near-duplicate
  copies. They are materialised from the BASE commit, so a PR can neither
  weaken the rules that govern it nor rewrite the prompt that reviews it; a
  missing prompt fails the job rather than degrading into an unspecified
  review that could look clean.
* Candidates cross the stage boundary as a workspace FILE, never string
  interpolation -- model output must not reach YAML or a shell argument, and a
  file has no arg-length ceiling.
* The fix-scope rule changes behaviour instead of being deleted: a finding
  whose only remedy lies outside the changed lines is reported as advisory
  rather than dropped, because the author cannot land the remedy here but the
  signal is still real. A regression the diff itself introduces still blocks,
  since reverting the hunk is an in-diff fix.

Job name, required check, tool surface and the marker contract are unchanged.
The fork lane keeps its no-shell posture (its diff is pre-fetched from
GitHub's compare endpoint).

Measured locally on the same corpus, running these exact prompt files:

* 8 negative-control runs (PRs where BOTH production lanes reviewed the same
  SHA and both emitted zero findings): 33 candidates in, 0 BLOCKING out, 7
  advisory out. The validator removed 79%. No run would have turned the
  required check red on a PR two reviewers passed.
* Positive controls (a GPT blocking the old Opus lane missed): recovered on
  #2109 (blocking, same guard GPT named, 2/2) and reported on #2169
  (advisory, 4/4, where the old lane said nothing). #2152's GPT finding was
  generated by discovery but dropped by the validator.

Three tests that asserted on prompt text follow it to its new home rather
than being deleted, and TestOpusTwoStageArchitecture locks the split in place
-- including a cross-check that the marker the gate greps equals the marker
the validation prompt is told to emit, because a typo either side fails every
PR closed and silently.

Known and deliberately not addressed here: the two stages are sequential, so
end-to-end latency roughly doubles under the same 90-minute runaway backstop.
That number can only be measured on a real CI run.

Stacked on the prompt-file bootstrap PR. Because the extraction reads the base
commit, the prompts have to exist on main before this can run -- and for the
same reason, a future PR that EDITS a prompt is still reviewed by the OLD one.
The workflow header documents that.
CrysisDeu pushed a commit that referenced this pull request Aug 10, 2026
The Opus lane had a 2.7% blocking rate against GPT's 18% across 73 PRs, and
63 of 73 posted comments contained literally "No findings." A controlled
experiment on this repo found the cause is not the model and not the
architecture -- it is three clauses in the prompt, each independently
sufficient to silence a defect the same model reports 3/3 times without them:
the closed-list reading of the residual defect classes, the certainty
threshold, and "drop the finding if the fix touches untouched code".

Splitting the call while keeping those clauses in the first half did NOT help:
the discovery pass produced zero candidates, so the filter had nothing to
keep. Precision enforcement has to sit downstream of discovery, which is also
what Anthropic's own code-review plugin does (parallel discovery agents, then
a per-candidate validation agent with a confidence floor).

What changes:

* Stage 1 (discovery) generates candidates with generous recall and no
  precision gates. Its output is never posted and gates nothing.
* Stage 2 (validation) is an independent call that re-derives input / call
  path / observable outcome for every candidate from code it opens itself,
  keeps only those it scores >= 80, and only then applies the closed blocking
  list. It keeps `id: review`, so the existing transcript capture, comment
  upsert and fail-closed gate are unchanged.
* The prompts move out of the YAML into .github/review-prompts/ and are shared
  by the same-repo and fork lanes, which previously carried near-duplicate
  copies. They are materialised from the BASE commit, so a PR can neither
  weaken the rules that govern it nor rewrite the prompt that reviews it; a
  missing prompt fails the job rather than degrading into an unspecified
  review that could look clean.
* Candidates cross the stage boundary as a workspace FILE, never string
  interpolation -- model output must not reach YAML or a shell argument, and a
  file has no arg-length ceiling.
* The fix-scope rule changes behaviour instead of being deleted: a finding
  whose only remedy lies outside the changed lines is reported as advisory
  rather than dropped, because the author cannot land the remedy here but the
  signal is still real. A regression the diff itself introduces still blocks,
  since reverting the hunk is an in-diff fix.

Job name, required check, tool surface and the marker contract are unchanged.
The fork lane keeps its no-shell posture (its diff is pre-fetched from
GitHub's compare endpoint).

Measured locally on the same corpus, running these exact prompt files:

* 8 negative-control runs (PRs where BOTH production lanes reviewed the same
  SHA and both emitted zero findings): 33 candidates in, 0 BLOCKING out, 7
  advisory out. The validator removed 79%. No run would have turned the
  required check red on a PR two reviewers passed.
* Positive controls (a GPT blocking the old Opus lane missed): recovered on
  #2109 (blocking, same guard GPT named, 2/2) and reported on #2169
  (advisory, 4/4, where the old lane said nothing). #2152's GPT finding was
  generated by discovery but dropped by the validator.

Three tests that asserted on prompt text follow it to its new home rather
than being deleted, and TestOpusTwoStageArchitecture locks the split in place
-- including a cross-check that the marker the gate greps equals the marker
the validation prompt is told to emit, because a typo either side fails every
PR closed and silently.

Known and deliberately not addressed here: the two stages are sequential, so
end-to-end latency roughly doubles under the same 90-minute runaway backstop.
That number can only be measured on a real CI run.

Stacked on the prompt-file bootstrap PR. Because the extraction reads the base
commit, the prompts have to exist on main before this can run -- and for the
same reason, a future PR that EDITS a prompt is still reviewed by the OLD one.
The workflow header documents that.
CrysisDeu pushed a commit that referenced this pull request Aug 10, 2026
The Opus lane had a 2.7% blocking rate against GPT's 18% across 73 PRs, and
63 of 73 posted comments contained literally "No findings." A controlled
experiment on this repo found the cause is not the model and not the
architecture -- it is three clauses in the prompt, each independently
sufficient to silence a defect the same model reports 3/3 times without them:
the closed-list reading of the residual defect classes, the certainty
threshold, and "drop the finding if the fix touches untouched code".

Splitting the call while keeping those clauses in the first half did NOT help:
the discovery pass produced zero candidates, so the filter had nothing to
keep. Precision enforcement has to sit downstream of discovery, which is also
what Anthropic's own code-review plugin does (parallel discovery agents, then
a per-candidate validation agent with a confidence floor).

What changes:

* Stage 1 (discovery) generates candidates with generous recall and no
  precision gates. Its output is never posted and gates nothing.
* Stage 2 (validation) is an independent call that re-derives input / call
  path / observable outcome for every candidate from code it opens itself,
  keeps only those it scores >= 80, and only then applies the closed blocking
  list. It keeps `id: review`, so the existing transcript capture, comment
  upsert and fail-closed gate are unchanged.
* The prompts move out of the YAML into .github/review-prompts/ and are shared
  by the same-repo and fork lanes, which previously carried near-duplicate
  copies. They are materialised from the BASE commit, so a PR can neither
  weaken the rules that govern it nor rewrite the prompt that reviews it; a
  missing prompt fails the job rather than degrading into an unspecified
  review that could look clean.
* Candidates cross the stage boundary as a workspace FILE, never string
  interpolation -- model output must not reach YAML or a shell argument, and a
  file has no arg-length ceiling.
* The fix-scope rule changes behaviour instead of being deleted: a finding
  whose only remedy lies outside the changed lines is reported as advisory
  rather than dropped, because the author cannot land the remedy here but the
  signal is still real. A regression the diff itself introduces still blocks,
  since reverting the hunk is an in-diff fix.

Job name, required check, tool surface and the marker contract are unchanged.
The fork lane keeps its no-shell posture (its diff is pre-fetched from
GitHub's compare endpoint).

Measured locally on the same corpus, running these exact prompt files:

* 8 negative-control runs (PRs where BOTH production lanes reviewed the same
  SHA and both emitted zero findings): 33 candidates in, 0 BLOCKING out, 7
  advisory out. The validator removed 79%. No run would have turned the
  required check red on a PR two reviewers passed.
* Positive controls (a GPT blocking the old Opus lane missed): recovered on
  #2109 (blocking, same guard GPT named, 2/2) and reported on #2169
  (advisory, 4/4, where the old lane said nothing). #2152's GPT finding was
  generated by discovery but dropped by the validator.

Three tests that asserted on prompt text follow it to its new home rather
than being deleted, and TestOpusTwoStageArchitecture locks the split in place
-- including a cross-check that the marker the gate greps equals the marker
the validation prompt is told to emit, because a typo either side fails every
PR closed and silently.

Known and deliberately not addressed here: the two stages are sequential, so
end-to-end latency roughly doubles under the same 90-minute runaway backstop.
That number can only be measured on a real CI run.

Stacked on the prompt-file bootstrap PR. Because the extraction reads the base
commit, the prompts have to exist on main before this can run -- and for the
same reason, a future PR that EDITS a prompt is still reviewed by the OLD one.
The workflow header documents that.
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Aug 10, 2026
CrysisDeu pushed a commit that referenced this pull request Aug 10, 2026
The Opus lane had a 2.7% blocking rate against GPT's 18% across 73 PRs, and
63 of 73 posted comments contained literally "No findings." A controlled
experiment on this repo found the cause is not the model and not the
architecture -- it is three clauses in the prompt, each independently
sufficient to silence a defect the same model reports 3/3 times without them:
the closed-list reading of the residual defect classes, the certainty
threshold, and "drop the finding if the fix touches untouched code".

Splitting the call while keeping those clauses in the first half did NOT help:
the discovery pass produced zero candidates, so the filter had nothing to
keep. Precision enforcement has to sit downstream of discovery, which is also
what Anthropic's own code-review plugin does (parallel discovery agents, then
a per-candidate validation agent with a confidence floor).

What changes:

* Stage 1 (discovery) generates candidates with generous recall and no
  precision gates. Its output is never posted and gates nothing.
* Stage 2 (validation) is an independent call that re-derives input / call
  path / observable outcome for every candidate from code it opens itself,
  keeps only those it scores >= 80, and only then applies the closed blocking
  list. It keeps `id: review`, so the existing transcript capture, comment
  upsert and fail-closed gate are unchanged.
* The prompts move out of the YAML into .github/review-prompts/ and are shared
  by the same-repo and fork lanes, which previously carried near-duplicate
  copies. They are materialised from the BASE commit, so a PR can neither
  weaken the rules that govern it nor rewrite the prompt that reviews it; a
  missing prompt fails the job rather than degrading into an unspecified
  review that could look clean.
* Candidates cross the stage boundary as a workspace FILE, never string
  interpolation -- model output must not reach YAML or a shell argument, and a
  file has no arg-length ceiling.
* The fix-scope rule changes behaviour instead of being deleted: a finding
  whose only remedy lies outside the changed lines is reported as advisory
  rather than dropped, because the author cannot land the remedy here but the
  signal is still real. A regression the diff itself introduces still blocks,
  since reverting the hunk is an in-diff fix.

Job name, required check, tool surface and the marker contract are unchanged.
The fork lane keeps its no-shell posture (its diff is pre-fetched from
GitHub's compare endpoint).

Measured locally on the same corpus, running these exact prompt files:

* 8 negative-control runs (PRs where BOTH production lanes reviewed the same
  SHA and both emitted zero findings): 33 candidates in, 0 BLOCKING out, 7
  advisory out. The validator removed 79%. No run would have turned the
  required check red on a PR two reviewers passed.
* Positive controls (a GPT blocking the old Opus lane missed): recovered on
  #2109 (blocking, same guard GPT named, 2/2) and reported on #2169
  (advisory, 4/4, where the old lane said nothing). #2152's GPT finding was
  generated by discovery but dropped by the validator.

Three tests that asserted on prompt text follow it to its new home rather
than being deleted, and TestOpusTwoStageArchitecture locks the split in place
-- including a cross-check that the marker the gate greps equals the marker
the validation prompt is told to emit, because a typo either side fails every
PR closed and silently.

Known and deliberately not addressed here: the two stages are sequential, so
end-to-end latency roughly doubles under the same 90-minute runaway backstop.
That number can only be measured on a real CI run.

Stacked on the prompt-file bootstrap PR. Because the extraction reads the base
commit, the prompts have to exist on main before this can run -- and for the
same reason, a future PR that EDITS a prompt is still reviewed by the OLD one.
The workflow header documents that.
CrysisDeu pushed a commit that referenced this pull request Aug 10, 2026
The Opus lane had a 2.7% blocking rate against GPT's 18% across 73 PRs, and
63 of 73 posted comments contained literally "No findings." A controlled
experiment on this repo found the cause is not the model and not the
architecture -- it is three clauses in the prompt, each independently
sufficient to silence a defect the same model reports 3/3 times without them:
the closed-list reading of the residual defect classes, the certainty
threshold, and "drop the finding if the fix touches untouched code".

Splitting the call while keeping those clauses in the first half did NOT help:
the discovery pass produced zero candidates, so the filter had nothing to
keep. Precision enforcement has to sit downstream of discovery, which is also
what Anthropic's own code-review plugin does (parallel discovery agents, then
a per-candidate validation agent with a confidence floor).

What changes:

* Stage 1 (discovery) generates candidates with generous recall and no
  precision gates. Its output is never posted and gates nothing.
* Stage 2 (validation) is an independent call that re-derives input / call
  path / observable outcome for every candidate from code it opens itself,
  keeps only those it scores >= 80, and only then applies the closed blocking
  list. It keeps `id: review`, so the existing transcript capture, comment
  upsert and fail-closed gate are unchanged.
* The prompts move out of the YAML into .github/review-prompts/ and are shared
  by the same-repo and fork lanes, which previously carried near-duplicate
  copies. They are materialised from the BASE commit, so a PR can neither
  weaken the rules that govern it nor rewrite the prompt that reviews it; a
  missing prompt fails the job rather than degrading into an unspecified
  review that could look clean.
* Candidates cross the stage boundary as a workspace FILE, never string
  interpolation -- model output must not reach YAML or a shell argument, and a
  file has no arg-length ceiling.
* The fix-scope rule changes behaviour instead of being deleted: a finding
  whose only remedy lies outside the changed lines is reported as advisory
  rather than dropped, because the author cannot land the remedy here but the
  signal is still real. A regression the diff itself introduces still blocks,
  since reverting the hunk is an in-diff fix.

Job name, required check, tool surface and the marker contract are unchanged.
The fork lane keeps its no-shell posture (its diff is pre-fetched from
GitHub's compare endpoint).

Measured locally on the same corpus, running these exact prompt files:

* 8 negative-control runs (PRs where BOTH production lanes reviewed the same
  SHA and both emitted zero findings): 33 candidates in, 0 BLOCKING out, 7
  advisory out. The validator removed 79%. No run would have turned the
  required check red on a PR two reviewers passed.
* Positive controls (a GPT blocking the old Opus lane missed): recovered on
  #2109 (blocking, same guard GPT named, 2/2) and reported on #2169
  (advisory, 4/4, where the old lane said nothing). #2152's GPT finding was
  generated by discovery but dropped by the validator.

Three tests that asserted on prompt text follow it to its new home rather
than being deleted, and TestOpusTwoStageArchitecture locks the split in place
-- including a cross-check that the marker the gate greps equals the marker
the validation prompt is told to emit, because a typo either side fails every
PR closed and silently.

Known and deliberately not addressed here: the two stages are sequential, so
end-to-end latency roughly doubles under the same 90-minute runaway backstop.
That number can only be measured on a real CI run.

Stacked on the prompt-file bootstrap PR. Because the extraction reads the base
commit, the prompts have to exist on main before this can run -- and for the
same reason, a future PR that EDITS a prompt is still reviewed by the OLD one.
The workflow header documents that.
@buluoray
buluoray force-pushed the fix/terminal-output-redaction branch from 71c1112 to 5e12b18 Compare August 10, 2026 17:33
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Aug 10, 2026
CrysisDeu pushed a commit that referenced this pull request Aug 10, 2026
The Opus lane had a 2.7% blocking rate against GPT's 18% across 73 PRs, and
63 of 73 posted comments contained literally "No findings." A controlled
experiment on this repo found the cause is not the model and not the
architecture -- it is three clauses in the prompt, each independently
sufficient to silence a defect the same model reports 3/3 times without them:
the closed-list reading of the residual defect classes, the certainty
threshold, and "drop the finding if the fix touches untouched code".

Splitting the call while keeping those clauses in the first half did NOT help:
the discovery pass produced zero candidates, so the filter had nothing to
keep. Precision enforcement has to sit downstream of discovery, which is also
what Anthropic's own code-review plugin does (parallel discovery agents, then
a per-candidate validation agent with a confidence floor).

What changes:

* Stage 1 (discovery) generates candidates with generous recall and no
  precision gates. Its output is never posted and gates nothing.
* Stage 2 (validation) is an independent call that re-derives input / call
  path / observable outcome for every candidate from code it opens itself,
  keeps only those it scores >= 80, and only then applies the closed blocking
  list. It keeps `id: review`, so the existing transcript capture, comment
  upsert and fail-closed gate are unchanged.
* The prompts move out of the YAML into .github/review-prompts/ and are shared
  by the same-repo and fork lanes, which previously carried near-duplicate
  copies. They are materialised from the BASE commit, so a PR can neither
  weaken the rules that govern it nor rewrite the prompt that reviews it; a
  missing prompt fails the job rather than degrading into an unspecified
  review that could look clean.
* Candidates cross the stage boundary as a workspace FILE, never string
  interpolation -- model output must not reach YAML or a shell argument, and a
  file has no arg-length ceiling.
* The fix-scope rule changes behaviour instead of being deleted: a finding
  whose only remedy lies outside the changed lines is reported as advisory
  rather than dropped, because the author cannot land the remedy here but the
  signal is still real. A regression the diff itself introduces still blocks,
  since reverting the hunk is an in-diff fix.

Job name, required check, tool surface and the marker contract are unchanged.
The fork lane keeps its no-shell posture (its diff is pre-fetched from
GitHub's compare endpoint).

Measured locally on the same corpus, running these exact prompt files:

* 8 negative-control runs (PRs where BOTH production lanes reviewed the same
  SHA and both emitted zero findings): 33 candidates in, 0 BLOCKING out, 7
  advisory out. The validator removed 79%. No run would have turned the
  required check red on a PR two reviewers passed.
* Positive controls (a GPT blocking the old Opus lane missed): recovered on
  #2109 (blocking, same guard GPT named, 2/2) and reported on #2169
  (advisory, 4/4, where the old lane said nothing). #2152's GPT finding was
  generated by discovery but dropped by the validator.

Three tests that asserted on prompt text follow it to its new home rather
than being deleted, and TestOpusTwoStageArchitecture locks the split in place
-- including a cross-check that the marker the gate greps equals the marker
the validation prompt is told to emit, because a typo either side fails every
PR closed and silently.

Known and deliberately not addressed here: the two stages are sequential, so
end-to-end latency roughly doubles under the same 90-minute runaway backstop.
That number can only be measured on a real CI run.

Stacked on the prompt-file bootstrap PR. Because the extraction reads the base
commit, the prompts have to exist on main before this can run -- and for the
same reason, a future PR that EDITS a prompt is still reviewed by the OLD one.
The workflow header documents that.
@github-actions github-actions Bot removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention labels Aug 10, 2026
@CrysisDeu CrysisDeu reopened this Aug 10, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Aug 10, 2026
The web terminal scanned every 4096-byte PTY read for credentials before
forwarding it. That protected nothing and corrupted output.

It protected nothing because the panel renders into the authenticated
operator's own browser, showing bytes their own shell just wrote. Any threat
that reaches it reaches the terminal application beside it, and the scrollback
ring buffer is replayed only to that same browser. The one path by which
terminal output reaches a model is the selection hand-off, and
POST /api/terminal/redact already scans that unconditionally — over the whole
contiguous selection, which is also the only input the regex redactors can be
accurate on, since a secret split across two reads is invisible to a per-chunk
scan by construction. A source guard now pins the redactors to that single call
site so a scan cannot reappear on the streaming path.

It corrupted output because scanning required decoding, and a PTY read ends
wherever the kernel had bytes. A multi-byte character split across two reads
became two U+FFFD, permanently destroying CJK, emoji and a TUI's box-drawing
glyphs, with no way for the client to recover the original bytes. Forwarding
bytes moves reassembly to xterm.js, which runs its own incremental decoder, so
the server holds no decoder state that can desynchronize from the client's.

The scan also cost real correctness on secrets printed deliberately: it hid the
output of `gh auth token`, swallowed entire device-code login and presigned S3
URLs mid-flow, and mis-fired on npm `integrity sha512-…` lines.

Separately, the read loop now captures the WebSocket into a local and
revalidates it after taking the send lock. `sess.ws` is set to None by the WS
handler on disconnect, and AttributeError is not caught by the loop's
`except OSError`, so dereferencing it after a suspension point killed the
reader task and stopped PTY draining and scrollback capture for a session the
client could still reconnect to.
@buluoray
buluoray force-pushed the fix/terminal-output-redaction branch from 5e12b18 to a5c91a9 Compare August 10, 2026 19:11
@buluoray buluoray changed the title fix(terminal): decode PTY reads incrementally, allow redaction opt-out fix(terminal): stop scanning PTY output on the way to the browser Aug 10, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Aug 10, 2026
@buluoray
buluoray merged commit c4e80b9 into main Aug 10, 2026
58 of 60 checks passed
@buluoray
buluoray deleted the fix/terminal-output-redaction branch August 10, 2026 19:45
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants