close
Skip to content

fix(prepare-pr): re-mirror the local review gate onto the current CI contract - #2456

Merged
iamwhatever merged 1 commit into
mainfrom
ci/prepare-pr-mirror-4.8
Aug 10, 2026
Merged

fix(prepare-pr): re-mirror the local review gate onto the current CI contract#2456
iamwhatever merged 1 commit into
mainfrom
ci/prepare-pr-mirror-4.8

Conversation

@iamwhatever

Copy link
Copy Markdown
Collaborator

Problem

prepare-pr no longer mirrors CI. Its whole value is that local-green predicts server-green — and four things had drifted since the CI review sprint, so the local gate was mirroring a contract CI stopped running.

The model was the reported symptom, but it was the smallest of the four:

prepare-pr said CI actually runs
Opus model claude-opus-5 us.anthropic.claude-opus-4-8 (#2339)
Blocking budget ≤2 BLOCKING 5 BLOCKING + 6 advisory FINDING (#2322, #2379)
Blocking classes 2 (AUTOSDE + residual) 5, incl. correctness-on-normal-path (#2379)
Quality dimensions not mentioned 7 dimensions + consequence-chain bar (#2379)

Why it matters

Two of these actively degrade the gate:

  • Wrong model — the local reviewer was reviewing with a different model than the gate it claims to mirror, so local findings had no particular relationship to server findings.
  • Budget of 2 vs 5 — a local cap of 2 systematically under-predicts the server round. That is precisely the "one push turns into ten" failure the skill's own "Common mistakes" section warns about.

Fix

Model: claude-opus-4.8, fallback claude-opus-4.7. CI has no fallback (#2339 removed it); the local tier stays because local model availability varies — and the charter now says that, so the difference reads as intentional rather than as drift.

Budget: ≤5 BLOCKING, ≤6 advisory FINDING.

Blocking contract: all 5 CI classes — blocking: true AUTOSDE rule, reachable security hole, crash/data-loss/corruption, removed guard with no replacement, and a correctness defect that is unconditional wrong behaviour on the NORMAL path.

Dimensions: the charter now names the consequence-chain bar (cause → mechanism → consequence, dropped rather than downgraded if it cannot complete) and the seven review dimensions. Previously the local reviewer had no instruction to look at correctness, resource/lifecycle, scope fidelity, consistency, maintainability, or observability at all.

GPT charter: now states its ≤5 BLOCKING budget and that CI runs the lane as two passes (discovery + authoritative falsification), so a single local pass knows to apply the same falsification bar.

Tests

test_opus_profile_model_matches_the_ci_workflow — a drift guard that parses --model out of claude-review.yml and compares it to the profile pin, normalizing across the two id namespaces on purpose (CI uses the Bedrock regional inference profile, the local harness uses the kiro-cli id).

Anchored to the real claude_args line: an unanchored regex matched the prose "--model below" in the comment above the job and produced a false failure — caught while writing it.

Positive control: reverting the pin to claude-opus-5 makes the guard fail with both ids named; restoring it passes. 18/18 pass, flake8 + isort clean.

Manual verification

  • Diffed the profile and SKILL.md against origin/main's claude-review.yml / codex-review.yml to enumerate the drift rather than assuming only the model moved.
  • Confirmed claude-opus-4.7 is a served kiro-cli id before pinning it as the fallback.

Screenshots

N/A — skill/profile text and a test.


Note: the installed copy under ~/.kiro/crew/skills/kirocrew-dev/prepare-pr/ is a real directory, not a symlink to the repo, so it refreshes on the next install/update rather than immediately from this merge.

@iamwhatever
iamwhatever requested a review from a team as a code owner August 10, 2026 01:59
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Aug 10, 2026
@github-actions

github-actions Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 4bf5ee2ce6870ff7da565fe7941416ac9675039d — this comment is updated in place on each push.

Review details

All new test assertions hold against the actual workflows (--model us.anthropic.claude-opus-4-8 normalizes to claude-opus-4.8; both budget lines are 5 BLOCKING / 6 advisory). The change is skill/profile prose plus a profile pin correction and drift-guard tests, all internally consistent. No semantic defects on changed lines.

No findings.

[OPUS-REVIEWED] 4bf5ee2

Verdict parsed from the review's SHA-scoped output markers for commit 4bf5ee2ce6870ff7da565fe7941416ac9675039d.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 4bf5ee2ce6870ff7da565fe7941416ac9675039d: <one-sentence reason>

@github-actions

github-actions Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Advisory design-level review of 4bf5ee2ce6870ff7da565fe7941416ac9675039d — updated in place on each push; does not block merge.

I've verified the PR's claims against the repo: CI does pin us.anthropic.claude-opus-4-8 at claude-review.yml:209, and both workflows carry the 5 BLOCKING / 6 advisory FINDING budgets the charter now quotes. The diff matches the description bidirectionally, and the drift-guard tests convert the previously prose-only mirror into a pinned contract — the same pattern AGENTS.md already uses for the denied-rule count.

Design-Verdict: PASS

Real drift, fixed at the cause: the mirror claim is now test-pinned to the workflows instead of trusted prose, so recurrence fails loudly.

Suggestions

  • The drift guard covers only the opus model pin; the gpt-5.6-sol pin can drift from codex-review.yml the same way — worth a matching guard here or as a follow-up.

[DESIGN-REVIEWED] 4bf5ee2

@github-actions

github-actions Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 4bf5ee2ce6870ff7da565fe7941416ac9675039d and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 4bf5ee2

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 4bf5ee2ce6870ff7da565fe7941416ac9675039d: <one-sentence reason>

@iamwhatever
iamwhatever force-pushed the ci/prepare-pr-mirror-4.8 branch from 6f80b7b to 90bdfc9 Compare August 10, 2026 02:02
@iamwhatever
iamwhatever enabled auto-merge (squash) August 10, 2026 02:08
@iamwhatever
iamwhatever force-pushed the ci/prepare-pr-mirror-4.8 branch from 90bdfc9 to c746ea5 Compare August 10, 2026 02:16
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Aug 10, 2026
…contract

prepare-pr's entire value is that local-green predicts server-green. Four
things had drifted since the CI review sprint, so the local gate was
mirroring a contract CI no longer runs.

Model (the reported symptom):
- profile pinned `claude-opus-5`; claude-review.yml moved to
  `us.anthropic.claude-opus-4-8` in #2339. The local gate was reviewing with
  a DIFFERENT model than the gate it claims to mirror.
- Now `claude-opus-4.8`, fallback `claude-opus-4.7`. CI has no fallback
  (#2339 removed it); the local tier stays because local model availability
  varies, and that difference is now stated in the charter instead of
  looking like drift.

Three further drifts, all understating CI:
- Budget said "≤2 BLOCKING". CI is 5 BLOCKING + 6 advisory FINDING (#2322,
  #2379). A local cap of 2 systematically under-predicts the server round --
  the precise cause of the "one push turns into ten" failure this skill
  warns about.
- Blocking contract said 2 classes (AUTOSDE blocking:true, residual-class).
  CI enumerates 5, including a correctness defect that is unconditional
  wrong behaviour on the NORMAL path.
- The charter never mentioned the consequence-chain bar or the quality
  dimensions added in #2379, so the local reviewer had no instruction to
  look at correctness, resource/lifecycle, scope fidelity, consistency,
  maintainability, or observability at all.

The GPT charter also now states its ≤5 BLOCKING budget and that CI runs the
lane as two passes (discovery + authoritative falsification), so a single
local pass knows to apply the same falsification bar.

Drift guard: test_opus_profile_model_matches_the_ci_workflow parses
--model out of claude-review.yml and compares it to the profile pin,
normalizing across the two id namespaces on purpose -- CI uses the Bedrock
regional inference profile (us.anthropic.claude-opus-4-8), the local harness
uses the kiro-cli id (claude-opus-4.8). Anchored to the real claude_args
line: an unanchored regex matched the prose "--model below" in the comment
above the job and produced a false failure.

Positive control: reverting the pin to claude-opus-5 makes the guard fail
with the two ids named; restoring it passes.

18/18 test_prepare_pr_profiles.py pass. flake8 + isort clean.
@iamwhatever
iamwhatever force-pushed the ci/prepare-pr-mirror-4.8 branch from c746ea5 to 4bf5ee2 Compare August 10, 2026 02:32
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Aug 10, 2026
@iamwhatever

Copy link
Copy Markdown
Collaborator Author

Re: Design Review 🟡 CONCERNS (c746ea57f)

Watch — the drift guard pins only the model, so the hand-copied prose contract can silently re-drift: FIXED in 4bf5ee2ce.

The concern is correct and it was a hole in this PR's own thesis: I argued drift is the problem, then guarded one of the four things that drifted. Took your first suggestion (extend the guard) rather than the second (slim the charter), because the charter's restated numbers are what make the local reviewer's instructions self-contained — a subagent that has to go read the workflow to learn its own budget is a worse contract, not a better one.

Added test_charter_budgets_match_the_ci_workflows, which parses the authoritative BUDGET: lines out of both review workflows and asserts SKILL.md quotes those numbers:

  • at most (\d+) BLOCKING from claude-review.yml → the charter must say ≤5 BLOCKING, ≤6 advisory FINDING
  • At most (\d+) advisory FINDING from claude-review.yml → same assertion
  • at most (\d+) BLOCKING from codex-review.yml → the gpt charter must say ≤5 BLOCKING

Positive control: mutating the charter to ≤2 BLOCKING (the exact stale value this PR removes) fails with the opus charter's budget no longer matches claude-review.yml (5 BLOCKING / 6 advisory); restoring it passes. So the next budget change on either side fails CI instead of silently re-opening the local-green/server-green gap.

19/19 pass, flake8 + isort clean, brand gate green.

Not pinned, deliberately: the five blocking classes and the seven dimensions are prose enumerations with no single parseable line to anchor to, so a regex guard there would be brittle rather than protective. The budget numbers were the cheaply-parseable facts you identified, and those are now covered.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Aug 10, 2026
@iamwhatever
iamwhatever merged commit 9b3c55a into main Aug 10, 2026
51 checks passed
@iamwhatever
iamwhatever deleted the ci/prepare-pr-mirror-4.8 branch August 10, 2026 04:08
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Aug 10, 2026
encomjp pushed a commit to encomjp/kirocrew-customapi that referenced this pull request Aug 22, 2026
…contract (kirodotdev#2456)

prepare-pr's entire value is that local-green predicts server-green. Four
things had drifted since the CI review sprint, so the local gate was
mirroring a contract CI no longer runs.

Model (the reported symptom):
- profile pinned `claude-opus-5`; claude-review.yml moved to
  `us.anthropic.claude-opus-4-8` in kirodotdev#2339. The local gate was reviewing with
  a DIFFERENT model than the gate it claims to mirror.
- Now `claude-opus-4.8`, fallback `claude-opus-4.7`. CI has no fallback
  (kirodotdev#2339 removed it); the local tier stays because local model availability
  varies, and that difference is now stated in the charter instead of
  looking like drift.

Three further drifts, all understating CI:
- Budget said "≤2 BLOCKING". CI is 5 BLOCKING + 6 advisory FINDING (kirodotdev#2322,
  kirodotdev#2379). A local cap of 2 systematically under-predicts the server round --
  the precise cause of the "one push turns into ten" failure this skill
  warns about.
- Blocking contract said 2 classes (AUTOSDE blocking:true, residual-class).
  CI enumerates 5, including a correctness defect that is unconditional
  wrong behaviour on the NORMAL path.
- The charter never mentioned the consequence-chain bar or the quality
  dimensions added in kirodotdev#2379, so the local reviewer had no instruction to
  look at correctness, resource/lifecycle, scope fidelity, consistency,
  maintainability, or observability at all.

The GPT charter also now states its ≤5 BLOCKING budget and that CI runs the
lane as two passes (discovery + authoritative falsification), so a single
local pass knows to apply the same falsification bar.

Drift guard: test_opus_profile_model_matches_the_ci_workflow parses
--model out of claude-review.yml and compares it to the profile pin,
normalizing across the two id namespaces on purpose -- CI uses the Bedrock
regional inference profile (us.anthropic.claude-opus-4-8), the local harness
uses the kiro-cli id (claude-opus-4.8). Anchored to the real claude_args
line: an unanchored regex matched the prose "--model below" in the comment
above the job and produced a false failure.

Positive control: reverting the pin to claude-opus-5 makes the guard fail
with the two ids named; restoring it passes.

18/18 test_prepare_pr_profiles.py pass. flake8 + isort clean.

Co-authored-by: Joe Guo <zejiangg@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants