feat(cloud): durable launch jobs + owner-only API for EC2 crew setup - #2059
Conversation
Design Review (Fable 5) — 🟡 CONCERNSAdvisory design-level review of I have enough to render a verdict. The design is thorough — durable job store, injected engine, owner-only routes, extensive rollback paths. The remaining risks cluster around billing-visible edges. Design-Verdict: CONCERNS Sound durable-job design; the residual risks are billing-shaped: a restart-orphaned stack has no dashboard recovery path, and stable size keys silently repriced. Watch
Suggestions
[DESIGN-REVIEWED] f35e099 |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 4.8 Review — ✅ no blocking findingsReviewed Review detailsNo findings. The change is well-scoped and heavily tested. I verified the security- and correctness-sensitive paths:
[OPUS-REVIEWED] f35e099 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
1493c63 to
d4c8cf5
Compare
d4c8cf5 to
56ca7c3
Compare
|
Dispositions for the two blocking findings on 1. 2. Two follow-on test corrections worth flagging, since they came out of the fix rather Local gates on |
|
Dispositions for the advisory 1. "Durable-on-disk but not durable-in-control" — FIXED, and the claim corrected. I also removed the "survives a gateway restart" claim from the PR body and from 2. No already-running guard on 3. Your framing of the tradeoff is fair and I am not rebutting the arithmetic. What this PR |
56ca7c3 to
0cb2d2a
Compare
0cb2d2a to
c1f3e5b
Compare
|
Round 5 dispositions for SAST (Semgrep) Suppressed with a rule-scoped Backend Tests (Windows) shard 1 — FIXED. Mine, and a real portability defect in the Also replaced my hand-rolled atomic write with the repo's GPT 5.6 Review — "review incomplete", no verdict produced for Local gates on |
|
Round 11 dispositions for 1. One consequence worth adding, because it makes this more than a mislabelled row: if the snapshot was Fixed as suggested: re-read the job after finding no event and terminalize only that fresh snapshot. 2. The behaviour report is accurate: I did not pass the cancel signal into Local gates on Note on the brand gate: my local run flagged |
6f3c8e1 to
fb94d7a
Compare
UX Review (Fable 5) — 🟡 CONCERNSAdvisory UX-level review of I have everything I need — diff, strings, screenshots, and sibling code. Composing the review. UX-Verdict: CONCERNS Solid flow overall, but the sign-in recovery button lies about its action, and the armed "Confirm delete" never disarms — both on billable paths. Watch
Suggestions
[UX-REVIEWED] f35e099 |
|
Dispositions for the findings raised across 1. I did not take "wait for deletion and raise" literally, because waiting before 2. Moving it broke two of my own tests, which is worth recording: they patched 3. Widened beyond the prescribed fix: Pattern worth flagging to a human reviewer: three separate findings on this PR have Local gates on Disclosure about the local review gate: this session has no |
|
Answering both advisory UX Review 🟡 CONCERNS1. "Stop is a one-way door" — FIXED. Verified and correct: 2. "Disconnect vs Stop side by side are indistinguishable" — ACCEPTED-AND-DEFERRED. 3. "The success badge says Connect" — ACCEPTED-AND-DEFERRED. Correct that 4. "'Smaller and x86_64 sizes' breaks its promise" — REBUTTED (copy is accurate), 5. "Screenshot 03-setup-sizes.png is stale" — VALID, and I have removed it rather Design Review 🟡 CONCERNS6. "A restart recreates the same harm: reap_orphans marks the job failed but a I am not fixing that here because the honest fix changes the launch's ordering 7. "Reused tier keys silently reprice ~2.4×" — ACCEPTED, deliberately, and now 8. "Cloud-crew identity derives from correlating registry entries against launch-job Local gates on |
|
Dispositions for GPT 5.6 —
|
|
Dispositions for the advisory verdicts on UX Review 🟡 CONCERNS1. "Start gives zero feedback; the Stop key interpolates an object" — VALID, FIXED, 2. "Stop and Start render side-by-side with no instance-state indicator" — 3. "Finished launch is badged 'Connect'" — ACCEPTED-AND-DEFERRED. Confirmed: the 4. "'Smaller and x86_64 sizes' contains nothing smaller" — REBUTTED on the facts, 5. "Armed 'Confirm delete' has no way to back out" — ACCEPTED-AND-DEFERRED, and the 6. "Tab badge counts 7. " Design Review 🟡 CONCERNS8. "Serve the size ladder from
If a maintainer prefers the endpoint I will build it; I am not claiming the gate is 9. "Restart recovery contradicts the PR's own goal — a mid-provision restart strands a I am still not fixing it here, for one reason: the durable fix is to register immediately 10. " Summary of what is deliberately left unfixedItems 2, 3, 5, 6, 7 are a UI/copy pass (state indicator, terminal-state badge, Local gates on Disclosure: this session has no |
|
Dispositions for the verdicts on GPT 5.6 —
|
|
Disposition for GPT 5.6 —
|
|
Dispositions for GPT 5.6 —
|
|
Disposition for GPT 5.6 —
|
|
Round update on A launched crew (kc-90a9df) sat on "Pane failed to load" for hours. On-box: the gateway Fix ( End-to-end validation is a fresh cloud launch from a gateway running this branch (the Local gates on |
|
Follow-up on the dashboard-build guard (surfaced by a real launch on this branch). The guard worked: a fresh crew detected the missing SPA, failed the launch, and rolled the stack back so it stops billing -- no more silent dead crew. But it exposed two gaps the guard alone didn't close:
Fix: the cloud bootstrap now sets Local gates on |
|
Disposition for GPT 5.6 —
|
|
Disposition for GPT 5.6 —
|
Creating a remote crew on EC2 was CLI-only, and the CLI drives an interactive device login — so the gateway could list remote crews but never create one. Backend: durable launch jobs (`cloud/launch_job.py`) that persist every state transition to disk, so progress survives navigating away, plus ten owner-only `/api/cloud/*` routes and a `RealLaunchEngine` that binds to the existing `ec2` / `login` / `connect` modules rather than adding new AWS logic. A cancel after provisioning rolls the stack back so a half-built crew cannot bill silently, and a restart terminalizes orphaned jobs instead of leaving a card that can never advance. Frontend: one Remote Crew settings page with two tabs — "Your crews" (manage, default) and "Set up a new one" (AWS prerequisite checklist + size picker + launch progress with the device code). The account inputs sit inside the prerequisites card, above the rows they produce, and the card names the profile and region it probed. Sizes are re-laddered to 16 / 32 (default) / 64 GB, described by the number of parallel sub-agents each supports, because the cap is CPU-bound.
|
Disposition for GPT 5.6 BLOCKING —
|
Problem
Creating a remote crew on EC2 is CLI-only, and the CLI drives an interactive
device login. The dashboard therefore had no way to offer it: a web request cannot
sit and wait while a human approves a code in a browser, and an HTTP handler that
blocks for the minutes a CloudFormation stack takes would be dead on arrival. So
the gateway could list remote crews but never create one.
Why it matters
The only path to a remote crew was "read a guide, run CLI commands, and get the AWS
prerequisites right on the first try". Everything needed already exists in
cloud/— it just had no second caller. Users who want a bigger machine for parallel
sub-agents drop out at the CLI step.
Fix (symptom → root cause → change)
Symptom: the dashboard can list crews but not create one.
Root cause: two properties of the CLI path are incompatible with a
request/response cycle — a multi-minute provision, and a human-in-the-loop device
code.
Change: move the long-running work off the request and make the device code a
piece of durable state the UI can read.
cloud/launch_job.py— a launch is a durable job persisted after every statetransition, so progress survives navigating away. The device code is job state,
not a blocking prompt.
cloud/launch_engine.py—RealLaunchEnginebinds to the existingec2.deploy/login.start_device_login/connect.register_instance. No new AWS logic; theengine is injected so tests never touch AWS.
dashboard/handlers_cloud.py— ten owner-only/api/cloud/*routes (rejectingSlack origin, app tokens, and non-POSIX hosts). Every error response carries a
machine-readable
code.settings/RemoteCrewPanel.tsx— one Remote Crew page, two tabs: Your crews(manage, default) and Set up a new one (prerequisite checklist, size picker,
launch progress with the device code).
Making a launched crew actually reachable and accounted for
Manual testing surfaced several ways a paid instance could end up outside the
dashboard's control. Each is fixed here:
local_port == remote_port(theremote gateway only trusts Origins on its own configured port) and hard-fails when
that port is busy. Registering every crew on the default 5476 produced a crew that
could never be connected — the operator's own gateway usually owns 5476. The
launcher now allocates a port (deterministically, skipping ports the registry
already uses) and applies it to both ends: a new
DashboardPortstackparameter sets
KIROCREW_PORTin the systemd unit and the bootstrap healthprobe, and the same value is recorded as the crew's
remote_port.source.repo_root()fails closedwhen there is no checkout (it must never tar up
site-packages), which madeone-click setup impossible from a wheel/app install. A non-raising
find_repo_root()now decides: checkout → ship local source; otherwise install bycloning the public repo, which is the fallback the template already carried.
registered last, so a cancel after provisioning left a running, billing instance
that never appeared in the crew list. It now deletes the stack it created, acking
the cancelled state first and only reporting "Removed" once AWS confirms —
a delete that lands in
DELETE_FAILEDsays so instead.connect.register_instanceisbest-effort by contract (returns
None), so a registry failure used to mark thelaunch
donewith no crew. It now fails the job and names the instance.that is awaiting sign-in persists the device-login URL and code — a credential
that completes the Kiro sign-in. It was written under
<config_dir>/cloud/, whichis NOT on the shared sensitive-path floor, so a prompt-injected same-UID agent
could read and exfiltrate it via ordinary file tools (0600 perms do not help — the
agent runs as the same user). Jobs now live under
<config_dir>/run/cloud-launch-jobs/;run/is onsecurity._SENSITIVE_HOME_DIRS, so agent file tools refuse it, whilethe gateway's own writer opens the path directly and is unaffected.
POST /api/cloud/launchhandler persisted the new job with a synchronousstore.create()(mkdir + temp-write +os.replace) directly on the aiohttploop, while its sibling
listcall right above it was already offloaded via_in_executor-- so on a slow disk the create could stall every other gatewayrequest and the heartbeat behind it.
create()now runs in the executor too(still under the single-launch lock). Regression:
test_create_persists_off_the_event_loop.copy-pasteable install command we show for the local host downloaded the AWS
package to a fixed
/tmp/session-manager-plugin.{pkg,deb}and then ransudo installer/sudo dpkg -ion it. On a shared host a local user couldpreplant or swap that path and have the privileged install execute their scripts
as root. The command now downloads into a private
mktemp -d(0700, owner-only)and removes it afterward; the rpm (
dnf install <url>) and Homebrew branchesnever touched a temp file and are unchanged.
_path()validated theid charset but not its length, so a charset-valid but over-long
{id}onGET/cancel/signin /api/cloud/launch/{id}(e.g. 300 hex chars) reachedPath.exists()and raisedENAMETOOLONG-> HTTP 500. The store now requires theexact generated shape (12 hex chars) before any filesystem access, so a bad id is
a clean 404 (
get()maps the ValueError to None -> the handler's not-found path).retry/exposure above lives in install.sh, which travels with the CLONED source --
so on the default
KirocrewRef=mainlaunch (no local checkout shipped) the crewruns main's non-fatal install.sh and the real npm error was lost. The template's
dist-check (which travels with the launching gateway) now greps the setup log for
the npm/vite/tsc error and folds it into the WaitCondition reason, so the failure
names its cause regardless of which install.sh ran.
install.sh treated a build failure as a non-fatal warning (exit 0), so the cloud
bootstrap's retry never fired for it and the real npm/vite error was swallowed
behind "legacy fallback". The bootstrap now sets
KIROCREW_REQUIRE_FRONTEND=1,making install.sh exit non-zero on a failed build: the existing install retry
re-runs it on the warm box (first-boot contention -- the common cause -- self-heals),
and if it still fails the build-log tail is printed into the WaitCondition reason
instead of a bare "build failed". Local CLI installs stay non-fatal.
install.sh treats a frontend-build failure as a non-fatal warning (legacy
fallback) -- right for a local CLI user, wrong for a cloud crew whose whole point
is the remote dashboard. Without the built SPA the gateway serves a ~782-byte
"not built" stub that still returns HTTP 200, so the bootstrap health probe
(curl 127.0.0.1:/) passed and the stack reached CREATE_COMPLETE while the
crew's pane could never load -- a green launch handing over a dead, billing box.
The bootstrap now verifies src/kiro_crew/static/dist/index.html exists after
install and fails the WaitCondition (folding the build log into the reason) when
it does not, so a failed build rolls the stack back with a real error.
ec2.deploycreates the stack andthen blocks until healthy, so a transient failure after the stack exists (e.g. a
post-create
DescribeStackserror) marked the job FAILED while the instance keptrunning, unregistered and invisible. A STEP_PROVISION failure now best-effort
tears the stack down (mirroring the cancel path) before recording the failure;
scoped to that step so a later-step failure — where the crew is created and named
for recovery — is never deleted out from under the user.
only to jobs that still carried a sign-in prompt, so a launch that FAILED or was
reaped on restart (no prompt) rendered nothing after a reload — hiding its error
and the "check your crews, it may still be running" warning for a possibly-billing
stack. It now falls back to the newest persisted job.
Slack-origin and app tokens but then admitted any request carrying a
userwith anempty
app— and a dashboard session token is also minted for every allowed Slackuser (
!dashboard), which is exactly that shape. A non-owner could therefore reacha control plane that creates, stops and terminates billable AWS resources on the
owner's account. The guard now matches the configured owner via the shared
is_owner_dashboard_requestpredicate (the same definitionask_questionand thesource-provider routes use); a single-owner setup with no owner configured is
unaffected (the owner's own local token still matches).
while starting or resuming the device login used to propagate out of the
sign-in handle, failing the job before registration and leaving a paid instance
absent from the crew list. Both the
start_device_logincall in the handle'sconstructor and the daemon resume in
wait()now degrade to "sign-inunconfirmed": the crew still registers, and the device code (when there is one) is
preserved so the user can finish from the dashboard.
device code alive when sign-in could not be confirmed (it clears it only once
sign-in succeeds), but the card gated that block on
status === 'awaiting_signin'and the remount fallback only adopted in-progress jobs — so the surviving code was
unreachable the moment the job went terminal, making "finish it from the dashboard"
a dead end. The card now shows the code (with an explicit "could not confirm"
message) whenever a terminal job still carries a prompt.
the whole
{tag, coords}variables object, yieldingstop:[object Object]— a key norow could match, so the label never changed for the duration of the request. It stayed
invisible because the row still disables (that only tests
!!busy). Start is alsowired into the same computation, which it was missing entirely.
advance.
(
cleanup: "pending"); the registry row is dropped minutes later by the backgroundwatcher once AWS confirms
DELETE_COMPLETE. The row now shows a Deleting…state and the crew list polls until it disappears, instead of reappearing
unchanged so the click read as a no-op.
Set-up tab usability (found by manual testing)
and the card names the profile and region it probed — showing the verdict first and
the inputs later inverted cause and effect.
usePersistedString); losing them meantthe next probe silently tested the AWS CLI default profile.
isFetching(withisLoading, a refetch of analready-populated query showed no feedback at all).
session-manager-plugininstall command is served by the backend, whichknows the gateway host's OS (macOS pkg/brew, Linux deb/rpm) — a hardcoded
Homebrew line was unusable on every Linux gateway.
sub-agents each supports, because the cap is CPU-bound (
floor(vCPU × 0.8)). Notethis makes
--size balancedresolve to a larger, pricier instance; the tier keysare deliberately unchanged.
Tests
Backend (197): job lifecycle and durability; cancel-rollback including an
unconfirmed delete not being reported as removed; orphan reaping vs a freshly
created job; owner-only guards on all ten routes — including a non-owner allowed Slack
user (not only an app token) being rejected;
instance_idderived server-siderather than trusted from the caller; the allocated gateway port reaching both the
stack and the registry, and skipping ports already in use; source-mode selection
with and without a checkout; the plugin install command per platform/arch; an
unconfirmable sign-in returning "not signed in" instead of raising — from both
the daemon resume and a failure starting the device login; the
error-code contract ratchet; and a size-table parity gate asserting the panel's
SIZE_TIERS/X86_TIERSfacts matchcloud/sizes.pytier-for-tier, so the twosources of truth can no longer drift silently.
Frontend (19 panel tests): cloud vs hand-added vs unidentifiable crews and which
destructive action each may offer; the device code surviving a remount; the crew list
refreshing when a launch finishes; the profile surviving a remount and driving the
first probe; Re-check showing progress; the server-supplied install command
rendering; size-card interpolation; a delete showing a Deleting… state once the
request is accepted; and copy guards that the progress card promises
only navigation persistence and that inputs precede their checks.
Manual verification
Found via live debugging of a launched crew (kc-90a9df): the box was healthy and
served HTTP 200, but the journal showed "Dashboard dist/ not found" -- the
public-clone launch had not built the SPA, so the pane timed out. End-to-end
validation is a fresh cloud launch from a gateway running this branch: the crew now
either comes up with a working dashboard, or fails the launch with the explicit
"dashboard frontend build missing" error and rolls back.
Built and run from the feature worktree against a real AWS account (116101834266).
The prerequisite checklist correctly reported expired credentials, went green after
aws login, and correctly stayed red for a missingsession-manager-plugin. Thatsession is what surfaced the input ordering, the profile-persistence trap, the
invisible Re-check, the macOS-only install command, and the source-shipping failure
— all fixed here.
cfn-lintpasses on the template.Screenshots
Launch flow (10.7s)
Full launch sequence
Notes for reviewers
config/infrasignal: the flagged file iscloud/templates/kirocrew-ec2.yaml— a newDashboardPortparameter (default5476, so a direct deploy is unchanged) wired into the systemd unit's
KIROCREW_PORTand the bootstrap health probe.app/wheel install it clones the public repo at
main. That means a DMG user's crewtracks upstream
main, not the version inside their app — deliberate for now.Selectable stable/nightly/own-S3 channels are a follow-up.
main, so this singlePR carries the whole feature (backend + frontend + evidence).