fix(markdown): stop CJK punctuation being swallowed into bare URLs

[CrysisDeu]

What it looks like

Captured by website/scripts/capture-cjk-autolink.mjs — the REAL built SPA
behind the repo's shared fixture server, so the bubble goes through the actual
remark/rehype pipeline. One run from this branch, one from a dist built with
origin/main's MarkdownRenderer.tsx. Three lines in one frame: the reported
message, the bracket rule, and a real article URL containing （公司）
that must survive untouched in BOTH builds.

Before (origin/main)

After (this branch)

Light theme: before · after

Read the frames line by line:

line	before	after
reported message	href runs through ， and eats the code span's opening backtick; readiness: passed renders with literal backticks	href stops at /2137; both code spans render as code
（详见 …/pull/2137）	the closing ） and the rest of the sentence land inside the href	）后面还有正文。 is prose again
…/wiki/苹果（公司）	correct	identical — no opener is pending, so there is no evidence and nothing is cut

Not pictured, deliberately: …/wiki/モーニング娘。紹介``. A sentence-ender
directly before a backtick is left alone, so it renders the same in both builds.
That trade is stated below and locked by a unit test rather than photographed.

Problem

Reported from the dashboard chat transcript. This message:

**#2137 — review-ready**（https://github.com/kirodotdev/KiroCrew/pull/2137，`96ed647b`）：`readiness: passed`，45 绿 0 红

renders as:

（<a href="https://github.com/kirodotdev/KiroCrew/pull/2137%EF%BC%8C%6096ed647b%60%EF%BC%89">
   https://github.com/kirodotdev/KiroCrew/pull/2137，`96ed647b`）：`readiness</a>: passed`，45 绿 0 红

GFM's autolink-literal extension ends a bare https://… run only at ASCII
whitespace or <. CJK prose writes punctuation directly after a URL with no
space, so ，96ed647）：readiness` all lands inside the href.

Why it matters

The broken href is the smaller half of the damage. The run also eats the
opening backtick of the code span that follows, which shifts every later
backtick pairing in the paragraph by one — so prose renders as inline code
(the whole tail of the message goes monospace) and real code renders with
literal backticks. One missing space corrupts the rest of the message, and
every Chinese/Japanese message that cites a URL hits it.

Fix (symptom → root cause → change)

Symptom is a wrong href plus a corrupted paragraph. Root cause is a tokenizer
boundary: the run has no end until whitespace. So the change gives it one —
fixCjkAutolinkBoundaries runs on the markdown source, beside the existing
fixCodeFences / stripStrayTags preprocessors, and re-emits the URL head as an
angle autolink <url> (explicit end, identical rendering).

Source level is not a shortcut: re-splitting the link node on the mdast fixes
the href but cannot restore the code-span pairing, because that pairing is
decided while micromark tokenizes the whole paragraph. The boundary has to exist
before parsing.

Two questions have to be answered, and they are answered by different means.

Which regions are off-limits — read off remark's own parse

autolinkLiteralSpans() parses the source with the same plugin set the render
pipeline uses (remark-parse + remark-gfm + remark-math with
singleDollarTextMath: false) and returns the source offsets of GFM
autolink-literal nodes only.

Everything that must not be rewritten — fenced code, indented code, inline-code
spans including multi-line ones, existing links and images, angle autolinks, raw
HTML tags, math — never becomes such a node, so it is excluded by
construction. An earlier revision hand-rolled a masking scanner for this; three
review rounds found six real defects in it (tab-stop indentation, blockquoted
fences, multi-line code spans, closing-fence validity, …), which is a signal
about the approach rather than the cases. Deleting the scanner in favour of the
parser closes the whole class, and picked up two behaviours the mask never had:
math spans are protected, and list continuation paragraphs are no longer
skipped by a blanket "any 4-column-indented line is code" rule.

Angle autolinks and [text](url) links can also satisfy text === url, so the
literal test is on the source text at the node's start, not on node shape alone.
A parse failure returns no spans — a malformed message renders unfixed rather
than not at all.

Where the URL ends — evidence, not characters

CJK punctuation is not by itself proof that a URL ended — it reaches real
URLs raw (…/wiki/苹果（公司）, …/wiki/我，机器人, …/wiki/モーニング娘。 are
all live articles that GFM links correctly today). So a cut needs one of two
pieces of evidence:

1. A CJK closing bracket that closes an opener SURROUNDING the URL — one
   left unclosed in the prose before the URL on its line, and not opened
   inside the run. Prose is the operative word: the parse also yields a mask of
   every non-prose node (inlineCode, code, non-literal links, images, html,
   math, link definitions), and those characters are blanked out of the prefix, so
   a （ in a code sample or an HTML attribute cannot supply the opener.

   Autolink literals are not pre-masked, because a greedy run's span is not
   uniformly URL — （https://a/1）和【https://b/2】 is ONE run whose ）和【 is
   real prose. Instead each URL's own characters are masked as the scan consumes
   them: a run that yields no cut is masked whole (so a （ in its query string
   still cannot pose as context), while a run that yields a cut masks only the
   head, leaving the inter-URL prose visible to the next URL's bracket balance. This
   is GFM's own ASCII paren-balancing rule, generalised. （https://x.com/a） and
   （见 https://x.com/a） cut. https://x.com/苹果（公司） does not (the opener
   is inside the URL), and neither does https://x.com/search?q=foo） — nothing
   there opened a bracket, so it is plausibly part of the query and GFM links it
   correctly today.

2. A SEPARATOR-class CJK mark IMMEDIATELY followed by a BACKTICK. The backtick is the
   one character RFC 3986 excludes (browsers percent-encode it), so unlike *,
   [ or ] — all legal and common in query strings (?q=foo，*test,
   ?filter[name]=x) — it cannot plausibly be inside a raw-written URL. It is
   also the character whose loss does the real damage: the run eats an opening
   code-span delimiter and every later backtick pairing in the paragraph shifts.
   Directly-after (not anywhere-later) matters: …/wiki/我，机器人简介 has the markup in the same whitespace-delimited run, but the comma is followed by more *title*, so it must not cut. A contiguous punctuation run counts as one boundary and the cut lands at its start, so `…/a、，`c does not leave 、 inside the href.

Sentence-enders are excluded from rule 2. 。．！？…｡ end real
page titles and reach the URL raw — …/wiki/モーニング娘。,
…/wiki/魔法先生ネギま！ — so cutting there would point the anchor at the
wrong article. Separators (， 、 ； ： · ～ “) do not end titles,
so they stay eligible, which is what keeps the reported case fixed. A mixed run
resolves conservatively: …/a。，c yields `<…/a。>，`c, leaving the
possibly-title 。 inside the href.

Vercel's @streamdown/cjk@1.0.3 takes the aggressive side of this same trade —
first mark in a fixed 20-character set, no evidence test. Measured, it breaks
…/wiki/苹果（公司）, …/wiki/モーニング娘。 and …/wiki/我，机器人,
cannot repair code-span pairing (it runs after tokenization), and mis-splits
explicit <…> autolinks. Independent evidence that the unconditional cut is the
wrong default.

Documented limitation: …/pull/1，然后回来 — a bare CJK sentence running off
a URL with no space and no markup — is left alone, and so is …/a，**注意**. It is character-for-character
indistinguishable from a legitimate …/wiki/我，机器人, so it keeps today's
behaviour rather than risking a correct link. A test locks that in so it reads as
a decision, not an oversight.

Other guards

• Never invents a link. The head must clear the same bar GFM applies (host
  contains a dot; neither of the last two labels contains _), so
  http://localhost:5476/a，c and `http://a_b.com/x，`c are left alone.
• Matches GFM's trailing-punctuation trim (?!.,:*_~ plus an unbalanced
  )), which an angle autolink would otherwise keep.
• Runs AFTER fixCodeFences. That preprocessor does not only escape N.
  lines — later passes create code blocks the raw source did not have (blank
  line before a fence glued to preceding text; splitting a closing fence glued to
  trailing text). Rewriting first would judge such a region as prose and leave a
  literal <…> inside what ends up displayed as code. Both directions are
  asserted in a test.
• Skipped entirely when sourcePos is on. That surface maps a DOM selection
  back to source coordinates through data-sourcepos for inline commenting;
  inserting two characters shifts every later column on the line and would
  anchor a comment to the wrong occurrence. The inline-comment surface keeps the
  unfixed but coordinate-accurate render.
• Second URL inside one autolink node still gets a boundary —
  （https://a/1）和（https://b/2） is ONE run, so the scan resumes after each head.
• A https:// nested inside another URL's path (?u=https://…) is never cut
  separately.

Scope: http(s):// only. Scheme-less www. literals have the same flaw but
cannot be closed with <…>, which requires a scheme — noted in the code.

Dependencies

unified and remark-parse become declared dependencies of website/. Both
were already resolved in the tree as react-markdown transitives, so
package-lock.json gains 2 lines and no new package is downloaded.

Tests

website/src/test/cjkAutolinkBoundaries.test.tsx — 34 tests in four groups:

• cuts on evidence: the reported case, each bracket family, an opener that
  sits earlier on the line rather than adjacent, a second URL whose opener sits in
  the prose INSIDE the same greedy run, every
  non-bracket punctuation class with a markdown-active follower, emphasis and
  link followers, contiguous punctuation runs, multi-URL runs, GFM tail trimming,
  balanced ASCII parens.
• refuses to cut without evidence: the three live CJK-titled article URLs,
  the CJK-titled URL with a code span attached in the same run, the
  bare-sentence limitation, every protected region (fenced, blockquoted fenced,
  fence-content-that-looks-like-a-closer, indented, quoted-indented,
  tab-indented, mixed-space-tab-indented, single-line and multi-line inline
  code, math, existing links/images/autolinks/definitions), and the two
  must-not-linkify hosts, the three URL shapes that carry * / [ / ]
  legitimately, three shapes where the only （ lives in a non-prose region
  (earlier URL query, inline code, HTML attribute), and three unmatched-closer
  shapes (no opener at all,
  a different bracket type, and an opener already closed before the URL). Two
  positive controls prove prose inside a blockquote and inside an indented list
  continuation is still fixed.
• pipeline order: a glued fence opener (text```sh) proves the rewrite must
  run after fixCodeFences, asserting both that the URL survives and that no
  <https:// is inserted.
• rendered output: href is clean, both code spans come back
  (container.textContent contains no literal backtick), and the
  モーニング娘。 article still resolves to the right percent-encoded page.

Manual verification

N/A — the defect is textual (href value plus code-span pairing) and the
rendered-DOM assertions pin both exactly. No UI surface changed.

[github-actions]

UX Review (Fable 5) — ✅ PASS

Advisory UX-level review of 5bef225fdffd29999d13299f1df2e6e781d61a1d — updated in place on each push; does not block merge.

UX-Verdict: PASS

Invisible-plumbing fix that only repairs broken renders: CJK chat messages stop corrupting, screenshots prove both themes, and legitimate URLs like …/wiki/苹果（公司） survive untouched.

[UX-REVIEWED] 5bef225

[github-actions]

Design Review (Fable 5) — ✅ PASS

Advisory design-level review of 5bef225fdffd29999d13299f1df2e6e781d61a1d — updated in place on each push; does not block merge.

Design-Verdict: PASS

Real, evidenced rendering corruption fixed at the only layer that can restore code-span pairing, with conservative evidence-gated cuts, measured rejection of the aggressive alternative, and clean reversibility.

Suggestions

• The ~330-line boundary subsystem is self-contained (own parser, own constants, exported pure function); extracting it from the 2400-line MarkdownRenderer.tsx into its own module would keep the renderer a composition seam rather than a heuristics host — mechanical follow-up, not a blocker.

[DESIGN-REVIEWED] 5bef225

[github-actions]

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 5bef225fdffd29999d13299f1df2e6e781d61a1d and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 5bef225

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 5bef225fdffd29999d13299f1df2e6e781d61a1d: <one-sentence reason>

[github-actions]

Opus 4.8 Review — ✅ no blocking findings

Reviewed 5bef225fdffd29999d13299f1df2e6e781d61a1d — this comment is updated in place on each push.

Review details

I've reviewed the diff thoroughly. Let me verify one correctness-critical aspect: that the module-level AUTOLINK_PARSER.parse() actually produces GFM autolink-literal nodes (the whole fix depends on it), and that insert ordering can't be violated.

The rendered-output tests (lines 861-886) actually render through the pipeline and assert on href/code element contents, which would fail in CI if .parse() didn't apply the gfm micromark/fromMarkdown extensions — so that dependency is validated by the gated suite. Insert ordering is guaranteed by DFS pre-order traversal yielding literals in ascending offset, with consumedTo guarding nested https:// and the two per-URL inserts (< at at, > at at+head.length) being monotonic within a run.

I examined:

• cjkCutIndex bracket-balance and separator/sentence-ender logic — bounded loops, no out-of-range.
• prosePrefix/nonProse masking — line-scoped, in-bounds fills.
• The sourcePos gate correctly skipping the fix to preserve data-sourcepos column accuracy.
• Pipeline order (fixCodeFences before fixCjkAutolinkBoundaries) matching the documented rationale and the pipeline-order test.
• Parse-failure fallback returning empty spans (unfixed render, not a crash).
• frontend-security blocking rule: no innerHTML, no dangerouslySetInnerHTML, no HTML-string linkification — the fix emits markdown source (<url>) re-parsed by the sanitized pipeline.

No semantic defect completes a consequence chain to user- or system-visible harm on the changed lines.

No findings.

[OPUS-REVIEWED] 5bef225

Verdict parsed from the review's SHA-scoped output markers for commit 5bef225fdffd29999d13299f1df2e6e781d61a1d.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 5bef225fdffd29999d13299f1df2e6e781d61a1d: <one-sentence reason>

[CrysisDeu]

Both blocking findings on 4f3404d598cb5a1e6e4f069e4c985f8ad0267ac9 were legitimate. Fixed in 03002a87.

1. Valid punctuation inside URL paths is truncated — fixed.
https://ja.wikipedia.org/wiki/モーニング娘。 is a real article and the ideographic full stop is part of the title; so are …/wiki/苹果（公司） and …/wiki/我，机器人. The unconditional character-class cut was wrong.

Replaced with an evidence-based cut (cjkCutIndex), which requires one of:

• a CJK closing bracket with no opener inside the run — GFM's own ASCII paren-balancing rule generalised, so （https://x.com/a） cuts but …/苹果（公司） does not;
• CJK punctuation followed, later in the same run, by a markdown-active character (` * [ ]) — the destructive case, and the one where "this is prose" is actually evidenced.

Accepted residual, locked in by a test rather than left implicit: …/pull/1，然后回来 is not cut. It is character-for-character indistinguishable from …/wiki/我，机器人, so it keeps today's behaviour instead of risking a correct link. Three live CJK-titled article URLs are now regression tests, including a rendered-DOM assertion that モーニング娘。 still resolves to the right percent-encoded page.

2. Source rewriting corrupts inline-comment coordinates — fixed.
fixCjkAutolinkBoundaries is now bypassed when sourcePos is enabled, exactly as suggested. That surface keeps the unfixed but coordinate-accurate render.

Also fixed — Design Review suggestion (advisory). FENCE_OPEN_RE never matched inside a blockquote, so a URL in a > ```` fenced block was rewritten — the conservative bias inverted. Blockquote markers are now stripped before block detection. Finding the bug also surfaced that the prefix pattern must require an actual >: a {0,3}-space-only match ate the indent that identifies an indented code block, which a > `-quoted-indented-code test now covers.

Local gates on 03002a87: tsc -b clean, 19/19 in the new suite, 10857 passed across the full frontend suite (the one failure, src/i18n/format.test.ts asserting Intl.DurationFormat is undefined, is a local Node 24 artifact — this PR's Frontend Tests check was green on CI's Node).

[CrysisDeu]

Both blocking findings on 03002a8790bd493849dbe43f0fc23b5cea40c963 were legitimate. Fixed in fce708ac.

1. Valid CJK URLs truncated before later markup — fixed.
The evidence test scanned the whole remainder of the run, so https://zh.wikipedia.org/wiki/我，机器人 with a code span attached (…我，机器人\简介`) cut at the comma and pointed the anchor at /wiki/我`.

cjkCutIndex now requires the markdown-active character to sit directly after the punctuation, as suggested. Refinement over the literal suggestion: it walks the contiguous punctuation run first, so …/a、，\c`is one boundary and the cut lands at its **start** rather than leaving、 inside the href. The reported case still cuts (，is immediately followed by a backtick);我，机器人`简介`` no longer does, and is now a regression test.

2. Tab-indented code rewritten — fixed.
/^ {4,}\S/ counted characters, so a leading tab — already 4 columns under CommonMark's multiple-of-4 tab stops — was treated as prose. Replaced with an indentWidth() helper that expands tabs to the next stop, plus an explicit blank-line guard the old \S anchor was providing. Tests cover \t and the mixed \t case, alongside the existing 4-space and > -quoted variants.

Local gates on fce708ac (rebased onto current main): tsc -b clean, 21/21 in the new suite.

[CrysisDeu]

Both blocking findings on fce708ac3fc52b9fe24d2481920d502071905b14 were legitimate. Fixed in 16d6942b — but not by patching the two cases, because they were the third consecutive round of the same class.

Rounds 1-3 each surfaced two real defects, all in one place: the hand-rolled maskNonAutolinkRegions scanner re-deriving CommonMark block and inline rules. Tab-stop indentation, blockquoted fences, multi-line code spans, closing-fence validity — that list has no natural end, so the scanner is the wrong shape, not the individual cases.

Structural fix: read the protected regions off remark's own parse.
maskNonAutolinkRegions, indentWidth, and the four region regexes are deleted. autolinkLiteralSpans() parses the source with the same plugin set the render pipeline uses (remark-parse + remark-gfm + remark-math with singleDollarTextMath: false) and returns the source offsets of GFM autolink-literal nodes only. Anything inside a fence, an indented block, an inline-code span (multi-line included), an existing link or image, an angle autolink, a raw HTML tag, or a math span never becomes such a node, so it is now unreachable by construction instead of by a mask that has to be right about every block rule.

• Finding 1 (multi-line code spans) — fixed. The parser pairs a `` opener with a closer on a later line; a per-line mask cannot. Regression test added.
• Finding 2 (invalid fence closers) — fixed. ~~~not-a-close is fence content, and the parser knows it. Regression test added.
• Angle autolinks and [text](url) links can also satisfy text === url, so the literal test is on the source text at the node's start, not on node shape alone.
• Parse failure returns no spans, so a malformed message renders unfixed rather than not at all.

Two bonus effects worth flagging: a math span is now protected (it was not before), and the old mask's conservative "blank any 4-column-indented line" rule no longer skips list continuation paragraphs — both are tests.

unified and remark-parse are now declared dependencies. Both were already resolved in the tree as react-markdown transitives, so package-lock.json gains 2 lines and no new package is downloaded.

Local gates on 16d6942b (rebased onto current main): tsc -b clean, 26/26 in the new suite, eslint no new findings. Full frontend suite: 10867 passed. Two unrelated failures, neither in a touched file and both passing standalone: src/i18n/format.test.ts asserts Intl.DurationFormat is undefined (true on CI's Node, not on local Node 24), and src/test/App.test.tsx's foreign-agent import gate times out on findByRole only under full-suite parallelism.

[CrysisDeu]

The blocking finding on 0219fbdef7dd0e1c8082ce30763c3d6cdfda5afe — "backtick heuristic truncates valid CJK URL suffixes", https://ja.wikipedia.org/wiki/モーニング娘。\紹介`→ anchor targets/wiki/モーニング娘— is **accepted as legitimate**. Resolved infd85640` by narrowing rather than removing the rule, per the author's decision.

Why not the literal suggested fix. The suggestion was to drop backtick adjacency as boundary evidence entirely. That would remove the fix for the case this PR exists to solve: the reported message is …/pull/2137，\96ed647`）, whose boundary IS a CJK mark immediately followed by a backtick. Only the bracket rule would remain, and it does not cover ，`hash``.

What changed. Rule 2 now applies only to separator-class marks. Sentence-enders 。．！？…｡ are excluded, because real page titles end in them and reach the URL raw — …/wiki/モーニング娘。, …/wiki/魔法先生ネギま！, …/wiki/そして誰もいなくなった…. Separators (，、、、；、：、·、～、“) do not end titles, so they stay eligible.

A mixed run resolves the conservative way with no extra code: the loop skips the sentence-ender and reaches the separator on a later iteration, so …/a。，\c`yields<…/a。>，`c`— the possibly-title。` stays inside the href.

Accepted residual, locked in by a test rather than left implicit. A genuine prose 。 directly before a code span now keeps today's behaviour. That is the same class as the already-documented …/pull/1，然后回来: character-for-character indistinguishable from a legitimate title, so the safe direction is not to cut.

Discriminating power verified. Removing the one-line narrowing makes exactly the two new tests fail (leaves a sentence-ender alone…, cuts at the separator in a run that starts with a sentence-ender) and leaves the other 32 passing — so the tests measure the narrowing, not just the surrounding behaviour.

Also relevant to this thread. @streamdown/cjk@1.0.3 (Vercel's Streamdown) ships autolink handling for the same problem and takes the aggressive side of exactly this trade: it splits at the first character in a fixed 20-mark set with no evidence test. Measured against it — it breaks …/wiki/苹果（公司）, …/wiki/モーニング娘。 and …/wiki/我，机器人, cannot repair code-span pairing because it runs after tokenization, and mis-splits explicit <…> autolinks. Independent evidence that the unconditional cut is the wrong default.

Rebased onto current main (level, 0 behind). Local gates on fd85640d: tsc -b clean, 34/34 in this suite, full frontend suite 877 files / 11829 tests, zero failures, eslint no new findings.