Components: Fix FormTokenField validation preventing default behavior

[rushikeshmore]

What?

Fixes #69252

Why?

FormTokenField with __experimentalValidateInput and tokenizeOnSpace has two bugs:

1. Space key always prevented: When typing a value that fails validation and pressing space, addCurrentToken() sets preventDefault = true before knowing whether addNewToken() succeeded. This prevents the space character from being typed into the input, making it impossible to type multi-word values that don't yet pass validation.

2. Paste bypasses validation: When pasting comma/space-separated values, addNewTokens() (called via onInputChangeHandler) never runs tokens through __experimentalValidateInput, allowing invalid tokens to be added.

How?

• addNewToken() now returns boolean (true on success, false on validation failure)
• addCurrentToken() uses the return value to decide preventDefault for typed input (selected suggestions still always prevent default)
• addNewTokens() filters tokens through __experimentalValidateInput

Testing Instructions

1. Render a FormTokenField with tokenizeOnSpace and __experimentalValidateInput={ (v) => /^[A-Z]/.test(v) }
2. Type hello (lowercase, space) → the space should appear in the input (not tokenized)
3. Type Hello (uppercase, space) → should create a "Hello" token
4. Type Apple,banana,Cherry, → only "Apple" and "Cherry" should be added as tokens

Testing Instructions for Keyboard

• Verify space key types normally when validation fails with tokenizeOnSpace
• Verify Enter key still works for valid tokens
• Verify comma key still prevents default (comma is always a separator)

Use of AI Tools

Claude Code was used to assist with this bug fix. All changes were reviewed, tested, and verified manually.

[github-actions]

👋 Thanks for your first Pull Request and for helping build the future of Gutenberg and WordPress, @rushikeshmore! In case you missed it, we'd love to have you join us in our Slack community.

If you want to learn more about WordPress development in general, check out the Core Handbook full of helpful information.

[github-actions]

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: rushikeshmore <rushikeshmore@git.wordpress.org>
Co-authored-by: ciampo <mciampini@git.wordpress.org>
Co-authored-by: Mamaduka <mamaduka@git.wordpress.org>
Co-authored-by: alshakero <alshakero@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

[ciampo]

Some tests are still failing, indicating that the Space key fix is insufficient: onInputChangeHandler still tokenizes on space, clearing the input even when validation fails (see packages/components/src/form-token-field/index.tsx:287–298)

More in general, I appreciate the willingness to help the project. At the same time, please make sure to check the proposed changes and their correctness (such as having related tests passing) before asking for review. Especially now with AI tools assisting, review time is precious.

[rushikeshmore]

Fixed in 27aad02. Two issues addressed:

1. onInputChangeHandler now preserves tokens that fail validation in the input instead of clearing them. When separator-delimited text is processed, only valid tokens are removed from the input -- invalid ones stay visible.

2. Paste test was using user.type which sends comma keystrokes through handleCommaKey (in onKeyPress), not through onInputChangeHandler. Switched to user.paste which correctly exercises the separator-splitting logic in onInputChangeHandler.

All 70 tests pass locally. Sorry about the earlier incomplete fix -- I should have traced both code paths (onKeyDown + onInputChangeHandler) and verified tests before pushing.

[ciampo]

Things are mostly looking good. Apart for the last comment, we'll also need to rebase on top of latest trunk and fix CHANGELOG conflicts.

[rushikeshmore]

Thanks for the careful review @ciampo. Force-pushed:

Duplicate edge case: Carved duplicates out of the failed-segments filter as you suggested. A segment now only stays in the input when it actually fails __experimentalValidateInput. Empty transforms, segments accepted earlier in the same batch (addedTokens), and segments already present in value (via valueContainsToken) are intentional skips, not failures.

Added the two regression tests from your diff (with and without a custom validator). All 74 tests in form-token-field pass locally.

CHANGELOG: Moved the entry to Unreleased > Bug Fixes and reworded to your suggested phrasing.

Rebase: Rebased on top of latest trunk and resolved the CHANGELOG conflict.

[ciampo]

I also suggest rebasing again, in case it fixes the Static Analysis CI check failures.

[rushikeshmore]

Rebased on trunk (5 commits preserved, CHANGELOG conflict resolved). Hopefully that clears the static-analysis failures. Let me know if anything else needs adjusting.

[ciampo]

@rushikeshmore I believe there is still unaddressed feedback from the last round of review

[rushikeshmore]

@ciampo apologies for missing the round-2 feedback in the rebase. Just pushed a commit addressing all four threads:

• Paste rejoin refactor with willFailValidation precheck, commits trailing valid segment, ',' rejoin
• 'all segments fail on paste' test (your suggested test verbatim)
• 'commit trailing valid segment' test for the no-trailing-separator case
• wasAdded -> passedValidation rename + JSDoc on addNewToken
• handleCommaKey comment expanded to your suggested wording

One side-effect noted in the thread reply for the rejoin: the tokenizeOnSpace+typing test had to drop its 'hello ' assertion to 'hello' since the trailing empty tokenValue no longer round-trips back. Same trailing-separator philosophy as your paste example. Let me know if you'd rather preserve the trailing in the typing path.

[rushikeshmore]

Pushed a follow-up commit reworking the rejoin per your suggestion (separator derived from text, trailing preserved), reverting the three test assertions, and dropping the explicit boolean return on addNewToken.

[ciampo]

LGTM 🚀

Thank you for working on this!