NFC_MAX_CODEPOINTS is LBUF_SIZE * 2, so NFCCodePoint cps[] is
65,536 entries of 8 bytes (UTF32 cp; int ccc) -- 524,288 bytes, and
the whole of the function's 524,496-byte frame. It is the slow path
of a routine every piece of string handling reaches, so entering it
cost half a megabyte of stack.
Frame drops 524,496 -> 288 bytes.
This is the largest fixable frame outside the engine. The larger ones
in libmux are all in color_ops.c -- co_splice at 589,856, co_reverse at
524,304, four more at 262,208 -- and those cannot be fixed this way:
mux/rv64/Makefile compiles exactly three objects into the freestanding
blob (softlib.o, color_ops.o, unicode_tables.o), which has no malloc
and no .bss. a7a816ea8 left co_reverse's 512 KB alone for that reason.
utf8_normalize.cpp is not one of the three, so it may use .bss.
Safe as a plain function-local static, with no claim guard, because
there is exactly one claim site: utf8_normalize_nfc_slow neither
recurses (zero self-calls) nor evaluates softcode, and none of
DecomposeOne, CanonicalOrder or CanonicalCompose call back into it, so
no two activations are ever live on one thread at once.
make test: 32 passed, 1 skipped (stubslave, not configured), 0 failed.
Coverage here is indirect -- there is no unit test for normalization in
tests/ -- but the combining-character cases reach this path: graphemes
26 cases, comp 33, chr 13, all succeeding. The change is a storage
class only; no logic is touched.
utf8_is_nfc() ran GetCCCandNFCQC() -- a DFA traversal -- for every code
point, including plain ASCII. net.cpp:436 calls utf8_is_nfc() on every
command line from every player, and those lines are overwhelmingly ASCII.
Every code point U+0000..U+007F has CCC = 0 and NFC_QC = Yes, so for ASCII
the traversal cannot change the answer. Verified exhaustively against this
file's own tables rather than from outside knowledge: all 128 return
ccc = 0, qc = 0, while 114 of U+0080..U+03FF do not, so the property is
specific to ASCII rather than vacuously true of everything.
utf8_is_nfc() on pure-ASCII input, aarch64, best-of-7, 300k iterations:
len 16 90.4 ns -> 12.1 ns
len 43 228.9 ns -> 23.7 ns
len 200 1035.4 ns -> 100.3 ns
len 512 2640.3 ns -> 233.1 ns
Roughly 5.2 ns/char before, 0.5 ns/char after.
Behaviour-preserving: a 1,000,000-input differential fuzz (ASCII, combining
marks, Latin precomposed, Hangul syllables and jamo, CJK, SMP, injected
invalid UTF-8) produces byte-identical output before and after, and the ICU
72.1 cross-check reports the same count on both sides.
test_unicode_icu 51/51, smoke 1596/0, and test-ganl, test-netaddr, test-dbt
and test-comsys-conformance all pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CanonicalCompose() applied the UAX #15 blocking rule with an extra clause
that exempted exactly the case that must be blocked:
bool blocked = (lastCCC != -1 && lastCCC >= ccc && ccc != 0);
D115 states that B is blocked from starter A if there is some C between
them with CCC(C) = 0 or CCC(C) >= CCC(B). There is no exemption for
CCC(B) = 0 -- the opposite holds, since when B is itself a starter every
intervening mark satisfies CCC(C) >= 0 and blocks it. The `ccc != 0`
clause therefore removed protection from precisely those sequences.
NFC consequently composed Hangul jamo across an intervening mark:
U+B3C4 U+032B U+11C1 normalized to U+B3DE U+032B, consuming the jongseong
and changing the syllable. ICU 72.1 and Python unicodedata both leave the
sequence alone, as does the rule applied by hand.
lastCCC is only ever assigned from a non-zero ccc, so `lastCCC != -1`
already means an intervening mark was seen; adjacent starters still
compose normally.
Verified by differential fuzz against ICU 72.1 over 1,000,000 randomized
inputs (ASCII, combining marks, Latin precomposed, Hangul syllables and
jamo, CJK, SMP, injected invalid UTF-8), deterministic corpus:
master bb28a852c 904,353 compared, 54 mismatched
with fix 904,353 compared, 0 mismatched
The 54 is the negative control -- the harness demonstrably fires on the
unfixed build. test_unicode_icu 51/51, smoke 1596/0, and test-ganl,
test-netaddr, test-dbt and test-comsys-conformance all pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
utf8_normalize_nfc() declared its NFD working buffer as a local:
NFCCodePoint cps[NFC_MAX_CODEPOINTS]; // LBUF_SIZE*2 * 8 = 512 KB
A frame that large makes GCC's -fstack-clash-protection -- enabled by
default on Debian/Ubuntu, so it appears in no CFLAGS listing -- emit a
prologue loop that touches all 128 of its 4 KB pages on entry:
lea -0x80000(%rsp),%r11
loop:
sub $0x1000,%rsp
orq $0x0,(%rsp)
cmp %r11,%rsp
jne loop
A prologue runs before any early-out in the body, so the already-NFC
fast path at the top of the function paid all 128 probes for a buffer it
never touches. That path is the common case: chr() output, and every
command line from every player via net.cpp:439.
Measured on a live server with perf (999 Hz, DWARF unwind): a single
+jobs cost 1.53 CPU-seconds, of which utf8_normalize_nfc was 51.19% --
and perf annotate put 92.98% of that function's samples on the single
`cmp %r11,%rsp` above. The time was in the prologue, not in
normalization; utf8_is_nfc did not appear in the profile at all.
Move the buffer into a noinline slow-path helper so only calls that
actually normalize pay for it. Microbenchmark (2M iterations):
already-NFC 1 char 633.9 ns -> 15.7 ns (~40x)
already-NFC 44 chars 886.9 ns -> 340.7 ns (~2.6x)
decomposed (slow path) 658.8 ns -> 669.1 ns (unchanged, within noise)
NFC_MAX_CODEPOINTS is deliberately unchanged: LBUF_SIZE * 2 is the
worst-case decomposition bound and lowering it would be a correctness
regression.
Verified with testcases/tools/test_unicode_icu against ICU 74.2 --
51 passed, 0 failed, identical before and after. Smoke: 1594 passed,
1 failed both before and after (TC003 cansee, which needs WOD_REALMS or
REALITY_LVLS compiled in; unrelated to this change).
The same shape appears elsewhere -- 103 functions in the shipped
binaries carry probe loops, 14 at >=256 KB and three at ~1.5 MB
(fun_strunion/strinter/strdiff). Those are unmeasured and left alone
here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Merge the two separate DFA lookups (Canonical Combining Class and
NFC Quick Check) into one combined DFA that returns ccc*3+nfcqc.
This eliminates one full DFA traversal per non-ASCII codepoint
in utf8_is_nfc.
Pipeline: gen_ccc_nfcqc.pl merges tr_ccc.txt + tr_nfcqc.txt,
integers builds the combined tr_ccc_nfcqc DFA table. Fully
automated via Makefile — no hand-edited data files.
GetCCC still works via combined/3. GetCCCandNFCQC returns both
values from a single traversal for the is_nfc hot path.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
