SaveWidths() in buildFiles.cpp now checks General_Category before
East Asian Width: Mn/Me (nonspacing/enclosing marks) and Cf (format
chars like ZWJ, ZWNJ, ZWSP) are explicitly zero-width instead of
inheriting width 1 from their EAW property.
Move string_desc typedef from generated utf8tables.h into externs.h
so the utf/ pipeline is fully self-contained. Update Makefile.in to
auto-generate file preambles (header guards, includes) for both
utf8tables.h.txt and utf8tables.cpp.txt.
Regenerate DFA tables from corrected tr_widths.txt. Update sort_fn
test hashes to match the new collation-aware width tables.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Update CL_PRINT DFA to accept Unicode combining marks (Mn, Mc, Me
categories) instead of rejecting them at the network boundary. This
allows composed input like e + U+0301 to pass through to the normalizer.
Hook utf8_normalize_nfc() into save_command() so all UTF-8 input is
normalized to NFC before entering the command queue. The quick check
(utf8_is_nfc) short-circuits for ASCII-only input with zero overhead.
Regenerated CL_PRINT DFA: 155,134 code points included (was 154,200),
314 states / 91 columns (was 335 / 93).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace UnicodeData.txt and EastAsianWidth.txt with Unicode 16.0 (2024)
data files and regenerate all DFA state machines. This adds 18,181 new
printable code points and 156 new case mapping entries.
Add Windows build support for the utf/ pipeline tools (autoconf_win.h,
build_win.cmd) and guard iconv-dependent code in buildFiles.cpp with
#ifdef HAVE_ICONV_H so the core pipeline runs without iconv on Windows.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
