buildFiles.cpp now loads SpecialCasing.txt and merges the 103 unconditional
full case mappings into tr_toupper/tolower/totitle.txt. This adds
length-changing mappings like ß→SS, ligature expansions (ff→FF, ffi→FFI),
and Greek iota-subscript uppercasing — all via the existing DFA/OTT pipeline.
The DFA tables grew (toupper OTT: 158→230, SBT type widened to unsigned short)
so iState was widened from unsigned char to unsigned short in stringutil.h and
unicode_tables_c.h to prevent wrap-around on states exceeding 255.
New standalone test harness in tests/color_ops/ exercises all 51 co_* functions
with deterministic edge cases, color interleaving, Unicode, and 20,000 fuzz
iterations checking round-trip invariants.
593/593 smoke tests pass. 118/118 co_* tests pass.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
SaveWidths() in buildFiles.cpp now checks General_Category before
East Asian Width: Mn/Me (nonspacing/enclosing marks) and Cf (format
chars like ZWJ, ZWNJ, ZWSP) are explicitly zero-width instead of
inheriting width 1 from their EAW property.
Move string_desc typedef from generated utf8tables.h into externs.h
so the utf/ pipeline is fully self-contained. Update Makefile.in to
auto-generate file preambles (header guards, includes) for both
utf8tables.h.txt and utf8tables.cpp.txt.
Regenerate DFA tables from corrected tr_widths.txt. Update sort_fn
test hashes to match the new collation-aware width tables.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Update CL_PRINT DFA to accept Unicode combining marks (Mn, Mc, Me
categories) instead of rejecting them at the network boundary. This
allows composed input like e + U+0301 to pass through to the normalizer.
Hook utf8_normalize_nfc() into save_command() so all UTF-8 input is
normalized to NFC before entering the command queue. The quick check
(utf8_is_nfc) short-circuits for ASCII-only input with zero overhead.
Regenerated CL_PRINT DFA: 155,134 code points included (was 154,200),
314 states / 91 columns (was 335 / 93).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace UnicodeData.txt and EastAsianWidth.txt with Unicode 16.0 (2024)
data files and regenerate all DFA state machines. This adds 18,181 new
printable code points and 156 new case mapping entries.
Add Windows build support for the utf/ pipeline tools (autoconf_win.h,
build_win.cmd) and guard iconv-dependent code in buildFiles.cpp with
#ifdef HAVE_ICONV_H so the core pipeline runs without iconv on Windows.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
