1. Fix pairs DFA generator (pairs.cpp): always resolve undefined
transitions before optimization. Without this, MergeAcceptingStates
and RowsEqual collapse states incorrectly, causing NFC to
over-compose and DUCET contractions to false-match.
2. Regenerate NFC compose tables (129→1010 states) and DUCET
contraction tables (27→294 states) with the corrected generator.
The reglattr sort test was validating wrong order (BETA before
ALPHA) — fixed to match correct DUCET primary ordering.
3. Port NFC tiebreaker to collation: normalize to NFC before binary
comparison so canonically equivalent strings compare equal per UCA.
In TinyMUX all data is already NFC so this path rarely fires.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- color_ops.h: Add LIBMUX_API to all co_* declarations; fix CO_CS_NORMAL
compound literal for MSVC (both C and C++ modes)
- color_ops.c: Remove redundant (co_ColorState) casts that fail on MSVC C
- utf8tables.h/cpp: Add extern "C" wrapping for C/C++ linkage compatibility
- stringutil.h/cpp: extern "C" on utf8_FirstByte; LIBMUX_API on co_console_width
- unicode_tables_c.h: Guard extern declarations with #ifndef __cplusplus
(C++ gets them from utf8tables.h with proper LIBMUX_API and types);
add CO_OTT_CAST macro for string_desc/co_string_desc pointer conversion;
add proper LIBMUX_API fallback with dllexport/dllimport for Windows
- libmux.vcxproj: Add color_ops.c and unicode_tables.c as C compilation units
- ast.cpp: Use QueryPerformanceCounter for WIN32 astbench timing
- engine_com.cpp, functions.h, functions.cpp: Guard JIT-only code with TINYMUX_JIT
- utf/ generators: Add LIBMUX_API to all extern declarations in smutil.cpp,
strings.cpp, pairs.cpp; strings.cpp guards OTT externs with #ifndef __cplusplus;
preamble/postamble updated with CO_OTT_CAST and proper LIBMUX_API fallback
NOTE: Build not yet clean — ubuntu agent needs to regenerate auto-generated
files (utf8tables.h, utf8tables.cpp, unicode_tables_c.h) from updated
utf/ generators to pick up all LIBMUX_API and linkage fixes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add -o/-i flags to classify, integers, strings, and pairs tools so
they write to caller-specified paths instead of hardcoded .txt files.
Add -c flag to strings for C-mode output (co_string_desc, C casts).
Rewrite utf/Makefile.in to generate all output files directly in
their final locations:
mux/include/utf8tables.h — C++ header (libmux.so)
mux/lib/utf8tables.cpp — C++ table data (libmux.so)
mux/include/unicode_tables_c.h — C header (color_ops, rv64)
ragel/unicode_tables.h — symlink to unicode_tables_c.h
ragel/unicode_tables.c — C tables (case mapping)
mux/rv64/src/unicode_tables.c — C tables (case + widths + gcb)
The C header is assembled from preamble + generated DFA declarations +
postamble (inline DFA runner functions). No manual sync needed — the
upper_sbt debacle cannot recur.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Introduce pairs.cpp, a new utf/ pipeline tool that builds a DFA state
machine from two-code-point input sequences. The UTF-8 bytes of both
code points are concatenated and fed through the standard DFA builder,
with an indirection table mapping accepting states to result code points.
This is a fourth mapping type for the pipeline:
1. classify: code point -> {0,1} (set membership)
2. integers: code point -> integer
3. strings: code point -> code point sequence
4. pairs: (code point, code point) -> code point (NEW)
964 canonical NFC composition pairs produce a 1010-state DFA (~19 KB)
plus a 964-entry result table (~4 KB). Every pair is exhaustively
verified during generation.
Also adds gen_compose.pl to extract canonical composition pairs from
UnicodeData.txt (excluding Hangul and CompositionExclusions), and
Unicode 16.0 CompositionExclusions.txt.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
