A file with a PRE-EXISTING stray unbalanced '"' -- 1990s mudlib archives
ship many that never compiled -- was silently rewritten into garbage:
past the stray quote the tokenizer's string/code sense inverts, so real
string content lexes as code tokens (every CJK character a separate
'unknown' token the formatter space-separates, '\n' escapes torn into
'\ n') and everything the formatter renders from those tokens shreds.
200+ real files across a 91-mudlib corpus scan were corrupted this way.
Driver-lexer ground truth (src/compiler/internal/lexer.l): hitting EOF
inside a string, template literal, block comment, char literal, or text
block is a hard lexerror ("End of file in string" / "End of file in a
comment" / "End of file in template literal" / lpc_lex_char_error /
heredoc-terminator error). Such a file has NO well-defined token
stream, so per the documented safety contract (docs/lpc/formatter.md)
the only correct behavior is refusal: report, leave byte-identical,
exit nonzero. No speculative "fixed" formatting is invented.
Root cause of the silence: the corpus safety net (token-sequence
equivalence + literal byte-identity + idempotency) re-tokenizes the
output with the same tokenizer, and input and output mis-lex
IDENTICALLY -- the same self-check blind spot as the "(::" (d64c3fa4)
and "'''" (fd3f8b53) fixes; neither of those branches affects this bug
(verified: shredding reproduces with both merged). Unlike those two,
this is not a tokenizer decision that can be corrected -- the input
itself is lexically meaningless past the stray quote -- so the fix is a
new gate rather than a lexing change:
- tokenizer.mjs: any spanning token that reaches end-of-input without
its terminator (string/char/template/'/*' comment/text block) is
emitted with `unterminated: true`, mirroring the driver's <<EOF>>
lexerrors one-for-one. Directive-embedded unterminated quotes stay
unflagged (line-bounded by design; legal per the driver).
- format.mjs: formatLPC throws up front when any token carries the
flag ("unterminated string starting at line N ... refusing to
format"), which format-corpus.mjs already turns into FORMAT ERROR +
leave-untouched + nonzero exit. This closes the whole class for this
failure mode: no token comparison is involved, so identical mis-lex
on both sides can no longer pass.
- bin/format-corpus.mjs: the idempotency re-format of the candidate
OUTPUT is now inside a try -- a throw there is a per-file refusal,
not a crash of the whole run (input passing the gate does not prove
the output would).
- testsuite/format.sh: exclude the three deliberately-unterminated
EOF-lexerror fail fixtures (eof_in_string / eof_in_comment /
bad_at_block) alongside the two raw-byte bad-UTF-8 ones -- their
brokenness is the point and the formatter now refuses them by
design. --check goes from 779 to 776 files; errors (0) and the one
pre-existing wouldChange (testsuite/tmp_eval_file.c) are unchanged
vs master.
- test.mjs: regression tests -- a minimal unbalanced-quote fixture in
the real-world shape (missing close quote on a key string, CJK + \n
escape content following) must flag exactly the EOF-swallowing
string token and make formatLPC throw, its balanced sibling must
still format; all five EOF-swallowing constructs are refused the
same way, terminated siblings unflagged. The old "formatter is
stable on a source that swallows to EOF" check asserted the
superseded behavior and now asserts the refusal. All three fail
against the pre-fix code (verified by swapping master's files in).
- docs/lpc/formatter.md, README.md: document the clean-lex gate as
safety guarantee #1 and the fixture exclusions.
Verified on the three sample corruptions from the corpus scan
(beimeixiakexing2001 qiyuan3.lpc, xinkuangxiangkongjian2
god_weapon.lpc, xiyangzaixian3 vendor_sale.lpc -- 57/387/363 double
quotes, all odd counts): master shreds all three and reports the run
clean; with this fix all three are refused with FORMAT ERROR, left
byte-identical, exit 1.
node tools/lpc-syntax/test.mjs: all pass.
testsuite/format.sh --check: 776 files, 0 errors, 1 pre-existing
unrelated wouldChange (testsuite/tmp_eval_file.c, identical to
master's 779-file baseline modulo the three by-design exclusions).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VyQCUoTo1Z93Py9aVFHQi1
3bfb7483fe