fix(lpc-syntax): tokenizer mis-lexes unescaped quote-char literal '''

The unescaped quote character literal ''' -- an old MudOS-ism common
in 1990s Chinese mudlibs (`case ''':` for the say-shortcut quote key)
-- was mis-tokenized, and on the real-world shape

    case ''':	//'
        cmd = "say " + cmd[1..];

(the trailing //' comment's quote balances editors' highlighting)
the formatter merged the case label, the comment, AND the next line's
statement into one output line:

    case '' ':	//' cmd = "say " + cmd[1..];

The char literal came out torn in two (`'' '`), and the re-flowed `//`
comment now swallowed the following assignment -- on recompile the
driver comments out the statement, silently deleting it.

Driver-lexer ground truth (src/compiler/internal/lexer.l): a char
literal's body is EXACTLY ONE unit -- <SC_CHAR_BODY>[^\\] matches any
single raw byte, explicitly "including a literal quote" per the rule's
own comment, or one escape sequence -- and then <SC_CHAR_CLOSE>
requires the closing quote. So ''' is a VALID literal (body = the
quote char, value 39): the mudlib containing this file compiled and ran
natively before formatting. If the closing quote is missing, the driver
reports an error and pushes the offending byte back (LPC_YYLESS(0)) for
the next scan.

Root cause: tokenizer.mjs's skipCharSpan() (and a duplicated inline
scan in the main loop) scanned "to the next quote" string-style. On
''' that terminates immediately, emitting an empty '' token, and the
leftover third quote then opens a bogus literal that runs to the next
quote ANYWHERE on the line -- here the one inside the trailing //'
comment -- producing char tokens `''` and `':<tab>//'`. The formatter
then laid out those tokens tight on one line, and everything after the
embedded // became comment text to the real compiler.

Fix: rewrite skipCharSpan() to mirror lexer.l's grammar exactly -- one
raw body byte (any byte, including a quote) or one escape (with the
variable-length forms scanned per lexer.l: "\x"[0-9a-fA-F]+ hex,
"\"[0-7]+ octal, "\<CR><LF>"/"\<LF>" escaped newline, otherwise
backslash + one char), then the closing quote; a missing close ends the
span there so the offending byte re-lexes, mirroring the driver's
push-back recovery. The main tokenizer loop now calls skipCharSpan()
instead of duplicating the old string-style scan; findInterpEnd() and
directiveLineEnd() pick up the corrected rule for free. The sibling
escaped forms ('\'', '\"', '\\', '\x41', '\101') lex identically
before and after.

Same self-check blind spot as the "(::" parent-call fix (d64c3fa4): the
corpus safety net re-tokenizes the output with the same buggy tokenizer,
mis-lexes both sides identically, and reports the corrupted file clean
(token-equivalent AND idempotent). Considered an extra assertion ("a
trailing // comment token never grows"), but it cannot catch this
class: in the mis-lexed token stream there IS no comment token -- the
//' sat inside the bogus char token on both sides, so any assertion
built on the same tokenizer's comment tokens inherits the blind spot.
Not added; the tokenizer fix is the real repair.

Found in the same ~91-real-world-mudlib corpus scan as d64c3fa4; the
in-repo testsuite corpus has no ''' occurrence, which is why it
slipped past test.mjs and testsuite/format.sh.

Adds a regression test: tokenizer-level (''' is ONE 3-char token;
escaped siblings unchanged) and formatter-level (the real-world switch
formats with the comment intact and both statements surviving on their
own comment-free lines, idempotently). The new test fails against the
pre-fix tokenizer (verified) and passes now.

node tools/lpc-syntax/test.mjs: all pass.
testsuite/format.sh --check: 781 files, 0 errors, 1 pre-existing
unrelated wouldChange (testsuite/tmp_eval_file.c) -- byte-identical
result with master's tokenizer swapped in (verified both ways).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VyQCUoTo1Z93Py9aVFHQi1

tools/lpc-syntax/test.mjs

@@ -812,6 +812,54 @@ check('keyword-tail safety net: a reserved word never has the next token'
         }
         return true;
       })());
+// The unescaped quote-char literal `'''` (an old MudOS-ism, valid per
+// lexer.l: <SC_CHAR_BODY>[^\\] matches ANY raw byte including a literal
+// quote, then <SC_CHAR_CLOSE> takes the third quote) used to mis-lex:
+// the tokenizer scanned "to the next quote" string-style, reading `'''`
+// as an empty `''` plus a stray `'` that opened a bogus literal running
+// to the next quote anywhere on the line. In one real 1990s mudlib
+// (`case ''':	//'` -- a trailing `//'` comment whose quote balances
+// editors' highlighting) that bogus literal swallowed the `:` and the
+// comment opener, and the formatter merged the case label, comment, and
+// the NEXT line's statement into one line -- the `//` then commented
+// out the statement on recompile, silently deleting it. Caught in the
+// same ~91-mudlib corpus scan as the "(::" bug; equally invisible to
+// the token-equivalence self-check (same mis-lex on both sides).
+check("unescaped quote-char literal `'''` lexes as ONE char token (lexer.l:"
+      + ' one raw body byte -- even a quote -- then the closing quote), so a'
+      + " `case ''':` with a trailing `//'` comment never swallows the"
+      + ' following statement',
+      (() => {
+        // Tokenizer ground truth first: one 3-char token, not ''+stray.
+        if (kinds("c = ''';").join(',') !==
+            "identifier:c,operator:=,char:''',punctuation:;") return false;
+        // The sibling escaped forms must be unaffected.
+        if (kinds("c = '\\'';")[2] !== "char:'\\''") return false;
+        if (kinds("c = '\\\"';")[2] !== "char:'\\\"'") return false;
+        // The real-world shape: the comment must stay a comment token and
+        // the next line's statement must survive formatting on its own line.
+        const src = 'void f(string cmd) {\n'
+                  + '\tswitch(cmd[0]) {\n'
+                  + "\t\tcase ''':\t//'\n"
+                  + '\t\t\tcmd = "say " + cmd[1..];\n'
+                  + '\t\t\tbreak;\n'
+                  + '\t\tcase \'\\"\':\t//"\n'
+                  + '\t\t\tcmd = "tell " + cmd[1..];\n'
+                  + '\t\t\tbreak;\n'
+                  + '\t}\n'
+                  + '}\n';
+        const out = formatLPC(src);
+        if (formatLPC(out) !== out) return false;
+        // The case label keeps its literal and its trailing comment...
+        if (!out.includes("case ''':  //'\n")) return false;
+        if (!out.includes("case '\\\"':  //\"\n")) return false;
+        // ...and the statements survive, NOT on a comment-bearing line.
+        for (const stmt of ['cmd = "say " + cmd[1..];', 'cmd = "tell " + cmd[1..];']) {
+          const line = out.split('\n').find((l) => l.includes(stmt));
+          if (!line || line.includes('//')) return false;
+        }
+        return true;
+      })());
 check('a trailing line comment stays at the end of its line at EVERY'
       + ' flush site -- after a one-liner block, an empty block, and a'
       + ' multi-line block close -- and its length never triggers'

tools/lpc-syntax/tokenizer.mjs

@@ -38,17 +38,47 @@ function skipStringSpan(src, i) {
 }
 
 function skipCharSpan(src, i) {
-  // Scan to the CLOSING quote like skipStringSpan -- char literals carry
-  // variable-length escapes (`'\x41'`, `'\101'`), and the old fixed
-  // 2-char assumption made findInterpEnd swallow an interpolation's `}`
-  // right after such a literal, tearing the template apart (literal
-  // corruption on valid LPC).
+  // Mirror lexer.l's char-literal grammar EXACTLY: after the opening
+  // quote, the body is ONE unit -- either a single raw byte (any byte,
+  // *including a literal quote*, per <SC_CHAR_BODY>[^\\]) or one escape
+  // sequence -- and then a closing quote is required. Escapes are
+  // variable-length (`'\x41'` hex, `'\101'` octal), which is why this
+  // can't assume a fixed 2-char width (the old fixed width made
+  // findInterpEnd swallow an interpolation's `}`), but it must NOT
+  // "scan to the next quote" like skipStringSpan either: that misreads
+  // the valid MudOS-ism `'''` (quote char, body is a raw `'`) as an
+  // empty `''` plus a stray `'` that then opens a bogus literal running
+  // to the next quote anywhere on the line -- in one real mudlib that
+  // next quote sat inside a trailing `//'` comment, and the formatter
+  // merged the case label, the comment, and the following statement
+  // into one line, silently deleting the statement on recompile.
   let j = i + 1;
-  while (j < src.length && src[j] !== "'") {
-    if (src[j] === '\\') j++;
-    j++;
+  if (j >= src.length) return j;
+  if (src[j] === '\\') {
+    j++; // the escape introducer; now classify per lexer.l's rules
+    const e = src[j];
+    if (e === undefined) return j;
+    if (e === 'x') {
+      // "\\x"[0-9a-fA-F]+ (or bare "\\x", an error the driver still
+      // consumes as just the two chars before the close-quote check)
+      j++;
+      while (j < src.length && /[0-9A-Fa-f]/.test(src[j])) j++;
+    } else if (e >= '0' && e <= '7') {
+      // "\\"[0-7]+ octal, maximal munch
+      while (j < src.length && src[j] >= '0' && src[j] <= '7') j++;
+    } else if (e === '\r' && src[j + 1] === '\n') {
+      j += 2; // "\\\r\n" escaped newline
+    } else {
+      j++; // "\\." -- simple/unknown escapes are exactly one char
+    }
+  } else {
+    j++; // one raw body byte -- including a literal `'` or newline
   }
-  return Math.min(j + 1, src.length);
+  // Closing quote. If it's missing the driver reports an error and
+  // pushes the offending byte back for the next scan (LPC_YYLESS(0));
+  // mirror that by ending the span here so the byte re-lexes normally.
+  if (j < src.length && src[j] === "'") j++;
+  return j;
 }
 
 function findInterpEnd(src, start) {
@@ -276,17 +306,13 @@ export function tokenize(src) {
       continue;
     }
 
-    // char literal -- escapes can be longer than one char (\xHH hex,
-    // \NNN octal are variable-length), so scan for the closing quote
-    // the same way string literals do rather than assuming a fixed width.
+    // char literal -- one body byte or escape, then the closing quote,
+    // exactly as lexer.l scans it (see skipCharSpan for the full rule;
+    // notably `'''` is a VALID quote-char literal, not an empty `''`).
     if (c === "'") {
-      let j = i + 1;
-      while (j < src.length && src[j] !== "'") {
-        if (src[j] === '\\') j++;
-        j++;
-      }
-      push('char', src.slice(i, Math.min(j + 1, src.length)));
-      i = Math.min(j + 1, src.length);
+      const j = skipCharSpan(src, i);
+      push('char', src.slice(i, j));
+      i = j;
       continue;
     }