Fix compiler hang on source with an embedded NUL byte

Found by the new AFL++ compiler harness (fuzz_compile), minimized to 2
bytes: any non-NUL byte followed by a NUL. Pre-existing (confirmed present
at ec9b6a4, predating this branch's whole history) and reachable through
completely ordinary compilation -- any .lpc file on disk containing a raw
NUL (a stray upload, a bad editor, a crafted mudlib file) hangs the driver
on an ordinary load_object(), and the same in-memory source API lpcshell/
lpcc -e/load_object_from_source() use is exposed just as directly.

Root cause: the Flex-generated scanner's YY_END_OF_BUFFER handling
special-cases a NUL byte that isn't at the two-sentinel end-of-buffer
position (yy_try_NUL_trans / yy_get_previous_state, lexer.autogen.cc, both
untouched, unmodified Flex-generated boilerplate). For certain surrounding
byte patterns this leaves yy_get_previous_state()'s bounding pointer
(yy_c_buf_p) effectively unbounded, so its scan loop never returns --
confirmed with gdb: repeated stack samples sat inside that one loop at
increasing depth, not cycling through yylex() being re-entered.

LPC source is text; a real program never legitimately contains a raw NUL.
Rather than trying to make the generated scanner itself robust to one (a
change to sensitive, regenerated Flex machinery with no quick way to gain
confidence in a fix), reject it at the two places source bytes actually
enter the compiler: scratch_slurp_fd_prepared() (on-disk -- the main file
and every #include'd file share this one reader) and start_new_file()
(in-memory source). Both reuse the read-error contract every caller
already handles cleanly, turning the hang into an ordinary catchable
compile error.

Verified: the minimized reproducer, and all 85 hangs the fuzzing campaign
saved, no longer time out; the new regression test genuinely hangs the
pre-fix ec9b6a4 driver (confirmed via a ported build, 15s and counting)
and passes cleanly post-fix. Full LPC testsuite green on Debug+ASan/UBSan
(2x) and RelWithDebInfo.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

src/compiler/internal/compiler.cc

@@ -3122,7 +3122,11 @@ static bool prolog(std::string_view source, const char* name, void* scanner) {
       return false;
     }
   } else {
-    start_new_file(source, scanner);
+    if (!start_new_file(source, scanner)) {
+      yyerror(scanner, "source contains an illegal embedded NUL byte");
+      num_parse_error++;
+      return false;
+    }
   }
   return true;
 }

src/compiler/internal/lexer.h

@@ -216,8 +216,10 @@ char* get_f_name(int);
 // in-memory Flex base buffer via lpc_lex_set_source; a missing final
 // newline is appended); prolog()/stage_output drive it directly.
 // keep_macros=true retains the user #define table across chunks (REPL
-// persistence); the #if stack always resets.
-void start_new_file(std::string_view source, void* yyscanner, bool keep_macros = false);
+// persistence); the #if stack always resets. Returns false (no state
+// touched) if source contains a real embedded NUL byte -- see the
+// rationale in scratch_slurp_fd_prepared()'s matching check.
+bool start_new_file(std::string_view source, void* yyscanner, bool keep_macros = false);
 // Zero-copy variant: reads fd's content straight into the arena block
 // flex scans in place. Returns false on read error.
 bool start_new_file_fd(int fd, void* yyscanner, bool keep_macros = false);

src/compiler/internal/lexer_utils.cc

@@ -23,6 +23,7 @@
 #include <cstdio>    // for EOF
 #include <fcntl.h>   // for O_RDONLY etc
 #include <cstdlib>   // for exit(), FIXME
+#include <cstring>   // for memchr
 #include <cctype>    // for isspace
 #include <unistd.h>  // for read(), FIXME
 #include <vector>
@@ -1177,6 +1178,21 @@ std::pair<char*, size_t> scratch_slurp_fd_prepared(int fd) {
     if (n == 0) break;
     len += static_cast<size_t>(n);
   }
+  // A real embedded NUL byte anywhere in the source hangs the Flex-generated
+  // scanner: yylex()'s YY_END_OF_BUFFER handling special-cases a NUL that
+  // isn't at the two-sentinel end-of-buffer position (yy_try_NUL_trans), and
+  // for certain surrounding byte patterns yy_get_previous_state()'s bounding
+  // pointer (yy_c_buf_p) can end up set such that its scan loop effectively
+  // never terminates (found by AFL++ fuzzing the compiler, minimized to 2
+  // bytes: a non-NUL byte followed by NUL). LPC source is text; a real file
+  // never legitimately contains a raw NUL. Reject it here, at the one place
+  // every on-disk source (main file and #include'd files alike) is read,
+  // instead of trying to make the generated scanner robust to it -- this
+  // reuses the read-error contract every caller already handles cleanly
+  // (lexerror("Cannot read #include file") / "could not read source file").
+  if (memchr(base, '\0', len) != nullptr) {
+    return {nullptr, 0};
+  }
   if (len == 0 || base[len - 1] != '\n') base[len++] = '\n';
   base[len] = 0;
   base[len + 1] = 0;
@@ -1292,7 +1308,16 @@ void lpc_lex_teardown_active(void) {
 static void start_new_file_prepared(char* prepared_base, size_t prepared_body, void* yyscanner,
                                     bool keep_macros);
 
-void start_new_file(std::string_view source, void* yyscanner, bool keep_macros) {
+bool start_new_file(std::string_view source, void* yyscanner, bool keep_macros) {
+  // Reject a real embedded NUL before touching any compiler state -- see
+  // the matching check + rationale in scratch_slurp_fd_prepared() (the
+  // on-disk-file counterpart of this in-memory-source path: lpcshell,
+  // lpcc -e, load_object_from_source()). Failing here, before any of the
+  // resets below run, mirrors start_new_file_fd()'s failure (nothing is
+  // touched when scratch_slurp_fd_prepared() itself fails).
+  if (memchr(source.data(), '\0', source.size()) != nullptr) {
+    return false;
+  }
   // Prepare an arena block from the caller's view: copy + trailing-'\n'
   // guarantee + the two yy_scan_buffer sentinels.
   bool add_nl = !source.empty() && source.back() != '\n';
@@ -1303,6 +1328,7 @@ void start_new_file(std::string_view source, void* yyscanner, bool keep_macros)
   base[body] = 0;
   base[body + 1] = 0;
   start_new_file_prepared(base, body, yyscanner, keep_macros);
+  return true;
 }
 
 bool start_new_file_fd(int fd, void* yyscanner, bool keep_macros) {

testsuite/single/tests/compiler/embedded_nul_hang.lpc

@@ -0,0 +1,68 @@
+// Regression: a real embedded NUL byte in LPC source (anywhere other than
+// as the buffer's own end-of-scan sentinel) could hang the Flex-generated
+// scanner forever. yylex()'s YY_END_OF_BUFFER handling special-cases a NUL
+// that isn't at the two-sentinel end-of-buffer position via
+// yy_try_NUL_trans()/yy_get_previous_state() (lexer.autogen.cc); for
+// certain surrounding byte patterns this leaves yy_get_previous_state()'s
+// bounding pointer effectively unbounded, so its scan loop never
+// terminates -- found by AFL++ fuzzing the compiler front-end
+// (load_object_from_source()), minimized to 2 bytes: any non-NUL byte
+// followed by a NUL. Reachable through completely ordinary compilation:
+// any .lpc file on disk containing a raw NUL (a stray upload, a bad editor,
+// a crafted mudlib file) hangs the driver on load_object(), and the same
+// in-memory API lpcshell/lpcc -e and load_object_from_source() use is
+// exposed just as directly.
+//
+// LPC source is text; a real program never legitimately contains a raw
+// NUL. The fix rejects it at the two places source bytes actually enter
+// the compiler -- scratch_slurp_fd_prepared() (on-disk: the main file and
+// every #include'd file share this one reader) and start_new_file()
+// (in-memory source) -- turning the hang into an ordinary, catchable
+// compile error instead of trying to make the generated scanner itself
+// robust to it.
+//
+// This can only be driven through the file-loading / in-memory-source
+// entry points (compile_file()/load_object_from_source()), not through an
+// LPC string literal -- LPC string literals cannot contain a raw NUL byte
+// at the source-text level in the first place. Exercise it the same way
+// the compiler's own audit regressions do for a compile-time bug: write a
+// generated file to disk and load it.
+void do_tests() {
+  string path = "/data/embedded_nul_hang_gen";
+  rm(path + ".c");
+
+  // A real embedded NUL can't be expressed as an LPC string literal or via
+  // sprintf("%c", 0) (rejected as not-a-valid-UTF8-char) -- build the file
+  // byte-for-byte with write_buffer() instead (a fresh allocate_buffer()
+  // is zero-filled), exactly the way a corrupted/crafted real .lpc file
+  // could contain one even though save_object()'s own format never does.
+  buffer nul = allocate_buffer(1);
+
+  // The minimized 2-byte reproducer, verbatim: one ordinary byte then NUL.
+  write_buffer(path + ".c", 0, "0");
+  write_buffer(path + ".c", 1, nul);
+  ASSERT2(catch(load_object(path)),
+    "a source file with an embedded NUL must fail to compile cleanly, not hang");
+
+  // A NUL inside a longer, otherwise ordinary program -- same bug class,
+  // more representative of what a corrupted real file looks like.
+  rm(path + ".c");
+  string prog = "void create() { ";
+  string tail = " int x = 1; }\n";
+  write_buffer(path + ".c", 0, prog);
+  write_buffer(path + ".c", strlen(prog), nul);
+  write_buffer(path + ".c", strlen(prog) + 1, tail);
+  ASSERT2(catch(load_object(path)),
+    "a NUL byte inside an otherwise-valid program must fail cleanly, not hang");
+
+  rm(path + ".c");
+
+  // A well-formed compile right after must still work, proving the
+  // rejection doesn't leave any compiler-global scratch state dirty.
+  write_file(path + ".c", "int ok() { return 42; }\n");
+  object ob = load_object(path);
+  ASSERT2(ob, "a normal compile right after the rejections must still succeed");
+  ASSERT_EQ(42, ob->ok());
+  destruct(ob);
+  rm(path + ".c");
+}