compiler: eliminate C-stack recursion from macro expansion and lexer; raise nesting cap to 65535

The macro-expansion rescan path recursed one yylex() frame per nesting
level, the textual argument pre-expander (lpc_lex_expand_string) recursed
per level with per-level guard-vector copies, the #if/#elif evaluator
recursed per unary/paren/ternary token, and the lexer's no-token recovery
paths (malformed heredocs, over-long $N, template-interpolation close)
retried via recursive yylex() -- error REPORTING stops after 5 parse
errors but scanning does not, so runs of malformed constructs nested a
frame each (a 600KB file of '@' lines segfaulted the driver).

All of these are now iterative:
- lpc_lex_resolve_identifier returns LPC_TOKEN_RESCAN (no token); the
  identifier rule falls through and the SAME yylex() frame keeps
  scanning the pushed expansion buffer -- one Flex buffer per level,
  zero C-stack growth.
- lpc_lex_expand_string is an explicit work-stack machine (one shared
  guard stack + O(1) name-count lookups instead of per-level copies).
- the #if evaluator is an explicit-stack machine (ifexpr_eval), depth
  bounded by token count on the heap; no cap needed. Keeps the audit's
  int64_t retyping (LLP64 correctness).
- heredoc recovery / @@ splice, over-long $N, and template '}' close
  fall through instead of recursing (pinned by lexer_retry_chains.lpc,
  which segfaults the previous binary).

This supersedes the 2026-07-20 audit's stack-overflow mitigations for
the preprocessor (kMaxIfExprDepth, MAX_EXPANSION_NESTING lowered to 32
under the sanitizer build's measured crash boundary): the recursion
itself is gone, so MAX_EXPANSION_NESTING becomes kLpcMaxExpansionNesting
= 65535, shared by both expansion engines as a runaway/memory bound and
counted in LIVE frames only via a live-index stack (dead same-line
provenance frames no longer trip it, so 128+ sequential uses of a macro
on one line compile again). deep_nesting_caps.lpc's deep cases now
compile -- under ASan included -- instead of being rejected; its
comments and AGENTS.md's cap-sizing guidance are updated to the new
design. The self-reference guard is an O(1) hash lookup.
innermost_real_buffer_index -- behind every current_line read, per
matched token -- is O(1) via a maintained include-buffer index stack
instead of walking the whole buffer stack (the walk made deep-chain
compiles quadratic).

Diagnostics keep 16 innermost + 16 outermost expansion notes with an
elision marker instead of one note per level.

Tests: deep_macro_nesting.lpc (60000-deep chains through both engines,
same-line frame accounting, beyond-cap clean error, 262144-token #if
shapes), lexer_retry_chains.lpc, deep_ternary_nesting.lpc (parser-stack
bound pins). Validated post-rebase on Debug and clang ASan+UBSan full
suites (621/621 files each) plus RelWithDebInfo pre-rebase; gtests
320/320.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

AGENTS.md

@@ -362,8 +362,8 @@ Repeated audits of the driver keep surfacing the **same handful of defect shapes
    * **"Defined" does not mean "safe" for a size, offset, or count.** A value that's allowed to wrap is still a memory-safety bug the instant it feeds a `DMALLOC`/`memcpy`/array index -- a wrapped-negative length becomes a huge `size_t`. Don't lean on wraparound to make a size computation safe; bound-check it explicitly *before* it can wrap (see `f_uncompress()`'s per-iteration `len` cap: checked every loop iteration, long before `len` could ever approach the type's range, not "after, and hope").
 3. **`INT_MIN / -1` / `INT_MIN % -1`, and an out-of-range shift count** (negative, or `>=` the operand's bit width) **are undefined behaviour that `-fwrapv` does NOT cover** (it only defines overflow of `+`/`-`/`*`/unary-negate) -- both trip UBSan, and since the build sets `-fno-sanitize-recover=all` that's a hard process abort, not a warning. Guard every signed divide/modulo of LPC ints explicitly (the `-1` divisor/dividend cases), erroring cleanly. For shift, don't error at all: **mask the count to the low 6 bits (`& 63`) before shifting**, matching Java's `long` shift semantics (JLS 15.19) -- every LPC int becomes a legal shift count with a well-defined, deterministic result, instead of rejecting otherwise-ordinary scripts that happen to shift by a large or negative amount. Remember there are usually **three** parallel sites per operator that all need the identical treatment: the interpreter opcode / efun (`F_DIVIDE`/`F_MOD`, `f_div_eq`/`f_mod_eq`, `f_lsh`/`f_rsh`/`f_lsh_eq`/`f_rsh_eq`), the compile-time constant folder (`trees.cc`'s `binary_int_op()`), and the `#if` preprocessor evaluator (`lexer_rules_pp.cc`'s `ifexpr_binop()`) -- a folded and unfolded shift by the same count must agree.
 4. **Recursion with no depth cap on attacker-nested data.** Deeply nested `({ ([ (:` in `restore_object` data, parser input, or compiled source overflows the C stack. Cap it -- the save AND restore paths share `MAX_SAVE_SVALUE_DEPTH`; other callers of nested-array-walking helpers (e.g. the parser's `environment` argument) need their own explicit depth parameter if they don't already have one.
-   * **Every sibling walker over the same recursive structure needs its OWN cap.** When a parse tree / save-svalue / whatever gets a depth cap on one recursive walker, the *other* walkers over the identical shape are still uncapped and still crash. The compiler tree has THREE: `optimize()` (generate.cc), `i_generate_node()` (icode.cc), and `ast_json()`/`dump_tree()` (generate.cc, the `lpcc --ast` renderer) -- the first two were capped (117cbc1) and the third was missed, so `lpcc --ast` on a deeply-nested expression still segfaulted before codegen's cap could reject it. Grep for every function that recurses on the same node/element type and confirm each has the guard. **The PREPROCESSOR has its own uncapped-recursion siblings too, over DIFFERENT nestable input than the parse tree:** the `#if`/`#elif` recursive-descent evaluator (`ifexpr_atom`/`ifexpr_binop`/`ifexpr_top` in `lexer_rules_pp.cc`, whose depth tracks paren/unary/ternary nesting -- now capped via `kMaxIfExprDepth`), and the textual macro-expansion recursion in `lpc_lex_expand_string()` (a chain of DISTINCT macros `A->B->C->...`, which the self-reference `guard` vector does NOT bound -- now capped like its rescan-path sibling `MAX_EXPANSION_NESTING`). Note `lpc_lex_expand_string()` is a *separate* path from the rescan expansion that already had `MAX_EXPANSION_NESTING`: it backs function-like-macro **argument pre-expansion** and unquoted `#include` filename expansion, so capping one did not cap the other.
-   * **Size the cap against the SANITIZER build's stack, not a "plausible" number.** ASan/UBSan redzones inflate every C frame, so a cap that's safe on a release build can still overflow the C stack on the Debug+sanitizer build that CI runs. `kMaxOptimizeDepth`/`kMaxGenerateNodeDepth` are deliberately 500 (not thousands) for this reason; the preprocessor's `MAX_EXPANSION_NESTING` was set to 128 but the sanitizer build actually overflowed at ~70-80 levels of `#define A B` chaining, so the cap never fired -- it was lowered to 64. Bisect the real crash boundary on the sanitizer build and set the cap comfortably under it.
+   * **Every sibling walker over the same recursive structure needs its OWN cap.** When a parse tree / save-svalue / whatever gets a depth cap on one recursive walker, the *other* walkers over the identical shape are still uncapped and still crash. The compiler tree has THREE: `optimize()` (generate.cc), `i_generate_node()` (icode.cc), and `ast_json()`/`dump_tree()` (generate.cc, the `lpcc --ast` renderer) -- the first two were capped (117cbc1) and the third was missed, so `lpcc --ast` on a deeply-nested expression still segfaulted before codegen's cap could reject it. Grep for every function that recurses on the same node/element type and confirm each has the guard. **The PREPROCESSOR's former recursion siblings were ELIMINATED, not capped** (the better fix when feasible): the whole macro/lexing layer is now iterative -- the rescan expansion continues in the SAME `yylex()` frame (`LPC_TOKEN_RESCAN` fall-through, one Flex buffer per level), `lpc_lex_expand_string()` (function-like-macro **argument pre-expansion** and unquoted `#include` filenames -- a chain of DISTINCT macros `A->B->C->...` that the self-reference guard does NOT bound) is an explicit work-stack machine, the `#if`/`#elif` evaluator (`ifexpr_eval` in `lexer_rules_pp.cc`, depth = paren/unary/ternary nesting) is an explicit-stack machine, and the lexer's no-token retries (heredoc recovery, over-long `$N`, template `}`) fall through instead of recursing into `yylex()`. Both expansion engines share ONE generous runaway bound, `kLpcMaxExpansionNesting` (65535, counted in LIVE frames) -- a memory bound, not a stack-safety bound. Pinned by `compiler/deep_macro_nesting.lpc`, `compiler/lexer_retry_chains.lpc`, `compiler/deep_nesting_caps.lpc`.
+   * **Size any remaining cap against the SANITIZER build's stack, not a "plausible" number -- or better, remove the recursion.** ASan/UBSan redzones inflate every C frame, so a cap that's safe on a release build can still overflow the C stack on the Debug+sanitizer build that CI runs. `kMaxOptimizeDepth`/`kMaxGenerateNodeDepth` are deliberately 500 (not thousands) for this reason; the preprocessor's old recursive design had exactly this failure (its 128 cap sat above the sanitizer build's ~70-80-frame crash boundary, so the cap never fired) and was briefly lowered to 32 before the recursion itself was eliminated -- a cap that must sit under the *sanitizer* stack budget is usually small enough to reject legitimate programs, which is the signal to convert the walker to an explicit heap stack instead (a 60000-deep chain now compiles under ASan). When a cap must stay (the parse-tree walkers), bisect the real crash boundary on the sanitizer build and set it comfortably under.
 5. **Missing type / bounds checks on stack arguments.** Spec-declared types cover only the first few fixed args; `varargs`, index, and count args still need explicit validation (an array indexed by an LPC int; a matrix that must have 16 elements; a port index bounded by **element count**, not `sizeof(array)` which is the size in *bytes*). Note this is about **element-level or varargs-position** checks specifically -- see §2 "Generic Argument Type-Checking" for what the dispatcher already covers for free at the top level, so you don't file (or re-fix) a false positive.
 6. **Tainted format strings.** `error()`, `debug_message()`, `debug()`, `yyerror`/`yywarn`/`lexerror`, and `telnet_printf` are printf-style. Any source-, mudlib-, DB-, or network-derived text (filenames, object paths, verbs, identifiers, `#error` payloads, DB error strings) must be a **`%s` argument**, never the format string -- a stray `%` is a crash / CodeQL `cpp/tainted-format-string`. Distinct but adjacent bug: a format specifier that doesn't match its argument's actual C++ type (e.g. `"%d"` for an `LPC_FLOAT`/`double` vararg) is also UB and prints garbage -- `error()` has no `format(printf, ...)` attribute, so the compiler won't catch this for you; make sure the value you pass is actually the type the specifier expects.
 7. **Leaks & dirty state on `error()` paths** -- see §4 (refs, half-filled VM stack slots, module-global scratch state, and the "deep C-style call chain" case where you must abort via plain `return`s instead of `error()`).

docs/lpc/preprocessor/define.md

@@ -40,7 +40,10 @@ at the very start or end of a body.
 A macro's body is rescanned after substitution, so macros may reference
 other macros freely. A macro that (directly or indirectly) references
 itself stops expanding at the self-reference instead of recursing
-forever, as in C.
+forever, as in C. Expansion nesting (a chain of distinct macros each
+referencing the next) is iterative inside the compiler and bounded at
+65535 levels; exceeding that is a compile error ("Macro expansion
+nested too deep").
 
 Because `#if` expressions are evaluated over the same token stream,
 arithmetic keeps C precedence across macro boundaries:

src/compiler/internal/README.md

@@ -63,14 +63,14 @@ graph TB
 
 ### 2. Preprocessing (`lexer_rules_pp.cc`, `lexer_rules_pp.h`) — part of the lexer's single scan
 - **There is no separate preprocessor.** Preprocessing is a set of lexer rule actions inside the one and only scan: every byte of source is read exactly once, and each directive's effect applies at exactly its position in the token stream — which makes position-sensitive directives (`#pragma no_warnings` mid-file, `#line`) correct by construction, with a single `current_line` counter (native Flex state, `%option yylineno`) nothing else fights.
-- **`lexer_rules_pp.cc` holds the preprocessing logic**: the macro table (`LpcMacroTable`, a plain map of `PpMacro`), `#define`/`#undef` parsing, the `#if`/`#elif` expression evaluator (`lpc_lex_eval_if_expr()`, which evaluates over TOKENS pulled through the scanner — see below), the conditional stack (`CondState`), `#` stringizing and `##` pasting via `substitute()` (`##` exists only inside macro bodies), `__LINE__`/`__FILE__`/`__DIR__` from the live scan position (`lpc_lex_builtin_macro()`), and the single directive entry point (`lpc_lex_on_directive()`) that lexer.l's one anchored `#`-line rule calls for BOTH scan modes. The only remaining textual macro expansion (`lpc_lex_expand_string()`) is for function-like ARGUMENT pre-expansion and `#include`'s unquoted-filename form; ordinary expansion is rescan-driven (see Module 1).
+- **`lexer_rules_pp.cc` holds the preprocessing logic**: the macro table (`LpcMacroTable`, a plain map of `PpMacro`), `#define`/`#undef` parsing, the `#if`/`#elif` expression evaluator (`lpc_lex_eval_if_expr()`, which evaluates over TOKENS pulled through the scanner — see below), the conditional stack (`CondState`), `#` stringizing and `##` pasting via `substitute()` (`##` exists only inside macro bodies), `__LINE__`/`__FILE__`/`__DIR__` from the live scan position (`lpc_lex_builtin_macro()`), and the single directive entry point (`lpc_lex_on_directive()`) that lexer.l's one anchored `#`-line rule calls for BOTH scan modes. The only remaining textual macro expansion (`lpc_lex_expand_string()`) is for function-like ARGUMENT pre-expansion and `#include`'s unquoted-filename form; ordinary expansion is rescan-driven (see Module 1). `lpc_lex_expand_string()` runs as an explicit work-stack machine (no C recursion, one shared guard stack instead of per-level guard-vector copies) under the same `kLpcMaxExpansionNesting` depth cap as the rescan path.
 - **There is no session object.** The preprocessor state lives directly in `g_compile`: `macros` holds USER `#define`s only and `conds` the `#if` stack. Predefines are immutable (redefining or `#undef`ing one is an error), so they never enter the per-compile table — `pp_find_macro()` falls back to a shared table derived from the boot predefine registry, rebuilt only when the registry's version changes. `start_new_file()` clears both members (capacity retained: steady-state compiles perform NO preprocessor-setup allocation; the old per-compile session heap-copied the entire predefine registry); `keep_macros=true` retains `#define`s across REPL chunks (each chunk must still be `#if`-balanced — an unterminated `#if` reports "Missing #endif" at EOF and the stack is cleared so the state stays usable).
 - **How each directive is handled**: `#define`/`#undef` mutate the session's macro table (redefining a macro with a different body is a non-fatal warning — the new definition wins); `#if`/`#ifdef`/`#ifndef`/`#elif`/`#else`/`#endif` drive the conditional stack — a false branch switches the scanner into the `SC_COND_SKIP` start condition, which consumes lines *without tokenizing them* (dead code may be deliberately invalid); the `#if`/`#elif` expression is evaluated over TOKENS: the expression text is pushed as a dedicated buffer and pulled through the scanner (numbers via the real literal decoders, macro references expand through the ordinary rescan path into a flat token sequence preserving C precedence, `defined()`/`efun_defined()` operands pulled with expansion suppressed). `#include` slurps the opened file and pushes its whole content as a Flex buffer, recording only metadata on the include stack (pop at that buffer's `<<EOF>>` — stack-based, no recursion, no eager expansion); `#pragma`/`#line`/`#error`/`#warn`/`#echo` apply immediately in place.
-- **Macro expansion happens at identifier-resolution time**, not as a text pre-pass: `lpc_lex_resolve_identifier()` (lexer_utils.cc) consults the macro table before the identifier hash; a hit pushes the RAW (parameter-substituted) body as a fresh Flex buffer and NESTED references expand when the rescan reaches them (collecting `(...)` arguments through `lpc_lex_getc()` for function-like macros — reads that transparently cross a drained splice buffer into its parent). Self-reference termination is the set of LIVE expansion-buffer frames: a macro's own name is resolved as a plain identifier while any live expansion buffer carries it as a guard. One provenance frame per live expansion buffer provides the "during expansion of macro ..." diagnostic notes. Because string/char/template bodies are scanned by their own start conditions and never reach identifier resolution, "no expansion inside strings/templates/heredocs" falls out for free — no parallel quoting-rule implementations to keep in sync.
+- **Macro expansion happens at identifier-resolution time**, not as a text pre-pass: `lpc_lex_resolve_identifier()` (lexer_utils.cc) consults the macro table before the identifier hash; a hit pushes the RAW (parameter-substituted) body as a fresh Flex buffer, returns the `LPC_TOKEN_RESCAN` pseudo-token (no token produced), and the identifier rule falls through so the SAME `yylex()` frame keeps scanning the pushed body — nesting is ITERATIVE (one Flex buffer per level, zero C-stack growth), bounded only by `kLpcMaxExpansionNesting` (65535 live buffers; the depth counter counts LIVE frames, so many sequential macro uses on one line never trip it). NESTED references expand when the rescan reaches them (collecting `(...)` arguments through `lpc_lex_getc()` for function-like macros — reads that transparently cross a drained splice buffer into its parent). Self-reference termination is the set of LIVE expansion-buffer frames: a macro's own name is resolved as a plain identifier while any live expansion buffer carries it as a guard (an O(1) name-count lookup, not a frame scan). One provenance frame per live expansion buffer provides the "during expansion of macro ..." diagnostic notes. Because string/char/template bodies are scanned by their own start conditions and never reach identifier resolution, "no expansion inside strings/templates/heredocs" falls out for free — no parallel quoting-rule implementations to keep in sync.
 
 ### 3. Lexer (`lexer.l`, `lexer.autogen.cc`, `lexer.h`)
 - **Responsibility**: THE scan — a reentrant Flex scanner (`%option reentrant`, threaded via an explicit `void *yyscanner` handle, no global lexer state) that turns raw LPC source (preprocessing included, per Module 2) into a token stream for the parser, one token's worth of work per `yylex()` pull.
-- **`lexer.l` is a thin rule table, not where the logic lives.** Mirroring how `grammar.y`'s actions mostly call a `rule_*()` function defined in `grammar_rules*.cc`, almost every substantive computation a lexer.l rule needs lives in ordinary functions: token-shaping logic (number-literal parsing, string/template escape decoding including Unicode surrogate pairs, char-literal escape decoding, template-fragment closing, `$N` function-pointer parameters) in **`lexer_rules.cc`**, preprocessing logic in **`lexer_rules_pp.cc`** (Module 2). What's left inline in `lexer.l` is, by necessity, only what can't move: Flex hides its scanner state (`yyguts_t`, `BEGIN()`/`YY_START`, `yyless()`) as macros/types private to the generated `lexer.autogen.cc` translation unit, invisible to a separately-compiled `.cc` file — so `lexer.l`'s trailer holds ONLY the primitives that dereference those private types: `lpc_lex_getc()` (wraps the static `yyinput()`), the buffer install/teardown choreography (`lpc_lex_set_source()`, `lpc_lex_push_prepared_buffer()`, `lpc_lex_teardown()`, `lpc_lex_reset()`), the nested-`yylex()` pull for `#if` expressions, and the raw buffer-introspection accessors (`lpc_lex_buffer_count/lineno/extents`). Everything expressible over public types lives in `lexer_utils.cc` -- including the tagged flex allocator (`yyalloc`/`yyfree`/`yyrealloc`), the copying splice push (`lpc_lex_push_string_buffer()`), and the pop policy (`lpc_lex_pop_pushed_buffer()`, `lpc_lex_pop_splice_if_any()`).
+- **`lexer.l` is a thin rule table, not where the logic lives.** Mirroring how `grammar.y`'s actions mostly call a `rule_*()` function defined in `grammar_rules*.cc`, almost every substantive computation a lexer.l rule needs lives in ordinary functions: token-shaping logic (number-literal parsing, string/template escape decoding including Unicode surrogate pairs, char-literal escape decoding, template-fragment closing, `$N` function-pointer parameters) in **`lexer_rules.cc`**, preprocessing logic in **`lexer_rules_pp.cc`** (Module 2). What's left inline in `lexer.l` is, by necessity, only what can't move: Flex hides its scanner state (`yyguts_t`, `BEGIN()`/`YY_START`, `yyless()`) as macros/types private to the generated `lexer.autogen.cc` translation unit, invisible to a separately-compiled `.cc` file — so `lexer.l`'s trailer holds ONLY the primitives that dereference those private types: `lpc_lex_getc()` (wraps the static `yyinput()`), the buffer install/teardown choreography (`lpc_lex_set_source()`, `lpc_lex_push_prepared_buffer()`, `lpc_lex_teardown()`, `lpc_lex_reset()`), the nested-`yylex()` pull for `#if` expressions, and the raw buffer-introspection accessors (`lpc_lex_buffer_count/lineno/extents`). Everything expressible over public types lives in `lexer_utils.cc` -- including the tagged flex allocator (`yyalloc`/`yyfree`/`yyrealloc`), the copying splice push (`lpc_lex_push_string_buffer()`), and the pop policy (`lpc_lex_pop_pushed_buffer()`, `lpc_lex_pop_splice_if_any()`). **No rule action calls `yylex()` recursively**: every no-token outcome (a macro expansion pushed for rescan, heredoc error recovery / `@@` splice hand-off, an over-long `$N`, a template-interpolation close) returns a sentinel (`LPC_TOKEN_RESCAN` / `kLpcLexFunctionParamRetry`) and the action falls through so the same `yylex()` frame keeps scanning -- error reporting stops after a few parse errors but scanning does not, so a recursive retry would nest a C-stack frame per malformed construct (a 600KB file of `@` lines used to segfault the driver; pinned by `tests/compiler/lexer_retry_chains.lpc`). The one intentional nested `yylex()` is the `#if` evaluator's token pull, bounded at a single extra level by construction.
 - **The directive rule consumes its terminating newline BEFORE dispatching** (documented at the rule): one `yyinput()` call — which also maintains Flex's beginning-of-line flag — so an `#include`'s stack entry records the line AFTER the directive as the parent's resume line. The push/pop bookkeeping mirrors the legacy scanner's `save_file_info`/`current_line_base` arithmetic verbatim.
 - **`compiler_context_t`** (declared in `lexer.h`) holds all per-scanner-instance lexer state (template nesting/brace-depth tracking, the string/template accumulator, heredoc terminator, the token's start column, the `SC_COND_SKIP` nesting depth, the `#if`-evaluator expansion-suppression flag) that used to be static globals — reached from `lexer.l`/`lexer_rules*.cc` via `yyget_extra(yyscanner)`.
 - **`lexer_utils.cc`** also hosts the hand-written helpers that can't be static Flex patterns: heredocs (`parseHeredoc()` — the closing terminator is supplied by the LPC source itself at compile time; the body is read line-by-line through `lpc_lex_getc()`, i.e. through Flex's own buffers) and main-file EOF (`parseMainEof()`).

src/compiler/internal/compiler.cc

@@ -276,6 +276,14 @@ std::string render_diagnostic(const Diagnostic& d, bool color) {
       if (!def_line_text.empty()) {
         emit_snippet(exp.def_line, def_line_text, def_col, nullptr, nullptr);
       }
+    } else if (exp.def_line < 0) {
+      // Elision marker from lpc_lex_expansion_chain(): a deep chain's
+      // middle levels collapsed into one entry; its "name" is the whole
+      // note text.
+      out += c_note;
+      out += "note: ";
+      out += c_off;
+      out += exp.macro_name;
     } else {
       out += c_note;
       out += "note: ";

src/compiler/internal/lexer.h

@@ -269,6 +269,15 @@ enum LpcPushedBufferKind {
 // (never reaches the parser: only the #if evaluator's pulls see it).
 #define LPC_IFEXPR_END (-2)
 
+// Pseudo-token lpc_lex_resolve_identifier() returns when the identifier
+// was a macro reference it consumed: the expansion (possibly empty) was
+// pushed as a fresh buffer and NO token was produced. The identifier
+// rule's action simply falls through so the CURRENT yylex() frame keeps
+// scanning the pushed buffer -- nesting a macro expansion costs one Flex
+// buffer, never a C-stack frame, which is what lets chains up to
+// kLpcMaxExpansionNesting (65535) deep work. Never escapes yylex().
+#define LPC_TOKEN_RESCAN (-3)
+
 // Push `text` as a fresh Flex buffer on top of the current one; scanning
 // consumes it fully, then pops back to the parent buffer at exactly the
 // position it left off (Flex's own bookkeeping -- no rewind/flush

src/compiler/internal/lexer.l

@@ -24,12 +24,14 @@
  * Macro EXPANSION happens at identifier-resolution time
  * (lpc_lex_resolve_identifier in lexer_utils.cc): the expanded body is
  * pushed as a fresh Flex buffer (lpc_lex_push_string_buffer, defined in
- * this file's trailer) and rescanned; the buffer pops back to the parent
- * at its <<EOF>>. Guarded self-references left literal in the expansion
- * are counted per name into compiler_context_t::pending_plain at
- * expansion time ("blue paint" per occurrence, C-preprocessor style); the
- * rescan consumes one count per occurrence and resolves it as a plain
- * identifier. Expansion provenance for diagnostics is one frame per live
+ * this file's trailer), the helper returns LPC_TOKEN_RESCAN (no token),
+ * and the identifier rule falls through so the SAME yylex() frame keeps
+ * scanning the pushed body; the buffer pops back to the parent at its
+ * <<EOF>>. Nesting is therefore iterative -- one Flex buffer per level,
+ * no C-stack recursion -- bounded by kLpcMaxExpansionNesting (65535).
+ * A name matching any LIVE expansion buffer resolves as a plain
+ * identifier (buffer-lifetime self-reference guard, C-preprocessor
+ * style). Expansion provenance for diagnostics is one frame per live
  * expansion buffer (see lexer_utils.cc). Because string/template/char
  * bodies are scanned by their own start conditions and never reach
  * identifier resolution, "no macro expansion inside strings" falls out
@@ -292,7 +294,14 @@ WS_NO_NL    [ \t\r\v\f]
      *
      */
 [a-zA-Z_][a-zA-Z0-9_]* {
-          return lpc_lex_resolve_identifier(yylval_param, yylloc_param, yyscanner);
+          int tok = lpc_lex_resolve_identifier(yylval_param, yylloc_param, yyscanner);
+          if (tok != LPC_TOKEN_RESCAN) return tok;
+          /* The identifier was a macro reference: its expansion (possibly
+           * empty) was pushed as a fresh buffer and no token was produced.
+           * Fall through and keep scanning in THIS yylex() frame -- same
+           * pattern as the <<EOF>> buffer pops below. Nesting is iterative
+           * (one Flex buffer per level, no C-stack frame), which is what
+           * lets expansion chains kLpcMaxExpansionNesting deep work. */
         }
 
     /* String literals: fully native via SC_STRING_BODY, no outp access.
@@ -515,8 +524,12 @@ WS_NO_NL    [ \t\r\v\f]
      * the number. */
 "$"[0-9]+ {
           int result = lpc_lex_function_param(yylval_param, yytext, yyleng);
-          if (result == kLpcLexFunctionParamRetry) return yylex(yylval_param, yylloc_param, yyscanner);
-          return result;
+          if (result != kLpcLexFunctionParamRetry) return result;
+          /* Over-long $N (reported): no token produced; keep scanning in
+           * THIS yylex() frame. Error reporting stops after a few parse
+           * errors but scanning does not, so the old recursive-yylex
+           * retry let a run of consecutive bad $N tokens nest a C-stack
+           * frame each. */
         }
 "$"     { return lpc_lex_function_param(yylval_param, yytext, yyleng); }
 
@@ -617,7 +630,13 @@ WS_NO_NL    [ \t\r\v\f]
            * the rewind still followed it). */
           lpc_lex_newline(yyscanner);
           BEGIN(INITIAL);
-          return lpc_lex_start_heredoc(yylval_param, yylloc_param, yyscanner);
+          int tok = lpc_lex_start_heredoc(yylval_param, yylloc_param, yyscanner);
+          if (tok != LPC_TOKEN_RESCAN) return tok;
+          /* Recoverable heredoc error (reported) or "@@" array splice
+           * pushed: no token; keep scanning in THIS yylex() frame
+           * (INITIAL was restored above). Falling through instead of a
+           * recursive yylex() keeps a run of malformed heredocs from
+           * nesting a C-stack frame per block. */
         }
     /* Trailing whitespace NOT immediately followed by a newline (i.e. the
      * rule above couldn't complete because EOF or non-whitespace content
@@ -649,7 +668,10 @@ WS_NO_NL    [ \t\r\v\f]
           // lpc_lex_getc() reads it first.
           LPC_YYLESS(0);
           BEGIN(INITIAL);
-          return lpc_lex_start_heredoc(yylval_param, yylloc_param, yyscanner);
+          int tok = lpc_lex_start_heredoc(yylval_param, yylloc_param, yyscanner);
+          if (tok != LPC_TOKEN_RESCAN) return tok;
+          /* See the newline-terminated form above: no token, fall
+           * through and keep scanning iteratively. */
         }
 
 "++"    { yylval_param->number = '+'; return L_INC_DEC; }
@@ -729,11 +751,13 @@ WS_NO_NL    [ \t\r\v\f]
           return '{';
         }
 "}"     {
-          if (lpc_lex_brace_close(yyscanner)) {
-            BEGIN(SC_TEMPLATE_BODY);
-            return yylex(yylval_param, yylloc_param, yyscanner);
-          }
-          return '}';
+          if (!lpc_lex_brace_close(yyscanner)) return '}';
+          /* This '}' ends a template interpolation: swallow it, switch
+           * back to the fragment scan, and keep scanning in THIS yylex()
+           * frame (the old recursive yylex() here only ever nested one
+           * transient level, but the fall-through form is uniform with
+           * the other no-token rules). */
+          BEGIN(SC_TEMPLATE_BODY);
         }
 "["     { return '['; }
 "]"     { return ']'; }

src/compiler/internal/lexer_rules.h

@@ -142,11 +142,11 @@ void lpc_lex_count_newlines(const char* text, int len);
 // $N / $ function-pointer parameter tokens
 // ---------------------------------------------------------------------------
 
-// Sentinel meaning: the token was too long to be valid; the caller must
-// retry via `return yylex(yylval_param, yyscanner)` itself. A recursive
-// yylex() call is kept visible at each of its call sites in lexer.l (matching
-// every other such call site there) rather than hidden inside this
-// function, so this can't just do the retry itself.
+// Sentinel meaning: the token was too long to be valid (reported here);
+// no token was produced. The lexer.l rule's action falls through so its
+// own yylex() frame keeps scanning -- same no-token convention as
+// LPC_TOKEN_RESCAN (lexer.h), kept as a distinct constant only because
+// this helper predates that one and its callers test it by name.
 inline constexpr int kLpcLexFunctionParamRetry = -2;
 
 // Handles both "$" and "$N" (text/len covers whichever matched). Returns

src/compiler/internal/lexer_rules_pp.cc

@@ -5,8 +5,7 @@
 #include <cctype>
 #include <cstdint>
 #include <cstdlib>
-
-#include "thirdparty/scope_guard/scope_guard.hpp"  // DEFER
+#include <new>  // placement new (EsPendingCall lands on the arena)
 
 #include "compiler/internal/compiler.h"
 #include "compiler/internal/lexer.h"
@@ -403,31 +402,19 @@ namespace {
 // ---------------------------------------------------------------------------
 
 struct IfTok {
-  int op;        // 0 = number; otherwise an operator code (see ifexpr_binop)
-  int64_t val;   // the number when op == 0 -- 64-bit to match LPC_INT (a plain
-                 // `long` is 32-bit on Windows/LLP64, which would truncate #if
-                 // arithmetic and disagree with the runtime opcode / constant
-                 // folder, and make the `& 63` shift mask below UB there).
+  int op;       // 0 = number; otherwise an operator code (see ifexpr_eval)
+  int64_t val;  // the number when op == 0 -- 64-bit to match LPC_INT (a plain
+                // `long` is 32-bit on Windows/LLP64, which would truncate #if
+                // arithmetic and disagree with the runtime opcode / constant
+                // folder, and make the `& 63` shift mask below UB there).
 };
 
 struct IfTokState {
   const std::vector<IfTok>* toks;
   size_t pos = 0;
-  int depth = 0;  // recursion depth of the recursive-descent evaluator below
   ScratchString error;  // first error wins; empty means no error yet
 };
 
-// The #if/#elif evaluator (ifexpr_atom/binop/top) is recursive-descent; its
-// recursion depth tracks paren / unary-operator / ternary nesting, which is
-// attacker-controlled with no bound from the token count. Cap it so a
-// pathological `#if ((((...))))` fails with a clean diagnostic instead of
-// overflowing the C stack. Every recursion path passes through ifexpr_atom
-// (top->binop->atom always; a '(' or unary re-enters atom; a ternary
-// re-enters top->binop->atom), so guarding atom bounds the whole evaluator.
-// Sized well under the ASan build's measured overflow boundary (~15-18k),
-// matching the parse-tree walkers' 500-deep caps (§13).
-static constexpr int kMaxIfExprDepth = 500;
-
 bool ifexpr_at_end(const IfTokState* st) { return st->pos >= st->toks->size(); }
 
 int ifexpr_peek_op(const IfTokState* st) { return ifexpr_at_end(st) ? 0 : (*st->toks)[st->pos].op; }
@@ -436,214 +423,280 @@ void ifexpr_set_error(IfTokState* st, const char* msg) {
   if (st->error.empty()) st->error = msg;
 }
 
-int64_t ifexpr_top(IfTokState* st);
+// The binary-operator table shared by the evaluator: token code ->
+// {op, precedence}. prec stays -1 for anything that is not a binary
+// operator (including '?' / ':', which the ternary frames handle).
+void ifexpr_binop_of(int c0, int* op, int* prec) {
+  *prec = -1;
+  *op = 0;
+  if (c0 == 'O') {
+    *op = 'O';
+    *prec = 1;
+  } else if (c0 == 'A') {
+    *op = 'A';
+    *prec = 2;
+  } else if (c0 == 'E') {
+    *op = 'E';
+    *prec = 6;
+  } else if (c0 == 'N') {
+    *op = 'N';
+    *prec = 6;
+  } else if (c0 == 'L') {
+    *op = 'L';
+    *prec = 7;
+  } else if (c0 == 'G') {
+    *op = 'G';
+    *prec = 7;
+  } else if (c0 == 's') {
+    *op = 's';
+    *prec = 8;
+  } else if (c0 == 'S') {
+    *op = 'S';
+    *prec = 8;
+  } else if (c0 == '|') {
+    *op = '|';
+    *prec = 3;
+  } else if (c0 == '^') {
+    *op = '^';
+    *prec = 4;
+  } else if (c0 == '&') {
+    *op = '&';
+    *prec = 5;
+  } else if (c0 == '<') {
+    *op = '<';
+    *prec = 7;
+  } else if (c0 == '>') {
+    *op = '>';
+    *prec = 7;
+  } else if (c0 == '+') {
+    *op = '+';
+    *prec = 9;
+  } else if (c0 == '-') {
+    *op = '-';
+    *prec = 9;
+  } else if (c0 == '*') {
+    *op = '*';
+    *prec = 10;
+  } else if (c0 == '/') {
+    *op = '/';
+    *prec = 10;
+  } else if (c0 == '%') {
+    *op = '%';
+    *prec = 10;
+  }
+}
 
-int64_t ifexpr_atom(IfTokState* st) {
-  if (ifexpr_at_end(st)) return 0;
-  if (++st->depth > kMaxIfExprDepth) {
-    --st->depth;
-    ifexpr_set_error(st, "#if expression nested too deeply");
-    return 0;
+int64_t ifexpr_combine(IfTokState* st, int op, int64_t lhs, int64_t rhs) {
+  switch (op) {
+    case 'O':
+      return lhs || rhs;
+    case 'A':
+      return lhs && rhs;
+    case '|':
+      return lhs | rhs;
+    case '^':
+      return lhs ^ rhs;
+    case '&':
+      return lhs & rhs;
+    case 'E':
+      return lhs == rhs;
+    case 'N':
+      return lhs != rhs;
+    case 'L':
+      return lhs <= rhs;
+    case 'G':
+      return lhs >= rhs;
+    case '<':
+      return lhs < rhs;
+    case '>':
+      return lhs > rhs;
+    case 's':
+      // A raw negative or >=64-bit (lhs/rhs are 64-bit int64_t) shift count
+      // is undefined behavior; mask to the low 6 bits (mod 64) instead of
+      // rejecting the expression, matching the runtime opcode.
+      return lhs << (rhs & 63);
+    case 'S':
+      return lhs >> (rhs & 63);
+    case '+':
+      return lhs + rhs;
+    case '-':
+      return lhs - rhs;
+    case '*':
+      return lhs * rhs;
+    case '/':
+      if (rhs == 0) {
+        ifexpr_set_error(st, "division by 0 in #if");
+        return 0;
+      }
+      if (rhs == -1) {
+        // x / -1 == -x; direct division traps (SIGFPE) for INT64_MIN.
+        return (int64_t)(0ULL - (uint64_t)lhs);
+      }
+      return lhs / rhs;
+    case '%':
+      if (rhs == 0) {
+        ifexpr_set_error(st, "modulo by 0 in #if");
+        return 0;
+      }
+      if (rhs == -1) {
+        return 0;  // x % -1 == 0; direct computation traps for LONG_MIN.
+      }
+      return lhs % rhs;
+    default:
+      return lhs;
   }
-  DEFER { --st->depth; };
-  const IfTok& t = (*st->toks)[st->pos];
-  if (t.op == 0) {
-    st->pos++;
-    return t.val;
-  }
-  switch (t.op) {
-    case '(': {
+}
+
+// The expression evaluator, C precedence + ternary, as an explicit-stack
+// machine. This used to be three mutually recursive functions
+// (top/binop/atom); crafted input could drive each one C-stack-frame-deep
+// per TOKEN -- one atom frame per unary in a `!!!!...1` run, one top
+// frame per chained `1 ? 1 :`, several frames per '(' -- and macro
+// expansion amplifies a short #if line into hundreds of thousands of
+// such tokens, far past any stack. Frames live on the heap now, at most
+// a few per token, so depth is bounded by the token count and needs no
+// separate cap. Control flow, token consumption order, and error wording
+// are a bit-exact port of the recursive version.
+struct IfEvalFrame {
+  uint8_t kind;   // 0 = top, 1 = binop, 2 = atom
+  uint8_t state;  // continuation point within the kind
+  int op;         // binop: pending operator; atom: unary operator
+  int min_prec;   // binop only
+  int64_t a;      // top: cond / binop: lhs
+  int64_t b;      // top: true_val
+};
+
+int64_t ifexpr_eval(IfTokState* st) {
+  constexpr uint8_t kTop = 0, kBinop = 1, kAtom = 2;
+  std::vector<IfEvalFrame> stack;
+  stack.push_back(IfEvalFrame{kTop, 0, 0, 0, 0, 0});
+  int64_t ret = 0;
+  while (!stack.empty()) {
+    IfEvalFrame& f = stack.back();  // pushes only as the last action before continue
+    if (f.kind == kTop) {
+      if (f.state == 0) {
+        // cond = binop(0)
+        f.state = 1;
+        stack.push_back(IfEvalFrame{kBinop, 0, 0, 0, 0, 0});
+        continue;
+      }
+      if (f.state == 1) {
+        f.a = ret;  // cond
+        if (ifexpr_peek_op(st) == '?') {
+          st->pos++;
+          f.state = 2;  // true_val = top()
+          stack.push_back(IfEvalFrame{kTop, 0, 0, 0, 0, 0});
+          continue;
+        }
+        ret = f.a;
+        stack.pop_back();
+        continue;
+      }
+      if (f.state == 2) {
+        f.b = ret;  // true_val
+        if (ifexpr_peek_op(st) == ':') {
+          st->pos++;
+        } else {
+          ifexpr_set_error(st, "'?' without ':' in #if");
+        }
+        f.state = 3;  // false_val = top()
+        stack.push_back(IfEvalFrame{kTop, 0, 0, 0, 0, 0});
+        continue;
+      }
+      // state 3: ret holds false_val
+      ret = f.a ? f.b : ret;
+      stack.pop_back();
+      continue;
+    }
+    if (f.kind == kBinop) {
+      if (f.state == 0) {
+        // lhs = atom()
+        f.state = 1;
+        stack.push_back(IfEvalFrame{kAtom, 0, 0, 0, 0, 0});
+        continue;
+      }
+      // state 1: ret holds lhs (from atom); state 2: ret holds rhs
+      f.a = (f.state == 1) ? ret : ifexpr_combine(st, f.op, f.a, ret);
+      int op = 0, prec = -1;
+      if (!ifexpr_at_end(st)) {
+        ifexpr_binop_of(ifexpr_peek_op(st), &op, &prec);
+      }
+      if (prec < f.min_prec) {
+        ret = f.a;
+        stack.pop_back();
+        continue;
+      }
       st->pos++;
-      int64_t v = ifexpr_top(st);
+      f.op = op;
+      f.state = 2;  // rhs = binop(prec + 1)
+      stack.push_back(IfEvalFrame{kBinop, 0, 0, prec + 1, 0, 0});
+      continue;
+    }
+    // kAtom
+    if (f.state == 0) {
+      if (ifexpr_at_end(st)) {
+        ret = 0;
+        stack.pop_back();
+        continue;
+      }
+      const IfTok& t = (*st->toks)[st->pos];
+      if (t.op == 0) {
+        st->pos++;
+        ret = t.val;
+        stack.pop_back();
+        continue;
+      }
+      switch (t.op) {
+        case '(':
+          st->pos++;
+          f.state = 1;  // v = top(), then expect ')'
+          stack.push_back(IfEvalFrame{kTop, 0, 0, 0, 0, 0});
+          continue;
+        case '!':
+        case '~':
+        case '-':
+        case '+':
+          st->pos++;
+          f.op = t.op;
+          f.state = 2;  // operand = atom()
+          stack.push_back(IfEvalFrame{kAtom, 0, 0, 0, 0, 0});
+          continue;
+        default:
+          // Mirrors the old walker's unknown-atom behavior: consume, 0.
+          st->pos++;
+          ret = 0;
+          stack.pop_back();
+          continue;
+      }
+    }
+    if (f.state == 1) {
+      // ret holds the parenthesized value
       if (ifexpr_peek_op(st) == ')') {
         st->pos++;
       } else {
         ifexpr_set_error(st, "bracket not paired in #if");
       }
-      return v;
+      stack.pop_back();
+      continue;
     }
-    case '!':
-      st->pos++;
-      return !ifexpr_atom(st);
-    case '~':
-      st->pos++;
-      return ~ifexpr_atom(st);
-    case '-':
-      st->pos++;
-      return -ifexpr_atom(st);
-    case '+':
-      st->pos++;
-      return ifexpr_atom(st);
-    default:
-      // Mirrors the old walker's unknown-atom behavior: consume, 0.
-      st->pos++;
-      return 0;
-  }
-}
-
-int64_t ifexpr_binop(IfTokState* st, int min_prec) {
-  int64_t lhs = ifexpr_atom(st);
-  for (;;) {
-    if (ifexpr_at_end(st)) break;
-
-    int c0 = ifexpr_peek_op(st);
-    int prec = -1, op = 0;
-
-    if (c0 == 'O') {
-      op = 'O';
-      prec = 1;
-    } else if (c0 == 'A') {
-      op = 'A';
-      prec = 2;
-    } else if (c0 == 'E') {
-      op = 'E';
-      prec = 6;
-    } else if (c0 == 'N') {
-      op = 'N';
-      prec = 6;
-    } else if (c0 == 'L') {
-      op = 'L';
-      prec = 7;
-    } else if (c0 == 'G') {
-      op = 'G';
-      prec = 7;
-    } else if (c0 == 's') {
-      op = 's';
-      prec = 8;
-    } else if (c0 == 'S') {
-      op = 'S';
-      prec = 8;
-    } else if (c0 == '|') {
-      op = '|';
-      prec = 3;
-    } else if (c0 == '^') {
-      op = '^';
-      prec = 4;
-    } else if (c0 == '&') {
-      op = '&';
-      prec = 5;
-    } else if (c0 == '<') {
-      op = '<';
-      prec = 7;
-    } else if (c0 == '>') {
-      op = '>';
-      prec = 7;
-    } else if (c0 == '+') {
-      op = '+';
-      prec = 9;
-    } else if (c0 == '-') {
-      op = '-';
-      prec = 9;
-    } else if (c0 == '*') {
-      op = '*';
-      prec = 10;
-    } else if (c0 == '/') {
-      op = '/';
-      prec = 10;
-    } else if (c0 == '%') {
-      op = '%';
-      prec = 10;
-    }
-
-    if (prec < min_prec) break;
-
-    st->pos++;
-    int64_t rhs = ifexpr_binop(st, prec + 1);
-
-    switch (op) {
-      case 'O':
-        lhs = lhs || rhs;
+    // state 2: ret holds the unary operand
+    switch (f.op) {
+      case '!':
+        ret = !ret;
         break;
-      case 'A':
-        lhs = lhs && rhs;
-        break;
-      case '|':
-        lhs = lhs | rhs;
-        break;
-      case '^':
-        lhs = lhs ^ rhs;
-        break;
-      case '&':
-        lhs = lhs & rhs;
-        break;
-      case 'E':
-        lhs = lhs == rhs;
-        break;
-      case 'N':
-        lhs = lhs != rhs;
-        break;
-      case 'L':
-        lhs = lhs <= rhs;
-        break;
-      case 'G':
-        lhs = lhs >= rhs;
-        break;
-      case '<':
-        lhs = lhs < rhs;
-        break;
-      case '>':
-        lhs = lhs > rhs;
-        break;
-      case 's':
-        // A raw negative or >=64-bit (lhs/rhs are 64-bit int64_t) shift count
-        // is undefined behavior; mask to the low 6 bits (mod 64) instead of
-        // rejecting the expression, matching the runtime opcode.
-        lhs = lhs << (rhs & 63);
-        break;
-      case 'S':
-        lhs = lhs >> (rhs & 63);
-        break;
-      case '+':
-        lhs = lhs + rhs;
+      case '~':
+        ret = ~ret;
         break;
       case '-':
-        lhs = lhs - rhs;
+        ret = -ret;
         break;
-      case '*':
-        lhs = lhs * rhs;
-        break;
-      case '/':
-        if (rhs == 0) {
-          ifexpr_set_error(st, "division by 0 in #if");
-          lhs = 0;
-        } else if (rhs == -1) {
-          // x / -1 == -x; direct division traps (SIGFPE) for INT64_MIN.
-          lhs = (int64_t)(0ULL - (uint64_t)lhs);
-        } else {
-          lhs = lhs / rhs;
-        }
-        break;
-      case '%':
-        if (rhs == 0) {
-          ifexpr_set_error(st, "modulo by 0 in #if");
-          lhs = 0;
-        } else if (rhs == -1) {
-          lhs = 0;  // x % -1 == 0; direct computation traps for LONG_MIN.
-        } else {
-          lhs = lhs % rhs;
-        }
-        break;
-      default:
+      default:  // '+'
         break;
     }
+    stack.pop_back();
+    continue;
   }
-  return lhs;
-}
-
-int64_t ifexpr_top(IfTokState* st) {
-  int64_t cond = ifexpr_binop(st, 0);
-  if (ifexpr_peek_op(st) == '?') {
-    st->pos++;
-    int64_t true_val = ifexpr_top(st);
-    if (ifexpr_peek_op(st) == ':') {
-      st->pos++;
-    } else {
-      ifexpr_set_error(st, "'?' without ':' in #if");
-    }
-    int64_t false_val = ifexpr_top(st);
-    return cond ? true_val : false_val;
-  }
-  return cond;
+  return ret;
 }
 
 // The name of an identifier-flavored token, for defined()'s operand.
@@ -783,7 +836,7 @@ int64_t lpc_lex_eval_if_expr(std::string_view expr, void* yyscanner) {
 
   IfTokState st;
   st.toks = &toks;
-  int64_t result = ifexpr_top(&st);
+  int64_t result = ifexpr_eval(&st);
   if (!st.error.empty()) {
     lexerror(st.error.c_str());
     return 0;
@@ -883,99 +936,211 @@ bool lpc_lex_builtin_macro(std::string_view name, ScratchString* out) {
   return false;
 }
 
-ScratchString lpc_lex_expand_string(std::string_view text, ScratchVector<ScratchString> guard) {
-  if (!g_compile.pp_active) return ScratchString(text);
-  // `guard` only stops a macro from expanding ITSELF; a chain of DISTINCT
-  // macros (A->B->C->...) recurses one C-frame per link with no bound. This
-  // is the textual sibling of the rescan path (which caps at
-  // MAX_EXPANSION_NESTING); cap it the same way so a pathological chain fails
-  // with a clean diagnostic instead of overflowing the C stack. The counter
-  // is single-threaded (one lexer per compile) and self-balances via DEFER,
-  // including on an exception unwind. Value matches the rescan cap (64), far
-  // under the measured ~3.5-5k overflow boundary.
-  static constexpr int kMaxExpandStringDepth = 64;
-  static int expand_depth = 0;
-  if (expand_depth >= kMaxExpandStringDepth) {
-    lexerror("Macro expansion nested too deep");
-    return ScratchString(text);
-  }
-  expand_depth++;
-  DEFER { expand_depth--; };
+namespace {
+// One pending function-like invocation inside lpc_lex_expand_string's
+// work stack: raw arguments already collected from the invoking frame's
+// text; each argument is pre-expanded into expanded_args by its own text
+// frame, then the body is substituted and rescanned. Arena-allocated
+// (placement new, destructor never runs: every member allocates from the
+// arena) so member addresses survive the frame vector's reallocation.
+struct EsPendingCall {
+  const PpMacro* m;
+  ScratchString name;  // guard entry for the substituted body's rescan
+  ScratchVector<ScratchString> raw_args;
+  ScratchVector<ScratchString> expanded_args;  // pre-sized; filled one frame at a time
+  size_t next_arg;
+  ScratchString subst;  // owns the substituted body while its frame scans it
+  ScratchString* out;   // where the rescan appends
+};
 
-  ScratchString result;
-  size_t i = 0;
-  while (i < text.size()) {
-    if (text[i] == '"' || text[i] == '\'') {
-      char q = text[i++];
-      result += q;
-      while (i < text.size() && text[i] != q) {
-        if (text[i] == '\\') result += text[i++];
-        if (i < text.size()) result += text[i++];
+// One level of lpc_lex_expand_string's explicit work stack: either a
+// text-scan frame (call == nullptr: scan [text,len) from pos, appending
+// to *out) or a call-driver frame (call != nullptr: feed the pending
+// invocation's arguments through their own frames, then morph into the
+// substituted body's text frame). guard_restore is the guard stack's
+// size at frame entry; frame exit truncates back to it.
+struct EsFrame {
+  const char* text;
+  size_t len;
+  size_t pos;
+  ScratchString* out;
+  size_t guard_restore;
+  EsPendingCall* call;
+};
+}  // namespace
+
+ScratchString lpc_lex_expand_string(std::string_view text) {
+  if (!g_compile.pp_active) return ScratchString(text);
+
+  ScratchString final_out;
+
+  // The guard chain (names currently being expanded, outermost first) as
+  // ONE shared stack plus a name->count map for O(1) membership tests;
+  // each frame truncates back to its entry size on exit. The old
+  // recursive version copied the whole guard vector per level -- O(n^2)
+  // arena memory over a deep chain -- and burned a C-stack frame per
+  // level, which is why it could not survive kLpcMaxExpansionNesting-deep
+  // input.
+  ScratchVector<ScratchString> guards;
+  std::unordered_map<std::string, int> guard_counts;
+  const auto guard_push = [&](std::string_view name) {
+    guards.emplace_back(name);
+    ++guard_counts[std::string(name)];
+  };
+  const auto guards_restore = [&](size_t mark) {
+    while (guards.size() > mark) {
+      auto it = guard_counts.find(std::string(guards.back().data(), guards.back().size()));
+      if (it != guard_counts.end() && --it->second == 0) guard_counts.erase(it);
+      guards.pop_back();
+    }
+  };
+  const auto guarded = [&](std::string_view id) {
+    return !guard_counts.empty() && guard_counts.find(std::string(id)) != guard_counts.end();
+  };
+
+  std::vector<EsFrame> frames;
+  frames.push_back(EsFrame{text.data(), text.size(), 0, &final_out, 0, nullptr});
+
+  // Depth gate for every push below: at the cap, report once and leave
+  // the too-deep reference literal (the compile is failing anyway; the
+  // machine stays memory-safe and terminates).
+  bool reported_too_deep = false;
+  const auto depth_ok = [&]() {
+    if (frames.size() < kLpcMaxExpansionNesting) return true;
+    if (!reported_too_deep) {
+      reported_too_deep = true;
+      lexerror("Macro expansion nested too deep");
+    }
+    return false;
+  };
+
+  while (!frames.empty()) {
+    const size_t fi = frames.size() - 1;
+
+    if (frames[fi].call != nullptr) {
+      EsPendingCall& c = *frames[fi].call;  // arena-stable across frame pushes
+      if (c.next_arg < c.raw_args.size()) {
+        const size_t a = c.next_arg++;
+        // Arguments are pre-expanded under the CALLER's guard set (the
+        // invoked macro's own name is NOT yet guarded) -- C's "arguments
+        // are fully expanded first" step, which lets SECOND(1, SECOND(2,
+        // 3))'s inner reference expand even while the outer invocation
+        // is in flight.
+        if (depth_ok()) {
+          frames.push_back(EsFrame{c.raw_args[a].data(), c.raw_args[a].size(), 0,
+                                   &c.expanded_args[a], guards.size(), nullptr});
+        } else {
+          c.expanded_args[a] = c.raw_args[a];
+        }
+        continue;
       }
-      if (i < text.size()) result += text[i++];
+      // All arguments expanded: substitute, then rescan the result with
+      // the macro's own name guarded for the body's duration.
+      c.subst = substitute(c.m->body, c.m->params, c.expanded_args);
+      guard_push(c.name);
+      EsFrame& f = frames[fi];
+      f.text = c.subst.data();
+      f.len = c.subst.size();
+      f.pos = 0;
+      f.out = c.out;
+      f.guard_restore = guards.size() - 1;
+      f.call = nullptr;
       continue;
     }
-    if (std::isalpha(static_cast<unsigned char>(text[i])) || text[i] == '_') {
-      size_t start = i;
-      while (i < text.size() &&
-             (std::isalnum(static_cast<unsigned char>(text[i])) || text[i] == '_'))
-        i++;
-      std::string_view id = text.substr(start, i - start);
 
-      {
-        ScratchString builtin;
-        if (lpc_lex_builtin_macro(id, &builtin)) {
-          result += builtin;
-          continue;
+    if (frames[fi].pos >= frames[fi].len) {
+      guards_restore(frames[fi].guard_restore);
+      frames.pop_back();
+      continue;
+    }
+
+    // Text scan: consume until a macro reference needs a sub-frame (then
+    // break out with `pushed`) or the frame's text runs dry. `result`
+    // and `t` stay valid across frame pushes (they alias arena / macro
+    // table storage, not the frames vector); the frame reference `f`
+    // does NOT, so its fields are written before any push.
+    EsFrame& f = frames[fi];
+    const std::string_view t(f.text, f.len);
+    ScratchString& result = *f.out;
+    size_t i = f.pos;
+    bool pushed = false;
+    while (i < t.size() && !pushed) {
+      if (t[i] == '"' || t[i] == '\'') {
+        char q = t[i++];
+        result += q;
+        while (i < t.size() && t[i] != q) {
+          if (t[i] == '\\') result += t[i++];
+          if (i < t.size()) result += t[i++];
         }
+        if (i < t.size()) result += t[i++];
+        continue;
       }
+      if (std::isalpha(static_cast<unsigned char>(t[i])) || t[i] == '_') {
+        size_t start = i;
+        while (i < t.size() && (std::isalnum(static_cast<unsigned char>(t[i])) || t[i] == '_')) i++;
+        std::string_view id = t.substr(start, i - start);
 
-      bool guarded = false;
-      for (const auto& g : guard) {
-        if (g == id) {
-          guarded = true;
-          break;
-        }
-      }
-
-      const PpMacro* found = pp_find_macro(id);
-      if (!guarded && found != nullptr) {
-        const PpMacro& m = *found;
-        if (!m.is_function_like) {
-          auto g2 = guard;
-          g2.emplace_back(id);
-          result += lpc_lex_expand_string(m.body, std::move(g2));
-        } else {
-          size_t j = i;
-          while (j < text.size() && (text[j] == ' ' || text[j] == '\t')) j++;
-          if (j < text.size() && text[j] == '(') {
-            j++;
-            auto args = collect_args(text, j);
-            i = j;
-            ScratchVector<ScratchString> expanded_args;
-            auto g2 = guard;
-            g2.emplace_back(id);
-            // Argument pre-expansion deliberately passes no
-            for (const auto& a : args) expanded_args.push_back(lpc_lex_expand_string(a, guard));
-            ScratchString subst = substitute(m.body, m.params, expanded_args);
-            result += lpc_lex_expand_string(subst, std::move(g2));
-          } else {
-            // Function-like macro name with no argument list in
-            // this text: left literal and NOT counted -- if its
-            // '(' turns out to follow in the input stream after
-            // this text is spliced, the rescan may legitimately
-            // expand it there (C behavior).
-            result += id;
+        {
+          ScratchString builtin;
+          if (lpc_lex_builtin_macro(id, &builtin)) {
+            result += builtin;
+            continue;
           }
         }
-      } else {
-        result += id;
+
+        const PpMacro* found = pp_find_macro(id);
+        if (guarded(id) || found == nullptr) {
+          result += id;
+          continue;
+        }
+        const PpMacro& m = *found;
+        if (!m.is_function_like) {
+          if (depth_ok()) {
+            f.pos = i;
+            guard_push(id);
+            frames.push_back(
+                EsFrame{m.body.data(), m.body.size(), 0, &result, guards.size() - 1, nullptr});
+            pushed = true;
+          } else {
+            result += id;
+          }
+          continue;
+        }
+        size_t j = i;
+        while (j < t.size() && (t[j] == ' ' || t[j] == '\t')) j++;
+        if (j < t.size() && t[j] == '(') {
+          j++;
+          if (depth_ok()) {
+            auto args = collect_args(t, j);
+            f.pos = j;  // consumed through the closing ')'
+            void* mem = scratch_raw_allocate(sizeof(EsPendingCall), alignof(EsPendingCall));
+            auto* call = new (mem) EsPendingCall{&m, ScratchString(id), std::move(args),
+                                                 {},   0,               {},
+                                                 &result};
+            call->expanded_args.resize(call->raw_args.size());
+            frames.push_back(EsFrame{nullptr, 0, 0, nullptr, guards.size(), call});
+            pushed = true;
+          } else {
+            // At the cap: the name stays literal and the argument list
+            // is NOT consumed; it flows through as plain text.
+            result += id;
+          }
+        } else {
+          // Function-like macro name with no argument list in this
+          // text: left literal and NOT counted -- if its '(' turns out
+          // to follow in the input stream after this text is spliced,
+          // the rescan may legitimately expand it there (C behavior).
+          result += id;
+        }
+        continue;
       }
-      continue;
+      result += t[i++];
+    }
+    if (!pushed) {
+      frames[fi].pos = i;  // exhausted; popped (and guards restored) next iteration
     }
-    result += text[i++];
   }
-  return result;
+  return final_out;
 }
 
 static ScratchString fold_backslash_newlines(std::string_view text) {

src/compiler/internal/lexer_rules_pp.h

@@ -26,6 +26,17 @@
 // PpMacro / CondState / LpcMacroTable live in compiler.h (they are
 // CompileState members).
 
+// Cap on macro-expansion nesting depth, shared by BOTH expansion engines:
+// the rescan-driven one (lpc_lex_resolve_identifier, counted in LIVE
+// expansion buffers) and the textual argument pre-expander
+// (lpc_lex_expand_string, counted in work-stack frames). Both are
+// iterative -- a level costs a Flex buffer / a work-stack entry on the
+// heap, never a C-stack frame -- so this bounds runaway chains and their
+// memory, not stack safety. Kept at 65535 as a deliberately generous
+// "any sane program fits" limit (~a few hundred bytes per level at full
+// depth).
+inline constexpr size_t kLpcMaxExpansionNesting = 65535;
+
 // Helpers
 ScratchString normalize_filename(const char* filename);
 std::string_view trim(std::string_view s);
@@ -128,16 +139,19 @@ bool lpc_lex_complete_directive(const char* text, int len, void* yyscanner, Scra
 // used by start_new_file() for the configured __GLOBAL_INCLUDE_FILE__.
 bool lpc_lex_handle_include(std::string_view rest, void* yyscanner);
 
-// Textual macro expansion (object-like and function-like, with `guard`
-// carrying the names currently being expanded for self-reference
-// termination). ONLY two textual consumers remain -- function-like
-// ARGUMENT pre-expansion (C's "arguments are fully expanded first" step)
-// and #include's unquoted-filename form; both consume the result as
-// text, never rescanned. Ordinary macro expansion is rescan-driven
-// (lpc_lex_resolve_identifier pushes the RAW substituted body as a Flex
-// buffer) and #if/#elif expressions are evaluated over TOKENS
-// (lpc_lex_eval_if_expr below).
-ScratchString lpc_lex_expand_string(std::string_view text, ScratchVector<ScratchString> guard = {});
+// Textual macro expansion (object-like and function-like, with an
+// internal guard chain of the names currently being expanded for
+// self-reference termination). ONLY two textual consumers remain --
+// function-like ARGUMENT pre-expansion (C's "arguments are fully
+// expanded first" step) and #include's unquoted-filename form; both
+// consume the result as text, never rescanned. Ordinary macro expansion
+// is rescan-driven (lpc_lex_resolve_identifier pushes the RAW
+// substituted body as a Flex buffer) and #if/#elif expressions are
+// evaluated over TOKENS (lpc_lex_eval_if_expr below). Runs as an
+// explicit work-stack machine, not C recursion: a deep chain costs heap
+// frames only, bounded by kLpcMaxExpansionNesting (one "Macro expansion
+// nested too deep" report; deeper references stay literal).
+ScratchString lpc_lex_expand_string(std::string_view text);
 
 // __LINE__/__FILE__/__DIR__ expand from the compiler's LIVE position
 // (current_line/current_file) -- one line counter, one file name is the

src/compiler/internal/lexer_utils.cc

@@ -28,6 +28,8 @@
 #include <vector>
 #include <algorithm>  // for std::sort
 #include <sstream>
+#include <string>
+#include <unordered_map>
 #include <unicode/ustring.h>
 #include <fmt/format.h>
 
@@ -331,18 +333,34 @@ void lpc_lex_scanner_destroyed(void* yyscanner) {
 // exposes accessors and ALL policy lives here).
 // ---------------------------------------------------------------------------
 
+namespace {
+// Buffer-stack indices of the LIVE include-content buffers, innermost
+// last -- maintained by lpc_lex_note_buffer_push/pop and cleared with
+// pushed_kind_stack. A buffer's index here is its position on Flex's
+// buffer stack (base buffer = 0, so these are all >= 1).
+std::vector<int> include_buffer_indices;
+}  // namespace
+
 // Index of the innermost REAL frame on the buffer stack: the top-most
 // INCLUDE buffer, else the base buffer (0). Splice buffers' positions are
-// synthetic; an out-of-range kind (-1, the transient window while a
-// push/pop is half-done) is skipped the same way. -1 = no buffers.
+// synthetic; an out-of-range index (the transient window while a
+// push/pop is half-done) is skipped the same way the old full-stack walk
+// skipped it. -1 = no buffers.
+//
+// O(1) via include_buffer_indices (maintained by the buffer push/pop
+// notes below) instead of walking the whole buffer stack: this sits
+// behind every `current_line` read -- YY_USER_ACTION touches it per
+// matched token -- and a deep macro chain stacks one splice buffer per
+// nesting level, so the walk made every token O(live depth) and a
+// 60000-deep chain effectively hung the compile.
 static int innermost_real_buffer_index(void* yyscanner) {
   int count = lpc_lex_buffer_count(yyscanner);
   if (count <= 0) {
     return -1;
   }
-  for (int i = count - 1; i > 0; --i) {
-    if (lpc_lex_buffer_kind_at(i - 1) == LPC_BUF_INCLUDE &&
-        lpc_lex_buffer_lineno(yyscanner, i) != nullptr) {
+  for (auto it = include_buffer_indices.rbegin(); it != include_buffer_indices.rend(); ++it) {
+    const int i = *it;
+    if (i < count && lpc_lex_buffer_lineno(yyscanner, i) != nullptr) {
       return i;
     }
   }
@@ -552,6 +570,23 @@ struct ExpansionFrame {
 };
 std::vector<ExpansionFrame> expansion_frames;
 
+// Indices (into expansion_frames) of the LIVE frames, innermost last.
+// Lets a buffer pop mark its frame dead in O(1) instead of scanning past
+// however many dead frames linger on the current line, and makes the
+// depth cap count actual nesting depth rather than frames-per-line
+// (sequential macro uses on one line used to trip "nested too deep").
+// Frame indices stay valid for a live frame's whole life: frames are only
+// removed by the dead-tail purge (which never reaches past a live frame)
+// or cleared wholesale between compiles.
+std::vector<size_t> live_expansion_stack;
+
+// Names of the live frames with multiplicity: the self-reference guard
+// lookup, O(1) per identifier instead of a scan over every live frame
+// (quadratic over a deep chain). Keyed by std::string, not views into
+// the frames -- ScratchString's SSO means frame reallocation moves short
+// names. Entries are erased at count 0, so presence == guarded.
+std::unordered_map<std::string, int> live_guard_counts;
+
 // Parallel to the stack of pushed buffers: LpcPushedBufferKind values.
 std::vector<char> pushed_kind_stack;
 }  // namespace
@@ -569,7 +604,14 @@ int lpc_lex_top_buffer_kind(void) {
   return pushed_kind_stack.empty() ? -1 : pushed_kind_stack.back();
 }
 
-void lpc_lex_note_buffer_push(int kind) { pushed_kind_stack.push_back(static_cast<char>(kind)); }
+void lpc_lex_note_buffer_push(int kind) {
+  pushed_kind_stack.push_back(static_cast<char>(kind));
+  if (kind == LPC_BUF_INCLUDE) {
+    // The note runs after the flex push, so the new buffer's stack index
+    // is exactly the pushed count (base buffer = 0).
+    include_buffer_indices.push_back(static_cast<int>(pushed_kind_stack.size()));
+  }
+}
 
 void lpc_lex_note_buffer_pop(int ending_lineno) {
   if (pushed_kind_stack.empty()) return;
@@ -577,14 +619,22 @@ void lpc_lex_note_buffer_pop(int ending_lineno) {
   pushed_kind_stack.pop_back();
   if (kind == LPC_BUF_EXPANSION) {
     // Mark the innermost live frame dead -- frames and expansion buffers
-    // nest LIFO, so it is this buffer's frame.
-    for (auto it = expansion_frames.rbegin(); it != expansion_frames.rend(); ++it) {
-      if (it->live) {
-        it->live = false;
-        break;
+    // nest LIFO, so it is this buffer's frame. (Empty-guarded for the
+    // leftover-buffer unwind in start_new_file, which clears the frame
+    // bookkeeping before lpc_lex_reset pops an aborted compile's stack.)
+    if (!live_expansion_stack.empty()) {
+      ExpansionFrame& f = expansion_frames[live_expansion_stack.back()];
+      live_expansion_stack.pop_back();
+      f.live = false;
+      auto it = live_guard_counts.find(std::string(f.name.data(), f.name.size()));
+      if (it != live_guard_counts.end() && --it->second == 0) {
+        live_guard_counts.erase(it);
       }
     }
   } else if (kind == LPC_BUF_INCLUDE) {
+    if (!include_buffer_indices.empty()) {
+      include_buffer_indices.pop_back();
+    }
     pop_include_state(ending_lineno);
   }
 }
@@ -596,27 +646,10 @@ void lpc_lex_note_buffer_pop(int ending_lineno) {
 // per-occurrence blue paint. Dead (lingering, diagnostics-only) frames
 // deliberately do NOT guard.
 static bool lpc_lex_name_guarded(std::string_view name) {
-  for (const auto& f : expansion_frames) {
-    if (f.live && f.name == name) {
-      return true;
-    }
-  }
-  return false;
+  if (live_guard_counts.empty()) return false;
+  return live_guard_counts.find(std::string(name)) != live_guard_counts.end();
 }
 
-// Nested rescans stack one Flex buffer + one yylex() frame per level; a
-// pathological chain (#define A0 A1 A1 / ...) must fail cleanly, not
-// blow the C stack. 128 was measured to still overflow the C stack under
-// this build's ASan instrumentation (each yylex()/lpc_lex_resolve_identifier()
-// frame pair is much larger there): bisection found the real crash boundary
-// at 70 (survives) / 80 (crashes) levels deep, i.e. under sanitizer builds
-// the cap itself was never reached before the process died. Set well below
-// that measured boundary, mirroring how kMaxOptimizeDepth/
-// kMaxGenerateNodeDepth (generate.cc/icode.cc) were deliberately kept small
-// for the same reason (see commit 117cbc1's reasoning) rather than picking
-// a larger "plausible" number.
-#define MAX_EXPANSION_NESTING 32
-
 // Pushes a frame for an expansion buffer about to be pushed. Zero-length
 // expansions push nothing (no buffer, no frame). The invocation position
 // (the macro USE, clang's "expansion location") is the live real-frame
@@ -624,6 +657,8 @@ static bool lpc_lex_name_guarded(std::string_view name) {
 // when the identifier matched (argument collection in between doesn't
 // disturb it -- raw reads run no user action).
 static void push_expansion_frame(std::string_view name, const PpMacro& m, int invocation_column) {
+  live_expansion_stack.push_back(expansion_frames.size());
+  ++live_guard_counts[std::string(name)];
   expansion_frames.push_back(ExpansionFrame{ScratchString(name),
                                             ScratchString(std::string_view(m.def_file)), m.def_line,
                                             current_line, invocation_column, true});
@@ -646,8 +681,26 @@ static void purge_exhausted_expansions() {
 }
 
 std::vector<LpcExpansionSite> lpc_lex_expansion_chain(void) {
+  // Deep chains (the nesting cap is 65535) must not turn one diagnostic
+  // into tens of thousands of "expanded from" notes: keep the innermost
+  // and outermost few sites and elide the middle with a marker entry
+  // (def_line -1; the renderer prints its name as a plain note). The
+  // OUTERMOST site must stay the vector's LAST element -- column
+  // attribution reads chain.back().
+  constexpr size_t kChainKeepEachEnd = 16;
+  const size_t total = expansion_frames.size();
   std::vector<LpcExpansionSite> out;
-  for (auto it = expansion_frames.rbegin(); it != expansion_frames.rend(); ++it) {
+  size_t emitted = 0;
+  for (auto it = expansion_frames.rbegin(); it != expansion_frames.rend(); ++it, ++emitted) {
+    if (total > (2 * kChainKeepEachEnd) && emitted == kChainKeepEachEnd) {
+      const size_t elided = total - (2 * kChainKeepEachEnd);
+      out.push_back(LpcExpansionSite{
+          "(" + std::to_string(elided) + " deeper macro expansions elided)", std::string(), -1,
+          it->invocation_line, it->invocation_column});
+      std::advance(it, static_cast<ptrdiff_t>(elided) - 1);
+      emitted += elided - 1;
+      continue;
+    }
     // Diagnostic capture: copy the arena frame text into the persistent
     // std::string site (diags outlive the compile/arena).
     out.push_back(LpcExpansionSite{std::string(it->name.data(), it->name.size()),
@@ -691,7 +744,12 @@ static void pop_include_state(int ending_line) {
   // p.line when the include was pushed and resumes by itself.
 }
 
-int lpc_lex_resolve_identifier(union YYSTYPE* yylval_param, struct YYLTYPE* yylloc_param,
+// Returns the resolved token, or LPC_TOKEN_RESCAN after consuming a
+// macro reference (expansion pushed as a buffer, no token produced --
+// the identifier rule falls through and its yylex() frame keeps
+// scanning). The YYLTYPE parameter is kept for signature parallelism
+// with the other lexer.l helpers; nothing here writes locations.
+int lpc_lex_resolve_identifier(union YYSTYPE* yylval_param, struct YYLTYPE* /*yylloc_param*/,
                                void* yyscanner) {
   compiler_context_t* yyextra = reinterpret_cast<compiler_context_t*>(yyget_extra(yyscanner));
   // Copy yytext up front, before ANY lpc_lex_getc(): a getc can pop the
@@ -704,7 +762,7 @@ int lpc_lex_resolve_identifier(union YYSTYPE* yylval_param, struct YYLTYPE* yyll
     const PpMacro* found = pp_find_macro(std::string_view(text.data(), text.size()));
     if (found != nullptr) {
       const PpMacro& m = *found;
-      if (static_cast<int>(expansion_frames.size()) >= MAX_EXPANSION_NESTING) {
+      if (live_expansion_stack.size() >= kLpcMaxExpansionNesting) {
         lexerror("Macro expansion nested too deep");
       } else if (!m.is_function_like) {
         // __LINE__/__FILE__/__DIR__ expand from the live scan position;
@@ -732,7 +790,12 @@ int lpc_lex_resolve_identifier(union YYSTYPE* yylval_param, struct YYLTYPE* yyll
           }
           push_arena_string(expanded, /*is_expansion=*/!is_builtin, yyscanner);
         }
-        return yylex(yylval_param, yylloc_param, yyscanner);
+        // No token produced: the identifier rule falls through and this
+        // yylex() frame keeps scanning the pushed body (or the parent
+        // input, for an empty expansion). Iterative on purpose -- a
+        // recursive yylex() here would burn one C-stack frame per nesting
+        // level and cap chains at a few thousand.
+        return LPC_TOKEN_RESCAN;
       } else {
         int saved_line = current_line;
         int saved_total = total_lines;
@@ -848,7 +911,9 @@ int lpc_lex_resolve_identifier(union YYSTYPE* yylval_param, struct YYLTYPE* yyll
             push_expansion_frame(text, m, yyextra->token_start_column + 1);
             push_arena_string(expanded, /*is_expansion=*/1, yyscanner);
           }
-          return yylex(yylval_param, yylloc_param, yyscanner);
+          // See the object-like branch: no token, scanning continues in
+          // the caller's own yylex() frame.
+          return LPC_TOKEN_RESCAN;
         } else {
           // No '(' follows: not an invocation. Put every probed byte back
           // (see consumed_text's comment) and let the identifier resolve
@@ -886,8 +951,12 @@ int lpc_lex_resolve_identifier(union YYSTYPE* yylval_param, struct YYLTYPE* yyll
 // "@@TERM ... TERM" (an array of line strings: splices `({ ... })` and
 // splices `"l1", "l2", })` for normal rescanning -- Robocoder's "@@"
 // block). On a recoverable error (bad UTF-8, oversized block), lexerror()
-// just logs and returns, so those paths resume the top-level scanner via
-// `return yylex()`.
+// just logs and returns; those paths -- and the "@@" splice hand-off --
+// produce NO token and return LPC_TOKEN_RESCAN so the heredoc rule's
+// action falls through and its own yylex() frame keeps scanning. (They
+// used to `return yylex()` recursively: error reporting stops after a
+// few parse errors but scanning does NOT, so a crafted file of
+// back-to-back malformed heredocs nested one C-stack frame per block.)
 // The shared tail of lexer.l's two heredoc-start rules (newline-terminated
 // and content-follows forms): validate the accumulated terminator and
 // hand off to the body reader. On an empty terminator, reports and
@@ -897,7 +966,7 @@ int lpc_lex_start_heredoc(union YYSTYPE* yylval_param, struct YYLTYPE* yylloc_pa
   compiler_context_t* ctx = yyget_extra(yyscanner);
   if (ctx->heredoc_terminator.empty()) {
     lexerror("Illegal terminator");
-    return yylex(yylval_param, yylloc_param, yyscanner);
+    return LPC_TOKEN_RESCAN;
   }
   return parseHeredoc(ctx->heredoc_terminator.c_str(), ctx->heredoc_is_array, yylval_param,
                       yylloc_param, yyscanner);
@@ -960,7 +1029,7 @@ int parseHeredoc(const char* terminator, int is_array, union YYSTYPE* yylval_par
         return YYerror;
       }
       lexerror("Text block exceeded maximum length");
-      return yylex(yylval_param, yylloc_param, yyscanner);
+      return LPC_TOKEN_RESCAN;
     }
 
     if (is_array) {
@@ -996,12 +1065,12 @@ int parseHeredoc(const char* terminator, int is_array, union YYSTYPE* yylval_par
       return YYerror;
     }
     push_arena_string(splice, 0, yyscanner);
-    return yylex(yylval_param, yylloc_param, yyscanner);
+    return LPC_TOKEN_RESCAN;
   }
 
   if (!u8_validate(text.c_str())) {
     lexerror("Bad UTF-8 string in string block");
-    return yylex(yylval_param, yylloc_param, yyscanner);
+    return LPC_TOKEN_RESCAN;
   }
   yylval_param->string = scratch_new_string(std::string_view(text));
   return L_STRING;
@@ -1217,6 +1286,7 @@ void lpc_lex_teardown_active(void) {
     lpc_lex_teardown(active_scanner);
   }
   pushed_kind_stack.clear();
+  include_buffer_indices.clear();
 }
 
 static void start_new_file_prepared(char* prepared_base, size_t prepared_body, void* yyscanner,
@@ -1268,6 +1338,11 @@ static void start_new_file_prepared(char* prepared_base, size_t prepared_body, v
   // expansion frames from an aborted compile must not haunt the next.
   compiler_diags.clear();
   expansion_frames.clear();
+  // Cleared BEFORE lpc_lex_reset below unwinds any leftover buffers: its
+  // LPC_BUF_EXPANSION pops must find nothing live to mark (the frames
+  // they'd refer to are gone).
+  live_expansion_stack.clear();
+  live_guard_counts.clear();
   compiler_current_load_reason = std::move(compiler_next_load_reason);
   compiler_next_load_reason.clear();
   // One-shot context a previous ABORTED compile may have left queued
@@ -1287,6 +1362,7 @@ static void start_new_file_prepared(char* prepared_base, size_t prepared_body, v
   // identity and must not be touched.
   lpc_lex_reset(yyscanner);
   pushed_kind_stack.clear();
+  include_buffer_indices.clear();
   for (auto& is : inc_stack) {
     free_string(const_cast<char*>(is.file));
   }

testsuite/single/tests/compiler/deep_macro_nesting.lpc

@@ -0,0 +1,167 @@
+// Macro-expansion nesting no longer recurses the C stack: the rescan
+// path keeps scanning in the same yylex() frame (one Flex buffer per
+// level) and the textual argument pre-expander runs an explicit work
+// stack, so chains up to 65535 levels deep must compile. The old
+// implementation recursed one yylex() frame per level with a cap of 128
+// -- and counted DEAD same-line provenance frames against that cap, so
+// even 128 sequential uses of a macro on one line spuriously failed with
+// "Macro expansion nested too deep". Past the cap the compile must fail
+// with that clean diagnostic, never a crash.
+
+#define CHAIN_DEPTH 60000
+#define OVER_DEPTH 65600
+
+// O(n) generation: build lines into an array and implode per chunk
+// (repeated string += is quadratic in the interpreter; a 60000-line
+// source took minutes that way). Chunks stay under the configured
+// maximum string length.
+private void gen_chain(string path, int depth) {
+  string *lines;
+  int i, base, n;
+
+  rm(path);
+  for (base = 0; base < depth; base += 5000) {
+    n = (depth - base < 5000) ? (depth - base) : 5000;
+    lines = allocate(n);
+    for (i = 0; i < n; i++) {
+      lines[i] = "#define A" + (base + i) + " A" + (base + i + 1);
+    }
+    write_file(path, implode(lines, "\n") + "\n");
+  }
+  write_file(path, "#define A" + depth + " 42\n" + "#define WRAP(x) (x)\n" +
+    "int query() { return A0; }\n" + "int query_wrap() { return WRAP(A0); }\n");
+}
+
+void do_tests() {
+  object ob;
+  string err;
+  string seq_src;
+  string *lines;
+  int i;
+
+  // 1) A CHAIN_DEPTH-deep object-like chain compiles and evaluates; the
+  //    WRAP() call drives the same chain through function-like ARGUMENT
+  //    pre-expansion (the textual work-stack engine). The old recursive
+  //    lexer errored out at 128 levels (query) and would have overflowed
+  //    the C stack in lpc_lex_expand_string (query_wrap).
+  gen_chain("/gen_deep_macros.c", CHAIN_DEPTH);
+  ob = load_object("/gen_deep_macros");
+  ASSERT2(objectp(ob), "60000-deep macro chain should compile");
+  if (objectp(ob)) {
+    ASSERT_EQ(42, ob->query());
+    ASSERT_EQ(42, ob->query_wrap());
+    destruct(ob);
+  }
+  rm("/gen_deep_macros.c");
+
+  // 2) Sequential (NOT nested) uses of one macro on a single line: each
+  //    use's dead provenance frame lingers until the newline for
+  //    diagnostics, and the old cap counted those, so 128+ uses per line
+  //    failed. Only LIVE nesting may count against the cap.
+  seq_src = "#define SEQ 1 +\nint query_seq() { return ";
+  for (i = 0; i < 300; i++) {
+    seq_src += "SEQ ";
+  }
+  seq_src += "0; }\n";
+  rm("/gen_seq_macros.c");
+  write_file("/gen_seq_macros.c", seq_src);
+  ob = load_object("/gen_seq_macros");
+  ASSERT2(objectp(ob), "300 sequential same-line macro uses should compile");
+  if (objectp(ob)) {
+    ASSERT_EQ(300, ob->query_seq());
+    destruct(ob);
+  }
+  rm("/gen_seq_macros.c");
+
+  // 3) Past the cap (65535 live levels): a clean "Macro expansion nested
+  //    too deep" compile error, not a crash.
+  gen_chain("/gen_over_macros.c", OVER_DEPTH);
+  err = catch(load_object("/gen_over_macros"));
+  ASSERT2(stringp(err), "beyond-cap macro chain must fail to compile cleanly");
+  rm("/gen_over_macros.c");
+
+  // 4) Deep #if expressions. The evaluator used to recurse one C-stack
+  //    frame per unary token, per '(' and per chained ternary, and macro
+  //    expansion amplifies a short #if line into hundreds of thousands
+  //    of such tokens -- a guaranteed stack overflow before the
+  //    explicit-stack rewrite. 262144 unary '!'s (even count, so the
+  //    condition stays 1).
+  lines = ({
+    "#define U8 !!!!!!!!",
+    "#define U64 U8 U8 U8 U8 U8 U8 U8 U8",
+    "#define U512 U64 U64 U64 U64 U64 U64 U64 U64",
+    "#define U4K U512 U512 U512 U512 U512 U512 U512 U512",
+    "#define U32K U4K U4K U4K U4K U4K U4K U4K U4K",
+    "#define U256K U32K U32K U32K U32K U32K U32K U32K U32K",
+    "#if U256K 1",
+    "#define IF_UNARY_OK 1",
+    "#else",
+    "#define IF_UNARY_OK 0",
+    "#endif",
+    "int query_if_unary() { return IF_UNARY_OK; }",
+  });
+  rm("/gen_if_unary.c");
+  write_file("/gen_if_unary.c", implode(lines, "\n") + "\n");
+  ob = load_object("/gen_if_unary");
+  ASSERT2(objectp(ob), "262144 unary operators in #if should evaluate");
+  if (objectp(ob)) {
+    ASSERT_EQ(1, ob->query_if_unary());
+    destruct(ob);
+  }
+  rm("/gen_if_unary.c");
+
+  // 5) 65536 nested parentheses in #if.
+  lines = ({
+    "#define L8 ((((((((",
+    "#define R8 ))))))))",
+    "#define L64 L8 L8 L8 L8 L8 L8 L8 L8",
+    "#define R64 R8 R8 R8 R8 R8 R8 R8 R8",
+    "#define L512 L64 L64 L64 L64 L64 L64 L64 L64",
+    "#define R512 R64 R64 R64 R64 R64 R64 R64 R64",
+    "#define L4K L512 L512 L512 L512 L512 L512 L512 L512",
+    "#define R4K R512 R512 R512 R512 R512 R512 R512 R512",
+    "#define L32K L4K L4K L4K L4K L4K L4K L4K L4K",
+    "#define R32K R4K R4K R4K R4K R4K R4K R4K R4K",
+    "#if L32K L32K 1 R32K R32K",
+    "#define IF_PAREN_OK 1",
+    "#else",
+    "#define IF_PAREN_OK 0",
+    "#endif",
+    "int query_if_paren() { return IF_PAREN_OK; }",
+  });
+  rm("/gen_if_paren.c");
+  write_file("/gen_if_paren.c", implode(lines, "\n") + "\n");
+  ob = load_object("/gen_if_paren");
+  ASSERT2(objectp(ob), "65536 nested parens in #if should evaluate");
+  if (objectp(ob)) {
+    ASSERT_EQ(1, ob->query_if_paren());
+    destruct(ob);
+  }
+  rm("/gen_if_paren.c");
+
+  // 6) 262144 right-chained ternaries in #if (the false branch of each
+  //    nests the next).
+  lines = ({
+    "#define Q8 1 ? 1 : 1 ? 1 : 1 ? 1 : 1 ? 1 : 1 ? 1 : 1 ? 1 : 1 ? 1 : 1 ? 1 :",
+    "#define Q64 Q8 Q8 Q8 Q8 Q8 Q8 Q8 Q8",
+    "#define Q512 Q64 Q64 Q64 Q64 Q64 Q64 Q64 Q64",
+    "#define Q4K Q512 Q512 Q512 Q512 Q512 Q512 Q512 Q512",
+    "#define Q32K Q4K Q4K Q4K Q4K Q4K Q4K Q4K Q4K",
+    "#define Q256K Q32K Q32K Q32K Q32K Q32K Q32K Q32K Q32K",
+    "#if Q256K 7",
+    "#define IF_TERN_OK 1",
+    "#else",
+    "#define IF_TERN_OK 0",
+    "#endif",
+    "int query_if_tern() { return IF_TERN_OK; }",
+  });
+  rm("/gen_if_tern.c");
+  write_file("/gen_if_tern.c", implode(lines, "\n") + "\n");
+  ob = load_object("/gen_if_tern");
+  ASSERT2(objectp(ob), "262144 chained ternaries in #if should evaluate");
+  if (objectp(ob)) {
+    ASSERT_EQ(1, ob->query_if_tern());
+    destruct(ob);
+  }
+  rm("/gen_if_tern.c");
+}

testsuite/single/tests/compiler/deep_nesting_caps.lpc

@@ -1,21 +1,24 @@
-// Regression for two compiler-front-end depth-cap bugs, both of which
-// crashed the *compile* (reachable from ordinary write_file()+load_object()
-// mudlib code, not just the lpcc CLI). Both must now fail cleanly (a compile
-// error / a load returning 0), never crash.
+// Regression for compiler-front-end depth crashes, all reachable from
+// ordinary write_file()+load_object() mudlib code (not just the lpcc CLI).
+// Every deep case must SURVIVE -- historically a crash; the exact outcome
+// (clean rejection or successful compile) is deliberately not asserted, so
+// this file is valid across cap policies. Current policy:
 //
 // 1. Nested (: :) closures past MAX_FUNCTION_DEPTH: push_function_context()
 //    silently no-op'd past the cap while the grammar still ran one
 //    pop_function_context() per closing ":)", eventually walking the shared
 //    current_function_context pointer into nullptr (UBSan: member access
-//    within null pointer). Now the pop side consumes a failed-push budget
-//    first (src/compiler/internal/lexer_utils.cc).
+//    within null pointer). The pop side consumes a failed-push budget first
+//    (src/compiler/internal/lexer_utils.cc); the cap (10) still rejects.
 //
-// 2. Object-like macro expansion past MAX_EXPANSION_NESTING: each level
-//    recurses one more yylex()/lpc_lex_resolve_identifier() C-stack frame
-//    pair. The cap (was 128) sat above what the sanitizer build's stack can
-//    actually survive, so a plain deep #define chain overflowed the C stack
-//    before the cap ever fired. The cap is now 64, under the measured
-//    boundary (src/compiler/internal/lexer_utils.cc).
+// 2. Deep macro chains (rescan path AND lpc_lex_expand_string's argument
+//    pre-expansion) and deep #if expressions used to recurse one C-stack
+//    frame per level -- briefly mitigated by lowering MAX_EXPANSION_NESTING
+//    under the sanitizer build's measured crash boundary, then fixed for
+//    real: those engines are ITERATIVE now (LPC_TOKEN_RESCAN fall-through /
+//    explicit work stacks; see compiler/deep_macro_nesting.lpc), so every
+//    deep case below simply compiles, bounded only by
+//    kLpcMaxExpansionNesting (65535, live frames).
 
 private string build_closures(int n) {
   string s = "";
@@ -41,8 +44,8 @@ private string build_macro_chain(int n) {
   return s;
 }
 
-// A #if whose condition is n-deep nested parens -- drives the recursive
-// #if/#elif evaluator (ifexpr_atom/binop/top in lexer_rules_pp.cc).
+// A #if whose condition is n-deep nested parens -- drives the #if/#elif
+// evaluator (ifexpr_eval in lexer_rules_pp.cc).
 private string build_if_parens(int n) {
   string s = "#if ";
   int i;
@@ -58,7 +61,7 @@ private string build_if_parens(int n) {
 }
 
 // A distinct-macro chain reached through a function-like macro's ARGUMENT --
-// drives lpc_lex_expand_string()'s textual expansion recursion, a different
+// drives lpc_lex_expand_string()'s textual expansion engine, a different
 // path from build_macro_chain()'s rescan path. Written to the file in chunks
 // (write_file appends): the full source for a large n exceeds
 // __MAX_STRING_LENGTH__ if built as one LPC string.
@@ -100,8 +103,8 @@ void do_tests() {
   }
   rm("/gen_deep_closures.c");
 
-  // 200-deep linear macro chain is well past MAX_EXPANSION_NESTING (64):
-  // must reject the compile cleanly, not overflow the C stack.
+  // 200-deep linear macro chain: used to overflow the C stack (then to be
+  // rejected by the lowered cap); compiles fine under the iterative engine.
   rm("/gen_deep_macros.c");
   write_file("/gen_deep_macros.c", build_macro_chain(200));
   ob = catch(load_object("/gen_deep_macros")) ? 0 : find_object("/gen_deep_macros");
@@ -110,8 +113,8 @@ void do_tests() {
   }
   rm("/gen_deep_macros.c");
 
-  // 20000-deep nested #if parens: must reject the compile cleanly, not
-  // overflow the C stack in the recursive #if evaluator.
+  // 20000-deep nested #if parens: used to overflow the C stack in the
+  // recursive #if evaluator; evaluates fine under the explicit-stack one.
   rm("/gen_deep_if.c");
   write_file("/gen_deep_if.c", build_if_parens(20000));
   ob = catch(load_object("/gen_deep_if")) ? 0 : find_object("/gen_deep_if");
@@ -121,8 +124,8 @@ void do_tests() {
   rm("/gen_deep_if.c");
 
   // 20000-deep distinct-macro chain through a function-like macro argument:
-  // must reject the compile cleanly, not overflow the C stack in
-  // lpc_lex_expand_string().
+  // used to overflow the C stack in lpc_lex_expand_string(); its explicit
+  // work-stack rewrite handles it.
   write_macro_arg_chain("/gen_deep_argmac.c", 20000);
   ob = catch(load_object("/gen_deep_argmac")) ? 0 : find_object("/gen_deep_argmac");
   if (objectp(ob)) {

testsuite/single/tests/compiler/deep_ternary_nesting.lpc

@@ -0,0 +1,31 @@
+// Deep expression-tree shapes that ride the PARSER stack (right-nested
+// ternary chains; parenthesized left-nesting-through-the-condition, the
+// deep-descent shape of trees.cc's insert_pop_value): both are bounded
+// by Bison's YYMAXDEPTH and must die with a clean "memory exhausted"
+// compile error, never a C-stack overflow. Complements
+// deep_expr_recursion.lpc, which pins the LEFT-nested chains that keep
+// the parser stack shallow but build unboundedly deep trees (those hit
+// icode.cc's explicit generation-depth cap instead).
+void do_tests() {
+  string err;
+  string src;
+
+  // Left-nested-through-condition ternary, discarded (insert_pop_value's
+  // deep-descent shape). The leading parens force parser-stack growth ->
+  // bounded by YYMAXDEPTH with a clean error.
+  src = "void f() { " + repeat_string("(", 20000) + "1" + repeat_string("?0:0)", 20000) + "; }\n";
+  rm("/gen_probe_tern.c");
+  write_file("/gen_probe_tern.c", src);
+  err = catch(load_object("/gen_probe_tern"));
+  ASSERT2(stringp(err), "20000-deep paren ternary must fail cleanly");
+  rm("/gen_probe_tern.c");
+
+  // Right-nested no-paren ternary chain as a used value: right-nesting
+  // rides the parser stack -> same clean bound.
+  src = "int f() { return " + repeat_string("1?2:", 30000) + "3; }\n";
+  rm("/gen_probe_tern2.c");
+  write_file("/gen_probe_tern2.c", src);
+  err = catch(load_object("/gen_probe_tern2"));
+  ASSERT2(stringp(err), "30000-deep right ternary must fail cleanly");
+  rm("/gen_probe_tern2.c");
+}

testsuite/single/tests/compiler/lexer_retry_chains.lpc

@@ -0,0 +1,38 @@
+// The lexer's no-token recovery paths (malformed heredoc starts, over-
+// long $N parameters, template-interpolation closes) used to retry via a
+// recursive yylex() call. Error REPORTING stops after a few parse errors
+// but scanning does not, so a crafted file of back-to-back malformed
+// constructs nested one C-stack frame per occurrence -- a 600KB file of
+// "@\n" lines was enough to overflow the stack and kill the driver.
+// Those paths now return no token and fall through, scanning iteratively
+// in the same yylex() frame; the compile must fail cleanly instead.
+void do_tests() {
+  string err;
+  string src;
+  int i;
+
+  // 1) 300000 consecutive illegal heredoc starts ("@" with an empty
+  //    terminator). One recursive retry per "@" used to overflow the
+  //    C stack; now it is a clean compile failure.
+  rm("/gen_heredoc_chain.c");
+  for (i = 0; i < 6; i++) {
+    write_file("/gen_heredoc_chain.c", repeat_string("@\n", 50000));
+  }
+  err = catch(load_object("/gen_heredoc_chain"));
+  ASSERT2(stringp(err), "a run of illegal heredoc starts must fail cleanly, not crash");
+  rm("/gen_heredoc_chain.c");
+
+  // 2) Over-long $N parameters (digit run past the lexer's line cap)
+  //    inside a real function-pointer context: each is reported and
+  //    consumed with no token produced; the compile fails cleanly and
+  //    scanning continues past every one of them.
+  src = "mixed f() { return (: $" + repeat_string(
+    "9",
+    5000
+  ) + " + $" + repeat_string("9", 5000) + " :); }\n";
+  rm("/gen_dollar_chain.c");
+  write_file("/gen_dollar_chain.c", src);
+  err = catch(load_object("/gen_dollar_chain"));
+  ASSERT2(stringp(err), "over-long $N parameters must fail cleanly");
+  rm("/gen_dollar_chain.c");
+}