mmo-ref/tinymux
1
0
Fork 0
mirror of https://github.com/brazilofmux/tinymux synced 2026-08-13 00:23:11 -04:00
Code Issues Projects Releases Packages Wiki Activity

Prototype parser-controlled lexer modes for noeval

Browse source
This commit is contained in:
Stephen Dennis 2026-03-23 12:32:35 -06:00
parent fc236515f9
commit fbeca3a7b7
9 changed files with 1453 additions and 40 deletions
Download patch file Download diff file Expand all files Collapse all files

231
docs/design-parser-controlled-lexer.md Normal file
View file

@ -0,0 +1,231 @@
# Parser-Controlled Lexer Modes
## Purpose
This note documents an alternative to the full "split lexer" design in
[docs/design-split-lexer.md](/home/sdennis/tinymux/docs/design-split-lexer.md).
The core idea is:
- keep the parser in control of structural boundaries
- make lexer behavior depend on parser-supplied `EVAL`/`NOEVAL` mode
- re-scan deferred regions later under a different mode
This is still an intentional violation of clean textbook lexer/parser
separation. It is less ugly than AST sibling-munging, but it is not a
pure front-end architecture.
## Problem Restated
The current 2.14 AST pipeline does this:
1. scan the entire expression once
2. form compound `%...` and `\X` tokens immediately
3. parse the frozen token stream into an AST
4. try to recover 2.13 `NOEVAL` behavior later
That loses information too early.
In 2.13 and PennMUSH, the semantic meaning of `%` depends on the eval
state *at the moment the character is read*, while backslash consumption
still happens even in `NOEVAL` paths.
The scanner therefore needs evaluation context. The parser is the part
that knows that context.
## Design Summary
Do not scan the whole expression once with a single global mode.
Instead:
1. parse structure with parser-controlled scan boundaries
2. scan content regions under a parser-supplied lexer mode
3. store deferred regions as raw source spans or raw text
4. when a deferred region is later selected, re-scan it under `EVAL`
This is extremely close to what PennMUSH already does, but made explicit
in the AST architecture.
## Modes
The scanner needs explicit modes:
- `ASTLEX_EVAL`
- `ASTLEX_NOEVAL`
- `ASTLEX_STRUCTURAL`
Meaning:
### `ASTLEX_EVAL`
- `%` is semantic
- `\` consumes the following character
- `%` forms may be grouped according to the active profile
### `ASTLEX_NOEVAL`
- `%` is not semantic
- `%` text passes through literally
- `\` still consumes the following character
This asymmetry is the important legacy rule.
### `ASTLEX_STRUCTURAL`
- recognizes brackets, braces, commas, parens, semicolons, spaces
- does only the minimum needed to let the parser find region boundaries
- does not commit `%` and `\` semantics more than necessary
This mode exists so the parser can discover the shape of function args
and deferred bodies without prematurely freezing all content semantics.
## Where The Context Comes From
The parser already knows when it is crossing semantically important
boundaries:
- entering a function call
- parsing comma-separated args
- seeing whether the callee is `FN_NOEVAL`
- entering brace groups and eval brackets
- later selecting a branch/body from a deferred arg
That means the parser can drive the scanner:
- normal function args: scan/parse in `ASTLEX_EVAL`
- `FN_NOEVAL` args: collect region boundaries structurally, but keep raw
source for later
- selected deferred arg/body: re-scan in `ASTLEX_EVAL`
## Concrete Model In This Tree
### Current relevant files
- [mux/modules/engine/ast_scan.rl](/home/sdennis/tinymux/mux/modules/engine/ast_scan.rl)
- [mux/modules/engine/ast.cpp](/home/sdennis/tinymux/mux/modules/engine/ast.cpp)
- [mux/include/ast.h](/home/sdennis/tinymux/mux/include/ast.h)
### Current fault line
`ast_tokenize()` in `ast_scan.rl` is a one-shot whole-input scanner. It
forms:
- `ASTTOK_PCT`
- `ASTTOK_ESC`
before the parser knows whether a region is `EVAL` or `NOEVAL`.
That is the decision that must move.
## Phase Plan
### Phase 1: API and naming groundwork
Add explicit concepts to the AST interface:
- lexer mode enum
- source-span type for deferred regions
- region-parse entrypoint that accepts a mode
At this stage, the implementation may still be a wrapper around the
existing one-shot scanner. The purpose is to make the future design
visible in code and stop hard-coding the assumption that all parsing
starts in one global scanner mode.
### Phase 2: Region parse entrypoints
Add parser/scanner entrypoints that work on a substring or source span:
- tokenize region with a supplied mode
- parse region into a subtree
Still acceptable at this stage:
- `ASTLEX_EVAL` and `ASTLEX_NOEVAL` may initially share most code
- `FN_NOEVAL` may still store raw text rather than raw pointers
### Phase 3: Parser-aware function args
During function-call parsing:
- resolve the function name early
- if the callee is `FN_NOEVAL`, do not fully semantic-tokenize arg
contents yet
- record raw arg regions for later
This is the key parser/lexer handshake.
### Phase 4: Replace legacy replay path
Replace the current noeval text-replay workaround with:
1. collect deferred region in `NOEVAL`
2. strip braces if required
3. parse selected region again in `EVAL`
4. evaluate resulting subtree
At that point, re-scanning is intentional and parser-directed rather
than an after-the-fact AST serialization trick.
### Phase 5: Optional full incremental scanner
Only if needed, replace the current batch tokenizer with a scanner
object that can:
- maintain position
- expose marks/spans
- switch modes mid-parse
This is not required to validate the design.
## Why This Is Better Than The Full Split Lexer
Compared to the design in
[docs/design-split-lexer.md](/home/sdennis/tinymux/docs/design-split-lexer.md),
this approach keeps the ugly part closer to the original disease.
Benefits:
- no AST sibling-mutation or sibling-consuming evaluation logic
- no global conversion of every `\` and `%` into atomic AST nodes
- deferred re-scan happens only at semantically real boundaries
- normal eval path can stay closer to the current architecture
Costs:
- parser and scanner are no longer cleanly separated
- function metadata influences parse behavior
- deferred args become region objects, not just already-parsed children
- some re-scan is now explicit design, not an accidental workaround
## Risk
This still is not "proper" in the usual compiler sense.
The risk is not conceptual novelty; the risk is getting the exact
boundaries wrong:
- which args are collected raw
- when braces are stripped
- whether nested brackets/braces are rescanned under the right mode
- whether the JIT/AST cache still sees stable parse units
This design should therefore be treated as:
- a compatibility mechanism
- limited to the smallest surface that must preserve 2.13 semantics
not as a general endorsement of context-sensitive lexing everywhere.
## Recommendation
Prefer this design over the full split-lexer tree-walk design if:
- the problem can be confined to deferred-eval boundaries
- parser-visible mode control is enough to reproduce the live-engine
behavior
- ordinary top-level eval can stay in the normal path
If those assumptions fail, the broader split-lexer design remains the
fallback.

253
docs/design-split-lexer.md Normal file
View file

@ -0,0 +1,253 @@
# Split Lexer Design: Deferred Backslash/Percent Semantics
## Problem
The 2.14 AST parser tokenizes backslash escapes (`\\`) and percent
substitutions (`%X`) as compound tokens during a single up-front
scanning pass. The scanner knows nothing about evaluation context
(EVAL vs NOEVAL).
In 2.13 (and PennMUSH), the scanner and evaluator are the same
character-at-a-time loop. The NOEVAL flag is live during scanning
and affects how characters are consumed:
- **Backslash handler**: runs unconditionally (no EV_EVAL guard).
`\\` consumes one `\`, outputs `\`.
- **Percent handler**: guarded by EV_EVAL. In NOEVAL, `%X` passes
through literally. In EVAL, `%X` is dispatched as a substitution.
This means that in a NOEVAL context, `\\% capacity`:
1. `\\` strips to `\` (backslash handler fires)
2. `% ` passes through as `% ` (percent handler skipped)
3. Result after NOEVAL pass: `\% capacity`
4. On re-evaluation: `\%` is a single escape unit, outputs `%`
5. Final result: `% capacity`
The 2.14 AST scanner produces `ESC("\\")` and `SUBST("% ")` as
independent tokens before any evaluation state is known. No amount
of tree-walking can reunite them into the `ESC("\%")` that the
re-evaluation pass needs, because the semantic decision about what
constitutes an escape sequence was already made during scanning.
This breaks real-world softcode. Myrddin's BBS (installed on
essentially every MU* game) uses `\\%` inside a `switch()` NOEVAL
branch to produce a literal `%` character.
## Why Tree Rewriting Alone Cannot Fix This
The `ast_noeval_pass()` function (ast.cpp) attempts to serialize
the AST back to a string with one layer of backslash stripping,
then re-tokenize and re-evaluate. This is the right idea, but
it operates on pre-formed tokens:
- `ESC("\\")` can emit `\` correctly.
- `SUBST("% ")` can pass through as `% ` correctly.
- The serialized result `\% capacity` is correct.
But when this string is fed back to the scanner for re-tokenization,
the scanner splits it into `ESC("\%")` + `LIT(" capacity")` and
evaluation produces `% capacity`. In principle this should work.
In practice, the re-tokenization is a second scan that must exactly
reconstruct what the 2.13 character-at-a-time evaluator would have
produced. Edge cases involving nested braces, eval brackets, and
multi-level escaping can cause the round-trip to diverge.
More fundamentally, the problem is that the scanner makes semantic
decisions (what constitutes an escape sequence, what constitutes a
percent substitution) without access to evaluation context. Patching
this after the fact is inherently fragile.
## Proposed Design: Split Lexer
Defer the semantic decisions about backslash and percent to the
tree evaluation phase, where EVAL/NOEVAL context is known.
### Phase 1: Scanner Changes (Ragel)
The scanner currently forms compound tokens:
```
'\\' any => { ASTTOK_ESC, text = "\X" }
'%' any => { ASTTOK_PCT, text = "%X" }
```
Change to emit atomic single-character tokens:
```
'\\' => { ASTTOK_BACKSLASH }
'%' => { ASTTOK_PERCENT }
```
These tokens carry no following character. They are markers that
say "a backslash/percent appeared here" without deciding what it
means. All other scanner rules remain unchanged: braces, brackets,
commas, parens, function names, literals.
The parser builds `AST_BACKSLASH` and `AST_PERCENT` nodes in the
tree. These are leaf nodes with no text payload beyond the single
character.
### Phase 2: Context-Sensitive Combination (Tree Walk)
The sequence evaluator (`AST_SEQUENCE` case) gains lookahead logic
for `AST_BACKSLASH` and `AST_PERCENT` nodes. This is the "second
lexical pass" — it does character-level combination during tree
evaluation, which is architecturally ugly but necessary.
```
for each child in sequence:
if child is AST_BACKSLASH:
// Backslash handler: NOT guarded by EV_EVAL.
// Consumes the first character of the next sibling.
// Exactly replicates 2.13 behavior.
//
char ch = consume_first_char(next_sibling)
safe_chr(ch, buff, bufc)
else if child is AST_PERCENT:
if (eval & EV_EVAL):
// Percent handler: guarded by EV_EVAL.
// Consumes following character(s) and dispatches
// as a substitution (same as current AST_SUBST).
//
dispatch_percent_subst(next_sibling, ...)
else:
// NOEVAL: output literal '%', do not consume.
//
safe_chr('%', buff, bufc)
else:
ast_eval_node(child, ...)
```
The `consume_first_char()` operation is the messy part. It must
handle several sibling node types:
- **AST_LITERAL**: consume first byte, shorten the literal. If the
literal becomes empty, skip the node.
- **AST_BACKSLASH**: consume the `\` character itself. The backslash
node is fully consumed and skipped.
- **AST_PERCENT**: consume the `%` character itself. The percent
node is fully consumed and skipped.
- **AST_SPACE**: consume the space. Node is skipped.
- **End of sequence**: trailing backslash with nothing following.
Output nothing (or `\` — match 2.13 behavior).
The `dispatch_percent_subst()` operation similarly consumes one or
more characters from the following sibling(s) to form the
substitution key (`%b`, `%0`, `%q<name>`, `%c<rgb>`, etc.).
### Phase 3: NOEVAL Branch Evaluation
With the split lexer, NOEVAL evaluation of `\\% capacity` proceeds:
1. BACKSLASH: consume first char of next sibling (BACKSLASH) → `\`
2. PERCENT: EV_EVAL is off → output `%` literally
3. LIT(" capacity"): output ` capacity`
4. Result: `\% capacity`
The re-evaluation pass sees `\% capacity` and (now with EV_EVAL on):
1. BACKSLASH: consume first char of next sibling (PERCENT) → `%`
2. LIT(" capacity"): output ` capacity`
3. Result: `% capacity`
This matches 2.13 exactly because the combination decisions are made
with the same EVAL/NOEVAL awareness that 2.13's entangled
scanner/evaluator had.
## Tradeoffs
### Costs
- **More tokens, more AST nodes.** Every `\` and `%` in the source
becomes its own node instead of being folded into a compound token.
The LRU parse cache mitigates re-scanning cost.
- **Lookahead in sequence evaluation.** The sequence evaluator becomes
a mini state machine that reaches across sibling boundaries. This
is lexer-level work happening during tree evaluation — exactly the
entanglement we tried to eliminate with the AST architecture.
- **Node mutation during evaluation.** `consume_first_char()` modifies
or skips sibling nodes, which means the AST is not purely read-only
during evaluation. (Alternative: use an index-advancement scheme
instead of mutation.)
- **Percent dispatch complexity.** Multi-character percent forms
(`%q<name>`, `%c<rgb>`) require consuming variable numbers of
characters from potentially multiple sibling nodes.
### Benefits
- **Exact 2.13 NOEVAL semantics.** The split lexer reproduces the
character-at-a-time interleaving of scanning and evaluation that
makes `\\%` work in NOEVAL contexts.
- **No serialization round-trip.** The tree walk operates directly
on nodes. No serialize-to-string-and-re-scan step that could
introduce divergences.
- **Contained change.** The scanner change is mechanical (remove
compound rules, add atomic rules). The evaluator change is
localized to the `AST_SEQUENCE` case. The rest of the AST
infrastructure (caching, JIT, function dispatch) is unaffected.
- **Testable.** The existing oracle corpus in
`parser/escape_oracle_cases.txt` and the Myrddin BBS case provide
concrete pass/fail criteria.
## Scope and Risk
This is a significant change to the scanner and the hottest path
in the evaluator. It should not be attempted without:
1. A comprehensive test matrix covering `\\`, `\%`, `\\%`, `%%%`,
and multi-level nesting in both EVAL and NOEVAL contexts.
2. Benchmarking to measure the token-count and evaluation overhead.
3. Careful review of all percent substitution forms to ensure the
tree-walk dispatcher handles variable-length consumption correctly.
The change is confined to:
- `ast_scan.rl` / `ast_scan.cpp` (scanner)
- `ast.cpp` (AST_SEQUENCE evaluator, percent dispatch)
- `ast.h` (new node types)
No changes to the JIT compiler, HIR lowering, or SSA pipeline are
required, though the JIT would need to be taught about the new node
types if it encounters them.
## Prior Art
No compiler textbook covers this because no designed language works
this way. MU* softcode evolved organically over 30 years with the
scanner and evaluator entangled by accident, and real-world softcode
depends on the resulting behavior.
PennMUSH has the same entanglement — `process_expression()` is both
scanner and evaluator, with `PE_EVALUATE` checked during character
scanning. See `parse.c` line 3113.
TinyMUX 2.13's `mux_exec()` has the same structure with `EV_EVAL`
checked during the character loop.
The split lexer is TinyMUX 2.14's way of reintroducing the minimum
necessary entanglement into an otherwise clean AST architecture.
## Alternative
There is now a second design direction in
[docs/design-parser-controlled-lexer.md](/home/sdennis/tinymux/docs/design-parser-controlled-lexer.md).
That approach keeps parser-owned structural boundaries and lets the
scanner run under parser-supplied `EVAL`/`NOEVAL` modes for deferred
regions. It is still intentionally impure, but it may preserve the AST
architecture with less evaluator-side AST mangling than the full split
lexer described here.
## Status
Design proposal. Not implemented.

38
mux/include/ast.h
View file

@ -43,12 +43,14 @@ struct ASTNode {
ASTNodeType type;
std::string text;
std::vector<std::unique_ptr<ASTNode>> children;
std::vector<std::string> raw_args;
bool parser_known_noeval;
bool has_close_paren; // FUNCCALL: true if ')' was found
bool has_close_bracket; // EVALBRACKET: true if ']' was found
bool has_close_brace; // BRACEGROUP: true if '}' was found
ASTNode(ASTNodeType t, std::string_view s = "")
: type(t), text(s), has_close_paren(true),
: type(t), text(s), parser_known_noeval(false), has_close_paren(true),
has_close_bracket(true), has_close_brace(true) {}
void addChild(std::unique_ptr<ASTNode> child) {
@ -56,6 +58,24 @@ struct ASTNode {
}
};
// ---------------------------------------------------------------
// Source regions and lexer modes
// ---------------------------------------------------------------
enum ASTLexMode {
ASTLEX_EVAL,
ASTLEX_NOEVAL,
ASTLEX_STRUCTURAL
};
struct ASTSourceSpan {
const UTF8 *input;
size_t nLen;
ASTSourceSpan(const UTF8 *p = nullptr, size_t n = 0)
: input(p), nLen(n) {}
};
// ---------------------------------------------------------------
// Token types (internal to parser, but exposed for testing)
// ---------------------------------------------------------------
@ -90,6 +110,16 @@ struct ASTToken {
//
std::vector<ASTToken> ast_tokenize(const UTF8 *input, size_t nLen);
// Tokenize a MUX expression region under an explicit lexer mode.
//
// Phase 1 note:
// This is currently a naming/API scaffold. The initial implementation
// may still share behavior with the legacy whole-input tokenizer until
// parser-controlled mode switching is wired through.
//
std::vector<ASTToken> ast_tokenize_mode(const UTF8 *input, size_t nLen,
ASTLexMode mode);
// Parse a token stream into an AST.
//
std::unique_ptr<ASTNode> ast_parse(const std::vector<ASTToken> &tokens);
@ -99,6 +129,12 @@ std::unique_ptr<ASTNode> ast_parse(const std::vector<ASTToken> &tokens);
//
std::unique_ptr<ASTNode> ast_parse_string(const UTF8 *input, size_t nLen);
// Parse a source region directly into an AST under an explicit lexer
// mode. This is the entrypoint intended for deferred-region reparsing.
//
std::unique_ptr<ASTNode> ast_parse_region(ASTSourceSpan span,
ASTLexMode mode);
// Reconstruct the raw source text from an AST subtree.
//
std::string ast_raw_text(const ASTNode *node);

207
mux/modules/engine/ast.cpp
View file

@ -51,6 +51,42 @@ private:
|| m_tokens[m_pos].type == ASTTOK_EOF;
}
static bool parser_lookup_builtin_noeval(std::string_view funcName)
{
UTF8 TempFun[LBUF_SIZE];
size_t nName = funcName.size();
if (nName >= LBUF_SIZE)
{
nName = LBUF_SIZE - 1;
}
memcpy(TempFun, funcName.data(), nName);
TempFun[nName] = '\0';
size_t nUpper;
UTF8 *pUpper = mux_strupr(TempFun, nUpper);
if (nUpper >= LBUF_SIZE)
{
nUpper = LBUF_SIZE - 1;
}
memcpy(TempFun, pUpper, nUpper);
TempFun[nUpper] = '\0';
std::vector<UTF8> name_key(TempFun, TempFun + nUpper);
const auto it = mudstate.builtin_functions.find(name_key);
return it != mudstate.builtin_functions.end()
&& (it->second->flags & FN_NOEVAL) != 0;
}
std::string rawTextFromTokens(size_t start, size_t end) const
{
std::string raw;
for (size_t i = start; i < end && i < m_tokens.size(); i++)
{
raw.append(m_tokens[i].text.data(), m_tokens[i].text.size());
}
return raw;
}
std::unique_ptr<ASTNode> parseSequence(
bool stopRP, bool stopRB, bool stopRC, bool stopCM)
{
@ -148,6 +184,7 @@ private:
{
ASTToken funcTok = advance();
auto call = std::make_unique<ASTNode>(AST_FUNCCALL, funcTok.text);
call->parser_known_noeval = parser_lookup_builtin_noeval(funcTok.text);
if (atEnd() || peek().type != ASTTOK_LPAREN)
{
@ -180,13 +217,23 @@ private:
//
bool inBracket = (m_bracketDepth > 0);
bool inBrace = (m_braceDepth > 0);
size_t argStart = m_pos;
auto arg = parseSequence(true, inBracket, inBrace, true);
if (call->parser_known_noeval)
{
call->raw_args.push_back(rawTextFromTokens(argStart, m_pos));
}
call->addChild(std::move(arg));
while (!atEnd() && peek().type == ASTTOK_COMMA)
{
advance();
argStart = m_pos;
arg = parseSequence(true, inBracket, inBrace, true);
if (call->parser_known_noeval)
{
call->raw_args.push_back(rawTextFromTokens(argStart, m_pos));
}
call->addChild(std::move(arg));
}
@ -250,12 +297,37 @@ std::unique_ptr<ASTNode> ast_parse(const std::vector<ASTToken> &tokens)
return parser.parse();
}
std::unique_ptr<ASTNode> ast_parse_string(const UTF8 *input, size_t nLen)
std::vector<ASTToken> ast_tokenize_mode(const UTF8 *input, size_t nLen,
ASTLexMode mode)
{
auto tokens = ast_tokenize(input, nLen);
if (mode == ASTLEX_NOEVAL)
{
for (auto &tok : tokens)
{
if (tok.type == ASTTOK_PCT)
{
tok.type = ASTTOK_LIT;
}
}
}
return tokens;
}
std::unique_ptr<ASTNode> ast_parse_region(ASTSourceSpan span,
ASTLexMode mode)
{
auto tokens = ast_tokenize_mode(span.input, span.nLen, mode);
return ast_parse(tokens);
}
std::unique_ptr<ASTNode> ast_parse_string(const UTF8 *input, size_t nLen)
{
return ast_parse_region(ASTSourceSpan(input, nLen), ASTLEX_EVAL);
}
// ---------------------------------------------------------------
// Utility functions
// ---------------------------------------------------------------
@ -503,6 +575,12 @@ static std::string ast_noeval_pass(const ASTNode *node)
return "";
}
// Forward declaration.
//
static void ast_eval_node(const ASTNode *node, UTF8 *buff, UTF8 **bufc,
dbref executor, dbref caller, dbref enactor,
int eval, const UTF8 *cargs[], int ncargs);
// Evaluate a selected argument from a FN_NOEVAL function using the
// 2.13-style noeval pass followed by reparse/re-eval.
//
@ -510,7 +588,8 @@ static std::string ast_noeval_pass(const ASTNode *node)
// Pass 1: noeval -- strip one layer of backslash (ast_noeval_pass)
// Pass 2: eval -- re-tokenize the result and evaluate it
//
static void ast_eval_noeval_legacy_arg(const ASTNode *node, UTF8 *buff,
static void ast_eval_noeval_legacy_arg(const ASTNode *node,
const std::string *rawText, UTF8 *buff,
UTF8 **bufc, dbref executor, dbref caller, dbref enactor,
int eval, const UTF8 *cargs[], int ncargs)
{
@ -519,31 +598,62 @@ static void ast_eval_noeval_legacy_arg(const ASTNode *node, UTF8 *buff,
return;
}
const ASTNode *inner = node;
if (node->type == AST_BRACEGROUP && !node->children.empty())
std::string text;
if (rawText)
{
inner = node->children[0].get();
auto noevalAst = ast_parse_region(
ASTSourceSpan(reinterpret_cast<const UTF8 *>(rawText->c_str()),
rawText->size()),
ASTLEX_NOEVAL);
if (!noevalAst)
{
return;
}
UTF8 *temp = alloc_lbuf("ast_noeval_region");
UTF8 *tp = temp;
ast_eval_node(noevalAst.get(), temp, &tp,
executor, caller, enactor,
((eval & ~(EV_EVAL | EV_TOP | EV_FMAND | EV_STRIP_CURLY | EV_FCHECK))
| EV_NOFCHECK),
cargs, ncargs);
*tp = '\0';
text.assign(reinterpret_cast<const char *>(temp), tp - temp);
free_lbuf(temp);
}
else
{
const ASTNode *inner = node;
if (node->type == AST_BRACEGROUP && !node->children.empty())
{
inner = node->children[0].get();
}
// Fallback path for callers that do not yet preserve a raw
// deferred region: produce text from the existing AST.
text = ast_noeval_pass(inner);
}
// Pass 1: produce text with one backslash layer stripped.
// Pass 2: re-tokenize the selected region in EVAL mode and
// evaluate the resulting subtree directly. This is the first
// concrete use of the parser-controlled region-parse API.
//
std::string text = ast_noeval_pass(inner);
auto reparsed = ast_parse_region(
ASTSourceSpan(reinterpret_cast<const UTF8 *>(text.c_str()), text.size()),
ASTLEX_EVAL);
if (!reparsed)
{
return;
}
// Pass 2: re-tokenize and evaluate.
//
mux_exec(reinterpret_cast<const UTF8 *>(text.c_str()), text.size(),
buff, bufc, executor, caller, enactor,
ast_eval_node(reparsed.get(), buff, bufc,
executor, caller, enactor,
(eval & ~(EV_TOP | EV_FMAND | EV_STRIP_CURLY))
| EV_EVAL | EV_FCHECK,
cargs, ncargs);
}
// Forward declaration.
//
static void ast_eval_node(const ASTNode *node, UTF8 *buff, UTF8 **bufc,
dbref executor, dbref caller, dbref enactor,
int eval, const UTF8 *cargs[], int ncargs);
static bool ast_is_malformed_qsubst(const ASTNode *node)
{
if ( !node
@ -1124,11 +1234,21 @@ static void ast_eval_subst(const ASTNode *node, UTF8 *buff, UTF8 **bufc,
// ##/#@/#$ are resolved natively as AST_SUBST nodes at eval time
// (from mudstate.itext/inum/switch_token).
//
static void ast_eval_branch(const ASTNode *child, UTF8 *buff, UTF8 **bufc,
static void ast_eval_branch(const ASTNode *callNode, int childIndex,
const ASTNode *child, UTF8 *buff, UTF8 **bufc,
dbref executor, dbref caller, dbref enactor,
int eval, const UTF8 *cargs[], int ncargs)
{
ast_eval_noeval_legacy_arg(child, buff, bufc,
const std::string *rawText = nullptr;
if ( callNode
&& callNode->parser_known_noeval
&& 0 <= childIndex
&& childIndex < static_cast<int>(callNode->raw_args.size()))
{
rawText = &callNode->raw_args[childIndex];
}
ast_eval_noeval_legacy_arg(child, rawText, buff, bufc,
executor, caller, enactor, eval, cargs, ncargs);
}
@ -1200,12 +1320,12 @@ static void ast_noeval_ifelse(const ASTNode *node, UTF8 *buff, UTF8 **bufc,
if (xlate(lbuff))
{
ast_eval_branch(node->children[1].get(), buff, bufc,
ast_eval_branch(node, 1, node->children[1].get(), buff, bufc,
executor, caller, enactor, eval, cargs, ncargs);
}
else if (nfargs >= 3)
{
ast_eval_branch(node->children[2].get(), buff, bufc,
ast_eval_branch(node, 2, node->children[2].get(), buff, bufc,
executor, caller, enactor, eval, cargs, ncargs);
}
@ -1253,7 +1373,7 @@ static void ast_noeval_switch(const ASTNode *node, UTF8 *buff, UTF8 **bufc,
reinterpret_cast<char *>(mbuff)) == 0)
{
free_lbuf(tbuff);
ast_eval_branch(node->children[i + 1].get(), buff, bufc,
ast_eval_branch(node, i + 1, node->children[i + 1].get(), buff, bufc,
executor, caller, enactor, eval, cargs, ncargs);
mudstate.switch_token = saved_switch;
free_lbuf(mbuff);
@ -1266,7 +1386,7 @@ static void ast_noeval_switch(const ASTNode *node, UTF8 *buff, UTF8 **bufc,
//
if (i < nfargs)
{
ast_eval_branch(node->children[i].get(), buff, bufc,
ast_eval_branch(node, i, node->children[i].get(), buff, bufc,
executor, caller, enactor, eval, cargs, ncargs);
}
@ -1315,7 +1435,7 @@ static void ast_noeval_switchall(const ASTNode *node, UTF8 *buff, UTF8 **bufc,
reinterpret_cast<char *>(mbuff)) == 0)
{
bMatched = true;
ast_eval_branch(node->children[i + 1].get(), buff, bufc,
ast_eval_branch(node, i + 1, node->children[i + 1].get(), buff, bufc,
executor, caller, enactor, eval, cargs, ncargs);
}
}
@ -1325,7 +1445,7 @@ static void ast_noeval_switchall(const ASTNode *node, UTF8 *buff, UTF8 **bufc,
//
if (!bMatched && i < nfargs)
{
ast_eval_branch(node->children[i].get(), buff, bufc,
ast_eval_branch(node, i, node->children[i].get(), buff, bufc,
executor, caller, enactor, eval, cargs, ncargs);
}
@ -1463,16 +1583,11 @@ static void ast_noeval_iter(const ASTNode *node, UTF8 *buff, UTF8 **bufc,
mudstate.inum[mudstate.in_loop - 1] = number;
}
// Evaluate the body subtree directly. ## and #@ resolve
// natively from mudstate.itext/inum.
//
// Note: iter() body is not a noeval branch — evaluate directly.
// ast_eval_branch would route through the two-pass noeval→eval
// path intended for switch/if branches.
//
ast_eval_node(node->children[1].get(), buff, bufc,
executor, caller, enactor,
eval|EV_STRIP_CURLY|EV_FCHECK|EV_EVAL, cargs, ncargs);
// iter() body is collected through the FN_NOEVAL arg path and
// then re-evaluated per item. Preserve that boundary by
// routing through the deferred branch helper as well.
ast_eval_branch(node, 1, node->children[1].get(), buff, bufc,
executor, caller, enactor, eval, cargs, ncargs);
}
mudstate.in_loop--;
@ -1840,7 +1955,16 @@ static void ast_eval_funccall(const ASTNode *node, UTF8 *buff, UTF8 **bufc,
if (i < nfargs - 1 || nParsed <= nfargs)
{
std::string raw = ast_raw_text(node->children[i].get());
std::string raw;
if ( node->parser_known_noeval
&& i < static_cast<int>(node->raw_args.size()))
{
raw = node->raw_args[i];
}
else
{
raw = ast_raw_text(node->children[i].get());
}
size_t len = raw.size();
if (len >= LBUF_SIZE) len = LBUF_SIZE - 1;
memcpy(fargs[i], raw.c_str(), len);
@ -1854,7 +1978,16 @@ static void ast_eval_funccall(const ASTNode *node, UTF8 *buff, UTF8 **bufc,
for (int j = i; j < nParsed; j++)
{
if (j > i) safe_chr(',', fargs[i], &bp);
std::string raw = ast_raw_text(node->children[j].get());
std::string raw;
if ( node->parser_known_noeval
&& j < static_cast<int>(node->raw_args.size()))
{
raw = node->raw_args[j];
}
else
{
raw = ast_raw_text(node->children[j].get());
}
safe_str(reinterpret_cast<const UTF8 *>(raw.c_str()),
fargs[i], &bp);
}

7
parser/Makefile
View file

@ -3,7 +3,7 @@ CXXFLAGS = -std=c++17 -Wall -Wextra -g -O2
HEADERS = mux_parse.h
all: tokenize parse eval
all: tokenize parse eval stream_passes
tokenize: tokenize.cpp $(HEADERS)
$(CXX) $(CXXFLAGS) -o $@ $<
@ -14,7 +14,10 @@ parse: parse.cpp $(HEADERS)
eval: eval.cpp $(HEADERS)
$(CXX) $(CXXFLAGS) -o $@ $<
stream_passes: stream_passes.cpp
$(CXX) $(CXXFLAGS) -o $@ $<
clean:
rm -f tokenize parse eval
rm -f tokenize parse eval stream_passes
.PHONY: all clean

6
parser/README.md
View file

@ -13,6 +13,9 @@ For end-to-end traced command behavior, see
- `test_corpus.txt` — Test expressions
- `escape_oracle_cases.txt` — focused cross-profile escape corpus
- `run_escape_oracle.sh` — runner for the focused escape corpus
- `borrowed-stream-semantics.md` — reduced 2.13/Penn parser mechanisms
- `stream_passes.cpp` — tiny pass-by-pass stream vs frozen-token model
- `hammer_refrozen.sh` — prints core deferred-boundary cases across models
- `Makefile` — Build rules
## Usage
@ -25,6 +28,9 @@ echo '[setq(0,hello)]%q0 world' | ./parse
echo '\\\\% capacity' | ./eval --profile mux214
echo '[switch(1,1,{\\\\% capacity})]' | ./eval --profile mux213
echo '%xg' | ./eval --profile penn
echo '\\\\% capacity' | ./stream_passes --profile mux213 --model stream --passes noeval,eval
echo '\\\\% capacity' | ./stream_passes --profile mux213 --model frozen --passes noeval,eval
./hammer_refrozen.sh
./run_escape_oracle.sh
```

182
parser/borrowed-stream-semantics.md Normal file
View file

@ -0,0 +1,182 @@
# Borrowed Stream Semantics
This note extracts the smallest useful parts of the legacy streaming
parsers for study.
The goal is not to port TinyMUX 2.13 or PennMUSH wholesale. The goal is
to keep only the mechanisms that explain why `%` and `\` behave
correctly there and incorrectly in the current AST pipeline.
## Scope
Only these mechanisms matter for the current compatibility problem:
1. How function arguments are collected for deferred-eval functions.
2. What `%` does when evaluation is off.
3. What `\` does when evaluation is off.
4. How a deferred argument is evaluated again later.
Everything else is noise for this specific study.
## TinyMUX 2.13: Minimal Mechanism
Source anchors:
- `mux2.13_12/src/eval.cpp:1528`
- `mux2.13_12/src/eval.cpp:1680`
- `mux2.13_12/src/eval.cpp:2438`
### 1. FN_NOEVAL argument collection changes flags
For `FN_NOEVAL` functions, 2.13 clears `EV_EVAL`, `EV_TOP`,
`EV_FMAND`, and `EV_STRIP_CURLY` before parsing the argument list:
```c
if (fp && (fp->flags & FN_NOEVAL)) {
feval = eval & ~(EV_EVAL|EV_TOP|EV_FMAND|EV_STRIP_CURLY);
} else {
feval = eval & ~(EV_TOP|EV_FMAND);
}
tstr = parse_arglist_lite(..., feval, fargs, ...);
```
That means the argument text is still scanned by `mux_exec()`, but with
percent evaluation disabled.
### 2. Percent handling is gated by `EV_EVAL`
In the `%` branch:
```c
if (!(eval & EV_EVAL)) {
*(*bufc)++ = '%';
iStr++;
*(*bufc)++ = pStr[iStr];
} else {
// full substitution dispatch
}
```
So in noeval collection, `%` is copied literally.
### 3. Backslash handling is not gated by `EV_EVAL`
In the `\` branch:
```c
iStr++;
if (pStr[iStr]) {
*(*bufc)++ = pStr[iStr];
} else {
iStr--;
}
```
There is no `EV_EVAL` check here. Every pass through `mux_exec()`
consumes one backslash layer.
### 4. Deferred branch is evaluated again later
The stored argument text is later passed back into `mux_exec()` with
`EV_EVAL` restored by the function implementation. That creates the
critical multi-pass behavior:
- pass 1: `\\% capacity` -> `\% capacity`
- pass 2: `\% capacity` -> `% capacity`
## PennMUSH: Matching Minimal Mechanism
Source anchors:
- `pennmush/src/parse.c:2355`
- `pennmush/src/parse.c:2848`
- `pennmush/src/parse.c:3106`
PennMUSH preserves the same structural asymmetry.
### 1. Deferred parsing still calls the same streaming parser
Function arguments are collected by recursively calling
`process_expression()` with adjusted flags. The parser is still live
while argument text is being collected.
### 2. Percent handling is gated by `PE_EVALUATE`
```c
if (!(eflags & PE_EVALUATE)) {
safe_chr('%', buff, bp);
(*str)++;
savec = **str;
safe_chr(savec, buff, bp);
(*str)++;
...
}
```
So `%` passes through literally in noeval collection.
### 3. Backslash handling is not gated by `PE_EVALUATE`
```c
if (eflags & PE_LITERAL) {
safe_chr('\\', buff, bp);
(*str)++;
break;
}
if (!(eflags & PE_EVALUATE))
safe_chr('\\', buff, bp);
(*str)++;
if (!**str)
goto exit_sequence;
/* FALL THROUGH */
default:
safe_chr(**str, buff, bp);
(*str)++;
```
Penn differs in exact literal-mode behavior and `% ` grammar, but it
still keeps scan-time evaluation state live and still couples
backslash-consumption to the character stream rather than to a prior
tokenization pass.
## What To Borrow
Borrow these ideas, not these codebases:
- Deferred-eval argument collection is still a scan pass.
- `%` semantics depend on eval state at the moment characters are read.
- `\` stripping happens per pass, not per final output.
- A later re-eval pass must be able to see adjacency that was preserved
through the earlier pass.
## What Not To Borrow
Do not pull in:
- the full substitution tables
- permission checks
- function dispatch
- recursion/accounting machinery
- buffer management details
- command parser behavior outside expression scanning
Those are engine concerns, not the minimal parser-semantics study.
## Immediate Design Pressure On 2.14 AST
The current AST/token model commits too early:
- `\\` becomes one escape token
- `% ` or `%x` becomes one substitution token
After that, later passes no longer have the original stream adjacency
that 2.13 and Penn relied on during deferred evaluation.
That is the specific behavior any compatibility design must restore,
whether by:
- a narrow deferred-pass streaming layer, or
- a broader split-lexer design
This note is the reduced reference for that work.

24
parser/hammer_refrozen.sh Executable file
View file

@ -0,0 +1,24 @@
#!/bin/bash
set -eu
run_case() {
local profile="$1"
local expr="$2"
echo "expr : $expr"
echo "profile : $profile"
./stream_passes --profile "$profile" --model stream "$expr"
./stream_passes --profile "$profile" --model frozen "$expr"
./stream_passes --profile "$profile" --model boundary "$expr"
echo
}
run_case mux213 '\\% capacity'
run_case mux213 '[switch(1,1,{\\% capacity})]'
run_case mux213 '[if(1,{\\% capacity})]'
run_case mux213 '[case(1,1,{\\% capacity})]'
run_case mux213 '[iter(a b,{\\% capacity})]'
run_case mux213 '[switch(1,1,\\% capacity)]'
run_case mux213 '[switch(1,1,{\\%b})]'
run_case mux213 '[iter(a b,{\\%b})]'
run_case penn '[iter(a b,{\\% capacity})]'

545
parser/stream_passes.cpp Normal file
View file

@ -0,0 +1,545 @@
#include <cctype>
#include <iostream>
#include <sstream>
#include <string>
#include <vector>
enum class Profile {
Mux213,
Penn
};
enum class PassMode {
Eval,
Noeval
};
enum class UnitType {
Text,
Escape,
Percent
};
struct Unit {
UnitType type;
std::string text;
};
static bool parse_profile(const std::string &s, Profile &profile)
{
if (s == "mux213") {
profile = Profile::Mux213;
return true;
}
if (s == "penn") {
profile = Profile::Penn;
return true;
}
return false;
}
static bool parse_pass(const std::string &s, PassMode &mode)
{
if (s == "eval") {
mode = PassMode::Eval;
return true;
}
if (s == "noeval") {
mode = PassMode::Noeval;
return true;
}
return false;
}
static std::vector<PassMode> parse_pipeline(const std::string &s)
{
std::vector<PassMode> out;
std::stringstream ss(s);
std::string item;
while (std::getline(ss, item, ',')) {
PassMode mode;
if (!parse_pass(item, mode)) {
out.clear();
return out;
}
out.push_back(mode);
}
return out;
}
static void gather_angle(const std::string &input, size_t &i, std::string &out)
{
if (i >= input.size() || input[i] != '<') {
return;
}
out.push_back(input[i++]);
while (i < input.size() && input[i] != '>') {
out.push_back(input[i++]);
}
if (i < input.size() && input[i] == '>') {
out.push_back(input[i++]);
}
}
static std::string gather_percent_unit(const std::string &input, size_t &i, Profile profile)
{
std::string out("%");
if (i >= input.size()) {
return out;
}
char ch = input[i];
char upper = static_cast<char>(std::toupper(static_cast<unsigned char>(ch)));
if (profile == Profile::Penn && ch == ' ') {
out.push_back(input[i++]);
return out;
}
if (ch >= '0' && ch <= '9') {
out.push_back(input[i++]);
} else if (upper == 'Q') {
out.push_back(input[i++]);
if (i < input.size() && input[i] == '<') {
gather_angle(input, i, out);
} else if (i < input.size()) {
out.push_back(input[i++]);
}
} else if (upper == 'V') {
out.push_back(input[i++]);
if (i < input.size() && std::isalpha(static_cast<unsigned char>(input[i]))) {
out.push_back(input[i++]);
}
} else if (profile == Profile::Penn && upper == 'W') {
out.push_back(input[i++]);
if (i < input.size() && std::isalpha(static_cast<unsigned char>(input[i]))) {
out.push_back(input[i++]);
}
} else if (upper == 'C' || upper == 'X') {
out.push_back(input[i++]);
if (profile == Profile::Penn) {
if (upper == 'X' && i < input.size()
&& std::isalpha(static_cast<unsigned char>(input[i]))) {
out.push_back(input[i++]);
}
} else if (i < input.size()) {
if (input[i] == '<') {
gather_angle(input, i, out);
} else {
out.push_back(input[i++]);
}
}
} else if (ch == '=') {
out.push_back(input[i++]);
if (i < input.size() && input[i] == '<') {
gather_angle(input, i, out);
}
} else if (upper == 'I') {
out.push_back(input[i++]);
if (i < input.size() && std::isdigit(static_cast<unsigned char>(input[i]))) {
out.push_back(input[i++]);
} else if (profile == Profile::Penn && i < input.size()
&& std::toupper(static_cast<unsigned char>(input[i])) == 'L') {
out.push_back(input[i++]);
}
} else if (profile == Profile::Penn && ch == '$') {
out.push_back(input[i++]);
if (i < input.size() && (std::isdigit(static_cast<unsigned char>(input[i]))
|| std::toupper(static_cast<unsigned char>(input[i])) == 'L')) {
out.push_back(input[i++]);
}
} else {
out.push_back(input[i++]);
}
return out;
}
static std::vector<Unit> freeze_units(const std::string &input, Profile profile)
{
std::vector<Unit> units;
size_t i = 0;
while (i < input.size()) {
if (input[i] == '\\') {
std::string unit;
unit.push_back(input[i++]);
if (i < input.size()) {
unit.push_back(input[i++]);
}
units.push_back({UnitType::Escape, unit});
continue;
}
if (input[i] == '%') {
++i;
units.push_back({UnitType::Percent, gather_percent_unit(input, i, profile)});
continue;
}
std::string text;
while (i < input.size() && input[i] != '\\' && input[i] != '%') {
text.push_back(input[i++]);
}
units.push_back({UnitType::Text, text});
}
return units;
}
static std::string evaluate_percent(const std::string &unit, Profile profile, bool eval)
{
if (!eval) {
return unit;
}
if (unit == "%%") {
return "%";
}
if (unit == "%b" || unit == "%B") {
return " ";
}
if (unit == "%t" || unit == "%T") {
return "\t";
}
if (unit == "%r" || unit == "%R") {
return (profile == Profile::Penn) ? "\n" : "\r\n";
}
if (profile == Profile::Penn && unit == "% ") {
return "% ";
}
if (unit.size() >= 2) {
return std::string(1, unit[1]);
}
return "%";
}
static std::string run_stream_pass(const std::string &input, Profile profile, bool eval)
{
std::string out;
size_t i = 0;
while (i < input.size()) {
if (input[i] == '\\') {
++i;
if (i < input.size()) {
out.push_back(input[i++]);
}
continue;
}
if (input[i] == '%') {
++i;
std::string unit = gather_percent_unit(input, i, profile);
out += evaluate_percent(unit, profile, eval);
continue;
}
out.push_back(input[i++]);
}
return out;
}
static std::string run_frozen_pass(const std::vector<Unit> &units, Profile profile, bool eval)
{
std::string out;
for (const Unit &unit : units) {
switch (unit.type) {
case UnitType::Text:
out += unit.text;
break;
case UnitType::Escape:
if (unit.text.size() >= 2) {
out.push_back(unit.text[1]);
}
break;
case UnitType::Percent:
out += evaluate_percent(unit.text, profile, eval);
break;
}
}
return out;
}
static std::string quote_string(const std::string &s)
{
std::string out("\"");
for (char ch : s) {
switch (ch) {
case '\\':
out += "\\\\";
break;
case '"':
out += "\\\"";
break;
case '\n':
out += "\\n";
break;
case '\r':
out += "\\r";
break;
case '\t':
out += "\\t";
break;
default:
out.push_back(ch);
break;
}
}
out.push_back('"');
return out;
}
static const char *unit_name(UnitType type)
{
switch (type) {
case UnitType::Text:
return "TEXT";
case UnitType::Escape:
return "ESC";
case UnitType::Percent:
return "PCT";
}
return "?";
}
static void print_usage()
{
std::cerr
<< "usage: ./stream_passes --profile mux213|penn"
<< " --model stream|frozen|refrozen|boundary"
<< " --passes noeval,eval [text]\n";
}
static bool starts_with(const std::string &s, const std::string &prefix)
{
return s.size() >= prefix.size() && s.compare(0, prefix.size(), prefix) == 0;
}
static bool ends_with(const std::string &s, const std::string &suffix)
{
return s.size() >= suffix.size()
&& s.compare(s.size() - suffix.size(), suffix.size(), suffix) == 0;
}
static bool extract_switch_body(const std::string &input, std::string &body)
{
const std::string prefix = "[switch(1,1,";
const std::string suffix = ")]";
if (!starts_with(input, prefix) || !ends_with(input, suffix)) {
return false;
}
body = input.substr(prefix.size(), input.size() - prefix.size() - suffix.size());
return true;
}
static bool extract_if_body(const std::string &input, std::string &body)
{
const std::string prefix = "[if(1,";
const std::string suffix = ")]";
if (!starts_with(input, prefix) || !ends_with(input, suffix)) {
return false;
}
body = input.substr(prefix.size(), input.size() - prefix.size() - suffix.size());
return true;
}
static bool extract_case_body(const std::string &input, std::string &body)
{
const std::string prefix = "[case(1,1,";
const std::string suffix = ")]";
if (!starts_with(input, prefix) || !ends_with(input, suffix)) {
return false;
}
body = input.substr(prefix.size(), input.size() - prefix.size() - suffix.size());
return true;
}
static bool extract_iter_body(const std::string &input, std::string &body, int &count)
{
const std::string prefix = "[iter(";
const std::string suffix = ")]";
if (!starts_with(input, prefix) || !ends_with(input, suffix)) {
return false;
}
std::string inner = input.substr(prefix.size(), input.size() - prefix.size() - suffix.size());
const std::string list_prefix = "a b,";
if (!starts_with(inner, list_prefix)) {
return false;
}
body = inner.substr(list_prefix.size());
count = 2;
return true;
}
static std::string strip_one_brace_layer(const std::string &s)
{
if (s.size() >= 2 && s.front() == '{' && s.back() == '}') {
return s.substr(1, s.size() - 2);
}
return s;
}
int main(int argc, char **argv)
{
Profile profile = Profile::Mux213;
std::string model = "stream";
std::vector<PassMode> passes;
std::string input;
for (int i = 1; i < argc; ++i) {
std::string arg(argv[i]);
if (arg == "--profile" && i + 1 < argc) {
if (!parse_profile(argv[++i], profile)) {
print_usage();
return 1;
}
} else if (arg == "--model" && i + 1 < argc) {
model = argv[++i];
} else if (arg == "--passes" && i + 1 < argc) {
passes = parse_pipeline(argv[++i]);
if (passes.empty()) {
print_usage();
return 1;
}
} else if (!input.empty()) {
input += " ";
input += arg;
} else {
input = arg;
}
}
if (passes.empty()) {
passes.push_back(PassMode::Eval);
}
if (input.empty()) {
if (!std::getline(std::cin, input)) {
return 1;
}
}
std::cout << "input : " << quote_string(input) << "\n";
if (model == "stream") {
std::string current = input;
for (size_t i = 0; i < passes.size(); ++i) {
bool eval = passes[i] == PassMode::Eval;
current = run_stream_pass(current, profile, eval);
std::cout << "pass " << (i + 1) << " : "
<< (eval ? "eval " : "noeval ")
<< "-> " << quote_string(current) << "\n";
}
return 0;
}
if (model == "frozen") {
std::vector<Unit> units = freeze_units(input, profile);
std::cout << "units :";
for (const Unit &unit : units) {
std::cout << " " << unit_name(unit.type) << "(" << quote_string(unit.text) << ")";
}
std::cout << "\n";
for (size_t i = 0; i < passes.size(); ++i) {
bool eval = passes[i] == PassMode::Eval;
std::string out = run_frozen_pass(units, profile, eval);
std::cout << "pass " << (i + 1) << " : "
<< (eval ? "eval " : "noeval ")
<< "-> " << quote_string(out) << "\n";
}
return 0;
}
if (model == "refrozen") {
std::string current = input;
for (size_t i = 0; i < passes.size(); ++i) {
bool eval = passes[i] == PassMode::Eval;
// Freeze the current string into tokens.
//
std::vector<Unit> units = freeze_units(current, profile);
std::cout << "units :";
for (const Unit &unit : units) {
std::cout << " " << unit_name(unit.type)
<< "(" << quote_string(unit.text) << ")";
}
std::cout << "\n";
// Evaluate the frozen tokens.
//
current = run_frozen_pass(units, profile, eval);
std::cout << "pass " << (i + 1) << " : "
<< (eval ? "eval " : "noeval ")
<< "-> " << quote_string(current) << "\n";
}
return 0;
}
if (model == "boundary") {
std::string body;
int iter_count = 0;
if (extract_switch_body(input, body)
|| extract_if_body(input, body)
|| extract_case_body(input, body)) {
std::cout << "boundary: deferred arg " << quote_string(body) << "\n";
std::string current = run_stream_pass(body, profile, false);
std::cout << "pass 1 : noeval -> " << quote_string(current) << "\n";
current = strip_one_brace_layer(current);
std::cout << "strip : braces -> " << quote_string(current) << "\n";
std::vector<Unit> units = freeze_units(current, profile);
std::cout << "units :";
for (const Unit &unit : units) {
std::cout << " " << unit_name(unit.type)
<< "(" << quote_string(unit.text) << ")";
}
std::cout << "\n";
current = run_frozen_pass(units, profile, true);
std::cout << "pass 2 : eval -> " << quote_string(current) << "\n";
return 0;
}
if (extract_iter_body(input, body, iter_count)) {
std::cout << "boundary: iter body " << quote_string(body) << "\n";
std::string collected = run_stream_pass(body, profile, false);
std::cout << "pass 1 : noeval -> " << quote_string(collected) << "\n";
std::string inner = strip_one_brace_layer(collected);
std::vector<Unit> units = freeze_units(inner, profile);
std::cout << "units :";
for (const Unit &unit : units) {
std::cout << " " << unit_name(unit.type)
<< "(" << quote_string(unit.text) << ")";
}
std::cout << "\n";
std::string one = run_frozen_pass(units, profile, true);
std::string out;
for (int i = 0; i < iter_count; ++i) {
if (i) {
out.push_back(' ');
}
out += one;
}
std::cout << "pass 2 : eval -> " << quote_string(out) << "\n";
return 0;
}
std::cout << "boundary: plain expression, no deferred boundary detected\n";
std::string current = run_stream_pass(input, profile, true);
std::cout << "pass 1 : eval -> " << quote_string(current) << "\n";
return 0;
}
print_usage();
return 1;
}