mirror of
https://github.com/fluffos/fluffos
synced 2026-08-12 18:26:06 -04:00
6b6f169952
* lpc-syntax: wire formatter into vscode extension, fix tokenizer/formatter bugs
- Register a DocumentFormattingEditProvider (Format Document / format-on-save)
backed by format.mjs, gated by a new lpc.format.enabled setting; never lets
a formatter error corrupt or block a save.
- Regenerate the grammar contract (grammar.y already had `ref` = '&' sugar
that lpc-grammar.json/grammar.ebnf hadn't picked up) and make operator-list
generation deterministic (secondary alphabetical sort key instead of
relying on Python's randomized string-hash set ordering).
- tokenizer.mjs: fix template-interpolation brace scanning to skip nested
strings/chars/comments/templates as opaque spans (a stray '}' inside e.g.
`${ ch == '}' }` previously ended the interpolation early); fix char
literals with variable-length \xHH/\NNN escapes being truncated.
- format.mjs: track array/mapping literal braces `({ ... })` separately from
block braces so they don't affect indentation depth; force a flush after a
trailing `//` comment so a second format pass can't swallow following code
into it; fix an off-by-one that mis-indented every nested block; stop
accumulating a blank line on re-format of a source that swallows to EOF.
- language-configuration.json: add onEnterRules for /** */ doc-comment
continuation.
- Extend test.mjs with regression coverage for all of the above (59 checks).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XUhzkBiuWxX2M9RX94BckT
* lpc-syntax: fix heredoc, mapping-literal, case-colon, and indexing spacing in formatter
Verified tokenizer/highlighter already model heredoc (@/@@ text blocks)
correctly per parseHeredoc() in lexer_utils.cc. Found and fixed four real
formatter bugs, all in format.mjs:
- Mapping literals `([ ... ])` never got the array-literal treatment
({ ... }) got last session -- only '{'/'}' was tracked, not '['/']'.
Generalized the brace-tracking into one combined stack covering both,
distinguishing array/mapping literals from blocks/indexing by whether
the bracket is immediately preceded by '('.
- Both array and mapping literals collapsed onto a single line even when
the source spread them across many lines, which mangles real mudlib
data tables. Multi-line literals now preserve their line breaks and
indent one level, while short single-line literals still collapse as
before.
- `case`/`default` labels rendered as "case 1 :" (space before the
colon) -- checked against testsuite convention (843:6 no-space vs
space) and fixed; ternary/mapping colons are unaffected.
- `a[0]`/`b[1..2]` rendered as "a [0]" / "b [1 .. 2]" (space before '['
and around the range operator) -- checked against testsuite
convention (1603:15, 197:7) and fixed; varargs '...' spacing is
unaffected.
Fixing the heredoc terminator to force a line break (matching the
documented @/@@ style, since the driver rescans trailing code after the
terminator on its own) exposed a latent bug: the ';'-triggered flush
computed paren depth over just the current line buffer, which goes
negative (never reaches the expected 0) once a forced mid-statement
flush leaves an unmatched ')' behind. Replaced it with a running
paren-nesting counter across the whole pass.
Re-verified via an independent 723-file sweep of testsuite/ (tokenizer
lossless reconstruction, formatter idempotency + literal-content
preservation, highlighter lossless reconstruction, lint false-positive
check on real files): all clean except one confirmed non-issue
(intentional trailing-whitespace trim on a directive line). Added 6
regression tests (65 total) and regenerated the vscode/lib/format.mjs copy.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XUhzkBiuWxX2M9RX94BckT
* lpc-syntax: fix highlighting gaps found by auditing against grammar.y and docs/lpc
Cross-referenced the highlighting pipeline (tokenizer.mjs kind classification,
highlight.mjs, generate_ebnf.py's TextMate grammar generation) against
lexer_utils.cc's reswords[] table and every page under docs/lpc/.
Verified already correct, no change: `inherited` is genuinely not a keyword
(any identifier before `::` is treated uniformly, matching docs/lpc/constructs/
inherit.md's own examples); the full type/modifier keyword lists match
reswords[] exactly; range/spread/optional-chaining/nullish operators already
have distinct scopes; `array` staying highlighted as a keyword despite
ARRAY_RESERVED_WORD being #undef'd by default is a pre-existing, low-impact
gap not worth a schema change to plumb through.
Real gaps fixed, all in generate_ebnf.py/highlight.mjs (never hand-edit the
generated lpc-grammar.json/lpc.tmLanguage.json themselves):
- "struct" was an undocumented reserved word (lexer_utils.cc maps both
"class" and "struct" to L_CLASS, both gated on unconditionally-defined
macros) but TOKEN_SPEC only listed "class" -- struct declarations
highlighted as a plain identifier. Added the second spelling.
- class/struct are type-introducing keywords, not control flow -- split them
out of keyword.control.lpc into their own storage.type.class.lpc scope,
matching how other C-family TextMate grammars color struct/class.
- The function-call heuristic (identifier immediately before '(') had no
guard against matching a reserved word, relying only on TextMate's
same-position rule-order tie-break. Added an explicit negative lookahead
over the full keyword/type/modifier set so `if (`/`new (`/etc. can never
be misscoped as entity.name.function.lpc.
- $1/$2 closure params had no visual distinction in the HTML highlighter
(they intentionally still tokenize as plain 'identifier', since format.mjs
keys spacing off that kind) -- fixed at the highlight.mjs layer with a
dedicated lpc-param class, matching the TextMate grammar's existing
dollar-params rule.
- Illegal/unknown characters rendered with no visual flag in the HTML
highlighter -- added an lpc-unknown class so invalid syntax is visible.
Re-verified via an independent 723-file sweep of testsuite/ (highlighter
lossless reconstruction: 0 crashes, 0 mismatches) and the full test suite
(70 checks, all passing). tokenizer.mjs, lint.mjs, and format.mjs are
untouched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XUhzkBiuWxX2M9RX94BckT
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|---|---|---|
| .. | ||
| vscode | ||
| format.mjs | ||
| generate_ebnf.py | ||
| grammar.ebnf | ||
| grammar_lexical.ebnf.in | ||
| highlight.mjs | ||
| lint.mjs | ||
| lpc-grammar.json | ||
| README.md | ||
| test.mjs | ||
| tokenizer.mjs | ||
lpc-syntax — grammar-driven LPC tokenizer, highlighter, formatter
Dependency-free JavaScript (Node ≥ 16, ESM) tooling driven by
lpc-grammar.json — a machine-readable contract generated from the
driver's own grammar.y + lexer.l reserved-word/operator tables. The
tooling can never drift from the compiler: regeneration fails loudly
if grammar.y grows a token the spec doesn't categorize.
Files
| File | What |
|---|---|
generate_ebnf.py |
The generator: composes grammar.ebnf, emits lpc-grammar.json, and generates the VS Code extension's assets (vscode/syntaxes/lpc.tmLanguage.json, vscode/lib/*). Run via the generate_ebnf CMake target. |
grammar_lexical.ebnf.in |
Hand-authored Lexical + Preprocessor EBNF layers (terminals, escapes, templates, text blocks, directive grammar). |
grammar.ebnf |
Generated — the three-layer EBNF (hand-authored layers + bison-derived Syntax layer from src/compiler/internal/grammar.y). |
lpc-grammar.json |
Generated — keywords, type/modifier keywords, operators (longest-match ordered), punctuation, preprocessor directives, token categories, and all grammar productions. Never hand-edit. |
tokenizer.mjs |
Grammar-driven scanner: strings/escapes, template literals with nested ${} interpolation (interpolated code is recursively tokenized), @/@@ text blocks, comments, # directive lines with \ continuations, numbers (hex/binary/underscores/reals), keyword classification, longest-match operators. |
highlight.mjs |
highlightLPC(source) → HTML with <span class="lpc-..."> runs; defaultCss export. |
format.mjs |
formatLPC(source) → basic formatter: brace-depth reindentation, statement-per-line, operator spacing, directives at column 0, comments/strings/text blocks verbatim. Deterministic and idempotent. Powers the VS Code extension's Format Document / format-on-save. |
lint.mjs |
lintLPC(source) → structural diagnostics (illegal characters, unterminated literals, unbalanced brackets, mismatched conditionals) with 1-based positions. Powers the VS Code extension's as-you-type diagnostics. |
vscode/ |
VS Code extension: declarative highlighting (generated TextMate grammar), structural diagnostics as you type, Format Document / format-on-save, real lpcc compiler errors on save. See vscode/README.md. |
test.mjs |
node tools/lpc-syntax/test.mjs — 49 assertions, no dependencies (tokenizer, highlighter, formatter, linter, generated VS Code assets). |
Regenerating the grammar contract
cmake --build build --target generate_ebnf
This regenerates grammar.ebnf (the three-layer EBNF: hand-authored
Lexical and Preprocessor layers from grammar_lexical.ebnf.in,
plus the bison-derived Syntax layer), lpc-grammar.json, and the VS
Code extension's generated assets (vscode/syntaxes/lpc.tmLanguage.json
plus the self-contained copies under vscode/lib/). Rerun it whenever
grammar.y, the reserved-word table, the lexical layers, or
tokenizer.mjs/lint.mjs change — an uncategorized new token aborts
generation with an error naming it.
Usage
import { tokenize } from './tokenizer.mjs';
import { highlightLPC, defaultCss } from './highlight.mjs';
import { formatLPC } from './format.mjs';
const html = highlightLPC('int f() { return `v=${x}`; }');
const pretty = formatLPC('int f(){return 1;}');