mmo-ref/fluffos
1
0
Fork 0
mirror of https://github.com/fluffos/fluffos synced 2026-08-12 18:26:06 -04:00
Code Issues Projects Releases Packages Wiki Activity
1959 commits 93 branches 142 tags 54 MiB
Commit graph

6 commits

Author SHA1 Message Date
Yucong Sun
6b6f169952
lpc-syntax: wire formatter into vscode extension, fix tokenizer/formatter bugs (#1259) 
* lpc-syntax: wire formatter into vscode extension, fix tokenizer/formatter bugs

- Register a DocumentFormattingEditProvider (Format Document / format-on-save)
  backed by format.mjs, gated by a new lpc.format.enabled setting; never lets
  a formatter error corrupt or block a save.
- Regenerate the grammar contract (grammar.y already had `ref` = '&' sugar
  that lpc-grammar.json/grammar.ebnf hadn't picked up) and make operator-list
  generation deterministic (secondary alphabetical sort key instead of
  relying on Python's randomized string-hash set ordering).
- tokenizer.mjs: fix template-interpolation brace scanning to skip nested
  strings/chars/comments/templates as opaque spans (a stray '}' inside e.g.
  `${ ch == '}' }` previously ended the interpolation early); fix char
  literals with variable-length \xHH/\NNN escapes being truncated.
- format.mjs: track array/mapping literal braces `({ ... })` separately from
  block braces so they don't affect indentation depth; force a flush after a
  trailing `//` comment so a second format pass can't swallow following code
  into it; fix an off-by-one that mis-indented every nested block; stop
  accumulating a blank line on re-format of a source that swallows to EOF.
- language-configuration.json: add onEnterRules for /** */ doc-comment
  continuation.
- Extend test.mjs with regression coverage for all of the above (59 checks).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XUhzkBiuWxX2M9RX94BckT

* lpc-syntax: fix heredoc, mapping-literal, case-colon, and indexing spacing in formatter

Verified tokenizer/highlighter already model heredoc (@/@@ text blocks)
correctly per parseHeredoc() in lexer_utils.cc. Found and fixed four real
formatter bugs, all in format.mjs:

- Mapping literals `([ ... ])` never got the array-literal treatment
  ({ ... }) got last session -- only '{'/'}' was tracked, not '['/']'.
  Generalized the brace-tracking into one combined stack covering both,
  distinguishing array/mapping literals from blocks/indexing by whether
  the bracket is immediately preceded by '('.
- Both array and mapping literals collapsed onto a single line even when
  the source spread them across many lines, which mangles real mudlib
  data tables. Multi-line literals now preserve their line breaks and
  indent one level, while short single-line literals still collapse as
  before.
- `case`/`default` labels rendered as "case 1 :" (space before the
  colon) -- checked against testsuite convention (843:6 no-space vs
  space) and fixed; ternary/mapping colons are unaffected.
- `a[0]`/`b[1..2]` rendered as "a [0]" / "b [1 .. 2]" (space before '['
  and around the range operator) -- checked against testsuite
  convention (1603:15, 197:7) and fixed; varargs '...' spacing is
  unaffected.

Fixing the heredoc terminator to force a line break (matching the
documented @/@@ style, since the driver rescans trailing code after the
terminator on its own) exposed a latent bug: the ';'-triggered flush
computed paren depth over just the current line buffer, which goes
negative (never reaches the expected 0) once a forced mid-statement
flush leaves an unmatched ')' behind. Replaced it with a running
paren-nesting counter across the whole pass.

Re-verified via an independent 723-file sweep of testsuite/ (tokenizer
lossless reconstruction, formatter idempotency + literal-content
preservation, highlighter lossless reconstruction, lint false-positive
check on real files): all clean except one confirmed non-issue
(intentional trailing-whitespace trim on a directive line). Added 6
regression tests (65 total) and regenerated the vscode/lib/format.mjs copy.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XUhzkBiuWxX2M9RX94BckT

* lpc-syntax: fix highlighting gaps found by auditing against grammar.y and docs/lpc

Cross-referenced the highlighting pipeline (tokenizer.mjs kind classification,
highlight.mjs, generate_ebnf.py's TextMate grammar generation) against
lexer_utils.cc's reswords[] table and every page under docs/lpc/.

Verified already correct, no change: `inherited` is genuinely not a keyword
(any identifier before `::` is treated uniformly, matching docs/lpc/constructs/
inherit.md's own examples); the full type/modifier keyword lists match
reswords[] exactly; range/spread/optional-chaining/nullish operators already
have distinct scopes; `array` staying highlighted as a keyword despite
ARRAY_RESERVED_WORD being #undef'd by default is a pre-existing, low-impact
gap not worth a schema change to plumb through.

Real gaps fixed, all in generate_ebnf.py/highlight.mjs (never hand-edit the
generated lpc-grammar.json/lpc.tmLanguage.json themselves):

- "struct" was an undocumented reserved word (lexer_utils.cc maps both
  "class" and "struct" to L_CLASS, both gated on unconditionally-defined
  macros) but TOKEN_SPEC only listed "class" -- struct declarations
  highlighted as a plain identifier. Added the second spelling.
- class/struct are type-introducing keywords, not control flow -- split them
  out of keyword.control.lpc into their own storage.type.class.lpc scope,
  matching how other C-family TextMate grammars color struct/class.
- The function-call heuristic (identifier immediately before '(') had no
  guard against matching a reserved word, relying only on TextMate's
  same-position rule-order tie-break. Added an explicit negative lookahead
  over the full keyword/type/modifier set so `if (`/`new (`/etc. can never
  be misscoped as entity.name.function.lpc.
- $1/$2 closure params had no visual distinction in the HTML highlighter
  (they intentionally still tokenize as plain 'identifier', since format.mjs
  keys spacing off that kind) -- fixed at the highlight.mjs layer with a
  dedicated lpc-param class, matching the TextMate grammar's existing
  dollar-params rule.
- Illegal/unknown characters rendered with no visual flag in the HTML
  highlighter -- added an lpc-unknown class so invalid syntax is visible.

Re-verified via an independent 723-file sweep of testsuite/ (highlighter
lossless reconstruction: 0 crashes, 0 mismatches) and the full test suite
(70 checks, all passing). tokenizer.mjs, lint.mjs, and format.mjs are
untouched.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XUhzkBiuWxX2M9RX94BckT

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-07-12 15:42:14 -04:00
Yucong Sun
fd705f0634 docs: reconcile FFI plan + preprocessor/diagnostics docs with implementation 
Doc-only review pass checking every branch-authored doc against the
shipped code. Fixes where docs described intent rather than the result:

- docs/driver/ffi-plan.md (began life as a plan, drifted from the
  shipped package): ffi_status() returns a mapping, not mixed*;
  callbacks are implemented and in-scope (ffi_callback/_addr/_free added
  to the efun surface and moved out of "v2 deferred"); the DEBUGMALLOC
  section now describes the actual std::unordered_map + TAG_BUFFER scheme
  (no TAG_FFI / mark hook exists); valid_ffi gates load/symbol/prepare/
  callback; testing is the 20-file LPC suite + tools/ffi/test.py (there
  is no GTest fixture).
- docs/lpc/preprocessor/index.md: document the #warn directive.
- docs/lpc/diagnostics.md + preprocessor/pragma.md: show_error_context is
  a legacy flag that no longer changes clang-style compiler diagnostics
  (render_diagnostic ignores PRAGMA_ERROR_CONTEXT; only the runtime
  smart_log path still reads it).
- tools/lpc-syntax/README.md: test.mjs is 49 assertions, not 47.

Verified accurate, no change needed: AGENTS.md, testsuite/README.md,
compiler/internal/README.md, and the source-files / float / strings /
text_blocks / define / conditionals / inherit / include docs, sprintf %g,
and the tools/ffi + vscode READMEs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-07-09 20:48:48 -04:00
Yucong Sun
55e47edc07 Float exponent notation: 2.5e2, 1e3, 2E-5 are float literals now 
Neither master's hand-written scanner nor the flex lexer ever accepted
exponents ("2.5e2" lexed as REAL(2.5) IDENT(e2)) -- but the hand-written
EBNF already documented the form and it should exist. Two lexer rules
add it: an optional exponent on the fraction form, and a bare-exponent
form ("1e3"); "1e" without exponent digits stays NUMBER(1) IDENT(e),
"1..5" stays a range, hex "0x1E3" is untouched, and '_' separators work
in all parts. strtod() already converted exponents, so only the
patterns changed.

Pinned at every layer: syntax_literals.lpc (2.5e2, 2.5e-2, 1.5E+2,
1e3, 1_2e4), the JS tokenizer + generated tmLanguage (with node tests
for the exponent forms and the "1e" non-exponent), the lexical EBNF
(realLiteral | exponent rules, replacing a half-aspirational entry),
and docs/lpc/types/float.md (which also wrongly claimed single
precision -- LPC_FLOAT is a double).

Verified: testsuite x3 + ctest 297 (ASan Debug), RelWithDebInfo full
ctest, lpc-syntax node tests (50).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-09 20:48:48 -04:00
Yucong Sun
62e29ab5f2 Syntax coverage matrix; fix name::fn() super calls into .lpc parents 
Four new compiler test files pin the language surface end-to-end (121
checks): syntax_literals (integer bases, digit separators, float forms
including the trailing-dot spelling, char/string escapes, adjacent
concatenation, in-string line continuation, constant folding),
syntax_operators (C-precedence pins across the merged token families,
every compound assignment including <<= >>= &&= ||= ??=, ?? vs 0,
short-circuiting, ++/-- lvalue forms, comma expressions, all range
spellings, casts), syntax_control (do/while, for with comma clauses and
empty header, dangling else, string switches, open-ended case ranges,
fallthrough, catch and time_expression in expression AND block forms),
and syntax_functions (:: and named-scope super calls, varargs + call
spread, expression defaults, function-typed lambdas, new(class X,
field: value) initializers).

Writing them caught a real regression: the named-scope super-call
matcher (compiler.cc) hardcoded a 2-byte ".c" suffix when comparing
`name::fn()` against inherited program names, so every named super
call into an .lpc-compiled parent failed to resolve -- and the entire
testsuite had no named-scope call to notice. The matcher now strips
the real extension. fail/lambda_typed_return.lpc pins that lambdas
accept only the 'function' return type.

The review also surfaced three language facts the JS tooling had
wrong or the tests mis-assumed, now pinned: LPC floats have NO
exponent notation (tokenizer.mjs and the generated tmLanguage
accepted "2.5e2" as one number; the hand-authored EBNF was already
right), there is no open-low index range "[..n]" (only case labels
support it), and casts are type assertions, not conversions.

Verified: testsuite x3 + ctest 297 (ASan Debug), clang RelWithDebInfo
sanitizer full suite, RelWithDebInfo full ctest, lpc-syntax node
tests (48).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-09 20:48:48 -04:00
Yucong Sun
a0f6c74561 Grammar contract moves to tools/lpc-syntax; add VS Code extension 
generate_ebnf.py, grammar_lexical.ebnf.in and the generated
grammar.ebnf relocate from src/compiler/internal/ to tools/lpc-syntax/,
putting every grammar-contract asset in one place (the CMake
generate_ebnf target and all doc references follow).

New tools/lpc-syntax/vscode/ extension, driven by the same contract:
- syntax highlighting via a TextMate grammar GENERATED from
  lpc-grammar.json (keywords/types/modifiers/operators can never drift
  from grammar.y), plus templates with ${} interpolation, @/@@ text
  blocks with backreference terminators, directives, (: :) functionals
- structural diagnostics as you type from the new lint.mjs (illegal
  characters, unterminated strings/templates/comments/text blocks,
  unbalanced brackets, mismatched #if/#elif/#else/#endif)
- optional REAL compiler errors on save: with lpc.lpcc.path and
  lpc.lpcc.configFile set, the file is compiled by lpcc and its
  clang-style file:line:col diagnostics are mapped inline (verified
  against actual lpcc output), including into #include'd files

generate_ebnf.py now also emits the extension's generated assets
(syntaxes/lpc.tmLanguage.json and self-contained vscode/lib/ copies of
tokenizer/lint/grammar -- a packaged extension cannot reach ../).
test.mjs grows to 47 assertions covering the linter and the generated
assets.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-09 20:48:48 -04:00
Yucong Sun
a78f25eeaa Grammar as a machine contract: 3-layer EBNF + lpc-grammar.json + JS 
tokenizer/highlighter/formatter

The checked-in grammar.ebnf was stale (still referenced 8 tokens deleted
in the token diet, none of the merged ones, and defined no terminals).
Plan executed, landed as one commit:

1. REFRESH: the Syntax layer regenerates from today's grammar.y via the
   existing bison --xml pipeline (token diet, '(' '{' pairs, functional
   productions all current).
2. LAYER: generate_ebnf.py now composes grammar.ebnf from three layers
   -- a hand-authored LEXICAL layer (identifiers, numbers with
   hex/binary/underscores, strings + every escape form, template
   literals with ${} interpolation, @/@@ text blocks, comments, keyword
   families, functional (: :) forms) and a PREPROCESSOR layer (all 14
   directives, function-like macros with # and ##, token-based #if
   expressions, line continuations, builtin macros), both living in
   grammar_lexical.ebnf.in so regeneration can never lose them, plus
   the generated Syntax layer.
3. MACHINE CONTRACT: the same run emits tools/lpc-syntax/
   lpc-grammar.json -- keywords, type/modifier keywords, operators in
   longest-match order, punctuation, directives, token categories, and
   all 265 productions. The generator ASSERTS every grammar.y terminal
   is categorized: a new token without a spec entry fails regeneration
   loudly, so the artifacts cannot silently go stale again. Sources of
   truth: lexer.l's operator rules and lexer_utils.cc's reserved-word
   table.
4. JS TOOLING (tools/lpc-syntax, dependency-free ESM): a grammar-driven
   tokenizer (templates with recursively tokenized ${} interpolation,
   text blocks, directive lines with continuations, longest-match
   operators), an HTML syntax highlighter (highlightLPC + defaultCss),
   and a basic idempotent formatter (brace reindentation, statement
   splitting, operator spacing, directives at column 0, strings/
   comments verbatim). 27 assertions in test.mjs, run with plain node.

CMake generate_ebnf target produces both artifacts; docs in
tools/lpc-syntax/README.md, AGENTS.md section 11, and the compiler
README. Verified: node test suite (27/27), full driver battery (287
tests under ASan/UBSan).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
 2026-07-09 20:48:48 -04:00