mirror of
https://github.com/fluffos/fluffos
synced 2026-08-12 18:26:06 -04:00
11 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Claude
|
3bdbe9676e |
www: real ws backpressure fix, teardown leak fix, src/www docs, websocket smoke test in CI
The lws output wedge addressed in the previous commit was incompletely understood: under genuine backpressure (peer slow or paused, kernel send buffer full) a connection still froze permanently. Root cause, established by tracing the writeable-request plumbing end to end: lws_send_pipe_choked() is true not only when lws holds a truncated send (lws re-arms the writeable callback itself then) but also when a zero-timeout poll(POLLOUT) reports the socket simply full -- and in that case every lws_write() has fully succeeded, lws has nothing pending, and nobody re-arms anything. Fix, per lws README.coding.md: whenever the drain loop exits with data still queued in pss->buffer, request the next writeable callback. Queued data always has a callback requested, so no exit path can strand output. Also: - LWS_CALLBACK_CLOSED frees the session evbuffer unconditionally: on driver-initiated closes (e.g. the mudlib destructing the interactive) close_user_websocket() nulls pss->user first, and the old early return leaked the buffer every time. - src/www/README.md + src/www/AGENTS.md: architecture doc and agent checklist for the web terminal pages (xterm.js/telnet.js layering, vendor policy, packaging, testing, the wedge mechanism). - tools/ws-smoke.js, wired into CI on the Clang Debug matrix entries (with and without sanitizers): a dependency-free node websocket client that boots the real driver and exercises the http mount, telnet + ascii subprotocols through the shared src/www/telnet.js, SGA char-mode switching, live TUI streaming, TLS, and -- the actual regression gate -- forced-backpressure bursts (paused socket, ~4.8MB) on the plain and TLS ports plus a destruct-while-choked teardown check. All three backpressure checks fail on the unfixed driver; neither GTest nor the LPC suite exercises any websocket client traffic. - Fix stale src/www/wasm/vendor/ path references left from the vendor directory move (src/wasm/README.md, docs/build-wasm.md, release.yml). Validated on the ASan Debug build: forced-backpressure repros recover the full burst on both subprotocols, destruct-while-choked clean under ASan, ws-smoke 17/17, GTest 312/312, LPC testsuite clean. |
||
|
Claude
|
f45461cb97 |
web terminals on xterm.js; TUI library fixes; emsdk pin; lws output-wedge fix
One coherent change: make the mudlib TUI library (/std/tui) work in the browser on BOTH web terminals, harden everything the work surfaced, and pin the toolchain that broke it. ## Web terminals: xterm.js + a shared telnet client * Vendor @xterm/xterm 6.0.0 + @xterm/addon-fit 0.11.0 (dist files byte-exact from the official npm tarballs, licenses included) under src/www/vendor/. xterm.js does the terminal emulation on both pages: rendering, SGR 16/256/truecolor, alternate screen, cursor state, wide characters, scrollback, mouse reporting, bracketed paste, and keyboard encoding (the whole dialect testsuite/std/tui/keys.lpc decodes, including C-_ undo). * src/www/telnet.js -- one telnet option engine shared by both pages (transport-agnostic; hooks for page-specific options): ECHO masks the password prompt, WILL/WONT SGA -- the driver's char-mode signal (set_charmode) -- automatically switches between the line-input bar and raw keystroke streaming, NAWS reports real terminal geometry from the fit addon and re-reports on resize (driving the window_size apply), TTYPE answers xterm-256color. * src/www/wasm/index.html (the wasm shell): output/input rides xterm.js; the page keeps the synchronous-bridge queueing (sends flush outside receive() -- the wasm bridge re-enters the parser otherwise), the error modal, and the jsbridge handlers. Also guards crypto.getRandomValues() against views backed by resizable ArrayBuffers (see the emsdk section below). * src/www/index.html (the websocket client for the native driver): rewritten on the same stack, replacing a parseANSI() that stripped all cursor sequences (no TUI possible) and a telnet layer whose option bytes leaked into the text stream and never answered negotiations. Passwords now mask, char mode works, GMCP/MSP kept (dead TelnetOverWebSocket/handler classes removed); ws frames arrive as arraybuffers (no Blob/FileReader path); UTF-8 and telnet sequences survive frame splits (streaming decode + stateful parser). * tools/wasm/pack-mudlib.sh and the release zip ship vendor/ and telnet.js next to index.html in both layouts. ## Driver: websocket output wedged permanently on multi-window bursts Re-arming the writeable event from inside LWS_CALLBACK_SERVER_WRITEABLE is lossy with the libevent event lib: after the user callback returns, lws core clears POLLOUT and its pollfd bookkeeping desyncs from the evlib watcher -- the request is dropped and every later lws_callback_on_writable() no-ops, freezing output on that connection for good. First bites on any burst larger than one 2048-byte window (e.g. a full-screen TUI frame; no test had ever pushed one through a ws client). The ws_telnet.cc/ws_ascii.cc handlers now drain the evbuffer in a loop gated on lws_send_pipe_choked(); a choked write is flushed by lws's own core-managed POLLOUT path, which fires the callback again. Found by the browser end-to-end run below; documented in AGENTS.md 14. ## TUI library (/std/tui): review fixes + features Fixes: wslice() dropped combining marks from every sliced render; ESC[1;mR (modified F3) misdecoded as a cursor position report; readline lost the left scroll marker when a line overflowed both viewport edges; stray mouse events cancelled incremental search; Tab on a unique already-complete match missed the trailing space; menu lines wider than the terminal wrapped and desynced the in-place repaint (width now fed by the glue and re-fed on NAWS resize); the menu overflow indicator only showed below the window; backward focus cycling from the initial state skipped the last widget; a terminal-initiated close (disconnect, tui_destroy) leaked the app clone and its widgets -- teardown now runs through a reentry-guarded app_quit() in both directions. Features (from the README's own deferred list): readline C-_ undo (per-keystroke snapshots); pterm-style type-to-filter in select/ multiselect (results index the original choices); mouse-wheel scrolling in list/table/tree/log (wheel no longer click-selects); table clicks honour the header offset; tree Left on a leaf jumps to its parent. All pinned by testsuite/single/tests/std/tui/fixes.lpc, including the app teardown cycle via a runtime-written mock terminal. ## Toolchain: pin emsdk, guard random_get() emsdk 6.0.2 defaulted GROWABLE_ARRAYBUFFERS=1, making every ALLOW_MEMORY_GROWTH build's heap a resizable ArrayBuffer (wasmMemory.toResizableBuffer()) in browsers shipping the wasm rab-integration -- and two emscripten runtime paths pass raw HEAPU8.subarray() views into Web APIs that reject resizable-backed views: random_get() -> crypto.getRandomValues() (threw at boot) and UTF8ToString() -> TextDecoder (broke jsbridge). 6.0.3 reverted the default AND fixed the string codegen (getUnsharedTextDecoderView -> getHeapViewOrCopy), but random_get() is still unguarded upstream. Reproduced and certified against real 6.0.2/6.0.3 toolchains in Chromium (--js-flags=--experimental-wasm-rab-integration): 6.0.2 unpatched throws the exact boot error, 6.0.2 + the page's getRandomValues wrapper passes, 6.0.3 passes. CI now installs a pinned emsdk-ver input (default 6.0.3) instead of "latest". ## Verification * Native Debug+ASan: GTest 312/312; full LPC suite (574 files, per-file ref-count checker) x3 across the work; TUI test dir x3 randomized. * LPC suite inside the wasm driver under node: 5274 checks, 574 files. * Browser e2e (Playwright + Chromium): wasm shell 21/21 checks (charts/SGR, char-mode auto-switch, readline editing + undo + history, select with filter, multiselect/confirm, full-screen app, dashboard live repaint + NAWS resize relayout); websocket client against the native driver 13/13 twice on one instance (plus an ascii-subprotocol multi-window burst) -- the flow that caught the lws wedge. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018UvX1HbmGk9zWGBsW4Rkcq |
||
|
Yucong Sun
|
6b6f169952
|
lpc-syntax: wire formatter into vscode extension, fix tokenizer/formatter bugs (#1259)
* lpc-syntax: wire formatter into vscode extension, fix tokenizer/formatter bugs
- Register a DocumentFormattingEditProvider (Format Document / format-on-save)
backed by format.mjs, gated by a new lpc.format.enabled setting; never lets
a formatter error corrupt or block a save.
- Regenerate the grammar contract (grammar.y already had `ref` = '&' sugar
that lpc-grammar.json/grammar.ebnf hadn't picked up) and make operator-list
generation deterministic (secondary alphabetical sort key instead of
relying on Python's randomized string-hash set ordering).
- tokenizer.mjs: fix template-interpolation brace scanning to skip nested
strings/chars/comments/templates as opaque spans (a stray '}' inside e.g.
`${ ch == '}' }` previously ended the interpolation early); fix char
literals with variable-length \xHH/\NNN escapes being truncated.
- format.mjs: track array/mapping literal braces `({ ... })` separately from
block braces so they don't affect indentation depth; force a flush after a
trailing `//` comment so a second format pass can't swallow following code
into it; fix an off-by-one that mis-indented every nested block; stop
accumulating a blank line on re-format of a source that swallows to EOF.
- language-configuration.json: add onEnterRules for /** */ doc-comment
continuation.
- Extend test.mjs with regression coverage for all of the above (59 checks).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XUhzkBiuWxX2M9RX94BckT
* lpc-syntax: fix heredoc, mapping-literal, case-colon, and indexing spacing in formatter
Verified tokenizer/highlighter already model heredoc (@/@@ text blocks)
correctly per parseHeredoc() in lexer_utils.cc. Found and fixed four real
formatter bugs, all in format.mjs:
- Mapping literals `([ ... ])` never got the array-literal treatment
({ ... }) got last session -- only '{'/'}' was tracked, not '['/']'.
Generalized the brace-tracking into one combined stack covering both,
distinguishing array/mapping literals from blocks/indexing by whether
the bracket is immediately preceded by '('.
- Both array and mapping literals collapsed onto a single line even when
the source spread them across many lines, which mangles real mudlib
data tables. Multi-line literals now preserve their line breaks and
indent one level, while short single-line literals still collapse as
before.
- `case`/`default` labels rendered as "case 1 :" (space before the
colon) -- checked against testsuite convention (843:6 no-space vs
space) and fixed; ternary/mapping colons are unaffected.
- `a[0]`/`b[1..2]` rendered as "a [0]" / "b [1 .. 2]" (space before '['
and around the range operator) -- checked against testsuite
convention (1603:15, 197:7) and fixed; varargs '...' spacing is
unaffected.
Fixing the heredoc terminator to force a line break (matching the
documented @/@@ style, since the driver rescans trailing code after the
terminator on its own) exposed a latent bug: the ';'-triggered flush
computed paren depth over just the current line buffer, which goes
negative (never reaches the expected 0) once a forced mid-statement
flush leaves an unmatched ')' behind. Replaced it with a running
paren-nesting counter across the whole pass.
Re-verified via an independent 723-file sweep of testsuite/ (tokenizer
lossless reconstruction, formatter idempotency + literal-content
preservation, highlighter lossless reconstruction, lint false-positive
check on real files): all clean except one confirmed non-issue
(intentional trailing-whitespace trim on a directive line). Added 6
regression tests (65 total) and regenerated the vscode/lib/format.mjs copy.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XUhzkBiuWxX2M9RX94BckT
* lpc-syntax: fix highlighting gaps found by auditing against grammar.y and docs/lpc
Cross-referenced the highlighting pipeline (tokenizer.mjs kind classification,
highlight.mjs, generate_ebnf.py's TextMate grammar generation) against
lexer_utils.cc's reswords[] table and every page under docs/lpc/.
Verified already correct, no change: `inherited` is genuinely not a keyword
(any identifier before `::` is treated uniformly, matching docs/lpc/constructs/
inherit.md's own examples); the full type/modifier keyword lists match
reswords[] exactly; range/spread/optional-chaining/nullish operators already
have distinct scopes; `array` staying highlighted as a keyword despite
ARRAY_RESERVED_WORD being #undef'd by default is a pre-existing, low-impact
gap not worth a schema change to plumb through.
Real gaps fixed, all in generate_ebnf.py/highlight.mjs (never hand-edit the
generated lpc-grammar.json/lpc.tmLanguage.json themselves):
- "struct" was an undocumented reserved word (lexer_utils.cc maps both
"class" and "struct" to L_CLASS, both gated on unconditionally-defined
macros) but TOKEN_SPEC only listed "class" -- struct declarations
highlighted as a plain identifier. Added the second spelling.
- class/struct are type-introducing keywords, not control flow -- split them
out of keyword.control.lpc into their own storage.type.class.lpc scope,
matching how other C-family TextMate grammars color struct/class.
- The function-call heuristic (identifier immediately before '(') had no
guard against matching a reserved word, relying only on TextMate's
same-position rule-order tie-break. Added an explicit negative lookahead
over the full keyword/type/modifier set so `if (`/`new (`/etc. can never
be misscoped as entity.name.function.lpc.
- $1/$2 closure params had no visual distinction in the HTML highlighter
(they intentionally still tokenize as plain 'identifier', since format.mjs
keys spacing off that kind) -- fixed at the highlight.mjs layer with a
dedicated lpc-param class, matching the TextMate grammar's existing
dollar-params rule.
- Illegal/unknown characters rendered with no visual flag in the HTML
highlighter -- added an lpc-unknown class so invalid syntax is visible.
Re-verified via an independent 723-file sweep of testsuite/ (highlighter
lossless reconstruction: 0 crashes, 0 mismatches) and the full test suite
(70 checks, all passing). tokenizer.mjs, lint.mjs, and format.mjs are
untouched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XUhzkBiuWxX2M9RX94BckT
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
Yucong Sun
|
6a25f5fe9b
|
wasm: shrink the driver to ~0.8MB over the wire (33.5MB -> 3.5MB raw) (#1243)
The wasm binary was dominated by the stock 30MB ICU data archive; the driver only reads break-iterator data from it. Also drops zlib and the last TLS reference from the target, and turns off MCCP/compress there. - build-deps.sh trims the ICU archive with icupkg to brkitr rules + the converter alias table (~780KB). ICU_DATA_FILTER_FILE cannot do this: it only applies when building ICU data from source, and the -src tarball ships a prebuilt .dat. Table charsets (GBK, Big5, ...) are gone on this target -- string_encode() etc. raise an LPC error; a new __WASM__ LPC predefine lets mudlibs and tests adapt, and ICU_KEEP re-adds charsets for mudlibs that need them. - zlib is not linked on wasm at all: a global HAVE_ZLIB (defined on every other platform) now gates the core's gzip'd file support -- compressed save_object degrades to a plain save, write_file flag 2 raises an error, and read_file/restore_object use stdio instead of transparent gzopen. That also surfaced a latent bug: core used gz* but only got zlib transitively via the compress package/libtelnet, so native now links ZLIB::ZLIB explicitly. - TLS is fully gone from the target: the one shared caller of the TLS interface (the sys_reload_tls efun) is excluded from the wasm efun table in core.spec (the fullspec is preprocessed with the TARGET compiler, so #ifndef __EMSCRIPTEN__ works there), which lets the net/tls_stub.cc shim be deleted outright. Websocket code was already native-only via the Transport split. - compress package + MCCP are off on wasm (compressing a byte stream to a client on the same page wastes CPU and size). - INITIAL_MEMORY 128MB -> 64MB now that the data segment is small. - Deps prefix is ICU-only; CI/release cache keys bumped to -v3 with the zlib pin removed. Docs (build-wasm.md guide, driver/wasm.md cookbook, src/wasm/README.md, README, AGENTS, sys_reload_tls efun page) updated. Result: fluffos.wasm 33.5MB -> 3.5MB raw, ~0.8MB brotli / ~1.0MB gzip; the full LPC testsuite passes inside the wasm driver. Claude-Session: https://claude.ai/code/session_01VVpphH3cgXyziRDCbjUVkb Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Yucong Sun
|
e6f0c377f8
|
wasm: run the full driver in the browser (Emscripten port) (#1231)
The driver cross-compiles to WebAssembly and runs a complete mudlib inside a webpage: compiler, VM, all portable efun packages, and the real telnet protocol layer -- the page is the telnet client, JavaScript is the wire. The LPC testsuite passes inside the wasm driver (browser and node). Architecture (src/wasm/README.md): - Transport interface (src/net/transport.h): each interactive_t owns an abstract byte pipe (write/flush/schedule_command/close). comm.cc and telnet.cc are transport-agnostic and compile unchanged on every target. Implementations select at link time: SocketTransport + WebsocketTransport (net/transport_libevent.cc, native) vs WasmConsoleTransport (src/wasm/comm_wasm.cc). - Inverted event loop: the shared gametick/event core stays in backend.cc; backend_libevent.cc (blocking loop) vs src/wasm/backend_wasm.cc (page-driven fluffos_tick(now_ms), walltime priority queue, capped catch-up). Other per-target singletons (TLS, DNS resolver, crash handler) follow the same link-time pattern -- no #ifdef __EMSCRIPTEN__ in shared logic files. - Mudlib rides in Emscripten's MEMFS via file_packager; driver file I/O needed zero changes. jsbridge package (WASM only): js_eval() (sync), js_call() (async page handlers with LPC callbacks), js_export() (page calls LPC via Module.fluffos.callLPC, Promise-resolved on a later tick) -- fetch, canvas/WebGL, storage, page UIs driving the game. Demo in testsuite/command/jsdemo.lpc + the bundled web terminal. Build/tooling: native-tools + wasm CMake presets (host codegen tools, then cross build); tools/wasm/build-deps.sh (ICU + zlib cross-builds, including the ICU genccode data quirk); tools/wasm/pack-mudlib.sh (any mudlib + driver -> static web bundle, also works standalone from the release zip); src/www/wasm/index.html (self-contained web terminal with a minimal telnet client). On emsdk >= 3.1.57 the build uses native wasm exceptions (-fwasm-exceptions); older toolchains fall back to -sDISABLE_EXCEPTION_CATCHING=0. CI/release: a wasm CI job (latest emsdk, deps cache keyed on resolved emcc version) gates PRs on the LPC testsuite running inside the wasm driver under node; release.yml ships fluffos-<version>-wasm.zip (driver + web terminal + pack-mudlib.sh). Fixed a stale-predefine bug: options.autogen.h now depends on packages.autogen.h + config.h. Docs: docs/build-wasm.md (end-to-end workflow), docs/driver/wasm.md (packer + jsbridge cookbook), docs/efun/jsbridge/*, README + AGENTS updated. Testsuite files for optional packages guard themselves with #ifdef __PACKAGE_*__. Claude-Session: https://claude.ai/code/session_01VVpphH3cgXyziRDCbjUVkb Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Yucong Sun
|
fd705f0634 |
docs: reconcile FFI plan + preprocessor/diagnostics docs with implementation
Doc-only review pass checking every branch-authored doc against the shipped code. Fixes where docs described intent rather than the result: - docs/driver/ffi-plan.md (began life as a plan, drifted from the shipped package): ffi_status() returns a mapping, not mixed*; callbacks are implemented and in-scope (ffi_callback/_addr/_free added to the efun surface and moved out of "v2 deferred"); the DEBUGMALLOC section now describes the actual std::unordered_map + TAG_BUFFER scheme (no TAG_FFI / mark hook exists); valid_ffi gates load/symbol/prepare/ callback; testing is the 20-file LPC suite + tools/ffi/test.py (there is no GTest fixture). - docs/lpc/preprocessor/index.md: document the #warn directive. - docs/lpc/diagnostics.md + preprocessor/pragma.md: show_error_context is a legacy flag that no longer changes clang-style compiler diagnostics (render_diagnostic ignores PRAGMA_ERROR_CONTEXT; only the runtime smart_log path still reads it). - tools/lpc-syntax/README.md: test.mjs is 49 assertions, not 47. Verified accurate, no change needed: AGENTS.md, testsuite/README.md, compiler/internal/README.md, and the source-files / float / strings / text_blocks / define / conditionals / inherit / include docs, sprintf %g, and the tools/ffi + vscode READMEs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Yucong Sun
|
9be761aa00 |
package_ffi: foreign function interface for LPC (libffi), with callbacks
Fully implements the docs/driver/ffi-plan.md design. LPC can now load native shared libraries, call C functions whose signatures are described at runtime, manage native memory, pass in/out parameters, and expose LPC function pointers to C as callbacks. Package (src/packages/ffi, option PACKAGE_FFI ON, libffi via pkg-config): - ffi_load/unload/symbol; ffi_prepare/ffi_call (ffi_prep_cif + ffi_call); ffi_alloc/free/sizeof/peek/address; ffi_read/write; ffi_struct_layout; ffi_callback/ffi_callback_addr/ffi_callback_free (libffi closures that re-enter the VM via safe_call_function_pointer); ffi_error/ffi_status. - Buffers are the currency for all pointer/byte data; raw pointer VALUES (returned pointers, buffer/callback addresses) are ints. LPC strings are UTF-8-native and never implicitly marshalled -- a char* is a buffer the caller encoded (pinned by ffi_string.lpc). - Native allocations are LPC buffers (GC-tracked); handle tables freed at shutdown (ffi_cleanup) and marked for DEBUGMALLOC (mark_ffi). Security: master apply valid_ffi(op, arg, caller) gates every load/symbol/prepare/callback (VALID_FFI added to the applies table); a missing apply denies by default. Optional "ffi allowed libraries" config allow-list (rc.cc + runtime_config.h + regenerated config.md, new Security category). __PACKAGE_FFI__ predefine added. tools/ffi/generate.py: turns a C header into LPC bindings (buffer params for char*, optional --string-convenience UTF-8 overloads) plus a struct layout include; reports+skips unsupported forms; --emit-json contract. Dependency-free test.py. Tests: 20 testsuite/single/tests/efuns/ffi_*.lpc (every efun, the qsort callback round trip, the generated-bindings end-to-end path), guarded by __PACKAGE_FFI__ with a libc-reachability probe. The efuns are VM-stack- based, so the LPC testsuite is the surface -- libffi's call/closure paths run there under ASan/UBSan and the per-file check_memory leak gate. The clang RelWithDebInfo sanitizer caught an error()-unwind leak: both ffi_prepare and ffi_callback allocated before a code_to_type() that can error() -- now unique_ptr/custom-deleter owned (AGENTS.md section 4). Verified: testsuite x3 (ASan Debug) + ctest 297, RelWithDebInfo suite x3 + ctest 298, clang RelWithDebInfo sanitizer (leak-clean), tools/ffi test.py. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Yucong Sun
|
55e47edc07 |
Float exponent notation: 2.5e2, 1e3, 2E-5 are float literals now
Neither master's hand-written scanner nor the flex lexer ever accepted
exponents ("2.5e2" lexed as REAL(2.5) IDENT(e2)) -- but the hand-written
EBNF already documented the form and it should exist. Two lexer rules
add it: an optional exponent on the fraction form, and a bare-exponent
form ("1e3"); "1e" without exponent digits stays NUMBER(1) IDENT(e),
"1..5" stays a range, hex "0x1E3" is untouched, and '_' separators work
in all parts. strtod() already converted exponents, so only the
patterns changed.
Pinned at every layer: syntax_literals.lpc (2.5e2, 2.5e-2, 1.5E+2,
1e3, 1_2e4), the JS tokenizer + generated tmLanguage (with node tests
for the exponent forms and the "1e" non-exponent), the lexical EBNF
(realLiteral | exponent rules, replacing a half-aspirational entry),
and docs/lpc/types/float.md (which also wrongly claimed single
precision -- LPC_FLOAT is a double).
Verified: testsuite x3 + ctest 297 (ASan Debug), RelWithDebInfo full
ctest, lpc-syntax node tests (50).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Yucong Sun
|
62e29ab5f2 |
Syntax coverage matrix; fix name::fn() super calls into .lpc parents
Four new compiler test files pin the language surface end-to-end (121 checks): syntax_literals (integer bases, digit separators, float forms including the trailing-dot spelling, char/string escapes, adjacent concatenation, in-string line continuation, constant folding), syntax_operators (C-precedence pins across the merged token families, every compound assignment including <<= >>= &&= ||= ??=, ?? vs 0, short-circuiting, ++/-- lvalue forms, comma expressions, all range spellings, casts), syntax_control (do/while, for with comma clauses and empty header, dangling else, string switches, open-ended case ranges, fallthrough, catch and time_expression in expression AND block forms), and syntax_functions (:: and named-scope super calls, varargs + call spread, expression defaults, function-typed lambdas, new(class X, field: value) initializers). Writing them caught a real regression: the named-scope super-call matcher (compiler.cc) hardcoded a 2-byte ".c" suffix when comparing `name::fn()` against inherited program names, so every named super call into an .lpc-compiled parent failed to resolve -- and the entire testsuite had no named-scope call to notice. The matcher now strips the real extension. fail/lambda_typed_return.lpc pins that lambdas accept only the 'function' return type. The review also surfaced three language facts the JS tooling had wrong or the tests mis-assumed, now pinned: LPC floats have NO exponent notation (tokenizer.mjs and the generated tmLanguage accepted "2.5e2" as one number; the hand-authored EBNF was already right), there is no open-low index range "[..n]" (only case labels support it), and casts are type assertions, not conversions. Verified: testsuite x3 + ctest 297 (ASan Debug), clang RelWithDebInfo sanitizer full suite, RelWithDebInfo full ctest, lpc-syntax node tests (48). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Yucong Sun
|
a0f6c74561 |
Grammar contract moves to tools/lpc-syntax; add VS Code extension
generate_ebnf.py, grammar_lexical.ebnf.in and the generated
grammar.ebnf relocate from src/compiler/internal/ to tools/lpc-syntax/,
putting every grammar-contract asset in one place (the CMake
generate_ebnf target and all doc references follow).
New tools/lpc-syntax/vscode/ extension, driven by the same contract:
- syntax highlighting via a TextMate grammar GENERATED from
lpc-grammar.json (keywords/types/modifiers/operators can never drift
from grammar.y), plus templates with ${} interpolation, @/@@ text
blocks with backreference terminators, directives, (: :) functionals
- structural diagnostics as you type from the new lint.mjs (illegal
characters, unterminated strings/templates/comments/text blocks,
unbalanced brackets, mismatched #if/#elif/#else/#endif)
- optional REAL compiler errors on save: with lpc.lpcc.path and
lpc.lpcc.configFile set, the file is compiled by lpcc and its
clang-style file:line:col diagnostics are mapped inline (verified
against actual lpcc output), including into #include'd files
generate_ebnf.py now also emits the extension's generated assets
(syntaxes/lpc.tmLanguage.json and self-contained vscode/lib/ copies of
tokenizer/lint/grammar -- a packaged extension cannot reach ../).
test.mjs grows to 47 assertions covering the linter and the generated
assets.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Yucong Sun
|
a78f25eeaa |
Grammar as a machine contract: 3-layer EBNF + lpc-grammar.json + JS
tokenizer/highlighter/formatter
The checked-in grammar.ebnf was stale (still referenced 8 tokens deleted
in the token diet, none of the merged ones, and defined no terminals).
Plan executed, landed as one commit:
1. REFRESH: the Syntax layer regenerates from today's grammar.y via the
existing bison --xml pipeline (token diet, '(' '{' pairs, functional
productions all current).
2. LAYER: generate_ebnf.py now composes grammar.ebnf from three layers
-- a hand-authored LEXICAL layer (identifiers, numbers with
hex/binary/underscores, strings + every escape form, template
literals with ${} interpolation, @/@@ text blocks, comments, keyword
families, functional (: :) forms) and a PREPROCESSOR layer (all 14
directives, function-like macros with # and ##, token-based #if
expressions, line continuations, builtin macros), both living in
grammar_lexical.ebnf.in so regeneration can never lose them, plus
the generated Syntax layer.
3. MACHINE CONTRACT: the same run emits tools/lpc-syntax/
lpc-grammar.json -- keywords, type/modifier keywords, operators in
longest-match order, punctuation, directives, token categories, and
all 265 productions. The generator ASSERTS every grammar.y terminal
is categorized: a new token without a spec entry fails regeneration
loudly, so the artifacts cannot silently go stale again. Sources of
truth: lexer.l's operator rules and lexer_utils.cc's reserved-word
table.
4. JS TOOLING (tools/lpc-syntax, dependency-free ESM): a grammar-driven
tokenizer (templates with recursively tokenized ${} interpolation,
text blocks, directive lines with continuations, longest-match
operators), an HTML syntax highlighter (highlightLPC + defaultCss),
and a basic idempotent formatter (brace reindentation, statement
splitting, operator spacing, directives at column 0, strings/
comments verbatim). 27 assertions in test.mjs, run with plain node.
CMake generate_ebnf target produces both artifacts; docs in
tools/lpc-syntax/README.md, AGENTS.md section 11, and the compiler
README. Verified: node test suite (27/27), full driver battery (287
tests under ASan/UBSan).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|