Phase 4 over-committed pre-entry setup failures (carg/depth/get_dbt) as
#-1 LUA JIT RUN FAIL; fall back to the interpreter when nothing ran.
Protect lua_getglobal and lua_compare via pcall so raising metamethods
cannot abort the server. Apply EQK-class type gates and NIL/BOOL kinds
to register-register CMP_RR equality, and decline mixed-type order.
After RunCompiled, TryJIT always commits the result (success, LUA ERROR,
POST-ENTRY residual, RUN FAIL, or CPU LIMITED) and never returns false to
re-run the chunk in the Lua VM. Pre-entry fallback remains: compile refuse,
empty dump, prior ineligible, cache miss/recompile.
run_cached_program with a Lua state never returns false after dbt_run for
unhandled status — commits #-1 LUA JIT RUN FAIL instead.
UNHANDLED_LUA_BRIDGE becomes a committed LUA ERROR. lua_ecall_decline
rewrites any residual soft decline into ECALL_LUA_ERROR so the poison
retry path cannot return by accident.
Harness still green (post_entry_loud=1: EFFECT_REFUSED only).
Add a STATE tier to tests/luajit that runs the #1751 exhibits as multi-step
setup/action/state probes:
e1_double_mutation — N must match across legs (not 2 vs 1); JIT may loud-fail
e2_coroutine_effect — effect count must match when both succeed; loud fail OK
Also count EFFECT_REFUSED via jit_lua_note_post_entry_decline so
lua_post_entry_decline advances when TryJIT commits the loud path outside
RunCompiled. Ratchet POST_ENTRY_LOUD_BUDGET to 7 (includes STATE loud).
A discarded compiled Lua run is not free once guest code may have run.
Wire every ECALL_DECLINE after entry to a committed error
(#-1 LUA JIT POST-ENTRY DECLINE (site)), a JIT/RETRY log line, and
jitstats lua_post_entry_decline. effect_refused follows the same rule.
Pre-entry fallback (compile refuse / cache miss) is unchanged.
Also fix lua_ecall_decline so it returns ECALL_DECLINE instead of
recursing (that hang turned budget/LIMITED cases into harness timeouts).
Harness: AGREE/NESTED_AGREE accept loud fails; ratchets POST_ENTRY_LOUD=5
and AGREE_DECLINE=2 from measured counts.
softcode entry points already call tier2_lazy_init; the Lua compile path
did not. With jit_eval_brackets 0 (required for Lua JIT under #1326)
softcode never hits jit_eval, so softlib.rv64 stayed unloaded and every
tier2_sym_addr("pow") decline made runtime ^ vacuous.
tier2_ensure() from compile_lua_bytecode loads the blob once. Stacked on
#1488 so integral ^ results render as "4.0"/"8.0" rather than "4"/"8".
EXEC pins for return 3.0, a+b, and runtime ^ keep both regressions closed.
Three rounds of per-site guards have landed for the same defect class --
#1440, #1449, #1457/#1470 -- and each round found producers the previous
one had missed. The guards are invisible at the point of danger: nothing
about `h.ty[p]` says p might be -1, so a fast path added ahead of an
existing guard is unprotected by construction. #1470 found exactly that,
and near-identical loops 80 lines apart had also drifted (#1457).
Move the check into the subscript. The 16 per-instruction arrays become
hir_slot<T, N>, whose operator[] refuses an out-of-range index: the access
lands on a scratch element instead of past the array, and is recorded so
refused_index() makes the whole program unusable. 828 access sites across
five files are unchanged -- there is no memset or array decay anywhere in
the family, so the wrapper is a true drop-in.
Both halves are load-bearing, and measured separately. Making the
subscript merely *defined* without recording it would convert a sanitizer
report into a silently wrong compile, which is worse than the bug; that is
case C below. Recording without a scratch element would leave the out-of-
bounds access in place.
refused_index() is read wherever h.overflowed already is. The two mean
the same thing to a caller -- this program is not safe to use -- and
differ only in what noticed: overflowed is set by the producer of a -1, so
it catches capacity limits and the #1242 unknown-node refusal, while
refused_index() is set by the consumer, so it also catches a refusal no
producer flagged. That is the half the per-site guards kept missing.
Also closes a gap on the Lua entry point: compile_lua_bytecode never
checked h->overflowed at all. It checked only the -1 returned by
hir_lower_lua_proto, so a refusal a consumer swallowed left the flag set
and unread while SSA, the optimizer and codegen ran over a program whose
lowering had stopped partway. compile_expression has had that check since
#859; the Lua path is the same pipeline.
Negative controls, all with an unguarded consumer injected at the #1470
site (h.known_int[-1] = true; h.ty[-1] = TY_INT; then read it back):
A wrapper checks, refusal wired 0 UBSan reports compile_fail 0->1
B subscript passes through 2 UBSan reports (see note)
C wrapper checks, refusal unwired 0 UBSan reports compile_fail 0->0
B reports exactly the class #1501 names -- "index -1 out of bounds for
type 'bool [4096]'" and "'hir_type [4096]'". Its compile_fail also moved,
but by accident: the out-of-bounds write lands on the preceding slot's
`refused` byte and sets it, which is its own argument for the change. C
is the one that matters -- the compile completed and was used
(compile_ok 1->3) over a program with a refused index, with no sanitizer
signal to show it. In all three the answer stayed correct, because the
AST evaluator is the fallback.
Verification:
- make test passes, both smoke routes 1509/1509, 316/316 dispatched
- sanitizer tree (--enable-sanitizers, aarch64): test-jit-qreg,
test-jit-ifelse, test-lua-ecall and the smoke leg (1509/1509,
315/315) all clean -- zero sanitizer findings, zero "index -1 out of
bounds", which is the acceptance test #1501 asks for
- make test-asan itself cannot reach those legs on any tree; it stops at
leg 2 on a pre-existing link failure in tests/netaddr, filed as #1522
and reproduced on unmodified master
- no measurable compile-path cost: smoke 16s vs 17-18s baseline, i.e.
within run-to-run noise
hir_slot's element parameter is ElemT rather than T because mux_nls.h
defines a function-style macro T(x); a parameter named T turns `T()` into
`(reinterpret_cast<const UTF8 *>())`. Two static_asserts pin HIR_NOP and
TY_VOID as the zero values, since a refused read returning a live-looking
kind or type could steer a consumer down a real branch before
refused_index() is consulted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TINYMUX_DUMP_HIR covered only the softcode JIT. On the Lua path nothing was
observable from outside: not the HIR, not the block layout, and not the more
basic question of whether a chunk compiled at all.
That last one is the expensive part. A declined chunk runs on the interpreter,
so it agrees with the interpreter trivially -- which reads as a pass when the
two are compared. Several numeric-for results I first read as "the compiled
path is correct" had simply never compiled. So this reports declines too, and
names the reason:
--- Lua JIT: @#1/T declined by eligibility: LUA_BC_HAS_CLOSURE ---
--- Lua JIT: @#1/T declined by lowering (10 bytecode insns) ---
and otherwise dumps each phase, matching what jit_compiler.cpp already does for
softcode:
--- Lua JIT Compilation: @#1/T (10 bytecode insns) ---
Phase 1: HIR Lowering
Phase 2: CFG (5 blocks)
Phase 2b: SSA Construction
Phase 3: SSA Optimization
Gated on the same environment variable, so it is silent unless asked for;
verified that an unset variable produces no output and smoke is unchanged
(1505/1505, 316/316 dispatched).
This immediately paid for itself on #1486, where the dump shows the branch and
the returned value disagreeing:
v4 = GT int v2, v3
v5 = BRC void v4 ? -> BLOCK 3 : BLOCK 2 BLOCK 2 = then, BLOCK 3 = else
Result: v7 v7 = SCONST "111", in BLOCK 2
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
jitstats(flush) returned an empty string instead of OK whenever the region
containing the call was JIT-compiled -- which, with jit_eval_brackets on by
default, is the documented usage from wizhelp.
s_compile_cache holds compiled_program by value and compile_cached() hands
back a pointer into it. run_cached_program() keeps that pointer across the
whole execution and harvests through it afterwards:
int rc = dbt_run(dbt, prog->entry_pc, ...);
...
uint64_t out_addr = resolve_runtime_out_addr(prog->out_addr, ...);
and jit_eval() reads prog->ecalls / prog->tier2_calls after that returns.
dbt_run is what issues ECALLs, and ECALL callees run the interpreter -- that
is how u() and ufuns work from compiled code -- so fun_jitstats could execute
with a live program on the stack. s_compile_cache.clear() then freed the
program mid-flight and the harvest read released memory. It surfaced as a
lost result rather than a crash, which is allocator luck, not safety.
s_run_cached_depth did not help: it refuses nested run_cached_program, but an
ECALL into the interpreter is a plain call, so the guard never fired.
The SQLite DELETE and the write-queue drop stay immediate -- neither frees
anything a running program points at. Only the in-memory caches wait, drained
at the top of compile_cached() and of the Lua RunCompiled/Compile entries:
those are the points where the caller does not yet hold a pointer, and depth
== 0 means no outer frame holds one either. Nothing can be served stale in
between, because every lookup goes through one of those drains.
jit_lua's s_lua_cache had the same shape -- RunCompiled passes &it->second to
run_cached_program -- so it drains on the same flag.
Verified on macOS arm64, both routes, same tree:
before direct= nested= warm= (empty, on the compiled route)
after direct=OK nested=OK tailmark warm=OK
and the flush still flushes: across it, cache_hit_mem is unmoved while
cache_miss rises by exactly the number of regions evaluated after, including
one whose text had just been compiled.
jitstats_fn.mux TC005 pins both halves. It fails on the unfixed engine with
the exact signature above, and asserts the counters so a flush that returned
OK without flushing would not pass. No tr.done handoff was added: the dolist
over tr.tc* already dispatches it, and chaining from tc004 ran it twice.
with fix 1487 passed, 0 failed, 314/314, both routes
without fix 1486 passed, 1 failed (TC005), 314/314
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Part 1 of #1316 (Lua counters in jitstats()) already landed via #1317.
This closes the remaining gap: CSQLiteDB::CodeCacheFlush() was prepared
and never called, so a persisted JIT program could not be cleared after
engine codegen changes (the #1315 A/B trap — blob_hash only covers the
Tier-2 blob).
jitstats(flush) drops pending OP_CODE_CACHE_PUT ops, DELETE FROM
code_cache, and clears softcode/Lua in-memory compile caches so nothing
re-serves the old native code. Wizard-only, same surface as reset.
The Lua compile path never memcpy'd rc.code into guest memory (softcode
did). Compact/materialize then installed zeros at entry_pc; the DBT
translated illegal opcode 0 as exit_with_pc(same) and spun at guest PC 0
— the mux.args arithmetic "hang".
Also:
- ATOI/STRCMP of runtime_ref SCONSTs use guest addresses at runtime
(empty sval is not a compile-time constant).
- Keep needs_jit when sref_addrs is non-empty so return mux.args[1]
does not take the empty folded path.
Verified: hello/42/mux.args[1]//args alone, A2→A1, no dbt dispatch limit.
Phase 3a bring-up for the never-run Lua HIR/DBT path:
- Keep the first RETURN value; Lua's trailing RETURN was overwriting
"hello"/42 with empty SCONST (empty folded results).
- Fold ICONST/FCONST returns to digit SCONSTs (return 42).
- Map mux.args[N] to CARGS srefs; classify sref_addrs for cargs_used.
- Do not const-fold ATOI of runtime_ref (CARGS) slots to 0.
- Do not double-advance pc past MMBIN (was skipping RETURN).
- Do not start a new block after RETURN (false multi_block + budget).
- Always hir_build_cfg so single-block programs set block_last.
- Refuse nested run_cached_program (softcode JIT → fun_lua → Lua JIT).
- Include rc.needs_jit when setting compiled_program::needs_jit.
Verified: return "hello"/42/1+2; A2 then A1 with and without softcode
bracket JIT. Residual: mux.args arithmetic still trips DBT dispatch
limit then falls back to the Lua interpreter (answers correct).
s_lua_jit_stats had six counters incremented in six places and read in
none, and jitstats() reported only s_jit_stats. So there was no way --
in-game or in a log -- to tell whether the Lua JIT was compiling,
running, or failing.
That blind spot matters more than usual here because every Lua JIT
failure falls back to the Lua interpreter and still returns the correct
answer. A Lua JIT that compiles and then never runs is indistinguishable
from a healthy one by observing softcode output alone.
Publish the counters through jit_lua_get_stats() and append them to
jitstats() as lua_*; jitstats(reset) now clears them too.
Verified on Windows (MSVC x64), which reads out the two known defects
directly:
master: lua_compile_ok=0 lua_compile_fail=1 lua_run_ok=0
-- #1278, the bytecode loader rejects every chunk
master + #1306: lua_compile_ok=1 lua_run_ok=0 lua_run_fail=2
-- #1315, compiles but the DBT code buffer is exhausted
In both cases lua() returned correct results throughout.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Comment-only; no behavior change.
- mail.cpp RemoveItem: drop the inaccurate "Mirror RemoveAll()" claim.
RemoveAll erases the map entry then clears the iterators; RemoveItem
must test lst.empty() first, so it clears then erases. The point of
the fix is that the iterators get cleared at all, not the ordering.
- dbt_compile.h bail_alarm: it is counted per aborted DBT run, not
per command -- JIT-in-JIT nesting (an ECALL into another compiled
program) can bump it more than once as each run on the stack unwinds.
The old "per-command" wording overstated it.
- jit_lua.cpp RunCompiled: document that a mid-run wall-clock alarm now
surfaces as a successful "#-1 CPU LIMITED" result, not MUX_E_FAIL.
That is deliberate: MUX_E_FAIL makes the Lua caller re-run the whole
chunk in the Lua VM -- work we must not do once the command is over
its CPU budget.
Smoke 1319/1319; JIT q-register oracle green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The comsys and mail modules already use std::atomic<uint32_t> for their
COM-style refcounts; the Lua module did not, leaving CLuaMod::m_cRef and
the Lua JIT chunk-cache key counter as plain integers.
- #723: CLuaMod::m_cRef -> std::atomic<uint32_t>, with AddRef/Release using
fetch_add(relaxed)/fetch_sub(acq_rel) exactly like CComsysMod. (CLuaMod
has no separate factory class; the registration entry point is a plain
function, so there is no second counter to convert.)
- #724: jit_lua.cpp s_next_key -> std::atomic<uint64_t> with
fetch_add(relaxed); the pre-increment value is preserved (first key = 1).
The engine currently runs single-threaded, so neither was exploitable
today; this brings the Lua module in line with the established
correct-by-design refcounting convention so it stays sound if module calls
are ever made off the main thread. No behavioral change; smoke 1064/0.
Closes#723Closes#724
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
compile_lua_bytecode() was missing the metadata fields (entry_pc,
code_size, str_pool_end, fargs_pool_end, out_pool_end) needed by
the compact storage format, and CompileLuaBytecode() was inserting
into s_lua_cache without compacting first. The compile-then-run
path executed from zeroed guest memory instead of the compiled code.
- Populate all pool metadata from rc_state in compile_lua_bytecode()
- Call jit_compact_program() after SQLite persistence in CompileLuaBytecode()
- Expose jit_compact_program() via engine_api.h for cross-TU use
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Lua JIT now uses sha1(bytecodes) as cache key into the SQLite code_cache
table (prefixed with "lua:"). On startup, previously compiled Lua
programs are loaded from SQLite instead of recompiling from bytecode,
eliminating JIT compilation latency after server restart.
Exposes jit_sha1_hex(), jit_store_to_sqlite(), jit_load_from_sqlite()
from jit_compiler.cpp for shared use by both softcode and Lua JIT
compilation paths. Cache invalidation uses the existing blob_version
mechanism — Tier 2 blob changes automatically orphan stale entries.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Added void *lua_state to eval_ctx, run_cached_program(), and
mux_IJITCompile::RunCompiled(). CLuaMod::TryJIT passes m_L through
the entire chain so ECALL handlers can call back into the Lua VM
for operations the JIT can't handle natively.
nullptr for softcode JIT (no change in behavior). Non-null for
Lua JIT — enables future ECALL handlers for table access, string
ops, generic calls, and upvalue reads.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fast O(n) scan rejects ineligible protos before allocating HIR/RV64
state. Checks proto header fields (code size ≤1024, stack ≤64,
params ≤8, no nested protos) then scans the instruction stream for
hard-reject opcodes (CLOSURE, VARARG, TBC, TAILCALL) and verifies
all opcodes are in the supported whitelist. VARARGPREP is allowed
(harmless no-op in main chunks).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Lua scripts compiled by lua_mod.so can now be JIT-compiled through
the existing HIR/RV64/x86-64 pipeline in engine.so. The bytecode
deserializer reads lua_dump() output without requiring Lua headers.
New COM interface mux_IJITCompile on engine.so with CompileLuaBytecode,
RunCompiled, IsCompiled, and Invalidate methods. lua_mod.so acquires
this interface and transparently attempts JIT before falling back to
the Lua VM.
Opcode coverage: data movement (MOVE/LOADI/LOADK/LOADNIL/etc.),
integer arithmetic (ADD/SUB/MUL/IDIV/MOD/UNM + immediate/constant
variants), comparisons (EQ/LT/LE/EQI/LTI/LEI/GTI/GEI), control flow
(JMP/TEST/TESTSET/FORPREP/FORLOOP), returns, and mux.* bridge calls
pattern-matched from GETTABUP+GETFIELD+CALL to engine API ECALLs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
