Root cause: comsys_mod/mail_mod lacked mux_CanUnloadNow, so ModuleLoad
never set bLoaded and CreateInstance always returned CLASSNOTAVAILABLE —
commands and softcode both used the engine path by accident.
With modules actually loaded:
- Bump module revision on every write-through; softcode reloads engine
maps from SQLite when the revision advances (select_channel, mail_fetch,
MailList, etc.).
- CComsysStorage::SyncChannel sets has_comsys so softcode reload works.
- Channel Sync uses ON CONFLICT DO UPDATE so updates do not CASCADE-wipe
channel_users/player_channels.
- mailsend() goes through IMailControl::SoftcodeSend; bodies and
mail_db_top are write-through for softcode mail_* fields.
The migration gate was never bumped when the v12 (max_func_depth, #1002)
and v13 (n_func_calls) migrations were added. An existing database at
schema 11 or 12 early-returned out of MigrateSchema, skipped the pending
ALTERs, and then PrepareStatements failed on the missing code_cache
columns: the server dies at boot with 'Couldn't open storage backend.'
Fresh databases run the full chain from version 0 and never see it,
which is why the fresh-db smoke stayed green. Verified on Windows by
warm-starting a 2.14.0.9-era v11 netmux.sqlite: pre-fix boot fails as
above; post-fix both migrations run and the game loads.
Closes#1073
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Store the static function-invocation watermark alongside max_func_depth
so @restart reconstructs it from SQLite instead of leaving n_func_calls
at 0 until each expression recompiles into the memory LRU. Pre-v13 rows
default to 0 (watermark skipped until recompile), matching the
max_func_depth progressive path.
Resolves the #1002 design question as a semantic contract, per the
watermark sketch: compiled code flattens function nests and maintains
no func_nest_lev, so a JIT run that could reach the limit now declines
to the AST, which reproduces "#-1 FUNCTION RECURSION LIMIT EXCEEDED"
exactly.
- compiled_program.max_func_depth: maximum AST_FUNCCALL nesting of the
parse tree, computed at compile_expression and persisted through the
write queue into the SQLite code cache (schema v12 adds the column;
pre-v12 rows read 0 but are invalidated by the blob-version stamp
regardless).
- jit_eval declines when live func_nest_lev + max_func_depth >=
mudconf.func_nest_lim — the LIVE values, so nested entry contexts
count and @admin limit changes apply to cached programs. The check
precedes the folded-result shortcut (folding pre-computes the nest
and must obey the same contract). run_cached_program repeats the
check for callers that bypass jit_eval (rvbench). New bail_depth
jitstats counter.
- Documented boundary: ECALL-dispatched callees do not bump
func_nest_lev, so cross-ECALL nesting carries a one-level skew;
probe-writing softcode should not assert at exactly limit-1 across
a u() boundary.
jit_parity_fn.mux TC019 locks it: depths 4 and 6 at limit 6 (the
unambiguous zones; depth 5 sits in the skew zone and is deliberately
unasserted), with the limit juggling and u() probes run at Makesmoke
bake time as #1 — @admin is God-only and a runtime-triggered attribute
executes as the object (the nested_depth.mux pattern; the first cut
tried runtime @admin and the assertion ran against the unchanged
limit).
Verified: nest4 ok4 via JIT, nest6 declines with bail_depth=1 and
returns the exact AST error, restore honored by cached programs.
Smoke 1319/1319 both toggles; oracle 9/9; sweeps standard and
utf8+longreg(SEED=7) 400/0 LOGIC; GANL 14/0; netaddr 33/0.
Closes#1002.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Audit of the DB load paths (flatfile reader, mail/malias loader, SQLite
attribute bulk-load) under the malicious/corrupt-database threat model --
the same class as #834-843. Five distinct bugs, all in code the June
hardening (#806/#808/#841/#843) did not reach; each is reachable only from
crafted or tampered database content, not from normal gameplay. Verified by
full rebuild + smoke suite (1264/1264, 0 crashes), which exercises flatfile
export/import and SQLite attribute reads end to end.
1. getstring_noalloc: static buffer overflow (lib/dbutil.cpp)
The escaped-string reader mis-accumulated its output-byte count:
`nOutput = pOutput - p` overwrote the count from prior escape emits
instead of adding to it. On the multi-fgets refill path this
under-decrements nBufferLeft, so the `nBufferLeft <= 0` guard never trips
and the next fgets writes past the 2*LBUF_SIZE+20 static buffer.
Legitimate attributes (<= LBUF) never take the refill path, so this only
fires on a crafted quoted string > ~64KB. Fix: accumulate (+=).
2. make_numlist / malias_read: stack buffer overflow (modules/engine/mail.cpp)
malias_read read the recipient count (numrecep) straight from the file
and pushed that many dbrefs into m->list with no cap, making
m->list.size() attacker-controlled. make_numlist then copied all of
m->list into the fixed stack array aRecip[(LBUF_SIZE+1)/2] with no bound
on nRecip -- a crafted mail.db with numrecep > 16384 overflows the stack
(with attacker-chosen dbrefs) the next time any player mails the alias.
Fix: clamp numrecep to (LBUF_SIZE+1)/2 at load (also bounds the reserve()
that could otherwise exhaust memory on an INT_MAX count), and defensively
bound the copy loop in make_numlist.
3. SQLite bulk-load attribute value: heap buffer overflow (modules/engine/sqlitedb.cpp)
GetAllAttributes/GetBuiltinAttributes passed the raw column blob length to
the cache with no clamp, unlike the write path (cache_put) and the
standalone read path (GetAttribute), both of which clamp to LBUF_SIZE. A
value blob written directly into the SQLite file therefore flows unclamped
to atr_get_str_LEN's `memcpy(s, buff, (*pLen)+1)` into a fixed LBUF_SIZE
buffer, overflowing the heap on first read of the attribute
(Name/look/examine/get). Fix: clamp len to LBUF_SIZE in both bulk-load
functions, mirroring the existing clamps.
4. get_list: infinite loop + unbounded log on truncated flatfile (modules/engine/db_rw.cpp)
get_list had no EOF case: at end-of-file getc() returns EOF, falls to
default, and calls getstring_noalloc(), which makes no progress at EOF
(ungetc(EOF) is a no-op, fgets returns NULL). The for(;;) then spins
forever, pegging a core and emitting log lines. Trigger: a flatfile whose
last object's attribute list is truncated before its '<' terminator. Fix:
add a case EOF that aborts the load.
5. getboolexp1: BOOLEXP subtree leak on malformed v2 lock (modules/engine/db_rw.cpp)
Three error paths in the v2 lock parser returned TRUE_BOOLEXP without
freeing the partially built node/subtree (bad connective, missing ')',
EOF mid lock-string). Bounded and single-shot (db_read aborts the whole
load on corruption; nesting capped at 1024), but still a leak on crafted
input. Fix: free the appropriate node/subtree at each site, matching the
partial-construction state.
Also traced and dropped a sixth candidate (SQLite attribute owner read
unvalidated): the attribute owner is never used to index db[] -- only
compared, read for its flags, or re-stored -- so a corrupt owner is harmless.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Migrating an existing game to the SQLite backend had two silent traps that
both surfaced as "old passwords/characters don't work":
1. dbconvert resolves <basename>.sqlite relative to cwd, but the server
reads data/<name>.sqlite. Running db_load from the wrong directory put
the database where the server never looks, so old characters appeared
to vanish while a fresh login still worked against the stock db.
2. A populated netmux.sqlite silently shadows the netmux.db flatfile at
boot, so dropping in an old flatfile did nothing.
Changes:
- db_load/db_unload are now cwd-independent: the .sqlite always lands in
the game's data/ dir (next to the script), file args are resolved
against the caller's dir, and the scripts echo the absolute path. Arg
handling is space-safe (set --) and POSIX sh.
- dbconvert prints the exact database file it opened (CSQLiteDB::GetPath).
- The "would overwrite" guard now names the file and gives two ways
forward; a new -f/force option lets a load replace an existing db (the
load already clears attributes/objects/attr-names cleanly).
- The server logs a line when it warm-starts from SQLite and the flatfile
was not consulted, making the precedence visible.
- New docs/importing-a-game.md documents the two-file model and import
steps.
Verified: db_load from an unrelated dir lands the .sqlite in data/; the
guard refuses without -f and replaces with -f; db_unload round-trip is
byte-identical (passwords preserved). Smoke: 1078 ok / 0 new failures
(TC001/TC009 pre-existing on master).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hardening of the SQLite backend against OOM-NULL column results and against
error/busy paths that were dropped or left statements in a bad state.
NULL column guards. sqlite3_column_blob()/_text() may return NULL with bytes > 0
under OOM; the previous code fed those straight into memcpy or a consumer that
dereferences them.
- GetAttribute (#727): NULL blob with blobLen > 0 is now treated as "not
found"; the memcpy is also skipped for a legitimate zero-length attribute
(NULL blob, blobLen == 0) which remains valid.
- GetAllAttributes (#728) and GetBuiltinAttributes (same pattern): skip a row
whose blob is NULL with len > 0 instead of passing NULL to the callback.
- LoadAllAttrNames (#732): skip rows with a NULL name column rather than
handing the callback a NULL C string.
CodeCacheFlush (#729): reset the statement *after* stepping, not before, so an
error return (SQLITE_IOERR, SQLITE_FULL, ...) cannot leave it holding error/lock
state that makes the next use fail spuriously.
RunMigration (#730): capture and log the ROLLBACK result on the migration
failure path. Previously a failing ROLLBACK (busy, I/O error) was discarded with
a NULL errmsg, hiding that the connection was left in an indeterminate
transaction state. (The referenced abort_dump_no_restart() does not exist in
this module; the migration already aborts via its false return.)
Checkpoint (#731): retry sqlite3_wal_checkpoint_v2() on SQLITE_BUSY with short
backoff (up to 5 attempts, 10-40 ms). Previously a single busy return made @dump
report success while the WAL kept growing; busy is rare in the single-threaded
engine but possible under dbconvert -m. A persistent failure is now logged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reorganize standalone test harnesses under tests/:
- tests/db/: SQLite/storage backend tests (moved from db/)
- Fix missing <ctime> include in sqlitedb.cpp
- Add stubs.cpp for attr_mod_count symbols
- Update Makefile paths for new location
- tests/libmux/: New unit test harness linking against libmux.so
- 29 tests covering mux_atol, mux_atof, mux_i64toa, safe buffer
writing, StringClone, mux_stricmp, mux_strupr/strlwr,
trim_spaces, mux_strncpy
- tests/color_ops/: Unchanged (already in tests/)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add route() softcode function with BFS-based shortest-path routing
over rooms marked NAVIGABLE. The routing table stores only the next-hop
exit for each (source, dest) pair, compressed via diagonal elimination,
adjacent marking, and row redundancy. Lazy rebuild on generation-counter
mismatch triggered by topology changes (@dig, @destroy, @link, @open,
@unlink) and NAVIGABLE flag changes. SQLite schema v10 adds route_nodes,
route_table, route_meta tables for future persistence phases.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The code cache is a cache — structural integrity matters but
durability per-write does not. Disabling auto-checkpoint eliminates
fsync from the hot path. The WAL accumulates writes during normal
operation; explicit checkpoint at @dump and shutdown handles
persistence.
Smoke suite: 2s → 1s (50% reduction in kernel time).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@dbclean now finds user-defined attribute names (attrnum >= 256) that
no object references, removes them from both SQLite and the in-memory
vattr maps, then runs ANALYZE to refresh query planner statistics.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Default 1000, 0 = unlimited. Wizard-owned objects are exempt.
Enforced in atr_add_raw_LEN with a fast cache_get existence check
(updates are free) and SQLite COUNT(*) for new attributes only.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When the JIT compiler sees u(obj/attr) or ulocal(obj/attr) with a
constant first argument, it resolves the attr at compile time, parses
the body, and lowers it inline into the caller's HIR.
Three tiers cooperating:
Tier 1: resolves obj/attr at compile time (parse_attrib + atr_pget)
Tier 2: body's pure functions run as guest Tier 2 calls
Tier 3: orchestrates inlining with helper ECALLs
Correctness requirements (all addressed):
- Runtime permission: _CHECK_U_PERM → BRC fallback to host fun_u
- CARGS scoping: _SAVE_CARGS / _WRITE_CARG / _SET_NCARGS / _RESTORE
- ULOCAL qregs: _SAVE_QREGS / _RESTORE_QREGS
- Cache staleness: deps vector with per-attr mod_count
- Inline depth limit: MAX_INLINE_DEPTH = 3
Also fixes schema bootstrap: v7 migration CREATE TABLE reverted to
original (no deps_blob), v9 ALTER TABLE handles all paths.
Known issue: certain complex inlined bodies (e.g., switch() with
multiple branches) cause compilation hangs under benchmark().
Single invocations work correctly. Needs investigation.
Smoke tests: 610/610 passing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fresh databases and v7 migration both created code_cache without
deps_blob, but the v9 prepared statements require it. Added
deps_blob to both the base CreateSchema and v7 migration CREATE
TABLE definitions so all paths land on the same schema.
Smoke tests: 610/610 passing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
compiled_program now carries a deps vector of {obj, attr_num,
mod_count} recording which attrs were inlined at compile time.
Schema v9: adds deps_blob BLOB column to code_cache table.
Dependencies are serialized as a packed binary array (12 bytes
per entry) for minimal storage and fast deserialization.
store_to_sqlite_cache persists deps alongside the compiled blob.
reconstruct_from_cache restores them on SQLite cache hit.
CURRENT_SCHEMA_VERSION bumped to 9.
Smoke tests: 610/610 passing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The UINT32_MAX sentinel wrapped to 0 on next write, breaking
monotonicity. Replace with:
- attr_mod_count_invalidate_object() now seeds all attr counters
from SQLite BEFORE deletion (via GetAllAttrModCounts), then
increments each by 1. Counters are strictly monotonic — no
sentinel, no wrapping, no reuse across delete/recreate cycles.
- Call sites in object.cpp moved BEFORE DelAllAttributes so the
SQLite rows are still readable when seeding.
- ClearAttributes() clears the in-memory map entirely (correct
for full database reload where JIT cache is also cleared).
Smoke tests: 610/610 passing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two bugs from code review:
1. attr_mod_count_inc ran before cache_put/cache_del, so failed
writes bumped the in-memory counter without changing SQLite.
Fix: increment AFTER successful storage. PutAttribute reads
the current counter and passes counter+1 to the upsert
independently.
2. DelAllAttributes and ClearAttributes bypassed mod_count tracking.
Fix: attr_mod_count_invalidate_object() sets all entries for a
destroyed object to UINT32_MAX (always stale). ClearAttributes()
calls attr_mod_count_invalidate_all() to clear the entire map.
Smoke tests: 610/610 passing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two bugs from code review:
1. In-memory counter started at 0 regardless of persisted value.
After restart, first write to an attr with mod_count=17 in SQLite
would set in-memory to 1 while SQLite advanced to 18.
Fix: attr_mod_count_inc() seeds from SQLite on first access.
2. Delete + recreate reused mod_count=1, making stale cache entries
look fresh. Fix: increment in-memory counter BEFORE deletion
(survives the SQLite row removal), and PutAttribute uses
MAX(excluded.mod_count, mod_count+1) so the SQLite value is
always >= the in-memory value.
Smoke tests: 610/610 passing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Foundation for Tier 3 u() inlining cache staleness detection.
Each attribute now tracks a modification counter that increments
on every atr_add/atr_clr. The JIT compiler will record mod_count
at compile time and check for staleness at runtime.
Schema changes:
- Bump to v8: ALTER TABLE attributes ADD COLUMN mod_count
- PutAttribute uses INSERT...ON CONFLICT DO UPDATE to atomically
increment mod_count on each write
- New GetAttrModCount() method for SQLite lookup
In-memory tracking:
- attr_mod_count_inc() incremented from atr_add_raw_LEN / atr_clr
- attr_mod_count_get() reads from in-memory map with SQLite fallback
- Survives LRU cache eviction (separate map, not in AttrCacheEntry)
Also fixes CURRENT_SCHEMA_VERSION (was 6, now 8).
Smoke tests: 610/610 passing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add code_cache table (schema v7) to persist compiled softcode.
On compile-cache miss, check SQLite before compiling from scratch.
After successful compilation, store to SQLite for future restarts.
Cache keyed by expression text + blob version (size:func_count).
Blob version mismatch → stale entry skipped (tier2 addresses change).
Memory blob stores code+strings+fargs (0-32KB per entry).
Reconstructed programs get tier2 blob reinstalled at load time.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SQLite schema v6 adds connlog table with indexes on player, connect_time,
and ipaddr. AnnounceConnect inserts a row; AnnounceDisconnect updates with
disconnect time and reason. connlog(player[,limit]) returns pipe-delimited
records; addrlog(pattern[,limit]) does SQL LIKE matching (wizard-only).
COM interface updated: IPlayerSession::AnnounceConnect gains int64_t*
pConnlogId out-param, AnnounceDisconnect gains int64_t connlogId in-param.
DESC struct carries connlog_id through the session lifetime.
Also fixes isalpha_fn.mux test hashes after DFA table rebuild.
444/444 smoke tests pass.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
