From ecae4df9de39ca342aed746ee18741b7948c6889 Mon Sep 17 00:00:00 2001 From: Stephen Dennis Date: Sun, 7 Jun 2026 13:23:12 -0500 Subject: [PATCH] dbload: mistake-proof the SQLite import path (#766) Migrating an existing game to the SQLite backend had two silent traps that both surfaced as "old passwords/characters don't work": 1. dbconvert resolves .sqlite relative to cwd, but the server reads data/.sqlite. Running db_load from the wrong directory put the database where the server never looks, so old characters appeared to vanish while a fresh login still worked against the stock db. 2. A populated netmux.sqlite silently shadows the netmux.db flatfile at boot, so dropping in an old flatfile did nothing. Changes: - db_load/db_unload are now cwd-independent: the .sqlite always lands in the game's data/ dir (next to the script), file args are resolved against the caller's dir, and the scripts echo the absolute path. Arg handling is space-safe (set --) and POSIX sh. - dbconvert prints the exact database file it opened (CSQLiteDB::GetPath). - The "would overwrite" guard now names the file and gives two ways forward; a new -f/force option lets a load replace an existing db (the load already clears attributes/objects/attr-names cleanly). - The server logs a line when it warm-starts from SQLite and the flatfile was not consulted, making the precedence visible. - New docs/importing-a-game.md documents the two-file model and import steps. Verified: db_load from an unrelated dir lands the .sqlite in data/; the guard refuses without -f and replaces with -f; db_unload round-trip is byte-identical (passwords preserved). Smoke: 1078 ok / 0 new failures (TC001/TC009 pre-existing on master). Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/importing-a-game.md | 85 +++++++++++++++++++++++++++++++ mux/game/data/db_load | 77 +++++++++++++++++++++++----- mux/game/data/db_unload | 64 ++++++++++++++++++----- mux/include/modules.h | 2 +- mux/include/sqlitedb.h | 9 ++++ mux/modules/engine/engine_com.cpp | 39 ++++++++++++-- mux/modules/engine/sqlitedb.cpp | 9 ++++ mux/src/driver.cpp | 18 +++++-- 8 files changed, 266 insertions(+), 37 deletions(-) create mode 100644 docs/importing-a-game.md diff --git a/docs/importing-a-game.md b/docs/importing-a-game.md new file mode 100644 index 000000000..4e2bb6c35 --- /dev/null +++ b/docs/importing-a-game.md @@ -0,0 +1,85 @@ +# Importing an existing game / migrating to the SQLite backend + +TinyMUX now stores the live game in a **SQLite database** rather than reading +the flatfile directly on every boot. This changes how you import an existing +game, and it is the source of a common "my old passwords don't work" symptom +when migrating from an older build. This page explains the model and the steps. + +## The two files in `game/data/` + +| File | What it is | Role | +|------|------------|------| +| `netmux.db` | The **flatfile** (human-readable text, starts with `+X...`) | Seed / export format only | +| `netmux.sqlite` | The **live database** the server actually runs from | Authoritative | + +The SQLite filename is derived from your configured `input_database` +(in `netmux.conf`) by replacing the `.db` suffix with `.sqlite`. So +`input_database data/netmux.db` means the server reads and writes +`data/netmux.sqlite`. + +## Boot precedence (read this twice) + +> **If `data/netmux.sqlite` exists and contains a game, the server warm-starts +> from it and the `netmux.db` flatfile is never read.** + +On a brand-new install, the first boot builds `netmux.sqlite` from the seed +flatfile `netmux.db`. After that, the flatfile is ignored. This is why simply +dropping your old game in as `netmux.db` does nothing once a `.sqlite` exists — +you keep logging into whatever is already in the SQLite database. The startup +log notes this with a "Warm-started from SQLite database; the flatfile ... was +not consulted" line. + +## Importing your old flatfile + +You have two equivalent options. Both assume the server is **stopped**. + +### Option A — let the server rebuild from a flatfile + +```sh +cd game +rm -f data/netmux.sqlite # remove the stale/seed database +cp /path/to/your_old_game.flat data/netmux.db +./bin/netmux # first boot rebuilds netmux.sqlite from it +``` + +### Option B — load the flatfile explicitly with db_load + +```sh +cd game/data +./db_load netmux /path/to/your_old_game.flat +cd .. +./bin/netmux +``` + +`db_load` always writes the `.sqlite` into the game's `data/` directory (next to +the script), regardless of which directory you run it from, so it can no longer +land somewhere the server doesn't read. It prints the exact file it loaded into. + +If `data/netmux.sqlite` already exists, `db_load` refuses to overwrite it and +tells you the file path. Either remove that file first, or force the replacement: + +```sh +./db_load -f netmux /path/to/your_old_game.flat +``` + +To bring across comsys and mail as well: + +```sh +./db_load netmux your_old_game.flat -C comsys.db -m mail.db +``` + +## Exporting back to a flatfile + +```sh +cd game/data +./db_unload netmux backup.flat # writes backup.flat in this directory +./db_unload netmux backup.flat -C comsys.db -m mail.db +``` + +## Passwords + +Password hashes (`$SHA1$...`, `$1$...`, etc.) are carried through import +unchanged, so old passwords continue to work once you are logging into the +correct database. If old passwords appear to fail, the cause is almost always +that the server is reading a different `netmux.sqlite` than the one your game +was imported into — see the boot precedence above. diff --git a/mux/game/data/db_load b/mux/game/data/db_load index 2873abae1..a7ec9674d 100755 --- a/mux/game/data/db_load +++ b/mux/game/data/db_load @@ -2,25 +2,74 @@ # # Load a MUX flatfile into the SQLite database. # -# Usage: db_load [-C ] [-m ] +# Usage: db_load [-f] [-C ] [-m ] # # Example: db_load netmux netmux.flat # db_load netmux netmux.flat -C comsys.db -m mail.db # +# The SQLite database (.sqlite) is always written next to this +# script, i.e. into the game's data/ directory, so it matches the server's +# configured input_database no matter which directory you run this from. +# Flatfile arguments are resolved relative to your current directory. +# +# Use -f to replace an existing SQLite database (the load otherwise refuses +# to overwrite a database that already contains a game). +# -BIN=../bin +# Directory this script lives in (the game's data/ directory). +SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) -LD_LIBRARY_PATH=$BIN; export LD_LIBRARY_PATH -DYLD_LIBRARY_PATH=$BIN; export DYLD_LIBRARY_PATH +# Directory the caller invoked us from, for resolving relative file arguments. +INVOKE_DIR=$(pwd) -case $# in - 2) $BIN/dbconvert -d "$1" -l -i "$2" ;; - 4) $BIN/dbconvert -d "$1" -l -i "$2" $3 "$4" ;; - 6) $BIN/dbconvert -d "$1" -l -i "$2" $3 "$4" $5 "$6" ;; - *) echo "Usage: $0 [-C ] [-m ]" - echo " e.g. $0 netmux netmux.flat" - echo " $0 netmux netmux.flat -C comsys.db -m mail.db" - exit 1 ;; -esac +abspath() { + case "$1" in + /*) printf '%s\n' "$1" ;; + *) printf '%s\n' "$INVOKE_DIR/$1" ;; + esac +} -exit 0 +usage() { + echo "Usage: $0 [-f] [-C ] [-m ]" + echo " e.g. $0 netmux netmux.flat" + echo " $0 netmux netmux.flat -C comsys.db -m mail.db" + echo " -f Replace an existing SQLite database." + exit 1 +} + +FORCE= +if [ "$1" = "-f" ]; then + FORCE=-f + shift +fi + +[ $# -ge 2 ] || usage +BASENAME="$1" +FLATFILE=$(abspath "$2") +shift 2 + +COMSYS= +MAIL= +while [ $# -ge 1 ]; do + case "$1" in + -C) [ $# -ge 2 ] || usage; COMSYS=$(abspath "$2"); shift 2 ;; + -m) [ $# -ge 2 ] || usage; MAIL=$(abspath "$2"); shift 2 ;; + *) usage ;; + esac +done + +BIN="$SCRIPT_DIR/../bin" +LD_LIBRARY_PATH="$BIN"; export LD_LIBRARY_PATH +DYLD_LIBRARY_PATH="$BIN"; export DYLD_LIBRARY_PATH + +cd "$SCRIPT_DIR" || exit 1 + +echo "Loading into: $SCRIPT_DIR/$BASENAME.sqlite" + +set -- -d "$BASENAME" -l -i "$FLATFILE" +[ -n "$FORCE" ] && set -- "$@" -f +[ -n "$COMSYS" ] && set -- "$@" -C "$COMSYS" +[ -n "$MAIL" ] && set -- "$@" -m "$MAIL" + +"$BIN/dbconvert" "$@" +exit $? diff --git a/mux/game/data/db_unload b/mux/game/data/db_unload index e607d2c85..ed18344bb 100755 --- a/mux/game/data/db_unload +++ b/mux/game/data/db_unload @@ -7,22 +7,60 @@ # Example: db_unload netmux netmux.flat # db_unload netmux netmux.flat -C comsys.db -m mail.db # +# The SQLite database (.sqlite) is always read from next to this +# script, i.e. from the game's data/ directory, so it matches the server's +# configured input_database no matter which directory you run this from. +# Output flatfile arguments are resolved relative to your current directory. +# # Note: The server should not be running during export. # -BIN=../bin +# Directory this script lives in (the game's data/ directory). +SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) -LD_LIBRARY_PATH=$BIN; export LD_LIBRARY_PATH -DYLD_LIBRARY_PATH=$BIN; export DYLD_LIBRARY_PATH +# Directory the caller invoked us from, for resolving relative file arguments. +INVOKE_DIR=$(pwd) -case $# in - 2) $BIN/dbconvert -d "$1" -u -o "$2" ;; - 4) $BIN/dbconvert -d "$1" -u -o "$2" $3 "$4" ;; - 6) $BIN/dbconvert -d "$1" -u -o "$2" $3 "$4" $5 "$6" ;; - *) echo "Usage: $0 [-C ] [-m ]" - echo " e.g. $0 netmux netmux.flat" - echo " $0 netmux netmux.flat -C comsys.db -m mail.db" - exit 1 ;; -esac +abspath() { + case "$1" in + /*) printf '%s\n' "$1" ;; + *) printf '%s\n' "$INVOKE_DIR/$1" ;; + esac +} -exit 0 +usage() { + echo "Usage: $0 [-C ] [-m ]" + echo " e.g. $0 netmux netmux.flat" + echo " $0 netmux netmux.flat -C comsys.db -m mail.db" + exit 1 +} + +[ $# -ge 2 ] || usage +BASENAME="$1" +FLATFILE=$(abspath "$2") +shift 2 + +COMSYS= +MAIL= +while [ $# -ge 1 ]; do + case "$1" in + -C) [ $# -ge 2 ] || usage; COMSYS=$(abspath "$2"); shift 2 ;; + -m) [ $# -ge 2 ] || usage; MAIL=$(abspath "$2"); shift 2 ;; + *) usage ;; + esac +done + +BIN="$SCRIPT_DIR/../bin" +LD_LIBRARY_PATH="$BIN"; export LD_LIBRARY_PATH +DYLD_LIBRARY_PATH="$BIN"; export DYLD_LIBRARY_PATH + +cd "$SCRIPT_DIR" || exit 1 + +echo "Exporting from: $SCRIPT_DIR/$BASENAME.sqlite" + +set -- -d "$BASENAME" -u -o "$FLATFILE" +[ -n "$COMSYS" ] && set -- "$@" -C "$COMSYS" +[ -n "$MAIL" ] && set -- "$@" -m "$MAIL" + +"$BIN/dbconvert" "$@" +exit $? diff --git a/mux/include/modules.h b/mux/include/modules.h index 37b064331..4228b87c0 100644 --- a/mux/include/modules.h +++ b/mux/include/modules.h @@ -844,7 +844,7 @@ public: // virtual MUX_RESULT DbConvert(const UTF8 *infile, const UTF8 *outfile, const UTF8 *basename, bool bCheck, bool bLoad, bool bUnload, - const UTF8 *comsys_file, const UTF8 *mail_file) = 0; + const UTF8 *comsys_file, const UTF8 *mail_file, bool bForce) = 0; // Query the static configuration basket. The driver calls this once // after LoadGame to get a snapshot of mudconf values it needs. diff --git a/mux/include/sqlitedb.h b/mux/include/sqlitedb.h index c9fde7aae..6274a33ea 100644 --- a/mux/include/sqlitedb.h +++ b/mux/include/sqlitedb.h @@ -15,6 +15,7 @@ #include #include #include +#include // When building the standalone test harness, define TINYMUX_TYPES_DEFINED // via compiler flags and provide these types. When building inside TinyMUX, @@ -40,6 +41,10 @@ public: void Close(); bool IsOpen() const { return nullptr != m_db; } + // Path of the currently-open database file (empty when closed). + // + const char *GetPath() const { return m_path.c_str(); } + // Object metadata operations. // These correspond to the db[] array and s_Location() etc. // @@ -294,6 +299,10 @@ public: private: sqlite3 *m_db; + // Path of the open database file, for diagnostics. + // + std::string m_path; + // Object metadata statements. // sqlite3_stmt *m_stmtObjInsert; diff --git a/mux/modules/engine/engine_com.cpp b/mux/modules/engine/engine_com.cpp index 32c581360..d6020c520 100644 --- a/mux/modules/engine/engine_com.cpp +++ b/mux/modules/engine/engine_com.cpp @@ -2610,7 +2610,7 @@ public: virtual MUX_RESULT Shutdown(void); virtual MUX_RESULT DbConvert(const UTF8 *infile, const UTF8 *outfile, const UTF8 *basename, bool bCheck, bool bLoad, bool bUnload, - const UTF8 *comsys_file, const UTF8 *mail_file); + const UTF8 *comsys_file, const UTF8 *mail_file, bool bForce); virtual MUX_RESULT GetConfig(DRIVER_CONFIG *pConfig); virtual MUX_RESULT MarkConnected(dbref player); virtual MUX_RESULT DumpChildExited(int child_pid); @@ -2984,6 +2984,16 @@ MUX_RESULT CGameEngine::LoadGame(const UTF8 *configFile, // bDoFlatfileLoad = false; bLoadedGameFromSQLite = true; + + // Make the precedence visible: a populated SQLite database + // shadows the configured flatfile, so a freshly dropped-in + // flatfile would otherwise appear to be silently ignored. + // + STARTLOG(LOG_ALWAYS, "INI", "LOAD") + log_text(T("Warm-started from SQLite database; the flatfile ")); + log_text(mudconf.indb); + log_text(T(" was not consulted. Remove the SQLite database to reload from a flatfile.")); + ENDLOG } else if (sqlite_load_rc < 0) { @@ -3660,7 +3670,7 @@ static void dbconvert_info(int fmt, int flags, int ver) MUX_RESULT CGameEngine::DbConvert(const UTF8 *infile, const UTF8 *outfile, const UTF8 *basename, bool bCheck, bool bLoad, bool bUnload, - const UTF8 *comsys_file, const UTF8 *mail_file) + const UTF8 *comsys_file, const UTF8 *mail_file, bool bForce) { int setflags, clrflags, ver; int db_ver, db_format, db_flags; @@ -3709,11 +3719,30 @@ MUX_RESULT CGameEngine::DbConvert(const UTF8 *infile, const UTF8 *outfile, mux_fprintf(stderr, T("Can\xE2\x80\x99t open SQLite database.\n")); return MUX_E_FAIL; } - else if (cc == HF_OPEN_STATUS_OLD) + + // Report exactly which file we opened so it is never a mystery where the + // database lives relative to the server's configured input_database. + // + const char *pDbPath = g_pSQLiteBackend->GetDB().GetPath(); + mux_fprintf(stderr, T("Database file: %s\n"), + (nullptr != pDbPath && '\0' != pDbPath[0]) ? pDbPath : "(unknown)"); + + if (cc == HF_OPEN_STATUS_OLD) { - if (setflags == OUTPUT_FLAGS) + if (setflags == OUTPUT_FLAGS && !bForce) { - mux_fprintf(stderr, T("Would overwrite existing SQLite database.\n")); + // The target SQLite database already holds a game. Loading would + // replace it, so refuse unless the caller explicitly forces it. + // Name the file and spell out both ways forward rather than + // leaving a dead-end that tempts running from the wrong directory. + // + mux_fprintf(stderr, + T("Refusing to overwrite the existing SQLite database:\n" + " %s\n" + "That file already contains a game. To replace it, either:\n" + " * remove the file shown above and re-run, or\n" + " * re-run this load with the -f (force) option.\n"), + (nullptr != pDbPath && '\0' != pDbPath[0]) ? pDbPath : "(unknown)"); CLOSE; return MUX_E_FAIL; } diff --git a/mux/modules/engine/sqlitedb.cpp b/mux/modules/engine/sqlitedb.cpp index 67eda23e1..1913048a8 100644 --- a/mux/modules/engine/sqlitedb.cpp +++ b/mux/modules/engine/sqlitedb.cpp @@ -132,6 +132,14 @@ bool CSQLiteDB::Open(const char *path) return false; } + // Remember the path we opened so callers can report exactly which file + // backs the database. + // + if (nullptr != path) + { + m_path.assign(path); + } + return true; } @@ -143,6 +151,7 @@ void CSQLiteDB::Close() sqlite3_close(m_db); m_db = nullptr; } + m_path.clear(); } bool CSQLiteDB::ConfigurePragmas() diff --git a/mux/src/driver.cpp b/mux/src/driver.cpp index c0b99d770..e9fbe3169 100644 --- a/mux/src/driver.cpp +++ b/mux/src/driver.cpp @@ -37,6 +37,7 @@ static bool standalone_load = false; static bool standalone_unload = false; static const UTF8 *standalone_comsys_file = nullptr; static const UTF8 *standalone_mail_file = nullptr; +static bool standalone_force = false; // dbconvert delegates to engine via mux_IGameEngine::DbConvert. // @@ -61,7 +62,8 @@ static void dbconvert(void) mr = pEngine->DbConvert(standalone_infile, standalone_outfile, standalone_basename, standalone_check, standalone_load, - standalone_unload, standalone_comsys_file, standalone_mail_file); + standalone_unload, standalone_comsys_file, standalone_mail_file, + standalone_force); pEngine->Release(); exit(MUX_SUCCEEDED(mr) ? 0 : 1); } @@ -157,6 +159,7 @@ void init_sql(void) #define CLI_DO_ERRORPATH CLI_USER+11 #define CLI_DO_COMSYS_FILE CLI_USER+12 #define CLI_DO_MAIL_FILE CLI_USER+13 +#define CLI_DO_FORCE CLI_USER+14 static bool bMinDB = false; static bool bSyntaxError = false; @@ -182,7 +185,8 @@ static CLI_OptionEntry OptionTable[] = { "C", CLI_REQUIRED, CLI_DO_COMSYS_FILE }, { "m", CLI_REQUIRED, CLI_DO_MAIL_FILE }, { "p", CLI_REQUIRED, CLI_DO_PID_FILE }, - { "e", CLI_REQUIRED, CLI_DO_ERRORPATH } + { "e", CLI_REQUIRED, CLI_DO_ERRORPATH }, + { "f", CLI_NONE, CLI_DO_FORCE } }; static void CLI_CallBack(CLI_OptionEntry *p, const char *pValue) @@ -256,6 +260,11 @@ static void CLI_CallBack(CLI_OptionEntry *p, const char *pValue) standalone_mail_file = reinterpret_cast(pValue); break; + case CLI_DO_FORCE: + g_bStandAlone = true; + standalone_force = true; + break; + case CLI_DO_USAGE: default: bSyntaxError = true; @@ -346,13 +355,14 @@ int DCL_CDECL main(int argc, char *argv[]) mux_fprintf(stderr, T("Version: %s" ENDLINE), g_version); if (g_bStandAlone) { - mux_fprintf(stderr, T("Usage: %s -d [-i ] [-o ] [-l|-u|-k] [-C ] [-m ]" ENDLINE), pProg); - mux_fprintf(stderr, T(" -d Basename." ENDLINE)); + mux_fprintf(stderr, T("Usage: %s -d [-i ] [-o ] [-l|-u|-k] [-f] [-C ] [-m ]" ENDLINE), pProg); + mux_fprintf(stderr, T(" -d Basename (the SQLite file is .sqlite, relative to the current directory)." ENDLINE)); mux_fprintf(stderr, T(" -i Input file." ENDLINE)); mux_fprintf(stderr, T(" -k Check." ENDLINE)); mux_fprintf(stderr, T(" -l Load (import flatfile into SQLite)." ENDLINE)); mux_fprintf(stderr, T(" -o Output file." ENDLINE)); mux_fprintf(stderr, T(" -u Unload (export SQLite to flatfile)." ENDLINE)); + mux_fprintf(stderr, T(" -f Force load over an existing SQLite database (replaces it)." ENDLINE)); mux_fprintf(stderr, T(" -C Comsys flatfile (import/export)." ENDLINE)); mux_fprintf(stderr, T(" -m Mail flatfile (import/export)." ENDLINE)); }