fluffos/docs/efun/objects/recompile_object.md
Yucong Sun 82d82f3beb
Add recompile_object() efun: in-place program update, state preserved (#1237)
* Add recompile_object() efun: in-place program update, state preserved

Recompiles a master copy's program from its source file and swaps the
fresh program into the LIVE master copy and every clone sharing it -
the "hot update" alternative to destruct+load_object: nothing is
destructed, so object identity (pointers held elsewhere, name,
inventory, shadows, interactive state, call_outs, heart_beat) is
untouched, and each object's global variables carry over BY NAME
inside the driver (private ones included): the new program's __INIT
runs first, then every surviving name gets its old value back. The
recompile behaves like a normal load - unloaded parents resolve
through the retry dance and the compile-time master applies are
consulted. Returns the number of objects updated.

Made possible by moving an object's variable block OUT of the object_t
allocation into its own (TAG_OBJ_VARS, always >= 1 svalue, wired into
the debug-malloc walkers): every access already went through
ob->variables[i], so a program with a different variable count can now
be swapped onto a live object.

Safety: refused while any object sharing the program is executing
anywhere on the call stack (live frames hold bytecode positions and
variable indices of the old layout), for clones (pass the master
copy), the simul_efun object, pending replace_program(), and nested
calls. Function pointers whose behavior depends on the owner's program
layout (FP_LOCAL, FP_FUNCTIONAL) go stale instead of corrupting:
objects carry a prog_generation stamp, funptrs snapshot it at
creation/bind, and call_function_pointer() errors cleanly on mismatch.

Fixing a latent asymmetry this exposed: make_lfun_funp incremented
func_ref on the creation-time program but dealloc_funp decremented the
owner's CURRENT program. FP_LOCAL pointers now store their program and
account against it symmetrically (checkmemory and %O formatting
updated to match) - caught by the debug-build memory checker in the
testsuite.

The hot-reload daemon's default (state-keeping) path now reloads
through recompile_object() - changed ancestors first, then the watched
program - so clones ride along automatically; a cooperative
hot_reload_state()/hot_reload_restore() pair takes the destruct+load
path with exactly the state it chooses, and watch(prog, 0) opts out
entirely. The daemon test demonstrates finding all live instances with
children()/clonep() and both clone behaviors (updated in place vs.
stragglers on the old program); single/tests/efuns/recompile_object.lpc
pins the efun semantics (master+clones count, per-object state incl.
private, initializers for new variables, removed variables, stale
funptrs, executing/clone/missing-source guards, call_out survival).

Docs: efun reference page, hot-reload guide step 5 rewritten around
the efun with the value-transfer technique kept as the manual
alternative, caveats updated (clone behavior per path, stale funptrs).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018DGhzJfPhDGJA94EPh1gEK

* recompile_object: support master/simul_efun targets; review fixes

The master object and the simul_efun object can now be recompiled
live. Both subsystems dispatch through cached name->runtime-index
tables (master_applies / simuls) whose entries point into the old
program's function table, so recompile_object() rebuilds them against
the new program immediately after the swap and BEFORE the new
program's __INIT runs (an error inside it would already route through
those tables). Simul_efun indices are preserved by NAME across the
rebuild - that table is deliberately unsorted for exactly this reason
- so simul calls compiled into every other program keep working, and
a simul removed by the new source fails with the usual "no longer a
simul_efun" runtime error. set_master()/set_simul_efun() only
ref/assign when the object actually changes, keeping the classic
destruct-driven replacement path intact. %O of a function pointer to
a since-removed simul now prints a placeholder instead of derefing
the null table entry.

Testsuite: the efun test recompiles the live simul_efun object
mid-run (the very next ASSERT dispatches through the rebuilt table),
pins the currently-executing guard on the master (master::flag() sits
on the call stack for the whole run), and re-runs the master recompile
from a post-run call_out where the master is idle - state carry-over
and apply dispatch are enforced by exiting nonzero.

Also from this self-review round (multi-agent):

* f_recompile_object crashed when the target destructed itself from
  its new program's __INIT: destruct sweeps the VM stack, so the efun
  glue's stack slot held a plain 0 by the time it tried to
  free_object() it. Reproduced by a review agent's probe; the glue
  now uses free_svalue(), and the scenario is pinned in the efun test
  (destructed targets drop out of the updated count).

* hot_reload daemon: ancestors() now returns the inherit closure
  DEEPEST-first - recompiling a middle parent bakes in whatever
  grandparent program is live at that moment, so a >=3-level chain
  with two changed ancestors permanently embedded the stale
  grandparent (reproduced by a review agent; pinned by a new
  kid/mid/grand scenario).

* hot_reload daemon: dep records were map_delete'd before the
  recompile and rebuilt by the applies during it - but a throw BEFORE
  compiling (currently-executing guard, unreadable file) left the
  object loaded with no records, blinding closure_changed() to
  include-file edits forever. Records are now restored when the
  recompile throws (pinned by a new watched-object-drives-the-pass
  scenario).

* docs: inheritance wording ("copies code" -> the child links against
  the exact parent program it was compiled with), the
  currently-executing guard also covers inheritors running inherited
  code, and the cooperative-pair opt-out triggers on
  hot_reload_state() alone.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018DGhzJfPhDGJA94EPh1gEK

* recompile_object: void mid-update replace_program; cover virtuals

Two additions from the C++ review round:

* A replace_program() registered DURING the update slipped past the
  pre-flight check: an earlier target's __INIT can call into a
  not-yet-swapped clone, whose OLD code registers a pending entry -
  computed against the very program the update is replacing. The
  backend sweep then ran that entry's variable-offset shuffle against
  the fresh program's differently-sized variable block (negative
  num_fewer, heap corruption; reproduced under ASan by a review
  agent's probe). recompile_object() now voids any pending entry for
  each target at its swap point - an entry registered AFTER the swap
  is computed against the new program and survives. Pinned in the
  efun test; the rest of the suite run doubles as the sweep detector.

* Virtual objects (materialized through master::compile_object) are
  covered and pinned: the virtual object carries the BACKING file's
  program, so the recompile targets that source and swaps it in with
  the virtual name, identity, flag and state untouched. The testsuite
  master gains a /data/hu/virt* fixture mapping; docs note the
  behavior and that the hot-reload daemon keys its records by
  compiled program name (watch virtuals via their backing file).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018DGhzJfPhDGJA94EPh1gEK

* docs: capture hot-reload/recompile_object knowledge in README and AGENTS

README: the hot-reload language bullet now describes what actually
ships (recompile_object with state carried by name, clones included),
and Features gains a Hot Reload section linking the guide.

AGENTS.md, for future agents working on this machinery: the object
variable block is a separate allocation (TAG_OBJ_VARS) and what that
enables; the new-DMALLOC-tag checklist (checkmemory walkers); the
destruct-sweeps-the-VM-stack rule for efun glue; testsuite harness
facts (fixtures outside tests/, unconditional teardown, master::flag
on the stack all run + the post-run call_out pattern, full -ftest
paths, suite side-effect files); the compile-time master applies; and
the recompile_object invariants (executing-frame guard, dispatch-table
rebuild before __INIT, voiding mid-update replace_program entries,
funptr generation staleness, FP_LOCAL func_ref symmetry).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018DGhzJfPhDGJA94EPh1gEK

* recompile_object: pin shadow/catch_tell/add_action/heart_beat survival

New test recompile_object2.lpc verifies the object-attached runtime
state that dispatches by name keeps working across the swap:
catch_tell routes into the new program while accumulated state stays;
a shadow chain survives updating the SHADOWED object (still
intercepted, new code underneath) and updating the SHADOW itself while
attached; add_action sentences registered by the old code still fire
their verb into the new program; the heart_beat registration persists.

Also two doc wording fixes from the docs review: the executing-guard
bullet now covers both halves of the guard (frames executing the
program's code AND frames belonging to an object of the program
running inherited code), and the guide's mode summary matches the
daemon (hot_reload_state alone selects the cooperative path).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018DGhzJfPhDGJA94EPh1gEK

* recompile_object: fix simul_efun/__INIT edge cases; cover callback surface

Four defects from the C++ review round, each probe-verified under ASan:

* Recompiling the simul_efun object to a program that defines no simuls
  FREEd the live dispatch table (simul_names/simuls) while other
  compiled programs still carry F_SIMUL_EFUN opcodes and FP_SIMUL
  funptrs with baked indices -> use-after-free on the next simul call.
  Keep the tombstoned arrays instead (remove_simuls() already nulls
  every func, which yields the clean "no longer a simul_efun" error and
  preserves the name->index mapping for re-adds).

* The debug memory checker did not mark IHE_ORPHAN idents as permanent,
  so any run that removed a simul via an update tripped a spurious
  "orphan permanent identifier" leak and failed the testsuite gate.
  Add IHE_ORPHAN to the mark mask (it is part of IHE_PERMANENT).

* The disassembler dereferenced simuls[].func unguarded in two places;
  after a simul removal, dump_prog() on a program referencing it would
  null-deref. Guard both, matching the sprintf %O fix.

* An error() thrown from a target's __INIT during the swap leaked this
  loop's held references (the per-target snapshot ref, new_prog's
  compile ref, the old variable block) and left the update half-applied.
  Wrap call___INIT per target in save_context/try/restore: on error the
  object is left committed to the new program with fresh initializers
  (carried-over state dropped, like a create() that throws during load),
  sibling targets still update, and nothing leaks.

Test coverage:
* recompile_object.lpc: an __INIT that errors -- blueprint and clone
  both recompile, neither is immortalized, the object stays usable on
  the new program, no ref/variable leak (the per-file memory checker is
  the detector). (The simul zero-function / removal paths can't be
  exercised against the shared /single/simul_efun mid-suite; verified
  out-of-band with a throwaway ASan probe that reduces then restores the
  file.)
* recompile_object2.lpc: call_outs armed before the swap fire after it --
  a name-based call_out dispatches into the new program, a funptr
  call_out is stale and is refused cleanly (its target never runs, no
  crash), verified from a post-run call_out. Rounds out the by-name
  callback survivors already covered (catch_tell, add_action, heart_beat,
  shadows).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018DGhzJfPhDGJA94EPh1gEK

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-07-11 11:24:30 -04:00

4.4 KiB

title
objects / recompile_object

recompile_object

NAME

recompile_object() - recompile a program and swap it into the live
master copy and all its clones, keeping variable state

SYNOPSIS

int recompile_object( object master_copy );

DESCRIPTION

Recompiles the master copy's program from its source file and swaps
the fresh program into the master copy AND every clone sharing it.
Nothing is destructed: object identity is preserved everywhere, so
pointers held by other objects, the object's name, inventory,
shadows, interactive state, heart_beat and pending call_outs all
stay intact. This is an in-place "hot update", as opposed to the classic
destruct + load_object() cycle (which resets state and cannot touch
clones).

Each updated object's global variables carry over BY NAME: the new
program's variable initializers run first, then every variable
whose name also existed in the old program gets its old value back
(private variables included - the transfer happens inside the
driver). Variables new in this version keep their initializers;
vanished names are dropped. create() is NOT called again.

The recompile behaves like a normal load: unloaded inherited
programs are loaded on demand, and the master applies
inherit_program(4), include_file(4), get_include_path(4) and
valid_read(4) are all consulted.

Virtual objects (materialized through the master's
compile_object(4) hook) update like any other object: the recompile
targets the BACKING program - the real file whose program the
virtual object carries - and the virtual name, identity and flag
are untouched. A virtual object whose backing program has no
on-disk source fails cleanly.

The master object and the simul_efun object can themselves be
recompiled: their cached dispatch tables (apply-name and
simul_efun-name to function) are rebuilt against the new program
before its initializers run. Simul_efun indices are preserved by
NAME across the rebuild, so simul calls compiled into every other
program keep working; a simul_efun removed by the new source fails
with the usual "no longer a simul_efun" runtime error. Note that
the currently-executing rule below applies as usual - the master
cannot be recompiled from code the master itself is running (e.g.
from inside one of its applies).

Returns the number of objects updated (the master copy plus its
clones).

ERRORS

The call fails with an error if:

- a clone is passed (pass the master copy; its clones are updated
  with it);
- any live frame is executing the program's code or belongs to an
  object running it, anywhere on the call stack - including an
  inheritor running one of its inherited functions, and an object
  of this program running an inherited parent's code (bytecode
  positions and variable indices in live frames are relative to the
  old layout). In particular an object cannot recompile itself;
- the source fails to compile (the objects are left untouched on
  the old program);
- a replace_program() is pending on the program, or another
  recompile_object() is already in progress.

CAVEATS

Function pointers made before the update whose behavior depends on
the owner's program layout (pointers to local functions, and
functionals / anonymous functions) become STALE: calling them after
a recompile_object of their owner raises a clean "Stale function pointer"
error instead of running mis-indexed code. Recreate them after the
update. Efun and simul_efun pointers are unaffected.

Programs that INHERIT the updated program are not recompiled - like
with the destruct/reload cycle, a child program stays bound to the
exact parent program it was compiled against. Update inheritors
separately (parents first). The testsuite's
/single/hot_reload.lpc daemon automates exactly that ordering from
the compile-time dependency graph.

EXAMPLE

```c
object ob = find_object("/obj/sword");

// ... edit /obj/sword.c on disk ...

int n = recompile_object(ob);
// every live sword (master copy + clones) now runs the new code,
// each keeping its own enchantment, wielder, condition, ...
write(sprintf("updated %d objects\n", n));
```

SEE ALSO

reload_object(3), children(3), clonep(3), destruct(3),
inherit_program(4), include_file(4)