diff --git a/tools/testgen/README.md b/tools/testgen/README.md new file mode 100644 index 000000000..96ec68c1d --- /dev/null +++ b/tools/testgen/README.md @@ -0,0 +1,211 @@ +# gen-insn-tests.pl + +Pseudorandom test-case generator for NASM instruction patterns. For every +non-`PSEUDO` mnemonic in `x86/insns.xda`, it generates a +`travis//` test directory (`.asm` source, `.json` descriptor, +golden `.bin16/32/64[.t]`/`.stderr16/32/64[.t]` files) that plugs directly +into the existing `tools/travis/nasm-t.py` harness. This gives every +instruction-set pattern regression coverage without hand-writing +thousands of test cases. + +The generated tree currently checked in lives at `travis/insns/` (2432 +mnemonic subdirectories as of the last full run). + +## Usage + +```sh +# Regenerate everything into travis/insns (run from repo root, after +# ./nasm has been built and x86/insns.xda is up to date): +perl tools/testgen/gen-insn-tests.pl --outdir travis/insns + +# Regenerate just one mnemonic, useful while iterating on the generator: +perl tools/testgen/gen-insn-tests.pl --outdir /tmp/tg-scratch --only vaddps --verbose + +# Validate the full existing+generated suite (there's no generated +# Makefile in a fresh checkout without running configure, so invoke the +# harness directly rather than via `make travis`): +python3 tools/travis/nasm-t.py --nasm=./nasm --directory=./travis run +``` + +Options (`perl tools/testgen/gen-insn-tests.pl --help` equivalent, see +the `GetOptions` block at the top of the script for the authoritative +list): + +| Option | Default | Meaning | +|---|---|---| +| `--xda FILE` | `x86/insns.xda` | source instruction-template file | +| `--nasm PATH` | `./nasm` | nasm binary used to probe/assemble and to run `nasm-t.py update` | +| `--outdir DIR` | `travis` | where to write `/` subdirectories | +| `--seed N` | `1` | base seed for the deterministic per-mnemonic PRNG | +| `--per-mnemonic N` | `4` | max distinct operand-arity templates sampled per mnemonic | +| `--variants N` | `2` | concrete random instruction instances generated per sampled template | +| `--only MNEMONIC` | (all) | restrict generation to a single mnemonic, for iterating on the generator | +| `--no-undoc` | off | exclude `UNDOC`-flagged templates (not recommended — see below) | +| `--verbose` | off | print per-mnemonic progress as it generates | + +Generation is deterministic (fixed default seed), so re-running with no +source changes reproduces byte-identical `.asm`/`.json` files. It should +only need to be re-run when `x86/insns.dat` (and hence the regenerated +`x86/insns.xda`) changes — day-to-day `travis` test *runs* use the +existing harness and scale via `make -jN travis` as normal; generation is +a separate, infrequent step. + +## Design: why `x86/insns.xda` and not `insnsa.c`/`insns.pl` hooks + +Three data sources were considered: + +1. **Parse the generated `insnsa.c`/`insnsb.c` directly.** Rejected: + these encode operands as opaque bitmask constants and index into the + shared `nasm_bytecodes[]` array, which `x86/bytecode.txt` explicitly + documents as unstable ("byte codes can be moved around and recycled + at any time"). Parsing this from outside `insns.pl` would be fragile + across NASM releases and high-effort for no real benefit. +2. **Add new hooks to `x86/insns.pl`** to emit a purpose-built + intermediate format. Rejected: it would duplicate information + `insns.xda` already provides, adding maintenance risk to a + build-critical code generator for no real gain. +3. **Parse `x86/insns.xda`** (chosen). This file is already generated by + `insns.pl` from `insns.dat` as part of a normal build, with macros + fully expanded, one instruction template per line, in a small stable + text grammar: + + ``` + MNEMONIC optype1,optype2,... [enc: bytecode...] FLAGS + ``` + + Crucially, **we never need the bytecode/encoding field** — NASM's own + assembler selects the encoding from mnemonic + operand syntax at + assemble time. So the generator only needs to produce syntactically + valid operand text per operand-type token; it doesn't need to + understand or replicate NASM's instruction encoding at all. This + means the generated tests are **regression tests** — validated + against a golden captured from a known-good `nasm` build — not + independent correctness oracles. That's the same philosophy already + used by the rest of `travis/`. + +## Handling of UNDOC / OBSOLETE / NEVER instructions + +These are **included**, not excluded (only `PSEUDO` templates — which +aren't real ISA instructions, e.g. internal directives — are skipped). +Verifying that `OBSOLETE`/`NEVER`-flagged instructions produce the +expected warning (typically `-w+obsolete-removed` at the assembler's +default/highest CPU level) is itself part of the intended test coverage, +not something to suppress. When the generator's probe step detects +nonempty stderr for a template, it adds a `"stderr"` target to the +`.json` descriptor so the warning text becomes part of the golden. +`UNDOC` alone does not currently trigger a warning by default (verified +against `SALC`), so most `UNDOC` mnemonics just get ordinary output-only +targets. + +## Condition-code mnemonic families + +`insns.xda` leaves condition-code instruction families as literal, +unexpanded placeholders: `Jcc`, `SETcc`, `CMOVcc`, `CFCMOVcc`. Unlike +`$bwdq`-style width macros (which `insns.pl` *does* pre-expand into +concrete lines in `.xda`), NASM's parser expands condition codes against +its own cc-table at parse time — so the literal placeholder mnemonic +isn't directly assemblable. The generator expands these four families +itself into concrete mnemonics (`JE`, `SETNZ`, `CMOVGE`, ...) using the +16 canonical condition-code suffixes +(`o no b ae e ne be a s ns p np l ge le g`). + +**Known gap:** the newer APX condition-suffix families (`CCMPscc`, +`CTESTscc`, `CMPccXADD`, `SETccZU`) are *not* expanded and are currently +dropped as unsupported. Extending the `@cc_suffixes` expansion loop to +cover these would close this gap. + +## Bit-width (16/32/64) handling + +Bit-mode support isn't derived from CPU/mode flags in `insns.xda` — the +same generated `.asm` text is simply probed under `--bits 16/32/64`, and +only the modes that actually assemble successfully become targets in the +final `.json` (mirroring the pre-existing `travis/jmpxx/`-style pattern). +Two wrinkles this created and how they're handled: + +- **64-bit-only tokens poisoning 16/32 probes.** A single sampled + template using a 64-bit-only operand (e.g. `reg64`, `imm64`) would + otherwise make the *whole file* fail to assemble at 16/32-bit, even + though other templates in the same mnemonic don't need 64-bit. The + generator classifies operand tokens via a `%needs64_token` table and + renders **two** asm bodies per mnemonic — a "narrow" one (16/32-safe) + and a "full" one (includes 64-bit-only forms) — reusing one file when + they'd be identical. +- **Memory/vsib addressing across bit widths.** A fixed base-register + name (`rax` vs `eax` vs `ax`) isn't valid syntax across all three bit + widths. The generator sidesteps this by using **base-register-free** + addressing: bare-displacement memory operands (`[0x1234]`) and + base-free scaled vsib index forms (`[xmm0*1]`). This sacrifices + coverage of true `[base+index*scale+disp]` forms — a known, + documented limitation, not a bug. + +## Integration with `nasm-t.py` + +Golden capture is delegated entirely to the existing harness: + +```sh +python3 tools/travis/nasm-t.py --nasm=./nasm update -t / +``` + +rather than hand-rolling nasm invocation, renaming output binaries, or +hand-writing stderr golden files. This matters because +`nasm-t.py`'s stderr/stdout comparison is an **exact byte-for-byte +string match with no path normalization** — NASM embeds the literal +source-path argument it was invoked with into warning/error text, so +hand-rolled golden capture is easy to get subtly wrong. `update` uses +its own internal, consistent path construction, so the generator's own +cheap throwaway pre-probe (used only to decide per-bit-width viability +and whether a `"stderr"` target is needed) doesn't need to match those +paths exactly. If `nasm-t.py update` fails for a mnemonic (should only +happen if the pre-probe's viability check was wrong), the generator +drops that mnemonic's directory entirely rather than committing a +broken test. + +## Output naming convention + +Multiple bit-width variants generated from one source share the +established repo convention: `.bin16` / `.bin32` / `.bin64` +(binary; see e.g. `travis/jmpxx/jmpxx-ox.bin16`), with `.stderr16` / +`.stderr32` / `.stderr64` analogously for expected-warning goldens — not +`16.bin`. + +## Known limitations / coverage gaps + +The generator prints a coverage summary at the end of each run +(mnemonics generated, mnemonics dropped, and a frequency table of +unsupported operand-type tokens), so gaps are self-documenting. As of +the last full run: + +- **2432 / 2452 mnemonics generate at least one assemblable template.** +- **20 mnemonics produce zero output and are dropped**, mostly: + - `HINT_NOP0` .. `HINT_NOP63` placeholder mnemonics, + - `LOADALL` / `LOADALL286`, + - a handful of exotic AMX-transpose / APX instructions whose full + operand grammar wasn't targeted (e.g. `T2RPNTLVWZ0*`, `TCONJT*`, + `TTMMULTF32PS`). +- APX `scc`-suffix condition-code families (`CCMPscc`, `CTESTscc`, + `CMPccXADD`, `SETccZU`) are not expanded (see above). +- Memory/vsib operands only exercise base-register-free addressing + forms (see above) — true base+index+scale+disp combinations aren't + covered by generated tests. +- ~70+ distinct operand-type tokens have generator support (see the + `%fixed`/`%gen` tables at the top of the script); any new/renamed + token introduced by a future `insns.dat` change that isn't in those + tables will show up in the "unsupported operand-type tokens" summary + and cause the affected templates (not necessarily the whole mnemonic) + to be skipped. + +Extending coverage for any of the above is a matter of adding entries to +the `%fixed`/`%gen` operand-generator tables or the `@cc_suffixes` +expansion loop — no changes to the parser or driver loop are needed. + +## Validation + +Full-suite validation after the last generation run (existing `travis/` +tests + `travis/insns/`): + +```sh +python3 tools/travis/nasm-t.py --nasm=./nasm --directory=./travis run +``` + +4225 tests total, 4224 PASS, 1 pre-existing SKIP (`time`, an +intentional/known skip unrelated to this tool), 0 FAIL, 0 ABORT.