nasm/tools/testgen
H. Peter Anvin (Intel) 081a097909 testgen: document %ifdef ERROR error-case coverage
Adds a "Error-case (%ifdef ERROR) coverage" section describing the
needs64/avoid64 dual-source design and the nasm-t.py json convention
reused from travis/ret/ret.json, notes on the two branch-operand bugs
discovered and fixed along the way, and updates the "Known
limitations" and "Validation" sections with the final 2612/10
mnemonic counts and 1976/2612 error-coverage numbers.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-07-04 13:43:20 -07:00
..
gen-insn-tests.pl testgen: add %ifdef ERROR error-case coverage, fix branch-operand bugs 2026-07-04 13:41:59 -07:00
README.md testgen: document %ifdef ERROR error-case coverage 2026-07-04 13:43:20 -07:00

gen-insn-tests.pl

Pseudorandom test-case generator for NASM instruction patterns. For every non-PSEUDO mnemonic in x86/insns.xda, it generates a travis/<mnemonic>/ test directory (.asm source, .json descriptor, golden .bin16/32/64[.t]/.stderr16/32/64[.t] files) that plugs directly into the existing tools/travis/nasm-t.py harness. This gives every instruction-set pattern regression coverage without hand-writing thousands of test cases.

The generated tree currently checked in lives at travis/insns/ (2612 mnemonic subdirectories as of the last full run).

Usage

# Regenerate everything into travis/insns (run from repo root, after
# ./nasm has been built and x86/insns.xda is up to date):
perl tools/testgen/gen-insn-tests.pl --outdir travis/insns

# Regenerate just one mnemonic, useful while iterating on the generator:
perl tools/testgen/gen-insn-tests.pl --outdir /tmp/tg-scratch --only vaddps --verbose

# Validate the full existing+generated suite (there's no generated
# Makefile in a fresh checkout without running configure, so invoke the
# harness directly rather than via `make travis`):
python3 tools/travis/nasm-t.py --nasm=./nasm --directory=./travis run

Options (perl tools/testgen/gen-insn-tests.pl --help equivalent, see the GetOptions block at the top of the script for the authoritative list):

Option Default Meaning
--xda FILE x86/insns.xda source instruction-template file
--nasm PATH ./nasm nasm binary used to probe/assemble and to run nasm-t.py update
--outdir DIR travis where to write <mnemonic>/ subdirectories
--seed N 1 base seed for the deterministic per-mnemonic PRNG
--per-mnemonic N 4 max distinct operand-arity templates sampled per mnemonic
--variants N 2 concrete random instruction instances generated per sampled template
--only MNEMONIC (all) restrict generation to a single mnemonic, for iterating on the generator
--no-undoc off exclude UNDOC-flagged templates (not recommended — see below)
--verbose off print per-mnemonic progress as it generates

Generation is deterministic (fixed default seed), so re-running with no source changes reproduces byte-identical .asm/.json files. It should only need to be re-run when x86/insns.dat (and hence the regenerated x86/insns.xda) changes — day-to-day travis test runs use the existing harness and scale via make -jN travis as normal; generation is a separate, infrequent step.

Design: why x86/insns.xda and not insnsa.c/insns.pl hooks

Three data sources were considered:

  1. Parse the generated insnsa.c/insnsb.c directly. Rejected: these encode operands as opaque bitmask constants and index into the shared nasm_bytecodes[] array, which x86/bytecode.txt explicitly documents as unstable ("byte codes can be moved around and recycled at any time"). Parsing this from outside insns.pl would be fragile across NASM releases and high-effort for no real benefit.

  2. Add new hooks to x86/insns.pl to emit a purpose-built intermediate format. Rejected: it would duplicate information insns.xda already provides, adding maintenance risk to a build-critical code generator for no real gain.

  3. Parse x86/insns.xda (chosen). This file is already generated by insns.pl from insns.dat as part of a normal build, with macros fully expanded, one instruction template per line, in a small stable text grammar:

    MNEMONIC   optype1,optype2,...   [enc: bytecode...]   FLAGS
    

    Crucially, we never need the bytecode/encoding field — NASM's own assembler selects the encoding from mnemonic + operand syntax at assemble time. So the generator only needs to produce syntactically valid operand text per operand-type token; it doesn't need to understand or replicate NASM's instruction encoding at all. This means the generated tests are regression tests — validated against a golden captured from a known-good nasm build — not independent correctness oracles. That's the same philosophy already used by the rest of travis/.

Handling of UNDOC / OBSOLETE / NEVER instructions

These are included, not excluded (only PSEUDO templates — which aren't real ISA instructions, e.g. internal directives — are skipped). Verifying that OBSOLETE/NEVER-flagged instructions produce the expected warning (typically -w+obsolete-removed at the assembler's default/highest CPU level) is itself part of the intended test coverage, not something to suppress. When the generator's probe step detects nonempty stderr for a template, it adds a "stderr" target to the .json descriptor so the warning text becomes part of the golden. UNDOC alone does not currently trigger a warning by default (verified against SALC), so most UNDOC mnemonics just get ordinary output-only targets.

Condition-code mnemonic families

insns.xda leaves condition-code instruction families as literal, unexpanded placeholders: Jcc, SETcc, CMOVcc, CFCMOVcc (the "cc" family) and the newer APX CCMPscc, CTESTscc, CMPccXADD, SETccZU (the "scc" family). Unlike $bwdq-style width macros (which insns.pl does pre-expand into concrete lines in .xda), NASM's parser expands condition codes against its own cc-table at parse time — and x86/insns.pl's own conditional_forms() performs the equivalent expansion for the C-code generators, but only after insns.xda has already been written by preinsns.pl, so the placeholders reach this tool unexpanded either way.

The condition-code table (%conds, plus the $c_cc/$c_scc masks distinguishing which suffixes are valid for which family) is shared between insns.pl and this generator via x86/insns-cc.ph, a small Perl module required by both — this guarantees the two expansions never drift out of sync, since there's only one copy of the suffix/mask data. cc_suffix_list($is_scc) (defined in that module) returns the applicable suffix list for either family, and the generator mirrors insns.pl's own detection ($mnem =~ /s?cc/, case-sensitive) and substitution ($mnem =~ s/s?cc/\U$suffix/) exactly, so e.g. Jcc expands to JE/JNE/JGE/... , CCMPscc expands to CCMPE/ CCMPNE/... (excluding the "cc"-only suffixes pe/po/p/np, which aren't valid for scc-family instructions), and SETccZU expands to SETNEZU/... . Any future new "cc"/"scc" family added to insns.dat is picked up automatically without changes to this script.

Optional (*/?) operand coverage

insns.xda marks (at most one, per template) operand as optional:

  • a trailing * marks an optional source operand — when omitted from the source, x86/insns.pl's relaxed_forms() (which implements the actual encoding semantics for the C generators) duplicates the previous operand's value in the encoding, e.g. IMUL reg32,rm32*, imm32 covers both the 3-operand form and the 2-operand form IMUL reg32,imm32 (where reg32 is implicitly reused as the rm32).
  • a trailing ? marks an optional destination operand — when omitted, it's entirely absent from the encoding, e.g. APX non-destructive-destination (NDD) forms like INC reg32?,rm32 alongside the plain legacy INC rm32.

From a pure operand-syntax generation point of view we don't need either marker's full encoding semantics — both simply mean "this operand can be dropped from the concrete instruction line". The generator locates the (single) marked operand index per template via optional_operand_index() and, alongside the normal full-arity instruction line, emits one additional reduced-arity line with that operand dropped, so both the full and omitted forms get captured as goldens. This is done once per template (not once per --variants instance) to keep output size growth linear rather than combinatorial.

High-register-number (r8-r15, r16-r31) coverage

Registers numbered 8 and above only exist in 64-bit mode: r8-r15 need a REX prefix, and r16-r31 (APX) need REX2/EVEX register- extension bits — none of which exist outside 64-bit mode. This applies uniformly to GPRs and to vector registers (xmm/ymm/zmm 8-15 and 16-31). By default the generator only draws from the "low" tier (0-7) for its normal instruction lines, which is always valid at every --bits width.

For every operand-class token whose register pool has a hireg (8-15) or apxreg (16-31) tier — see gpr_pool() / vec_pool() and %extendable_base — the generator additionally tries building one all-hireg and one all-apxreg rendition of each eligible template (once per template, not once per --variants instance, via build_variant_line()). It doesn't need to consult APX/EVEX iflags to decide if this is legal for a given mnemonic: NASM's own template matching transparently substitutes an alternate (e.g. APX/EVEX-encoded) pattern whenever the operand syntax calls for it, so the generator can simply attempt to assemble the candidate and keep whichever combination works, using the same "regression test, not correctness oracle" philosophy as the rest of the tool.

A handful of mnemonics (mostly NOAPX/NOLONG-flagged ones, e.g. ARPL) genuinely can't take extended-register operands at all. To avoid letting one bad candidate line cost the whole mnemonic its pre-existing 64-bit coverage, a staged probe (shared with the EVEX decorator candidates below — see "Staged, cumulative probing") tries adding the hireg and apxreg buckets, keeping each only if it still assembles. The probe also checks --bits 16/32 whenever a mnemonic's only templates require 64-bit-sized registers regardless of number (e.g. URDMSR/UWRMSR, whose sole operands are reg64) — in that case the "full" body is reused verbatim for every bit width (see below), so a hireg/apxreg-only-in-64-bit-mode line would otherwise silently break their 16/32-bit coverage too.

This work also fixed a related latent bug: xmmreg/ymmreg/zmmreg previously drew uniformly from registers 0-15 for all lines, not just the register-number-focused ones, but 8-15 needs 64-bit mode too — so a perfectly ordinary SSE/AVX mnemonic (e.g. ADDPD) could non-deterministically lose its 16/32-bit golden whenever the PRNG happened to land on register 8 or above for that mnemonic's seed. The low/hireg/apxreg tier split fixes this for good, since normal lines now only ever draw from the 0-7 "low" pool.

EVEX decorator ({k}/{z}/{1toN}/{sae}/{er}) coverage

insns.xda operand tokens carry |-suffixed markers (mask, z, b16/b32/b64, sae, er) recording which of NASM's brace-decorator syntaxes a given EVEX-encoded operand supports (see the stripping regex in x86/insns.pl and asm/parser.c's parse_decorators() for the authoritative semantics). Each is exercised as an independent candidate line, once per distinct template across a mnemonic's entire template set — not just the capped per-mnemonic sample, since EVEX/ AVX512 forms are frequently appended well after a mnemonic's plain SSE/ AVX forms in insns.xda (e.g. VMOVAPD's mask-on-memory-destination forms come after its four plain AVX forms), so relying on the sample alone would silently skip decorator coverage for many mnemonics:

  • mask: {k1}-{k7} appended directly to the marked register (or, for masked-store instructions, memory) operand's text, e.g. zmm30{k7}.
  • z (zeroing): only ever co-occurs with mask on the same operand in insns.xda (verified empirically — no counter-examples), so it's folded into a second, maskz-family candidate that appends {z} immediately after the mask suffix: zmm30{k7}{z}.
  • b16/b32/b64 (broadcast): {1toN} appended to a memory operand (broadcast is only legal from memory, never a register), where N = vector-width-in-bits (128/256/512, from the token's xmm/ymm/ zmm/explicit-size base) divided by the marker's element width (16, 32, or 64), e.g. QWORD [rcx]{1to8} for a 512-bit operand with b64.
  • sae/er (suppress-all-exceptions / embedded rounding): written as a wholly separate, comma-preceded trailing pseudo-operand after all real operands, regardless of which real operand carried the marker in insns.xda (always the last operand in every observed template) — {sae} for sae, one of {rn-sae}/{rd-sae}/{ru-sae}/ {rz-sae} for er (embedded rounding always specifies a mode). Only legal when the marked operand resolves to a register, and — this is a real gotcha — the register must match the token's own declared size: e.g. VCVTSI2SD's rm64|er template requires a 64-bit register (rax) for {rn-sae} to be accepted; using a 32-bit register (eax, which is what the sibling rm32 template — with no |er — would use) fails with "unsupported mode decorator for instruction" even in 64-bit mode.

Register-span markers (rs2/rs4, used by multi-register FMA instructions like V4FMADDPS) are a related but distinct concept (register count/alignment, not {...}-suffix decorator syntax) and are deliberately out of scope for this coverage item.

Staged, cumulative probing

Six independent "extra candidate line" buckets now exist: hireg, apxreg (see above), and mask, maskz, broadcast, saeer (this section). Trying every subset combination would be combinatorially expensive as buckets are added, so the generator instead probes each bucket independently and cumulatively: starting from the already-validated base line set, each non-empty bucket is tentatively merged in and kept only if the combined result still assembles at the relevant bit width(s), otherwise it's discarded and the next bucket is tried against the last-known-good set. This is linear in the number of buckets rather than exponential, and — because each merge re-probes the whole accumulated candidate set, not just the new bucket in isolation — still correctly rejects a bucket that only fails in combination with an earlier-accepted one.

Decorator candidate lines are marked needs64 => 0 (mask/maskz/ broadcast syntax genuinely assembles fine at 16/32-bit given low-tier 0-7 registers, verified empirically), but the existing "extra-line" architecture only ever merges buckets into the 64-bit "full" body, never into the base narrow_lines computation — so in practice decorator lines only get 16/32-bit exercise in the (uncommon) case where a mnemonic's narrow_lines set is already empty for other reasons (all of its plain templates need 64-bit anyway). This is an accepted simplification, not a bug: most AVX512 mnemonics have non-decorated low-tier lines that keep narrow_lines non-empty regardless.

Modrm-memory disp8/disp32 boundary coverage

mem_operand() deliberately emits bare-displacement addressing ([0xNNN], no base register) for bit-width portability, which always encodes with a disp32 (or disp16 in 16-bit mode) — disp8 forms (or, for EVEX, the disp8×N compressed-displacement encoding) are never exercised by the ordinary generated lines. Whether a given instruction even has a disp8-encodable form, and what the compressed-displacement scale factor N is, is instruction-specific (it depends on the EVEX tuple type), so rather than computing the exact boundary per instruction, every distinct template (again scanned across the mnemonic's entire template set, for the same "buried variant" reason as EVEX decorators) with a modrm-memory-capable operand (mem*, rm*, xmmrm*/ymmrm256/zmmrm512, mmxrm* — see %mem_sizebits) additionally gets two candidate lines using [eax+1] and [eax+64] in place of that operand: +1 is unambiguously disp8-encodable everywhere, and +64 lands past the disp8 boundary for byte-granular encodings while still being a clean multiple of the larger EVEX compressed-displacement scales.

[eax+N] (rather than a bare displacement, or a bit-width-specific base register like rax/eax/ax) is used because it's valid addressing syntax at every --bits width — 32-bit addressing works in 16-bit and 64-bit code too via the 0x67 address-size-override prefix, confirmed empirically (mov al,[eax+1] assembles under --bits 16, 32, and 64 alike). This is why these candidates don't need any bit-width-specific handling, unlike hireg/apxreg. They're still routed through the same staged, cumulative probe as the other "extra" buckets (rather than being pushed directly into @lines), since a handful of instructions may have memory-operand restrictions this generator doesn't model (alignment, tuple-type quirks, etc.) and one bad candidate line shouldn't cost a mnemonic its pre-existing coverage.

Implicitly-sized memory operand coverage

Operand-type tokens fall into two groups with respect to memory-operand sizing: tokens with no size baked into their own name (plain mem), and tokens whose name already encodes a fixed size (mem8/16/32/ 64/128/256/512, rm8/16/32/64, xmmrm8/16/32/64/ 128, ymmrm256, zmmrm512, mmxrm/mmxrm64 — the same %mem_sizebits table used for disp-boundary coverage). For the first group, mem_operand($rng, 0) already omits a size keyword unconditionally, so a bare [0x1234]-style operand with no dword/ oword/etc. prefix was already being generated and exercised before this feature existed (confirmed by inspecting already-committed .asm files for mnemonics like PDISTIB/MOVNTI, whose sole memory operand is untyped mem).

The second group is where the actual gap was: the generator always attached an explicit size keyword for these tokens, so NASM's SM-flag-driven implicit-size-inference path — where a memory operand's size is inferred from a paired, already-sized operand in the same instruction (a SM0-N-flagged template, per x86/iflags.ph), e.g. ADD reg32,rm32 or MOVBE reg32,mem32 never actually need a dword keyword — was never tested.

Rather than parsing and range-expanding the SM/AR flag families from insns.xda to determine precisely which operand positions support implicit sizing (effectively reimplementing NASM's own operand-size-disambiguation logic), build_implicitsize_line() takes the same probe-and-keep approach as the other coverage buckets: for the first operand whose base token has a nonzero %mem_sizebits entry, it emits a bare (unsized) memory operand instead of the usual explicitly-sized one, leaving every other operand as normally generated. The candidate line is routed through the shared cumulative staged probe (alongside hireg/apxreg/mask/maskz/broadcast/saeer/ disp-boundary) and is only kept if it actually assembles for that specific instruction — so an instruction whose size truly can't be inferred (no SM flag, ambiguous encoding, etc.) simply doesn't gain the line, without risking the rest of its coverage.

Bit-width (16/32/64) handling

Bit-mode support isn't derived from CPU/mode flags in insns.xda — the same generated .asm text is simply probed under --bits 16/32/64, and only the modes that actually assemble successfully become targets in the final .json (mirroring the pre-existing travis/jmpxx/-style pattern). Two wrinkles this created and how they're handled:

  • 64-bit-only tokens poisoning 16/32 probes. A single sampled template using a 64-bit-only operand (e.g. reg64, imm64) would otherwise make the whole file fail to assemble at 16/32-bit, even though other templates in the same mnemonic don't need 64-bit. The generator classifies operand tokens via a %needs64_token table and renders two asm bodies per mnemonic — a "narrow" one (16/32-safe) and a "full" one (includes 64-bit-only forms) — reusing one file when they'd be identical.
  • Memory/vsib addressing across bit widths. A fixed base-register name (rax vs eax vs ax) isn't valid syntax across all three bit widths. The generator sidesteps this by using base-register-free addressing: bare-displacement memory operands ([0x1234]) and base-free scaled vsib index forms ([xmm0*1]). This sacrifices coverage of true [base+index*scale+disp] forms — a known, documented limitation, not a bug.
  • 16/32-bit-only tokens poisoning the 64-bit probe. The mirror image of the first wrinkle: CALL/JMP's near-indirect targets accept rm16/rm32/rm64 operands, but only rm64 is valid in 64-bit mode (call cx/call ecx both fail to assemble under --bits 64, matching the NOLONG flag on those insns.xda templates) — unlike an ordinary reg16/reg32 operand elsewhere, which assembles fine at any bit width. branch_narrow_only() flags lines built from these tokens (avoid64, mirroring needs64) and excludes them from the 64-bit "full" body; see "Error-case (%ifdef ERROR) coverage" below for where they end up instead.

Error-case (%ifdef ERROR) coverage

Verifying that an instruction fails to assemble the way it should (and prints the expected diagnostic) is itself test coverage, not just verifying the ways it succeeds. Per the preferred convention, error cases don't get a separate source file: known-bad lines are appended to the same per-mnemonic .asm file under a %ifdef ERROR guard, and the file is assembled twice — once plainly (existing behavior, unaffected, since the guarded block is invisible) and once with -DERROR defined (where the guarded block is included and expected to fail). This mirrors the pre-existing hand-written convention already used by travis/ret/ret.json ("option": "-DERROR ...", "error": "expected").

Two kinds of already-known-bad lines feed this, both by-products of the bit-width handling above rather than newly invented content:

  • Lines flagged needs64 (64-bit-only operands, hireg/apxreg register-number lines) are appended to the 16/32-bit "narrow" body and probed with -DERROR at --bits 16/32.
  • Lines flagged avoid64 (CALL/JMP's rm16/rm32 near-indirect targets) are appended to the 64-bit "full" body and probed with -DERROR at --bits 64.

Each candidate block is probed with -DERROR before becoming a json entry, and is only kept for the specific bit width(s) where it actually fails — this generator doesn't model every mode restriction, so a block that unexpectedly does assemble at some width simply doesn't get an "expected error" entry for that width, rather than becoming a false test failure. A mnemonic whose only surviving coverage would be error entries (no working positive-path line at any bit width) is still dropped entirely, same as before — "this instruction never assembles" isn't meaningful regression coverage by itself.

While wiring this up, two latent bugs in the pre-existing branch-target operand handling surfaced and were fixed: gen_operand()'s branch-mode substitution used to replace every operand of a branch mnemonic (is_branch) with the .L1 local-label text, not just genuine relative-displacement targets — producing nonsense like loop .L1, .L1 (LOOP's address-size-override form takes a fixed cx/ecx/rcx register, not a branch target) and call .L1 for JMP/CALL's indirect (rm16/32/64) and far-pointer (imm16:imm16) operand forms. These bogus lines silently broke assembly for the whole mnemonic, and LOOP/LOOPE/LOOPNE/LOOPNZ/LOOPZ/JCXZ were silently dropped as a result. Restricting the substitution to base tokens matching /^imm(?:8|16|32|64)$/ fixed both bugs and recovered all six previously-dropped mnemonics.

Integration with nasm-t.py

Golden capture is delegated entirely to the existing harness:

python3 tools/travis/nasm-t.py --nasm=./nasm update -t <outdir>/<mnemonic>

rather than hand-rolling nasm invocation, renaming output binaries, or hand-writing stderr golden files. This matters because nasm-t.py's stderr/stdout comparison is an exact byte-for-byte string match with no path normalization — NASM embeds the literal source-path argument it was invoked with into warning/error text, so hand-rolled golden capture is easy to get subtly wrong. update uses its own internal, consistent path construction, so the generator's own cheap throwaway pre-probe (used only to decide per-bit-width viability and whether a "stderr" target is needed) doesn't need to match those paths exactly. If nasm-t.py update fails for a mnemonic (should only happen if the pre-probe's viability check was wrong), the generator drops that mnemonic's directory entirely rather than committing a broken test.

Output naming convention

Multiple bit-width variants generated from one source share the established repo convention: <mnemonic>.bin16 / .bin32 / .bin64 (binary; see e.g. travis/jmpxx/jmpxx-ox.bin16), with .stderr16 / .stderr32 / .stderr64 analogously for expected-warning goldens — not <mnemonic>16.bin.

Known limitations / coverage gaps

The generator prints a coverage summary at the end of each run (mnemonics generated, mnemonics dropped, and a frequency table of unsupported operand-type tokens), so gaps are self-documenting. As of the last full run:

  • 2612 / 2622 mnemonics generate at least one assemblable template (includes all concrete expansions of the cc/scc condition-code families — see above).
  • 10 mnemonics produce zero output and are dropped, mostly:
    • HINT_NOP0 .. HINT_NOP63 placeholder mnemonics,
    • LOADALL / LOADALL286,
    • a handful of exotic AMX-transpose / APX instructions whose full operand grammar wasn't targeted (e.g. T2RPNTLVWZ0*, TCONJT*, TTMMULTF32PS).
  • Memory/vsib operands only exercise base-register-free (or, for the disp-boundary lines, [eax+N]) addressing forms — true base+index*scale+disp combinations, and non-eax base registers, aren't covered by generated tests.
  • Decorator ({k}/{z}/{1toN}/{sae}/{er}) candidate lines are only exercised at --bits 16/32 in the uncommon case where a mnemonic's non-decorated lines are already 64-bit-only (see "Staged, cumulative probing" above) — otherwise they're only ever probed at --bits 64, even though the syntax itself works at any width.
  • Disp8/disp32 boundary coverage uses two fixed offsets (+1, +64) rather than computing each instruction's actual disp8-encodable range (which depends on its EVEX tuple type for compressed displacement) — a deliberate approximation, not exhaustive boundary-value coverage.
  • Register-span (rs2/rs4) operands used by multi-register FMA instructions (e.g. V4FMADDPS) aren't targeted by dedicated coverage.
  • Implicit-size coverage only tries omitting the size keyword on the first size-carrying memory operand in a template; templates with multiple independently-sizable memory operands aren't exhaustively covered.
  • Error-case coverage only exercises the two bit-width-incompatibility patterns the generator already tracks for other reasons (needs64 and avoid64 lines); it doesn't target other classes of assembly errors (e.g. invalid immediate ranges, disallowed operand combinations not tied to bit width, EVEX-decorator misuse).
  • ~70+ distinct operand-type tokens have generator support (see the %fixed/%gen tables at the top of the script); any new/renamed token introduced by a future insns.dat change that isn't in those tables will show up in the "unsupported operand-type tokens" summary and cause the affected templates (not necessarily the whole mnemonic) to be skipped.

Extending coverage for any of the above is a matter of adding entries to the %fixed/%gen operand-generator tables or the shared x86/insns-cc.ph condition-code table — no changes to the parser or driver loop are needed.

Validation

Full-suite validation after the last generation run (existing travis/ tests + travis/insns/):

python3 tools/travis/nasm-t.py --nasm=./nasm --directory=./travis run

7371 tests total, 7370 PASS, 1 pre-existing SKIP (time, an intentional/known skip unrelated to this tool), 0 FAIL, 0 ABORT. (Count grew substantially from 4427 with this update, both from the new hireg/apxreg targets and from previously-silently-dropped 16/32-bit coverage being restored by the xmmreg/ymmreg/zmmreg low-tier bug fix described above.)

Adding EVEX decorator coverage did not change these counts — decorator candidate lines are additive to already-passing templates (971 mnemonics gained at least one decorator line), verified by comparing per-mnemonic .json entry counts between a scratch regeneration and the committed travis/insns/ tree (identical, 6955/6955) before committing. make -j32 travis continues to pass in ~26s.

Adding modrm-memory disp8/disp32 boundary coverage likewise did not change these counts (1987 mnemonics gained at least one [eax+N] boundary line, verified via the same per-mnemonic .json entry-count comparison, identical 6955/6955). make -j32 travis continues to pass in ~26s.

Adding implicitly-sized memory operand coverage likewise did not change these counts (1922 mnemonics gained at least one implicit-size line, verified via the same per-mnemonic .json entry-count comparison, identical 6955/6955). make -j32 travis continues to pass in ~26s.

Adding %ifdef ERROR error-case coverage — together with the branch-operand bug fixes it surfaced (see above) — changed the mnemonic-level counts for the first time since the initial hireg/apxreg update: 2612/2622 mnemonics now generate (up from 2606, since LOOP/LOOPE/LOOPNE/LOOPNZ/LOOPZ/JCXZ are no longer silently dropped), verified by diffing per-mnemonic non-error .json entry counts between a scratch regeneration and the previously-committed tree (identical aside from the six newly-recovered mnemonics — confirming no regressions in existing coverage). 1976/2612 mnemonics gained at least one error-case entry (3951 error json entries total). Full nasm-t.py run on the scratch tree: 10918/10918 PASS, 0 FAIL. make -j32 travis on the regenerated travis/insns/ continues to pass in ~27s.