mirror of
https://github.com/netwide-assembler/nasm
synced 2026-08-26 16:23:04 -04:00
eee1384bf5
Add error-case ("negative test") coverage per the user's preferred
convention: rather than a separate source file, error-triggering
instruction lines are appended to the existing per-mnemonic .asm file
under a %ifdef ERROR guard, and the harness assembles the same file
twice -- once without -DERROR (existing positive-path coverage,
unaffected) and once with -DERROR (expected to fail, per nasm-t.py's
"error": "expected" json convention, matching the pre-existing
travis/ret/ret.json pattern).
The error material comes for free from lines the generator already
knows are bit-width-incompatible:
- Lines needing 64-bit encodings (reg64/imm64 operands, hireg r8-r15,
apxreg r16-r31, etc. -- %needs64_token / build_variant_line) are, by
construction, exactly the lines already excluded from the 16/32-bit
"narrow" body. They're appended to the narrow file under %ifdef
ERROR and probed at --bits 16/32 with -DERROR; only widths where the
block actually fails become json entries.
- Symmetrically, CALL/JMP near-indirect targets via rm16/rm32 (only
rm64 is valid in 64-bit mode -- confirmed empirically, matches the
NOLONG flag on those insns.xda templates) are appended to the full
(64-bit) body under %ifdef ERROR and probed at --bits 64 with
-DERROR.
Each candidate block is probed before being turned into a json entry,
so a line that unexpectedly *does* assemble at some width (this
generator doesn't model every mode restriction) doesn't turn into a
bogus "expected error" test; a mnemonic whose *only* surviving
coverage would be error entries is also rejected (see below), since
"this never assembles" isn't meaningful regression coverage on its
own.
While wiring this up, discovered and fixed two related bugs in the
existing branch-mnemonic handling (gen_operand()'s is_branch
substitution):
1. is_branch replaced *every* operand of a branch mnemonic with the
".L1" local-label text, not just genuine relative/near/short/abs
branch-displacement operands. This produced nonsensical lines like
"loop .L1, .L1" (LOOP's address-size-override form takes a fixed
"cx"/"ecx"/"rcx" second operand, not a branch target) and "call
.L1" for JMP/CALL's indirect (rm16/32/64) and far-pointer
(imm16:imm16) forms instead of an actual register/memory operand.
These bogus lines silently poisoned assembly for the whole
mnemonic, and LOOP/LOOPE/LOOPNE/LOOPNZ/LOOPZ/JCXZ were silently
dropped entirely as a result (present in the "16 dropped" list).
Restricting the substitution to base tokens matching
/^imm(?:8|16|32|64)$/ fixes both LOOP's operand and JMP/CALL's
indirect/far forms, and recovers all six previously-dropped
mnemonics with correct coverage.
2. Once (1) exposed genuine rm16/rm32 operand generation for CALL/JMP,
a new bit-width interaction appeared: rm16/rm32 near-indirect
targets are only valid in 16/32-bit mode (unlike ordinary reg16/32
operands elsewhere, which work at any bit width), so a line built
from one now broke 64-bit assembly for the whole mnemonic the same
way a needs64 line breaks 16/32-bit assembly. Added
branch_narrow_only() and a parallel avoid64 line flag (mirroring
needs64) to exclude these lines from the 64-bit "full" body -- this
is also what feeds the new symmetric 64-bit error-case coverage
described above.
Also added a safety-net to the existing "couldn't assemble in any
mode, drop the directory" check: a directory is now only kept if it
has at least one *non*-error json entry, preventing a future bug
symmetric to (1) from silently producing a directory whose only
content is error-case entries.
Verified via full scratch regeneration (2612 mnemonics generated / 10
dropped, up from 2606/16 thanks to the LOOP-family/JCXZ fix) +
nasm-t.py run (10918/10918 PASS, 0 FAIL) + per-mnemonic non-error
.json entry-count diff against the prior committed tree (identical
except the 6 newly-recovered mnemonics, confirming no regressions).
1976/2612 mnemonics gained at least one error-case entry (3951 error
json entries total). Regenerated travis/insns/ and validated via
'make -j32 travis' (all PASS, ~27s).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1194 lines
55 KiB
Perl
Executable file
1194 lines
55 KiB
Perl
Executable file
#!/usr/bin/perl
|
|
# SPDX-License-Identifier: BSD-2-Clause
|
|
# Copyright 2026 The NASM Authors - All Rights Reserved
|
|
#
|
|
# gen-insn-tests.pl
|
|
#
|
|
# Prototype pseudorandom test-case generator for NASM instruction
|
|
# patterns. For each mnemonic found in x86/insns.xda, emits one
|
|
# travis/<mnemonic>/ directory containing:
|
|
#
|
|
# <mnemonic>.asm - a handful of pseudorandomly-generated valid
|
|
# instances of the instruction's operand forms
|
|
# <mnemonic>.json - a tools/travis/nasm-t.py descriptor with one
|
|
# target per {16,32,64}-bit mode that actually
|
|
# assembles successfully (mirrors the existing
|
|
# travis/pushimm/pushimm.json bin16/32/64 pattern)
|
|
# <mnemonic>.log - the golden output for each mode
|
|
#
|
|
# Design notes (see ai/ or the accompanying design writeup for the
|
|
# full rationale) --- the short version:
|
|
#
|
|
# * Data source is x86/insns.xda, NOT insnsa.c/insnsb.c. insns.xda is
|
|
# already generated by insns.pl from insns.dat, with macros fully
|
|
# expanded, one template per line, in a small stable text grammar:
|
|
#
|
|
# MNEMONIC optype1,optype2,... [enc: bytecode...] FLAGS
|
|
#
|
|
# insnsa.c/insnsb.c instead encode operands as opaque bitmask
|
|
# constants and index into a shared, explicitly-unstable bytecode
|
|
# array (x86/bytecode.txt: "byte codes can be moved around and
|
|
# recycled at any time") -- parsing that from outside insns.pl would
|
|
# be high effort and fragile across releases. Hooking insns.pl to
|
|
# emit yet another intermediate format would duplicate what
|
|
# insns.xda already gives us "for free" while adding maintenance
|
|
# risk to a build-critical generator. insns.xda requires no code
|
|
# generator changes at all.
|
|
#
|
|
# * We only need valid operand *syntax* per operand-type token --- we
|
|
# deliberately do NOT need to decode the bytecode/encoding field.
|
|
# NASM's assembler chooses the encoding from mnemonic + operand
|
|
# syntax on its own. This makes the generated tests regression
|
|
# tests (they catch future encoder/output regressions against a
|
|
# golden captured from a known-good nasm build), not independent
|
|
# correctness oracles -- the same philosophy already used by the
|
|
# rest of travis/.
|
|
#
|
|
# * Bit-mode support (16/32/64) is not hand-modeled from CPU flags;
|
|
# instead the same generated .asm text is tried under
|
|
# --bits 16/32/64 and only the modes that actually assemble
|
|
# successfully become targets, exactly like the pre-existing
|
|
# travis/pushimm/pushimm.json.
|
|
#
|
|
# * Operand-type tokens we don't yet know how to generate syntax for
|
|
# are skipped (with the containing template dropped) and counted in
|
|
# a coverage report printed at the end, so extending coverage is a
|
|
# matter of adding table entries, not touching the parser.
|
|
|
|
use strict;
|
|
use warnings;
|
|
use File::Path qw(make_path);
|
|
use Getopt::Long;
|
|
|
|
my $xda_file = 'x86/insns.xda';
|
|
my $nasm_bin = './nasm';
|
|
my $out_dir = 'travis';
|
|
my $seed = 1; # deterministic by default
|
|
my $per_mnem = 4; # max templates sampled per mnemonic
|
|
my $variants = 2; # concrete random instances per template
|
|
my $only = undef; # restrict to one mnemonic, for testing
|
|
my $exclude_undoc = 0; # by default include UNDOC/OBSOLETE/NEVER templates:
|
|
# verifying their expected warnings is itself
|
|
# test coverage (only PSEUDO ops are excluded).
|
|
my $verbose = 0;
|
|
|
|
GetOptions(
|
|
'xda=s' => \$xda_file,
|
|
'nasm=s' => \$nasm_bin,
|
|
'outdir=s' => \$out_dir,
|
|
'seed=i' => \$seed,
|
|
'per-mnemonic=i' => \$per_mnem,
|
|
'variants=i' => \$variants,
|
|
'only=s' => \$only,
|
|
'no-undoc' => \$exclude_undoc,
|
|
'verbose' => \$verbose,
|
|
) or die "usage error\n";
|
|
|
|
#-----------------------------------------------------------------------
|
|
# 1. Parse insns.xda into a list of templates grouped by mnemonic.
|
|
#-----------------------------------------------------------------------
|
|
|
|
my %by_mnemonic; # mnemonic => [ { ops => [...], flags => {...} }, ... ]
|
|
|
|
# insns.xda leaves condition-code instruction *families* as literal
|
|
# "...cc"/"...scc"-suffixed placeholders (Jcc, SETcc, CMOVcc, CFCMOVcc,
|
|
# CCMPscc, CTESTscc, CMPccXADD, SETccZU) -- x86/insns.pl's own
|
|
# conditional_forms() expands these into one concrete pattern per
|
|
# applicable condition code, but that expansion happens *after*
|
|
# insns.xda is generated (insns.xda comes from preinsns.pl), so the
|
|
# placeholders reach us unexpanded. Mirror insns.pl's own detection
|
|
# (case-sensitive "/s?cc/" match) and expansion using the shared
|
|
# x86/insns-cc.ph condition-code tables, so concrete, testable
|
|
# mnemonics (JE, SETNZ, CMOVGE, CCMPNE, SETccZU -> SETNEZU, ...) get
|
|
# generated instead of the unusable literal placeholder, and any
|
|
# future new cc/scc family added to insns.dat is picked up
|
|
# automatically without touching this script.
|
|
require 'x86/insns-cc.ph';
|
|
|
|
open(my $xda, '<', $xda_file) or die "cannot open $xda_file: $!\n";
|
|
while (my $line = <$xda>) {
|
|
next if $line =~ /^\s*;/ || $line =~ /^\s*$/;
|
|
# Skip pseudo-op / bytecode-less lines (operand list literally
|
|
# "ignore", third field not "[...]").
|
|
next unless $line =~ /^(\S+)\s+(\S+)\s+\[([^\]]*)\]\s*(\S*)\s*$/;
|
|
my ($mnem, $opstr, $enc, $flagstr) = ($1, $2, $3, $4);
|
|
my %flags = map { $_ => 1 } split(/,/, $flagstr // '');
|
|
|
|
next if $flags{PSEUDO}; # DB/DW/EQU/RESB/... aren't real ISA instructions
|
|
next if ($flags{UNDOC} && $exclude_undoc);
|
|
|
|
my @ops = ($opstr eq 'void') ? () : split(/,/, $opstr);
|
|
|
|
if ($mnem =~ /s?cc/) { # case-sensitive, same as insns.pl
|
|
my $is_scc = ($mnem =~ /scc/);
|
|
for my $suf (cc_suffix_list($is_scc)) {
|
|
(my $concrete = $mnem) =~ s/s?cc/\U$suf/;
|
|
push @{ $by_mnemonic{$concrete} }, { ops => \@ops, flags => \%flags };
|
|
}
|
|
next;
|
|
}
|
|
|
|
push @{ $by_mnemonic{$mnem} }, { ops => \@ops, flags => \%flags };
|
|
}
|
|
close($xda);
|
|
|
|
if (defined $only) {
|
|
%by_mnemonic = ($only => $by_mnemonic{$only})
|
|
if exists $by_mnemonic{$only};
|
|
}
|
|
|
|
#-----------------------------------------------------------------------
|
|
# 2. Operand-type token -> pseudorandom concrete NASM syntax.
|
|
#
|
|
# Each generator is called as generator->($rng, $decorators) and
|
|
# returns a text fragment usable directly as an operand. Registers
|
|
# are deliberately drawn from "REX-free" legacy pools by default so
|
|
# that the same generated text has the best chance of assembling
|
|
# successfully in all of 16/32/64-bit mode (bit-mode support is
|
|
# ultimately decided empirically in step 4, not modeled here).
|
|
#-----------------------------------------------------------------------
|
|
|
|
my %fixed = (
|
|
void => sub { return undef; }, # handled specially
|
|
reg_al => sub { 'al' }, reg_ax => sub { 'ax' },
|
|
reg_eax => sub { 'eax' }, reg_rax => sub { 'rax' },
|
|
reg_cl => sub { 'cl' }, reg_dx => sub { 'dx' },
|
|
reg_bx => sub { 'bx' }, reg_cx => sub { 'cx' },
|
|
reg_ecx => sub { 'ecx' }, reg_edx => sub { 'edx' },
|
|
reg_cs => sub { 'cs' }, reg_ds => sub { 'ds' },
|
|
reg_es => sub { 'es' }, reg_fs => sub { 'fs' },
|
|
reg_gs => sub { 'gs' }, reg_ss => sub { 'ss' },
|
|
xmm0 => sub { 'xmm0' }, fpu0 => sub { 'st0' },
|
|
unity => sub { '1' },
|
|
);
|
|
|
|
#-----------------------------------------------------------------------
|
|
# Register-number "tier" pools.
|
|
#
|
|
# Registers 8+ are only ever valid in 64-bit mode (they require a REX
|
|
# prefix -- or, for r16-r31/vector-reg 16-31, REX2/EVEX register-
|
|
# extension bits -- neither of which exist outside 64-bit mode). Three
|
|
# tiers per extendable register class:
|
|
# low - numbers 0-7: valid at any --bits width; used for the
|
|
# normal (non-register-number-focused) instruction lines.
|
|
# hireg - numbers 8-15: needs a REX (GPR) prefix; 64-bit mode only.
|
|
# apxreg - numbers 16-31: needs REX2/EVEX register-extension bits
|
|
# (GPR) or an EVEX V'/X4 bit (vector regs); 64-bit mode only.
|
|
# NB the "low" GPR pools intentionally avoid rsp (and, for reg32na,
|
|
# rbp too) as before; the hireg/apxreg tiers have no such special-
|
|
# purpose register to dodge.
|
|
#-----------------------------------------------------------------------
|
|
my @reg8_lo = qw(al bl cl dl);
|
|
my @reg16_lo = qw(ax bx cx dx si di bp);
|
|
my @reg32_lo = qw(eax ebx ecx edx esi edi ebp);
|
|
my @reg32na_lo = qw(eax ebx ecx edx esi edi); # "no esp" pool
|
|
my @reg64_lo = qw(rax rbx rcx rdx rsi rdi rbp);
|
|
|
|
my @reg8_hi = map { "r${_}b" } (8 .. 15);
|
|
my @reg16_hi = map { "r${_}w" } (8 .. 15);
|
|
my @reg32_hi = map { "r${_}d" } (8 .. 15);
|
|
my @reg64_hi = map { "r$_" } (8 .. 15);
|
|
|
|
my @reg8_apx = map { "r${_}b" } (16 .. 31);
|
|
my @reg16_apx = map { "r${_}w" } (16 .. 31);
|
|
my @reg32_apx = map { "r${_}d" } (16 .. 31);
|
|
my @reg64_apx = map { "r$_" } (16 .. 31);
|
|
|
|
my %gpr_pool = (
|
|
8 => { low => \@reg8_lo, hireg => \@reg8_hi, apxreg => \@reg8_apx },
|
|
16 => { low => \@reg16_lo, hireg => \@reg16_hi, apxreg => \@reg16_apx },
|
|
32 => { low => \@reg32_lo, hireg => \@reg32_hi, apxreg => \@reg32_apx },
|
|
64 => { low => \@reg64_lo, hireg => \@reg64_hi, apxreg => \@reg64_apx },
|
|
'32na' => { low => \@reg32na_lo, hireg => \@reg32_hi, apxreg => \@reg32_apx },
|
|
);
|
|
sub gpr_pool { my ($bits, $variant) = @_; return $gpr_pool{$bits}{$variant // 'low'}; }
|
|
|
|
my @xmm_lo = map { "xmm$_" } (0 .. 7);
|
|
my @xmm_hi = map { "xmm$_" } (8 .. 15);
|
|
my @xmm_apx = map { "xmm$_" } (16 .. 31);
|
|
my @ymm_lo = map { "ymm$_" } (0 .. 7);
|
|
my @ymm_hi = map { "ymm$_" } (8 .. 15);
|
|
my @ymm_apx = map { "ymm$_" } (16 .. 31);
|
|
my @zmm_lo = map { "zmm$_" } (0 .. 7);
|
|
my @zmm_hi = map { "zmm$_" } (8 .. 15);
|
|
my @zmm_apx = map { "zmm$_" } (16 .. 31);
|
|
|
|
my %vec_pool = (
|
|
xmm => { low => \@xmm_lo, hireg => \@xmm_hi, apxreg => \@xmm_apx },
|
|
ymm => { low => \@ymm_lo, hireg => \@ymm_hi, apxreg => \@ymm_apx },
|
|
zmm => { low => \@zmm_lo, hireg => \@zmm_hi, apxreg => \@zmm_apx },
|
|
);
|
|
sub vec_pool { my ($kind, $variant) = @_; return $vec_pool{$kind}{$variant // 'low'}; }
|
|
|
|
my @sreg = qw(cs ds es fs gs ss);
|
|
my @creg = qw(cr0 cr2 cr3 cr4);
|
|
my @dreg = qw(dr0 dr1 dr2 dr3);
|
|
my @treg = qw(tr3 tr4 tr5 tr6 tr7);
|
|
my @fpureg = map { "st$_" } (0 .. 7);
|
|
my @mmxreg = map { "mm$_" } (0 .. 7); # only 8 MMX regs exist -- no hi/apx tier
|
|
my @bndreg = map { "bnd$_" } (0 .. 3); # only 4 BND regs exist -- no hi/apx tier
|
|
my @tmmreg = map { "tmm$_" } (0 .. 7); # only 8 TMM regs exist -- no hi/apx tier
|
|
my @kreg = map { "k$_" } (1 .. 7); # avoid k0 (== "no mask"); only 8 k regs exist
|
|
my %memsize = (8=>'byte', 16=>'word', 32=>'dword', 64=>'qword',
|
|
128=>'oword', 256=>'yword', 512=>'zword');
|
|
|
|
sub pick { my ($rng, @pool) = @_; return $pool[int($rng->() * scalar(@pool))]; }
|
|
|
|
sub mem_operand {
|
|
my ($rng, $sizebits) = @_;
|
|
# Deliberately base-register-free (pure displacement) addressing:
|
|
# the same generated text is reused across --bits 16/32/64 probes,
|
|
# and a GPR base name (e.g. "rax") is only legal in one of those
|
|
# modes. A bare displacement is valid addressing syntax in all
|
|
# three, at the cost of not exercising base+index forms here.
|
|
my $disp = 0x100 + int($rng->() * 0xf00);
|
|
my $txt = sprintf('[0x%x]', $disp);
|
|
$txt = "$memsize{$sizebits} $txt" if $sizebits && $memsize{$sizebits};
|
|
return $txt;
|
|
}
|
|
|
|
sub vsib_operand {
|
|
my ($rng, $vreg_pool) = @_;
|
|
# Base-free scaled-index (vsib) form, for the same bit-mode-
|
|
# portability reason as mem_operand() above.
|
|
my $vreg = pick($rng, @$vreg_pool);
|
|
return "[$vreg*1]";
|
|
}
|
|
|
|
# Wide/simd register-or-memory generator: mostly prefers the register
|
|
# form (always syntactically safe), occasionally emits a sized memory
|
|
# operand. $force_reg (used for the hireg/apxreg register-number-
|
|
# focused lines, where the whole point is to exercise that specific
|
|
# register, not a coin-flip chance of falling back to memory) always
|
|
# picks the register form.
|
|
sub regmem {
|
|
my ($rng, $regpool, $sizebits, $force_reg) = @_;
|
|
if ($force_reg || $rng->() < 0.7) {
|
|
return pick($rng, @$regpool);
|
|
} else {
|
|
return mem_operand($rng, $sizebits);
|
|
}
|
|
}
|
|
|
|
sub imm_operand {
|
|
my ($rng, $bits, %opt) = @_;
|
|
my $max = (1 << ($bits > 30 ? 30 : $bits)) - 1; # keep literals modest
|
|
my $v = int($rng->() * ($max > 0 ? $max : 1)) + 1;
|
|
return $opt{signed} ? sprintf('%d', $v - int($max/2)) : sprintf('0x%x', $v);
|
|
}
|
|
|
|
# base-token (decorators after '|' or trailing '*' stripped) -> coderef
|
|
# coderef->($rng, \%decorators) -> operand text, or undef if unsupported
|
|
my %gen = (
|
|
reg8 => sub { my ($r,$v) = @_; pick($r, @{gpr_pool(8,$v)}) },
|
|
reg16 => sub { my ($r,$v) = @_; pick($r, @{gpr_pool(16,$v)}) },
|
|
reg32 => sub { my ($r,$v) = @_; pick($r, @{gpr_pool(32,$v)}) },
|
|
reg32na => sub { my ($r,$v) = @_; pick($r, @{gpr_pool('32na',$v)}) },
|
|
reg64 => sub { my ($r,$v) = @_; pick($r, @{gpr_pool(64,$v)}) },
|
|
'reg64:reg64' => sub { my ($r,$v) = @_; my $p=gpr_pool(64,$v); pick($r,@$p).":".pick($r,@$p) },
|
|
reg_sreg => sub { my ($r) = @_; pick($r, @sreg) },
|
|
reg_creg => sub { my ($r) = @_; pick($r, @creg) },
|
|
reg_dreg => sub { my ($r) = @_; pick($r, @dreg) },
|
|
reg_treg => sub { my ($r) = @_; pick($r, @treg) },
|
|
fpureg => sub { my ($r) = @_; pick($r, @fpureg) },
|
|
mmxreg => sub { my ($r) = @_; pick($r, @mmxreg) },
|
|
bndreg => sub { my ($r) = @_; pick($r, @bndreg) },
|
|
tmmreg => sub { my ($r) = @_; pick($r, @tmmreg) },
|
|
kreg => sub { my ($r) = @_; pick($r, @kreg) },
|
|
kreg8 => sub { my ($r) = @_; pick($r, @kreg) },
|
|
kreg16 => sub { my ($r) = @_; pick($r, @kreg) },
|
|
kreg32 => sub { my ($r) = @_; pick($r, @kreg) },
|
|
kreg64 => sub { my ($r) = @_; pick($r, @kreg) },
|
|
krm8 => sub { my ($r) = @_; pick($r, @kreg) },
|
|
krm16 => sub { my ($r) = @_; pick($r, @kreg) },
|
|
krm32 => sub { my ($r) = @_; pick($r, @kreg) },
|
|
krm64 => sub { my ($r) = @_; pick($r, @kreg) },
|
|
rm8 => sub { my ($r,$v) = @_; regmem($r, gpr_pool(8,$v), 8, $v && $v ne 'low') },
|
|
rm16 => sub { my ($r,$v) = @_; regmem($r, gpr_pool(16,$v), 16, $v && $v ne 'low') },
|
|
rm32 => sub { my ($r,$v) = @_; regmem($r, gpr_pool(32,$v), 32, $v && $v ne 'low') },
|
|
rm64 => sub { my ($r,$v) = @_; regmem($r, gpr_pool(64,$v), 64, $v && $v ne 'low') },
|
|
rm_sel => sub { my ($r,$v) = @_; regmem($r, gpr_pool(16,$v), 16, $v && $v ne 'low') },
|
|
mem => sub { my ($r) = @_; mem_operand($r, 0) },
|
|
mem8 => sub { my ($r) = @_; mem_operand($r, 8) },
|
|
mem16 => sub { my ($r) = @_; mem_operand($r, 16) },
|
|
mem32 => sub { my ($r) = @_; mem_operand($r, 32) },
|
|
mem64 => sub { my ($r) = @_; mem_operand($r, 64) },
|
|
mem80 => sub { my ($r) = @_; mem_operand($r, 0) },
|
|
mem128 => sub { my ($r) = @_; mem_operand($r, 128) },
|
|
mem256 => sub { my ($r) = @_; mem_operand($r, 256) },
|
|
mem512 => sub { my ($r) = @_; mem_operand($r, 512) },
|
|
mem_offs => sub { my ($r) = @_; sprintf('[0x%x]', int($r->()*0x1000)) },
|
|
xmmreg => sub { my ($r,$v) = @_; pick($r, @{vec_pool('xmm',$v)}) },
|
|
ymmreg => sub { my ($r,$v) = @_; pick($r, @{vec_pool('ymm',$v)}) },
|
|
zmmreg => sub { my ($r,$v) = @_; pick($r, @{vec_pool('zmm',$v)}) },
|
|
xmmrm => sub { my ($r,$v) = @_; regmem($r, vec_pool('xmm',$v), 128, $v && $v ne 'low') },
|
|
xmmrm8 => sub { my ($r,$v) = @_; regmem($r, vec_pool('xmm',$v), 8, $v && $v ne 'low') },
|
|
xmmrm16 => sub { my ($r,$v) = @_; regmem($r, vec_pool('xmm',$v), 16, $v && $v ne 'low') },
|
|
xmmrm32 => sub { my ($r,$v) = @_; regmem($r, vec_pool('xmm',$v), 32, $v && $v ne 'low') },
|
|
xmmrm64 => sub { my ($r,$v) = @_; regmem($r, vec_pool('xmm',$v), 64, $v && $v ne 'low') },
|
|
xmmrm128 => sub { my ($r,$v) = @_; regmem($r, vec_pool('xmm',$v), 128, $v && $v ne 'low') },
|
|
ymmrm256 => sub { my ($r,$v) = @_; regmem($r, vec_pool('ymm',$v), 256, $v && $v ne 'low') },
|
|
zmmrm512 => sub { my ($r,$v) = @_; regmem($r, vec_pool('zmm',$v), 512, $v && $v ne 'low') },
|
|
mmxrm => sub { my ($r) = @_; regmem($r, \@mmxreg, 64) },
|
|
mmxrm64 => sub { my ($r) = @_; regmem($r, \@mmxreg, 64) },
|
|
xmem32 => sub { my ($r,$v) = @_; vsib_operand($r, vec_pool('xmm',$v)) },
|
|
xmem64 => sub { my ($r,$v) = @_; vsib_operand($r, vec_pool('xmm',$v)) },
|
|
ymem32 => sub { my ($r,$v) = @_; vsib_operand($r, vec_pool('ymm',$v)) },
|
|
ymem64 => sub { my ($r,$v) = @_; vsib_operand($r, vec_pool('ymm',$v)) },
|
|
zmem32 => sub { my ($r,$v) = @_; vsib_operand($r, vec_pool('zmm',$v)) },
|
|
zmem64 => sub { my ($r,$v) = @_; vsib_operand($r, vec_pool('zmm',$v)) },
|
|
imm => sub { my ($r) = @_; imm_operand($r, 7) }, # unqualified
|
|
# "imm" width is
|
|
# only apparent
|
|
# from the
|
|
# bytecode field
|
|
# (not parsed);
|
|
# keep it small
|
|
# enough to fit
|
|
# any encoding.
|
|
imm8 => sub { my ($r) = @_; imm_operand($r, 8) },
|
|
imm16 => sub { my ($r) = @_; imm_operand($r, 16) },
|
|
imm32 => sub { my ($r) = @_; imm_operand($r, 32) },
|
|
imm64 => sub { my ($r) = @_; imm_operand($r, 32) }, # keep in +ve range
|
|
imm_known8 => sub { my ($r) = @_; '1' },
|
|
spec4 => sub { my ($r) = @_; int($r->() * 16) },
|
|
udword64 => sub { my ($r) = @_; imm_operand($r, 32) },
|
|
sdword64 => sub { my ($r) = @_; imm_operand($r, 31, signed=>1) },
|
|
sbytedword32 => sub { my ($r) = @_; imm_operand($r, 8, signed=>1) },
|
|
sbytedword64 => sub { my ($r) = @_; imm_operand($r, 8, signed=>1) },
|
|
sbyteword16 => sub { my ($r) = @_; imm_operand($r, 8, signed=>1) },
|
|
'imm16:imm16' => sub { my ($r) = @_; imm_operand($r,16).":".imm_operand($r,16) },
|
|
'imm16:imm32' => sub { my ($r) = @_; imm_operand($r,16).":".imm_operand($r,32) },
|
|
'imm32:imm32' => sub { my ($r) = @_; imm_operand($r,32).":".imm_operand($r,32) },
|
|
);
|
|
|
|
# Base tokens whose generated syntax can only assemble in 64-bit mode
|
|
# (64-bit GPRs / RIP-relative-only forms are simply not encodable in
|
|
# 16/32-bit mode). Used to split the generated instruction lines into
|
|
# a "narrow" (16/32-safe) and "full" (64-bit) body so multi-bit-mode
|
|
# instructions still get 16/32-bit coverage instead of the whole file
|
|
# being disqualified by one 64-bit-only line.
|
|
my %needs64_token = map { $_ => 1 } qw(
|
|
reg64 reg64:reg64 reg_rax imm64 udword64 sdword64 sbytedword64
|
|
);
|
|
|
|
# Mnemonics whose sole/last "imm"-like operand is actually a branch
|
|
# target: substitute a local label instead of a literal so relative
|
|
# encodings have something concrete (and in-range) to point at.
|
|
my %branch_mnemonic = map { $_ => 1 } qw(
|
|
JMP JMPE CALL LOOP LOOPE LOOPZ LOOPNE LOOPNZ JECXZ JCXZ JRCXZ
|
|
JO JNO JB JC JNAE JNB JNC JAE JE JZ JNE JNZ JBE JNA JA JNBE
|
|
JS JNS JP JPE JNP JPO JL JNGE JGE JNL JLE JNG JG JNLE
|
|
);
|
|
|
|
#-----------------------------------------------------------------------
|
|
# 3. Emit one concrete instruction line for a template.
|
|
#-----------------------------------------------------------------------
|
|
|
|
my %unsupported; # token => count, for the coverage report
|
|
|
|
sub base_token {
|
|
my ($tok) = @_;
|
|
# Trailing '*' marks an optional source operand (duplicates the
|
|
# previous operand in the encoding when omitted from the concrete
|
|
# instruction); trailing '?' marks an optional destination operand
|
|
# (omitted entirely, e.g. APX NDD forms). Both strip to the plain
|
|
# base operand-type token for generation purposes -- see
|
|
# optional_operand_index() below for where the omitted-form
|
|
# coverage itself is generated.
|
|
$tok =~ s/[\*\?]$//;
|
|
$tok =~ s/\|.*$//; # |mask, |z, |bNN, |rs2 ... decorators
|
|
return $tok;
|
|
}
|
|
|
|
sub gen_operand {
|
|
my ($rng, $tok, $mnem, $is_branch, $variant) = @_;
|
|
my $base = base_token($tok);
|
|
# Only genuine relative/near/short/abs branch-displacement operands
|
|
# (plain imm8/16/32/64, before the trailing "|near"/"|short"/"|abs"
|
|
# qualifier that base_token() already stripped) get replaced with a
|
|
# local label -- NOT every operand of a "branch mnemonic". Some
|
|
# branch mnemonics also have indirect (rm16/32/64), far-pointer
|
|
# (colon-compound "imm16:imm16"), or fixed-register (reg_cx, for
|
|
# LOOP's address-size-override form) operand forms, none of which
|
|
# are relative-displacement targets, and would otherwise get
|
|
# nonsensically replaced with the branch label too (producing
|
|
# invalid syntax like "loop .L1, .L1" instead of "loop .L1, cx").
|
|
if ($is_branch && $base =~ /^imm(?:8|16|32|64)$/) {
|
|
return '.L1';
|
|
}
|
|
if (exists $fixed{$base}) {
|
|
return $fixed{$base}->();
|
|
}
|
|
if (exists $gen{$base}) {
|
|
return $gen{$base}->($rng, $variant);
|
|
}
|
|
$unsupported{$base}++;
|
|
return undef;
|
|
}
|
|
|
|
# CALL/JMP near-indirect targets (the only branch-mnemonic templates
|
|
# with rm*-typed operands) restrict the operand-size override: rm64 is
|
|
# the only form valid in 64-bit mode -- rm16/rm32 forms genuinely only
|
|
# assemble in 16/32-bit mode (confirmed empirically: "call cx"/
|
|
# "call ecx" both fail under --bits 64, matching the NOLONG flag on
|
|
# those insns.xda templates). This is unrelated to %needs64_token
|
|
# (which flags the opposite direction -- 64-bit-*only* operands), so a
|
|
# separate helper marks lines built from these tokens as excluded from
|
|
# the 64-bit "full" body, mirroring how needs64 lines are excluded from
|
|
# the 16/32-bit "narrow" body.
|
|
sub branch_narrow_only {
|
|
my ($is_branch, $base) = @_;
|
|
return ($is_branch && $base =~ /^rm(?:16|32)$/) ? 1 : 0;
|
|
}
|
|
|
|
# Base tokens whose register pool has hireg (r8-r15 range) / apxreg
|
|
# (r16-r31 range) tiers available -- i.e. templates containing at
|
|
# least one of these are candidates for the extra register-number-
|
|
# focused ("hireg"/"apxreg") coverage lines built by build_variant_line()
|
|
# below.
|
|
my %extendable_base = map { $_ => 1 } qw(
|
|
reg8 reg16 reg32 reg32na reg64 reg64:reg64
|
|
rm8 rm16 rm32 rm64 rm_sel
|
|
xmmreg ymmreg zmmreg
|
|
xmmrm xmmrm8 xmmrm16 xmmrm32 xmmrm64 xmmrm128 ymmrm256 zmmrm512
|
|
xmem32 xmem64 ymem32 ymem64 zmem32 zmem64
|
|
);
|
|
|
|
sub has_extendable_token {
|
|
my ($ops) = @_;
|
|
for my $tok (@$ops) {
|
|
return 1 if $extendable_base{ base_token($tok) };
|
|
}
|
|
return 0;
|
|
}
|
|
|
|
# Build one concrete instruction line with every extendable register
|
|
# operand drawn from the given tier ('hireg' or 'apxreg'); returns
|
|
# undef if any operand token is unsupported. These lines are always
|
|
# 64-bit-mode-only (register numbers 8+ don't exist outside 64-bit
|
|
# mode), regardless of the base token's own size-based needs64 status.
|
|
sub build_variant_line {
|
|
my ($rng, $mnem, $ops, $is_branch, $variant) = @_;
|
|
my @operands;
|
|
for my $tok (@$ops) {
|
|
my $val = gen_operand($rng, $tok, $mnem, $is_branch, $variant);
|
|
return undef unless defined $val;
|
|
push @operands, $val;
|
|
}
|
|
my $text = lc($mnem) . (@operands ? ' ' . join(', ', @operands) : '');
|
|
return { text => $text, needs64 => 1 };
|
|
}
|
|
|
|
# insns.xda marks at most one operand per template with a trailing '*'
|
|
# (optional source operand -- x86/insns.pl's relaxed_forms() duplicates
|
|
# the *previous* operand in the encoding when this one is omitted from
|
|
# the source, e.g. "IMUL reg32,reg32" is short for "IMUL
|
|
# reg32,reg32,reg32") or '?' (optional destination operand -- entirely
|
|
# absent from the encoding when omitted, e.g. APX NDD forms like "INC
|
|
# reg32,rm32" alongside plain "INC rm32"). Either way, from a pure
|
|
# operand-*syntax* point of view (which is all this generator needs --
|
|
# see the design-rationale header comment / tools/testgen/README.md),
|
|
# both cases reduce to: the marked operand can be dropped from the
|
|
# concrete instruction's operand list entirely. Returns the operand
|
|
# index to drop for the reduced-arity variant, or undef if this
|
|
# template has no optional operand.
|
|
sub optional_operand_index {
|
|
my ($ops) = @_;
|
|
for my $i (0 .. $#$ops) {
|
|
return $i if $ops->[$i] =~ /[\*\?]$/;
|
|
}
|
|
return undef;
|
|
}
|
|
|
|
#-----------------------------------------------------------------------
|
|
# EVEX decorator ({k1}/{z}/{1toN}/{sae}/{rn-sae}...) coverage.
|
|
#
|
|
# insns.xda marks operand tokens with the decorators that operand may
|
|
# carry (see x86/insns.pl's decorator-stripping regex,
|
|
# `s/^(b(16|32|64)|mask|z|er|sae)$//`, on the same '|'-separated operand
|
|
# sub-fields as the rest of the token). Semantics (confirmed against
|
|
# `asm/parser.c`'s parse_decorators() and test/avx512*.asm):
|
|
# mask - "{k1}".."{k7}" written directly after the register (or,
|
|
# for masked-store forms, memory) operand text; k0 is
|
|
# skipped, same as the plain kreg pool (means "no mask").
|
|
# z - (always paired with mask in insns.xda) adds "{z}"
|
|
# immediately after the mask suffix -- zeroing- vs merging-
|
|
# masking are two independent EVEX bits worth exercising
|
|
# separately, hence separate "mask" and "maskz" lines below.
|
|
# b16/32/64 - broadcast decorator "{1toN}", only meaningful when the
|
|
# marked (register-or-memory) operand resolves to *memory*;
|
|
# N = (operand's vector width in bits) / (16, 32, or 64).
|
|
# sae/er - "{sae}" or one of "{rn-sae}"/"{rd-sae}"/"{ru-sae}"/"{rz-sae}"
|
|
# (embedded rounding always specifies one of the 4 modes;
|
|
# plain SAE does not), only meaningful when the marked
|
|
# operand resolves to a *register* -- written as a separate
|
|
# trailing pseudo-operand after all real operands, e.g.
|
|
# "vaddpd zmm30,zmm29,zmm28,{rn-sae}".
|
|
#-----------------------------------------------------------------------
|
|
|
|
sub token_decorators {
|
|
my ($tok) = @_;
|
|
my %d;
|
|
$d{$1} = 1 while $tok =~ /\|(mask|z|sae|er|b16|b32|b64)\b/g;
|
|
return \%d;
|
|
}
|
|
|
|
# Vector width in bits implied by a base token name -- used to compute
|
|
# the broadcast count N in "{1toN}".
|
|
sub vec_bits_of_base {
|
|
my ($base) = @_;
|
|
return 128 if $base =~ /^xmm/;
|
|
return 256 if $base =~ /^ymm/;
|
|
return 512 if $base =~ /^zmm/;
|
|
return $1 + 0 if $base =~ /^(?:mem|rm)(\d+)$/;
|
|
return undef;
|
|
}
|
|
|
|
# Force-memory text for a decorator line's broadcast-marked operand
|
|
# (regmem()-based tokens default to mostly-register; the whole point of
|
|
# this line is to exercise the memory+broadcast form specifically).
|
|
sub force_memory_text {
|
|
my ($rng, $base) = @_;
|
|
return mem_operand($rng, $1 + 0) if $base =~ /^(?:mem|rm)(\d+)$/;
|
|
return mem_operand($rng, 128) if $base =~ /^xmmrm/;
|
|
return mem_operand($rng, 256) if $base eq 'ymmrm256';
|
|
return mem_operand($rng, 512) if $base eq 'zmmrm512';
|
|
return undef;
|
|
}
|
|
|
|
# Force-register text for a decorator line's sae/er-marked operand
|
|
# (sae/er are only legal when that operand is a register, never
|
|
# memory); always drawn from the 'low' (0-7) tier, and sized according
|
|
# to the token's own declared width (matters for e.g. VCVTSI2SD's
|
|
# rm32-vs-rm64|er templates -- {rn-sae} is only legal with the rm64/
|
|
# 64-bit-register form).
|
|
sub force_register_text {
|
|
my ($rng, $base) = @_;
|
|
return pick($rng, @{ gpr_pool($1 + 0, 'low') }) if $base =~ /^rm(8|16|32|64)$/;
|
|
return pick($rng, @{ gpr_pool(16, 'low') }) if $base eq 'rm_sel';
|
|
return pick($rng, @xmm_lo) if $base =~ /^xmmrm/;
|
|
return pick($rng, @ymm_lo) if $base eq 'ymmrm256';
|
|
return pick($rng, @zmm_lo) if $base eq 'zmmrm512';
|
|
return undef;
|
|
}
|
|
|
|
my @saeer_er_suffixes = qw(rn-sae rd-sae ru-sae rz-sae);
|
|
|
|
# Build one decorated instruction line for $family ('mask', 'maskz',
|
|
# 'broadcast', or 'saeer'), or undef if this template has no operand
|
|
# carrying that decorator family (or generation otherwise fails).
|
|
sub build_decorated_line {
|
|
my ($rng, $mnem, $ops, $is_branch, $family) = @_;
|
|
my @operands;
|
|
my $found = 0;
|
|
my $trailing;
|
|
for my $tok (@$ops) {
|
|
my $base = base_token($tok);
|
|
my $deco = token_decorators($tok);
|
|
my $text;
|
|
if (($family eq 'mask' || $family eq 'maskz') && $deco->{mask}) {
|
|
$text = gen_operand($rng, $tok, $mnem, $is_branch);
|
|
return undef unless defined $text;
|
|
$text .= '{' . pick($rng, @kreg) . '}';
|
|
$text .= '{z}' if $family eq 'maskz';
|
|
$found = 1;
|
|
} elsif ($family eq 'broadcast' && ($deco->{b16} || $deco->{b32} || $deco->{b64})) {
|
|
my $elem = $deco->{b16} ? 16 : $deco->{b32} ? 32 : 64;
|
|
$text = force_memory_text($rng, $base);
|
|
return undef unless defined $text;
|
|
my $vecbits = vec_bits_of_base($base);
|
|
return undef unless $vecbits;
|
|
$text .= '{1to' . int($vecbits / $elem) . '}';
|
|
$found = 1;
|
|
} elsif ($family eq 'saeer' && ($deco->{sae} || $deco->{er})) {
|
|
$text = force_register_text($rng, $base);
|
|
return undef unless defined $text;
|
|
$trailing = $deco->{er} ? pick($rng, @saeer_er_suffixes) : 'sae';
|
|
$found = 1;
|
|
} else {
|
|
$text = gen_operand($rng, $tok, $mnem, $is_branch);
|
|
return undef unless defined $text;
|
|
}
|
|
push @operands, $text;
|
|
}
|
|
return undef unless $found;
|
|
push @operands, "{$trailing}" if defined $trailing;
|
|
return { text => lc($mnem) . ' ' . join(', ', @operands), needs64 => 0 };
|
|
}
|
|
|
|
#-----------------------------------------------------------------------
|
|
# Modrm-memory disp8/disp32 boundary coverage.
|
|
#
|
|
# mem_operand() deliberately emits bare-displacement addressing
|
|
# ([0xNNN], no base register) for bit-width portability (see its
|
|
# comment), which always encodes as a disp32 (or disp16 in 16-bit
|
|
# mode) -- disp8 forms (or, for EVEX, the disp8*N compressed-
|
|
# displacement encoding) are never exercised. Whether a given
|
|
# instruction even has a disp8-encodable form, and what the
|
|
# compressed-displacement scale factor N is, is instruction-specific
|
|
# (depends on the EVEX tuple type), so rather than trying to compute
|
|
# the exact boundary per instruction, every modrm-memory-operand
|
|
# template additionally gets a `[eax+1]` (unambiguously disp8-
|
|
# encodable everywhere) and a `[eax+64]` (large enough to land past
|
|
# the disp8 boundary for byte-granular encodings, while still being a
|
|
# clean multiple of the larger EVEX compressed-displacement scales) line.
|
|
# `[eax+N]` (rather than a bare displacement) is used because it's
|
|
# valid addressing syntax in all of --bits 16/32/64 (32-bit address
|
|
# size works everywhere via the 0x67 prefix), so -- unlike a fixed
|
|
# 64-bit base register -- it doesn't need any bit-width-specific
|
|
# handling and can simply be added to the base per-mnemonic @lines.
|
|
my %mem_sizebits = (
|
|
mem => 0, mem8 => 8, mem16 => 16, mem32 => 32, mem64 => 64,
|
|
mem80 => 0, mem128 => 128, mem256 => 256, mem512 => 512,
|
|
rm8 => 8, rm16 => 16, rm32 => 32, rm64 => 64, rm_sel => 16,
|
|
xmmrm => 128, xmmrm8 => 8, xmmrm16 => 16, xmmrm32 => 32, xmmrm64 => 64,
|
|
xmmrm128 => 128, ymmrm256 => 256, zmmrm512 => 512,
|
|
mmxrm => 64, mmxrm64 => 64,
|
|
);
|
|
|
|
# Build one disp-boundary instruction line: the first modrm-memory-
|
|
# capable operand in $ops becomes "[eax+$disp]" (sized per its own
|
|
# token), every other operand is generated normally. Returns undef if
|
|
# the template has no modrm-memory-capable operand, or if any other
|
|
# operand fails to generate.
|
|
sub build_dispboundary_line {
|
|
my ($rng, $mnem, $ops, $is_branch, $disp) = @_;
|
|
my @operands;
|
|
my $used_mem = 0;
|
|
my $needs64 = 0;
|
|
my $avoid64 = 0;
|
|
for my $tok (@$ops) {
|
|
my $base = base_token($tok);
|
|
if (!$used_mem && exists $mem_sizebits{$base}) {
|
|
my $sizebits = $mem_sizebits{$base};
|
|
my $txt = "[eax+$disp]";
|
|
$txt = "$memsize{$sizebits} $txt" if $sizebits && $memsize{$sizebits};
|
|
push @operands, $txt;
|
|
$used_mem = 1;
|
|
$avoid64 = 1 if branch_narrow_only($is_branch, $base);
|
|
next;
|
|
}
|
|
my $val = gen_operand($rng, $tok, $mnem, $is_branch);
|
|
return undef unless defined $val;
|
|
$needs64 = 1 if $needs64_token{$base};
|
|
$avoid64 = 1 if branch_narrow_only($is_branch, $base);
|
|
push @operands, $val;
|
|
}
|
|
return undef unless $used_mem;
|
|
return { text => lc($mnem) . ' ' . join(', ', @operands), needs64 => $needs64, avoid64 => $avoid64 };
|
|
}
|
|
|
|
# Implicitly-sized memory operand coverage.
|
|
#
|
|
# For memory-capable operand tokens that carry an explicit size in
|
|
# their own name (mem8/16/32/.../xmmrm128/ymmrm256/zmmrm512/mmxrm...,
|
|
# i.e. every key in %mem_sizebits with a nonzero size), the generator
|
|
# always renders a size keyword ("byte"/"dword"/"oword"/...) on the
|
|
# memory form. But NASM permits omitting the keyword entirely whenever
|
|
# the instruction template declares the operand's size is implied --
|
|
# either a fixed size (x86/iflags.ph's SB/SW/SD/SQ/ST/SO/SY/SZ flags,
|
|
# e.g. plain "mem" tokens like CLFLUSH/MOVNTI, already unsized by
|
|
# %mem_sizebits and so not touched here) or a size *match* to another,
|
|
# already-sized operand in the same template (SM0-4 flags, e.g.
|
|
# "ADD reg32,rm32" / "MOVBE reg32,mem32" -- extremely common for
|
|
# ALU/data-movement instructions pairing a register with a same-width
|
|
# memory operand). Rather than parsing/expanding the SM/AR flag ranges
|
|
# from insns.xda to decide exactly when this is legal, this candidate
|
|
# simply tries the unsized form and lets the existing staged probe
|
|
# below keep it only if it actually assembles -- consistent with the
|
|
# rest of this tool's "regression test, not correctness oracle"
|
|
# philosophy, and far simpler than re-deriving NASM's own operand-size
|
|
# disambiguation logic.
|
|
sub build_implicitsize_line {
|
|
my ($rng, $mnem, $ops, $is_branch) = @_;
|
|
my @operands;
|
|
my $used_mem = 0;
|
|
my $needs64 = 0;
|
|
my $avoid64 = 0;
|
|
for my $tok (@$ops) {
|
|
my $base = base_token($tok);
|
|
if (!$used_mem && ($mem_sizebits{$base} // 0)) {
|
|
push @operands, mem_operand($rng, 0);
|
|
$used_mem = 1;
|
|
$avoid64 = 1 if branch_narrow_only($is_branch, $base);
|
|
next;
|
|
}
|
|
my $val = gen_operand($rng, $tok, $mnem, $is_branch);
|
|
return undef unless defined $val;
|
|
$needs64 = 1 if $needs64_token{$base};
|
|
$avoid64 = 1 if branch_narrow_only($is_branch, $base);
|
|
push @operands, $val;
|
|
}
|
|
return undef unless $used_mem;
|
|
return { text => lc($mnem) . ' ' . join(', ', @operands), needs64 => $needs64, avoid64 => $avoid64 };
|
|
}
|
|
|
|
sub make_rng {
|
|
my ($seedval) = @_;
|
|
# Small deterministic xorshift-ish PRNG so runs are reproducible
|
|
# without depending on Perl's global srand/rand state.
|
|
my $state = $seedval || 1;
|
|
return sub {
|
|
$state ^= ($state << 13) & 0xFFFFFFFF;
|
|
$state ^= ($state >> 17);
|
|
$state ^= ($state << 5) & 0xFFFFFFFF;
|
|
return ($state & 0xFFFFFFFF) / 0xFFFFFFFF;
|
|
};
|
|
}
|
|
|
|
sub seed_for {
|
|
my ($str) = @_;
|
|
my $h = 2166136261;
|
|
for my $c (split //, $str) { $h = (($h ^ ord($c)) * 16777619) & 0xFFFFFFFF; }
|
|
return ($h % 0x7fffffff) + $seed;
|
|
}
|
|
|
|
sub render_body {
|
|
my ($lines_ref, $is_branch, $default_rel) = @_;
|
|
my $body = '';
|
|
# 64-bit mode: bare-displacement memory operands (as emitted by
|
|
# mem_operand()/vsib_operand() above) are ambiguous between
|
|
# absolute and RIP-relative addressing and trigger a deprecation
|
|
# warning under the default (ABS) mode; RIP-relative is the
|
|
# modern/expected default for 64-bit code anyway.
|
|
$body .= "default rel\n" if $default_rel;
|
|
$body .= ".L1:\n" if $is_branch;
|
|
$body .= join('', map { "\t$_->{text}\n" } @$lines_ref);
|
|
return $body;
|
|
}
|
|
|
|
#-----------------------------------------------------------------------
|
|
# 4. Driver: build .asm text per mnemonic, probe --bits 16/32/64,
|
|
# write out travis/<mnemonic>/{.asm,.json,.log}.
|
|
#-----------------------------------------------------------------------
|
|
|
|
my $generated = 0;
|
|
my $skipped_empty = 0;
|
|
|
|
for my $mnem (sort keys %by_mnemonic) {
|
|
my $rng = make_rng(seed_for($mnem));
|
|
my @templates = @{ $by_mnemonic{$mnem} };
|
|
|
|
# Sample up to $per_mnem distinct operand-arity/type combinations,
|
|
# deterministically but varied across the available templates.
|
|
my %seen;
|
|
my @dedup_all; # every distinct operand-arity/type combination, uncapped
|
|
for my $t (@templates) {
|
|
my $key = join(',', @{ $t->{ops} });
|
|
next if $seen{$key}++;
|
|
push @dedup_all, $t;
|
|
}
|
|
my @sample = @dedup_all;
|
|
@sample = @sample[0 .. $per_mnem-1] if scalar(@sample) > $per_mnem;
|
|
|
|
my $is_branch = $branch_mnemonic{$mnem} ? 1 : 0;
|
|
my @lines; # { text => ..., needs64 => 0|1 }
|
|
my @extra_hireg; # candidate hireg-tier lines, one per eligible template
|
|
my @extra_apxreg; # candidate apxreg-tier lines, one per eligible template
|
|
my @extra_mask; # candidate {k1}-masked lines
|
|
my @extra_maskz; # candidate {k1}{z}-masked lines
|
|
my @extra_broadcast; # candidate {1toN}-broadcast lines
|
|
my @extra_saeer; # candidate {sae}/{rn-sae}-decorated lines
|
|
my @extra_dispboundary; # candidate [eax+1]/[eax+64] boundary lines
|
|
my @extra_implicitsize; # candidate unsized-memory-operand lines
|
|
for my $t (@sample) {
|
|
my $opt_idx = optional_operand_index($t->{ops});
|
|
my $extendable = has_extendable_token($t->{ops});
|
|
for my $variant_num (1 .. $variants) {
|
|
my @operands;
|
|
my $ok = 1;
|
|
my $needs64 = 0;
|
|
my $avoid64 = 0;
|
|
for my $tok (@{ $t->{ops} }) {
|
|
my $val = gen_operand($rng, $tok, $mnem, $is_branch);
|
|
if (!defined $val) { $ok = 0; last; }
|
|
my $base = base_token($tok);
|
|
$needs64 = 1 if $needs64_token{$base};
|
|
$avoid64 = 1 if branch_narrow_only($is_branch, $base);
|
|
push @operands, $val;
|
|
}
|
|
next unless $ok;
|
|
push @lines, {
|
|
text => lc($mnem) . (@operands ? ' ' . join(', ', @operands) : ''),
|
|
needs64 => $needs64,
|
|
avoid64 => $avoid64,
|
|
};
|
|
|
|
# Optional-operand ('*'/'?') coverage: also emit the
|
|
# reduced-arity form once per template (using the first
|
|
# successfully-generated variant's operand values), so the
|
|
# omitted-operand parsing/encoding path gets exercised too,
|
|
# not just the full form.
|
|
if (defined $opt_idx && $variant_num == 1) {
|
|
my @reduced = @operands;
|
|
splice(@reduced, $opt_idx, 1);
|
|
push @lines, {
|
|
text => lc($mnem) . (@reduced ? ' ' . join(', ', @reduced) : ''),
|
|
needs64 => $needs64,
|
|
avoid64 => $avoid64,
|
|
};
|
|
}
|
|
|
|
|
|
# Register-number coverage: once per template (not once per
|
|
# --variants instance), also try building an all-hireg
|
|
# (r8-r15/xmm8-15/...) and an all-apxreg (r16-r31/xmm16-31)
|
|
# rendition of this same template. These are candidates
|
|
# only -- whether they actually get included in the
|
|
# generated file is decided by the staged-fallback probe
|
|
# below, since a handful of mnemonics won't support the
|
|
# extended encoding at all (e.g. NOAPX/NOLONG-flagged ones)
|
|
# and we don't want one bad line to cost the mnemonic its
|
|
# entire 64-bit body.
|
|
if ($extendable && $variant_num == 1) {
|
|
my $hl = build_variant_line($rng, $mnem, $t->{ops}, $is_branch, 'hireg');
|
|
push @extra_hireg, $hl if defined $hl;
|
|
my $al = build_variant_line($rng, $mnem, $t->{ops}, $is_branch, 'apxreg');
|
|
push @extra_apxreg, $al if defined $al;
|
|
}
|
|
|
|
}
|
|
}
|
|
|
|
# EVEX decorator coverage: once per *distinct* template across the
|
|
# mnemonic's *entire* template set (not just the capped @sample --
|
|
# decorator-bearing EVEX forms of a mnemonic are frequently appended
|
|
# well after the plain SSE/AVX forms in insns.xda, e.g. VMOVAPD's
|
|
# AVX512 mask-on-memory-destination templates come after its first
|
|
# four (non-decorated) AVX templates, so relying on @sample alone
|
|
# would silently never exercise them), try each of the four
|
|
# independent decorator families this template's operand tokens
|
|
# advertise support for (see build_decorated_line() above).
|
|
# Candidates only -- whether they actually get included in the
|
|
# generated file is decided by the staged probe below, for the same
|
|
# reason as hireg/apxreg above.
|
|
for my $t (@dedup_all) {
|
|
for my $fam_pair (['mask', \@extra_mask], ['maskz', \@extra_maskz],
|
|
['broadcast', \@extra_broadcast], ['saeer', \@extra_saeer]) {
|
|
my ($fam, $bucket) = @$fam_pair;
|
|
my $dl = build_decorated_line($rng, $mnem, $t->{ops}, $is_branch, $fam);
|
|
push @$bucket, $dl if defined $dl;
|
|
}
|
|
}
|
|
|
|
# Modrm-memory disp8/disp32 boundary coverage (see
|
|
# build_dispboundary_line() above): once per *distinct* template
|
|
# across the mnemonic's entire template set, for the same "buried
|
|
# variant" reason as the EVEX decorator loop above. Candidates
|
|
# only, routed through the same staged probe as hireg/apxreg/
|
|
# decorators below -- even though `[eax+N]` addressing syntax is
|
|
# valid at every --bits width in the general case, a handful of
|
|
# instructions may have memory-operand restrictions (alignment/
|
|
# tuple-type/etc) this generator doesn't model, so one bad
|
|
# candidate shouldn't cost the mnemonic its pre-existing coverage.
|
|
for my $t (@dedup_all) {
|
|
for my $disp (1, 64) {
|
|
my $dl = build_dispboundary_line($rng, $mnem, $t->{ops}, $is_branch, $disp);
|
|
# avoid64 candidates (CALL/JMP rm16/rm32 near-indirect
|
|
# targets -- see branch_narrow_only()) are dropped here
|
|
# rather than fed into the bits=64 probe: they're
|
|
# guaranteed to fail at 64-bit, and folding them into this
|
|
# bucket would either cost the whole bucket its otherwise-
|
|
# valid candidates (probe rejects the merge) or need a
|
|
# second parallel probe path for no real benefit, since
|
|
# this supplementary boundary coverage isn't the base
|
|
# per-mnemonic content the 64-bit error-case block (below)
|
|
# is built from.
|
|
push @extra_dispboundary, $dl if defined $dl && !$dl->{avoid64};
|
|
}
|
|
}
|
|
|
|
# Implicitly-sized memory operand coverage (see
|
|
# build_implicitsize_line() above): once per *distinct* template
|
|
# across the mnemonic's entire template set, same reasoning as the
|
|
# two loops above. Candidates only, routed through the same staged
|
|
# probe -- omitting the size keyword is only legal when the
|
|
# instruction's flags declare a fixed or size-matched operand size,
|
|
# which this generator doesn't parse directly (see comment above).
|
|
for my $t (@dedup_all) {
|
|
my $dl = build_implicitsize_line($rng, $mnem, $t->{ops}, $is_branch);
|
|
push @extra_implicitsize, $dl if defined $dl && !$dl->{avoid64};
|
|
}
|
|
|
|
|
|
next unless @lines; # nothing we knew how to generate
|
|
|
|
my @narrow_lines = grep { !$_->{needs64} } @lines;
|
|
# Lines that only assemble in 16/32-bit mode (currently just
|
|
# CALL/JMP rm16/rm32 near-indirect targets -- branch_narrow_only())
|
|
# must symmetrically be excluded from the 64-bit "full" body: they
|
|
# aren't flagged needs64 (they don't require 64-bit -- the opposite
|
|
# holds, they're incompatible with it), so they'd otherwise poison
|
|
# bits=64 assembly for the *entire* file the same way a needs64
|
|
# line would poison bits=16/32.
|
|
my @lines64 = grep { !$_->{avoid64} } @lines;
|
|
|
|
my $dirname = "$out_dir/" . lc($mnem);
|
|
make_path($dirname);
|
|
# "./"-prefixed relative form for arguments that get embedded
|
|
# verbatim into nasm's diagnostics or that must match the exact
|
|
# path convention tools/travis/nasm-t.py's directory scan will use
|
|
# at real test-run time (see travis/README.md / local.md); left
|
|
# untouched for absolute $out_dir (used for scratch/dry-run testing
|
|
# outside the repo, where the "./" convention doesn't apply).
|
|
my $dirref = ($dirname =~ m{^/}) ? $dirname : "./$dirname";
|
|
|
|
# Warnings that are inherent to the intentionally-simple bare-
|
|
# displacement addressing style used by mem_operand()/vsib_operand()
|
|
# (see comments there) rather than being interesting instruction-
|
|
# specific results in their own right; suppressed consistently so
|
|
# they don't get captured into every stderr golden.
|
|
my $warn_opts = '-w-ea-absolute -w-implicit-abs-deprecated';
|
|
|
|
# Staged-probe: each "extra" candidate-line category (hireg,
|
|
# apxreg, mask, maskz, broadcast, saeer) is tried independently,
|
|
# cumulatively on top of whichever earlier categories already
|
|
# passed, and only kept if it actually assembles. This bounds the
|
|
# extra probing cost to a handful of throwaway nasm invocations per
|
|
# category (only paid for mnemonics/templates that have candidates
|
|
# for that category at all), while guaranteeing @lines alone --
|
|
# the pre-existing, already-validated generator behavior -- is
|
|
# always the floor if every category fails.
|
|
#
|
|
# Which bit widths need to be probed: always 64 (that's where the
|
|
# extra lines live), *and* 16/32 too whenever @narrow_lines is
|
|
# empty -- in that case the per-bits loop below reuses the same
|
|
# "full" file for every width (see the $asmfile ternary further
|
|
# down), so a candidate line that only 64-bit mode can encode would
|
|
# otherwise silently break 16/32-bit coverage for such mnemonics
|
|
# (all of whose *only* templates happen to need 64-bit-sized
|
|
# register operands, e.g. URDMSR/UWRMSR).
|
|
my @probe_bits = @narrow_lines ? (64) : (16, 32, 64);
|
|
my $probe_asm = "$dirname/.probe_" . lc($mnem) . '.asm';
|
|
my $probe_bin = "$probe_asm.bin";
|
|
my $probe_ok = sub {
|
|
my ($candidate) = @_;
|
|
open(my $pfh, '>', $probe_asm) or die "$probe_asm: $!\n";
|
|
print $pfh render_body($candidate, $is_branch, 1);
|
|
close($pfh);
|
|
my $ok = 1;
|
|
for my $pbits (@probe_bits) {
|
|
my $cmd = sprintf('%s --bits %d -f bin %s %s -o %s >/dev/null 2>&1',
|
|
$nasm_bin, $pbits, $warn_opts, $probe_asm, $probe_bin);
|
|
system($cmd);
|
|
if ($? != 0 || !-s $probe_bin) { $ok = 0; last; }
|
|
unlink($probe_bin);
|
|
}
|
|
unlink($probe_asm, $probe_bin);
|
|
return $ok;
|
|
};
|
|
|
|
my @full_lines = @lines64;
|
|
for my $cat (\@extra_hireg, \@extra_apxreg, \@extra_mask, \@extra_maskz,
|
|
\@extra_broadcast, \@extra_saeer, \@extra_dispboundary,
|
|
\@extra_implicitsize) {
|
|
next unless @$cat;
|
|
my $candidate = [@full_lines, @$cat];
|
|
@full_lines = @$candidate if $probe_ok->($candidate);
|
|
}
|
|
|
|
my $asmfile_full = lc($mnem) . '.asm';
|
|
open(my $fh, '>', "$dirname/$asmfile_full") or die "$dirname/$asmfile_full: $!\n";
|
|
print $fh render_body(\@full_lines, $is_branch, 1);
|
|
close($fh);
|
|
|
|
my $asmfile_narrow = $asmfile_full;
|
|
if (@narrow_lines && scalar(@narrow_lines) != scalar(@full_lines)) {
|
|
$asmfile_narrow = lc($mnem) . '_narrow.asm';
|
|
open(my $nfh, '>', "$dirname/$asmfile_narrow") or die $!;
|
|
print $nfh render_body(\@narrow_lines, $is_branch);
|
|
close($nfh);
|
|
}
|
|
|
|
# Error-case coverage: lines that only assemble in 64-bit mode
|
|
# (reg64/imm64 operands, hireg r8-r15, apxreg r16-r31, etc. -- see
|
|
# %needs64_token / build_variant_line) are, by construction, exactly
|
|
# the lines excluded from the 16/32-bit "narrow" body above. Rather
|
|
# than inventing a separate error-only source file, append them to
|
|
# the *same* narrow file under a %ifdef ERROR guard (per the user's
|
|
# preferred convention: one .asm file, assembled twice -- with and
|
|
# without -DERROR). Without -DERROR the block is invisible and
|
|
# narrow-mode coverage is unaffected; with -DERROR the block is
|
|
# included and attempting to assemble it at --bits 16/32 should
|
|
# fail with a mode/operand-mismatch diagnostic, which is exactly
|
|
# the negative-test coverage we want. Each bit width is probed
|
|
# independently below and only kept if it actually fails, so a line
|
|
# that unexpectedly *does* assemble at some width (this generator
|
|
# doesn't model every mode restriction) doesn't turn into a bogus
|
|
# "expected error" test.
|
|
my @error_lines = grep { $_->{needs64} } @full_lines;
|
|
my $has_narrow_file = ($asmfile_narrow ne $asmfile_full);
|
|
if ($has_narrow_file && @error_lines) {
|
|
open(my $efh, '>>', "$dirname/$asmfile_narrow") or die $!;
|
|
print $efh "\n%ifdef ERROR\n";
|
|
print $efh join('', map { "\t$_->{text}\n" } @error_lines);
|
|
print $efh "%endif\n";
|
|
close($efh);
|
|
}
|
|
|
|
# Symmetric case: lines that only assemble in 16/32-bit mode
|
|
# (CALL/JMP rm16/rm32 near-indirect targets -- branch_narrow_only())
|
|
# are excluded from @full_lines above; append them to the *full*
|
|
# (64-bit) body under the same %ifdef ERROR convention, so
|
|
# attempting to assemble them at --bits 64 with -DERROR is
|
|
# exercised as an expected-error case too.
|
|
my @error64_lines = grep { $_->{avoid64} } @lines;
|
|
if (@error64_lines) {
|
|
open(my $e6fh, '>>', "$dirname/$asmfile_full") or die $!;
|
|
print $e6fh "\n%ifdef ERROR\n";
|
|
print $e6fh join('', map { "\t$_->{text}\n" } @error64_lines);
|
|
print $e6fh "%endif\n";
|
|
close($e6fh);
|
|
}
|
|
|
|
my @json_entries;
|
|
for my $bits (16, 32, 64) {
|
|
my $asmfile = ($bits == 64 || !@narrow_lines) ? $asmfile_full : $asmfile_narrow;
|
|
my $binfile = lc($mnem) . ".bin$bits";
|
|
my $stderrfile = lc($mnem) . ".stderr$bits";
|
|
# Cheap throwaway probe (discards its own output) just to
|
|
# decide (a) whether this bit-width assembles at all, and
|
|
# (b) whether it prints anything on stderr (e.g. the
|
|
# obsolete-removed warning for OBSOLETE/NEVER instructions, or
|
|
# a number-overflow warning) that needs to be declared as a
|
|
# "stderr" target so `nasm-t.py update` captures it as an
|
|
# expected-warning golden instead of it going unclaimed.
|
|
my $relsrc = "$dirref/$asmfile";
|
|
my $cmd = sprintf('%s --bits %d -f bin %s %s -o %s.tmp 2>%s',
|
|
$nasm_bin, $bits, $warn_opts, $relsrc,
|
|
"$dirname/$binfile", "$dirname/$stderrfile");
|
|
system($cmd);
|
|
my $rc = $? >> 8;
|
|
my $ok = ($rc == 0 && -s "$dirname/$binfile.tmp");
|
|
my $has_stderr = -s "$dirname/$stderrfile" ? 1 : 0;
|
|
unlink("$dirname/$binfile.tmp");
|
|
unlink("$dirname/$stderrfile");
|
|
if ($ok) {
|
|
push @json_entries, {
|
|
id => lc($mnem) . $bits,
|
|
bits => $bits,
|
|
output => $binfile,
|
|
source => $asmfile,
|
|
opts => $warn_opts,
|
|
stderr => $has_stderr ? $stderrfile : undef,
|
|
};
|
|
}
|
|
}
|
|
|
|
# Error-case coverage: probe the %ifdef ERROR block(s) appended
|
|
# above with -DERROR defined, and only turn each into a json
|
|
# "expected error" entry for the (source file, bit width) pairs
|
|
# where it actually fails to assemble.
|
|
my $probe_error_bits = sub {
|
|
my ($src, @bitlist) = @_;
|
|
for my $bits (@bitlist) {
|
|
my $errbin = lc($mnem) . ".errprobe$bits.bin";
|
|
my $errstderr = lc($mnem) . ".stderr${bits}err";
|
|
my $relsrc = "$dirref/$src";
|
|
my $cmd = sprintf('%s --bits %d -DERROR -f bin %s %s -o %s.tmp 2>%s',
|
|
$nasm_bin, $bits, $warn_opts, $relsrc,
|
|
"$dirname/$errbin", "$dirname/$errstderr");
|
|
system($cmd);
|
|
my $rc = $? >> 8;
|
|
unlink("$dirname/$errbin.tmp");
|
|
my $fails = ($rc != 0);
|
|
unlink("$dirname/$errstderr");
|
|
if ($fails) {
|
|
push @json_entries, {
|
|
id => lc($mnem) . $bits . 'err',
|
|
bits => $bits,
|
|
source => $src,
|
|
opts => "-DERROR $warn_opts",
|
|
stderr => $errstderr,
|
|
is_error => 1,
|
|
};
|
|
}
|
|
}
|
|
};
|
|
$probe_error_bits->($asmfile_narrow, 16, 32) if $has_narrow_file && @error_lines;
|
|
$probe_error_bits->($asmfile_full, 64) if @error64_lines;
|
|
|
|
if (!grep { !$_->{is_error} } @json_entries) {
|
|
# Couldn't assemble in any mode -- drop the whole directory,
|
|
# don't leave a dangling/empty test case behind. (Error-case
|
|
# entries alone don't count: a directory whose only "coverage"
|
|
# is "this asm never assembles" isn't meaningful regression
|
|
# coverage and would only mask a real generation bug -- see
|
|
# the is_branch operand-substitution fix above, discovered via
|
|
# exactly this scenario.)
|
|
unlink("$dirname/$asmfile_full");
|
|
unlink("$dirname/$asmfile_narrow") if $asmfile_narrow ne $asmfile_full;
|
|
rmdir($dirname);
|
|
$skipped_empty++;
|
|
next;
|
|
}
|
|
|
|
open(my $jf, '>', "$dirname/" . lc($mnem) . '.json') or die $!;
|
|
print $jf "[\n";
|
|
my $first = 1;
|
|
for my $e (@json_entries) {
|
|
print $jf ",\n" unless $first;
|
|
$first = 0;
|
|
if ($e->{is_error}) {
|
|
printf $jf <<"JSON", $mnem, $e->{id}, $e->{source}, $e->{bits}, $e->{opts}, $dirname, $e->{stderr};
|
|
{
|
|
"description": "Pseudorandom error-case test for %s",
|
|
"id": "%s",
|
|
"format": "bin",
|
|
"source": "%s",
|
|
"option": "--bits %d %s -I./%s/",
|
|
"target": [
|
|
{ "stderr": "%s" }
|
|
],
|
|
"error": "expected"
|
|
}
|
|
JSON
|
|
next;
|
|
}
|
|
my $stderr_target = $e->{stderr}
|
|
? qq(,\n { "stderr": "$e->{stderr}" })
|
|
: '';
|
|
printf $jf <<"JSON", $mnem, $e->{id}, $e->{source}, $e->{bits}, $e->{opts}, $dirname, $e->{output}, $e->{output}, $stderr_target;
|
|
{
|
|
"description": "Pseudorandom test for %s",
|
|
"id": "%s",
|
|
"format": "bin",
|
|
"source": "%s",
|
|
"option": "--bits %d %s -I./%s/",
|
|
"target": [
|
|
{ "output": "%s", "match": "%s.t" }%s
|
|
]
|
|
}
|
|
JSON
|
|
}
|
|
print $jf "]\n";
|
|
close($jf);
|
|
|
|
# Let the existing, already-battle-tested harness do the actual
|
|
# golden capture (binary output *and* any declared stderr target),
|
|
# including its reproducible-build NASMENV knob -- instead of
|
|
# re-implementing that logic here.
|
|
my $rc = system('python3', 'tools/travis/nasm-t.py', "--nasm=$nasm_bin",
|
|
'update', '-t', "$dirref/" . lc($mnem));
|
|
if ($rc != 0) {
|
|
warn "warning: nasm-t.py update failed for $mnem, dropping\n";
|
|
unlink("$dirname/$asmfile_full");
|
|
unlink("$dirname/$asmfile_narrow") if $asmfile_narrow ne $asmfile_full;
|
|
unlink("$dirname/" . lc($mnem) . '.json');
|
|
rmdir($dirname);
|
|
$skipped_empty++;
|
|
next;
|
|
}
|
|
|
|
$generated++;
|
|
print "generated: $mnem (" . join(',', map { $_->{bits} } @json_entries) . ")\n" if $verbose;
|
|
}
|
|
|
|
print "\n== gen-insn-tests summary ==\n";
|
|
print "mnemonics with generated tests: $generated\n";
|
|
print "mnemonics with no assemblable output (dropped): $skipped_empty\n";
|
|
print "unsupported operand-type tokens (skipped templates), by frequency:\n";
|
|
for my $tok (sort { $unsupported{$b} <=> $unsupported{$a} } keys %unsupported) {
|
|
printf " %-20s %d\n", $tok, $unsupported{$tok};
|
|
}
|