The docs: target fed doc/Makefile.in directly to nmake, but that file
relies on several GNU make-only constructs nmake cannot parse at all:
the $^ automatic variable (not valid nmake macro syntax -- fatal
parse error), $< used outside of an inference rule (silently
unsupported, since nmake only defines $< within .SUFFIXES-style
rules), and the GNU-only "-include *.dep" optional wildcard include
directive (nmake only understands the unrelated "!include"
directive). Add tools/mkmsvcdocmak.pl, which rewrites just those
constructs to their explicit nmake-safe equivalents in a generated
copy (doc/Makefile.msvc), leaving doc/Makefile.in itself untouched for
the Unix/GNU make build. Verified by rewriting doc/Makefile.in this
way and building nasmdoc.pdf through it with GNU make standing in for
nmake.
Also, doc/Makefile.in expects doc/warnings.src, doc/perlbreq.src and
doc/pptok.src to have already been generated -- the Unix "doc" target
does this via the top-level Makefile.in before recursing into doc/,
but msvc.mak's docs: target had no equivalent step. warnings.src and
pptok.src already had msvc.mak rules (just not wired up as
prerequisites of docs:); perlbreq.src had no rule at all, since the
top-level Makefile.in generates it with a POSIX find/sed pipeline that
doesn't translate to Windows. Add tools/genperlbreq.pl, a portable
pure-Perl equivalent (using File::Find), and a doc\perlbreq.src rule
in msvc.mak that uses it; verified it produces output byte-identical
to the existing POSIX pipeline.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds an option to remap file prefixes in output object files. This is
analogous to the "-fdebug-prefix-map" option in GCC, and allows files to
be built in a reproducible manner regardless of the build directory.
[ hpa: this still needs to be documented in doc/running.src. ]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: H. Peter Anvin (Intel) <hpa@zytor.com>
Adds a "Error-case (%ifdef ERROR) coverage" section describing the
needs64/avoid64 dual-source design and the nasm-t.py json convention
reused from travis/ret/ret.json, notes on the two branch-operand bugs
discovered and fixed along the way, and updates the "Known
limitations" and "Validation" sections with the final 2612/10
mnemonic counts and 1976/2612 error-coverage numbers.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add error-case ("negative test") coverage per the user's preferred
convention: rather than a separate source file, error-triggering
instruction lines are appended to the existing per-mnemonic .asm file
under a %ifdef ERROR guard, and the harness assembles the same file
twice -- once without -DERROR (existing positive-path coverage,
unaffected) and once with -DERROR (expected to fail, per nasm-t.py's
"error": "expected" json convention, matching the pre-existing
travis/ret/ret.json pattern).
The error material comes for free from lines the generator already
knows are bit-width-incompatible:
- Lines needing 64-bit encodings (reg64/imm64 operands, hireg r8-r15,
apxreg r16-r31, etc. -- %needs64_token / build_variant_line) are, by
construction, exactly the lines already excluded from the 16/32-bit
"narrow" body. They're appended to the narrow file under %ifdef
ERROR and probed at --bits 16/32 with -DERROR; only widths where the
block actually fails become json entries.
- Symmetrically, CALL/JMP near-indirect targets via rm16/rm32 (only
rm64 is valid in 64-bit mode -- confirmed empirically, matches the
NOLONG flag on those insns.xda templates) are appended to the full
(64-bit) body under %ifdef ERROR and probed at --bits 64 with
-DERROR.
Each candidate block is probed before being turned into a json entry,
so a line that unexpectedly *does* assemble at some width (this
generator doesn't model every mode restriction) doesn't turn into a
bogus "expected error" test; a mnemonic whose *only* surviving
coverage would be error entries is also rejected (see below), since
"this never assembles" isn't meaningful regression coverage on its
own.
While wiring this up, discovered and fixed two related bugs in the
existing branch-mnemonic handling (gen_operand()'s is_branch
substitution):
1. is_branch replaced *every* operand of a branch mnemonic with the
".L1" local-label text, not just genuine relative/near/short/abs
branch-displacement operands. This produced nonsensical lines like
"loop .L1, .L1" (LOOP's address-size-override form takes a fixed
"cx"/"ecx"/"rcx" second operand, not a branch target) and "call
.L1" for JMP/CALL's indirect (rm16/32/64) and far-pointer
(imm16:imm16) forms instead of an actual register/memory operand.
These bogus lines silently poisoned assembly for the whole
mnemonic, and LOOP/LOOPE/LOOPNE/LOOPNZ/LOOPZ/JCXZ were silently
dropped entirely as a result (present in the "16 dropped" list).
Restricting the substitution to base tokens matching
/^imm(?:8|16|32|64)$/ fixes both LOOP's operand and JMP/CALL's
indirect/far forms, and recovers all six previously-dropped
mnemonics with correct coverage.
2. Once (1) exposed genuine rm16/rm32 operand generation for CALL/JMP,
a new bit-width interaction appeared: rm16/rm32 near-indirect
targets are only valid in 16/32-bit mode (unlike ordinary reg16/32
operands elsewhere, which work at any bit width), so a line built
from one now broke 64-bit assembly for the whole mnemonic the same
way a needs64 line breaks 16/32-bit assembly. Added
branch_narrow_only() and a parallel avoid64 line flag (mirroring
needs64) to exclude these lines from the 64-bit "full" body -- this
is also what feeds the new symmetric 64-bit error-case coverage
described above.
Also added a safety-net to the existing "couldn't assemble in any
mode, drop the directory" check: a directory is now only kept if it
has at least one *non*-error json entry, preventing a future bug
symmetric to (1) from silently producing a directory whose only
content is error-case entries.
Verified via full scratch regeneration (2612 mnemonics generated / 10
dropped, up from 2606/16 thanks to the LOOP-family/JCXZ fix) +
nasm-t.py run (10918/10918 PASS, 0 FAIL) + per-mnemonic non-error
.json entry-count diff against the prior committed tree (identical
except the 6 newly-recovered mnemonics, confirming no regressions).
1976/2612 mnemonics gained at least one error-case entry (3951 error
json entries total). Regenerated travis/insns/ and validated via
'make -j32 travis' (all PASS, ~27s).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
For instruction operand tokens that carry an explicit size in their own
name (mem8/16/32/64/..., rm8/16/32/64, xmmrm.../ymmrm256/zmmrm512,
mmxrm/mmxrm64), the generator previously always emitted an explicit size
keyword (e.g. "dword", "oword") for the memory operand. This never
exercised NASM's SM-flag-driven implicit-size-inference path, where the
size of an ambiguous memory operand is inferred from a paired
already-sized operand (typically a same-width register) in the same
instruction template (e.g. ADD reg32,rm32 or MOVBE reg32,mem32 don't
need an explicit size keyword).
Add build_implicitsize_line(), which replaces the first memory-capable
operand whose base token has a nonzero %mem_sizebits entry with a bare
(size-keyword-free) memory operand, leaving all other operands as
normally generated. As with the other coverage buckets (hireg, apxreg,
mask/maskz/broadcast/saeer, disp8/32 boundary), the candidate line is
routed through the shared cumulative staged probe and only kept if it
actually assembles for that instruction -- this avoids needing to parse
and replicate NASM's own SM/AR flag-driven operand-size-disambiguation
logic.
Note: for tokens with no size in their own name (plain "mem"), the
existing mem_operand($rng, 0) generator path already omits the size
keyword unconditionally, so that half of implicit-size coverage was
already exercised prior to this change; only explicitly-sized tokens
needed the new candidate.
Verified via full scratch regeneration (2606 mnemonics / 16 dropped,
unchanged) + nasm-t.py run (6955/6955 PASS, 0 FAIL) + per-mnemonic
.json entry-count diff against the prior committed tree (zero
differences, confirming no bit-width regressions). 1922/2606
mnemonics gained new implicit-size coverage lines. Regenerated
travis/insns/ and validated via 'make -j32 travis' (all PASS, ~26s).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extend gen-insn-tests.pl to synthesize, for every distinct template
across a mnemonic's entire template set that has a modrm-memory-
capable operand (mem*, rm*, xmmrm*/ymmrm256/zmmrm512, mmxrm*), two
additional candidate lines using [eax+1] and [eax+64] addressing in
place of that operand's usual bare-displacement or register form.
Whether an instruction has a disp8-encodable form (or, for EVEX, what
its compressed-displacement scale factor is) is instruction-specific,
so rather than computing the exact boundary per instruction these two
fixed offsets are used as a baseline: +1 is unambiguously disp8-
encodable everywhere, and +64 lands past the disp8 boundary for
byte-granular encodings while still being a clean multiple of the
larger EVEX compressed-displacement scales.
[eax+N] (rather than a bare displacement, or a bit-width-specific base
register) is used because it's valid addressing syntax at every
--bits width via the 0x67 address-size prefix, confirmed empirically.
Candidates are routed through the same staged, cumulative probe as the
hireg/apxreg/EVEX-decorator buckets, so one bad candidate (e.g. an
instruction with memory-operand restrictions this generator doesn't
model) can't cost a mnemonic its pre-existing coverage.
Regenerated travis/insns/ (2606 mnemonics, 16 dropped, unchanged from
before; 1987 mnemonics gained at least one disp-boundary line).
Validated: nasm-t.py run against a scratch regeneration is 6955/6955
PASS/0 FAIL with per-mnemonic bit-width success counts identical to
the previous committed baseline, and make -j32 travis passes in ~26s.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extend gen-insn-tests.pl to recognize the four independent EVEX
decorator families encoded in insns.xda operand tokens (|mask, |z,
|b16|b32|b64, |sae|er) and synthesize additional candidate lines
exercising each:
- mask: {k1}-{k7} appended to the marked register/memory operand
- maskz: mask + trailing {z} (zeroing), only where |z co-occurs
- broadcast: {1toN} on a memory operand, N derived from the token's
vector width (xmm/ymm/zmm or explicit mem/rm size) and
the b16/b32/b64 element-width marker
- sae/er: a separate trailing {sae} or {rn|rd|ru|rz-sae} pseudo-
operand, only legal when the marked operand resolves to
a register sized per the token's own declared width
(e.g. rm64|er needs a 64-bit register, not rm32)
Candidate lines are generated once per *distinct* template across each
mnemonic's entire template set, not just the capped per-mnemonic
sample -- EVEX/AVX512 forms are frequently appended well after a
mnemonic's plain SSE/AVX forms in insns.xda (e.g. VMOVAPD's mask-on-
memory-destination forms), so relying on the sample alone would
silently skip them for many mnemonics.
Generalized the item-2 hireg/apxreg staged-fallback probe into a
per-category, cumulative probing loop over all six extra-line buckets
(hireg, apxreg, mask, maskz, broadcast, saeer): each candidate bucket
is tentatively merged into the accepted line set and kept only if the
result still assembles, avoiding the combinatorial blowup of trying
every subset as the number of independent categories grows.
Regenerated travis/insns/ (2606 mnemonics, 16 dropped, unchanged from
before). Validated: nasm-t.py run against a scratch regeneration is
6955/6955 PASS/0 FAIL (identical per-mnemonic bit-width success counts
to the committed baseline), and make -j32 travis passes in ~26s.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add register-number-focused coverage for every operand-class token that
has a hireg (r8-r15 / xmm-zmm8-15) or apxreg (r16-r31 / xmm-zmm16-31)
tier: registers 8+ only exist in 64-bit mode (they need a REX prefix,
or REX2/EVEX register-extension bits for 16-31), so this coverage is
64-bit-only regardless of the base token's own size.
Per template with at least one such token, generate one extra all-hireg
and one extra all-apxreg instruction line (once per template, not once
per --variants instance). Since NASM's own template-matching engine
transparently selects an alternate (e.g. APX/EVEX-encoded) pattern when
the operand syntax calls for it, the generator doesn't need to special-
case APX/EVEX iflags to decide whether extended registers are legal for
a given mnemonic -- it can simply try assembling and keep whichever
combination works. A staged-fallback probe (try hireg+apxreg, then
hireg alone, then apxreg alone, then neither) avoids letting one
genuinely-incompatible extra line (e.g. NOAPX/NOLONG-only mnemonics)
cost the whole mnemonic its pre-existing 64-bit coverage; the probe
checks --bits 16/32 too whenever a mnemonic's *only* templates need
64-bit registers regardless of number (e.g. URDMSR/UWRMSR), since in
that case the generated 'full' file is reused for every bit width.
Also fixes a latent bug: xmmreg/ymmreg/zmmreg previously drew uniformly
from registers 0-15, but 8-15 needs 64-bit mode just like the GPR
case -- so a template could non-deterministically lose its 16/32-bit
coverage whenever the RNG happened to land on 8-15 (e.g. ADDPD was
missing bin16/bin32 goldens before this fix). Register pools are now
split into explicit low (0-7)/hireg (8-15)/apxreg (16-31) tiers per
register class, threaded through gen_operand() via a new
parameter (defaulting to the existing 'low' behavior at all pre-
existing call sites).
Regenerated travis/insns/ (2606 mnemonics, 16 dropped -- unchanged
counts). Full suite validated via make -j32 travis: all tests PASS in
~27s.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
x86/insns.xda marks operands with a trailing '*' (optional source,
duplicates the previous operand in the encoding when omitted --
insns.pl's relaxed_forms() implements the actual encoding semantics for
the C generators) or '?' (optional destination, entirely absent from
the encoding when omitted, e.g. APX NDD forms). Previously the
generator always emitted every operand in a template, so the
omitted-operand parsing/encoding path was never exercised, and
reg8?/reg16?/reg32?-marked templates fell back to the generic '?'-
unaware base_token() lookup and were silently dropped as unsupported
(only the reg64? case happened to have an explicit, redundant table
entry).
base_token() now strips a trailing '?' the same way it already stripped
'*', so all four reg#? variants resolve to their plain register
generator (the redundant explicit reg8?/16?/32?/64? %gen entries are
removed as now-dead code). A new optional_operand_index() locates the
(at most one, per insns.xda) marked operand in a template, and the
per-mnemonic driver emits one additional reduced-arity instruction line
per template (dropping that operand) alongside the existing full-arity
line, so both forms get captured as goldens.
Regenerated travis/insns/ (2606 mnemonics, 16 dropped -- unchanged counts,
since this only adds coverage lines to existing mnemonic asm files, not
new mnemonics). Full suite validated via make -j32 travis: all tests
PASS in ~26s.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extract the condition-code suffix table (%conds, @conds, and the
c_ccmask/c_nd/c_cc/c_scc bitmasks) out of x86/insns.pl's
conditional_forms() into a new shared x86/insns-cc.ph module, adding
cc_suffix_list($is_scc) to return the suffix list applicable to a
'...cc'-family (Jcc, SETcc, CMOVcc, CFCMOVcc) or '...scc'-family
(CCMPscc, CTESTscc, CMPccXADD, SETccZU) placeholder mnemonic. insns.pl
is refactored to use it (behavior-preserving: verified byte-identical
x86/insnsa.c, insnsb.c, insnsd.c, insnsi.h, insnsn.c, iflag.c,
iflaggen.h, asm/tokhash.c, asm/tokens.h before/after).
tools/testgen/gen-insn-tests.pl now requires the same module instead
of hand-rolling a 16-suffix 'cc'-only expansion table, closing the gap
where the APX 'scc'-suffix families (CCMPscc/CTESTscc/CMPccXADD/
SETccZU) were dropped as unsupported. The generator now detects and
expands any '...cc'/'...scc' placeholder mnemonic the same way
insns.pl's conditional_forms() does (case-sensitive /s?cc/ match and
substitution), so future new cc/scc families need no generator
changes. Mnemonics with generated tests: 2432 -> 2606 (+174, one per
newly-expanded scc-family condition); dropped: 20 -> 16.
Regenerated travis/insns/ (174 new mnemonic dirs for the APX scc
families; existing 2432 regenerated byte-identically, confirming
determinism). Full suite validated both via
tools/travis/nasm-t.py --directory=./travis run (4427 total, 4426
PASS, 1 pre-existing SKIP, 0 FAIL) and via make -j32 travis (all
tests PASS, ~28s).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add gen-insn-tests.pl, a prototype tool that generates one travis test
directory per non-pseudo instruction mnemonic (asm + json + golden
output), by parsing x86/insns.xda (the already-generated, macro-
expanded, one-template-per-line intermediate NASM produces from
insns.dat) rather than insnsa.c/insnsb.c or hooking insns.pl.
Rationale: insnsa.c/insnsb.c encode operands as opaque bitmask
constants and index into an explicitly-unstable, shared bytecode array
(x86/bytecode.txt: byte codes can be moved and recycled at any time),
making them fragile/high-effort to parse from outside insns.pl.
insns.xda already provides exactly the needed semantic info (mnemonic,
comma-separated operand-type tokens, flags) in a small stable text
grammar, with zero changes to insns.pl or the build. We only need
valid operand *syntax* per operand-type token -- NASM picks the
encoding itself from mnemonic + operand syntax -- so the generated
tests are regression tests against a golden captured from a known-good
nasm build, consistent with the rest of travis/.
Includes UNDOC/OBSOLETE/NEVER-flagged instructions (only PSEUDO
pseudo-ops are excluded): where these emit an expected warning (e.g.
-w+obsolete-removed), the generator declares a stderr target so the
warning text itself becomes part of the golden, verifying the warning
fires rather than skipping the coverage.
Delegates golden capture to tools/travis/nasm-t.py update instead of
re-implementing it, so goldens get the harness's existing correctness
properties (reproducible-build NASMENV, path-embedding conventions,
etc.) for free.
Generated travis/insns/<mnemonic>/ for all 2432 assemblable mnemonics
(16/32/64-bit variants as applicable). Full suite validated via
tools/travis/nasm-t.py: 4225 total, 4224 PASS, 1 pre-existing SKIP,
0 FAIL.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The travis tool itself runs all tests it is given in series. Use make
to parallelize running the tests, leaving a log in each subdirectory
in addition to the global log.
Signed-off-by: H. Peter Anvin (Intel) <hpa@zytor.com>
This is a mostly automated, partially AI-assisted migration of tests
from the test/ directory into the travis framework.
Running tests manually in the test/ directory is still supported, but
move common include files into travis/test and add a default -I option
to Makefile.in in the test/ directory.
The incbin test fails for pre-existing reasons; for now it contains an
stderr file with the errors. The problem is that INCBIN is both a
macro and a special instruction (not even a directive...), but there
currently is no way to handle prefixes, *especially* TIMES, in
multi-line macros. This is a separate problem and needs to be dealt
with as such.
Reorganize the travis directory so that each test or collection of
tests are in a separate subdirectory of travis/, and travis itself
lives in tools/travis to avoid creating deeper paths.
Add support for recording compression options in the travis .json
files, so that compressed files can be recreated with the same
options: because of the generally repetitive nature of the binary
output test files, the parameters used for xz compression can matter
enormously.
These are combined into a single huge commit to avoid adding large
binary files into the repository that then would immediately be
obsoleted, but still retained in git.
Signed-off-by: H. Peter Anvin (Intel) <hpa@zytor.com>
SPDX is an international standard for documenting software license
requirements. Remove the existing headers and replace with a brief
SPDX preamble.
See: https://spdx.dev/use/specifications/
The script used to convert the files is added to "tools", and the
file header templates in headers/ are updated.
Signed-off-by: H. Peter Anvin (Intel) <hpa@zytor.com>
A handful of debug messages in mkdep.pl were not guarded with
if ( $debug ), resulting in really annoying unnecessary verbosity.
Signed-off-by: H. Peter Anvin (Intel) <hpa@zytor.com>
The handling of "path" and "fullpath" was inconsistent, resulting in a
lot of missing dependencies regardless if a separate build directory
was in use.
Signed-off-by: H. Peter Anvin <hpa@zytor.com>
This target exists; use it, instead of effectively open-coding it into
the release script, causing bit rot and testing problems.
Signed-off-by: H. Peter Anvin <hpa@zytor.com>
Don't break Makefile.dep generation if run from a separate build
directory.
config/unconfig.h is a bit special; it is kept in the repository for
the benefit of non-configure users. Therefore, don't have "make
spotless" delete it.
Signed-off-by: H. Peter Anvin <hpa@zytor.com>
config/unconfig.h really is no different than the other perlreq files,
so move it into the Makefile. This has the extra advantage that
config/unconfig.h no longer needs to be kept in the source repository.
Signed-off-by: H. Peter Anvin (Intel) <hpa@zytor.com>
Add warnings.src to PERLREQ. If this isn't done, warnings.src ends up
in the xdoc tarball rather than the release tarball, which means that
unless the user can "make warnings" manually (which requires Perl)
then they can't build the documentation.
As this only affects the documentation build, it probably does not
warrant a point release.
Add a few more extensions to the list of recognized binary extensions
for the purpose of generating a .zip file with DOS line ending
conventions.
Signed-off-by: H. Peter Anvin (Intel) <hpa@zytor.com>
There is no reason to include both nasmdoc.pdf.xz and nasmdoc.pdf. For
the -xdoc file, the one with minimal dependencies is the proper one.
Signed-off-by: H. Peter Anvin (Intel) <hpa@zytor.com>
Clean up some perl warnings, some of which were legitimate (apparently
undef doesn't actually take a list of arguments, a common enough
mistake that it is mentioned in the man page!, and a list of variables
after "my" can be cantankerous), and some of which were nuisance but
were easy enough to clean up.
Maybe this can resolve the problems with very old version of Perl?
Signed-off-by: H. Peter Anvin (Intel) <hpa@zytor.com>
1. The mkdep.pl program didn't handle excluded dependencies correctly,
causing it to error out due to config/config.h not existing.
2. NMAKE is sensitive to the order suffixes appear in .SUFFIXES,
causing it to try to use the builtin rule .c.exe instead of
.c.obj -> .obj.exe.
3. NMAKE doesn't handle the && operator between commands.
4. The !ifdef jungle around dependency generation was wrong.
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
Make it possible to keep dependency information separate from the
Makefiles, so we don't have to deal with it noisifying the git logs.
Signed-off-by: H. Peter Anvin <hpa@linux.intel.com>
make really doesn't like something called MAKE in the environment, so
call it makej like buildall.sh on the server.
Signed-off-by: H. Peter Anvin <hpa@zytor.com>
