* librz/core,cmd: add pf-aware autocompletion arg types
The `pf` family of commands (`pf`, `pf-`, `pfa`, `pfc`, `pfd`, `pf.`,
`pfn`, `pfo`, `pfs`, `pfv`, `pfw`) takes either a registered named
format, a `<format>.<field>[<idx>]...` path, or a Format Definition
File. None of these were autocompletable: the arg type was always
`RZ_CMD_ARG_TYPE_STRING`, so tab on `pf. <TAB>` did nothing and the
user had to remember every format and field name by hand.
Add three new arg types and wire them to the cmd_descs:
* `RZ_CMD_ARG_TYPE_PF_FORMAT_NAME` enumerates the named formats
from `rz_type_db_format_all()` and filters by the partial prefix.
Used by `pf-`, `pfa`, `pfc`, `pfd`, `pfn`, `pfs`, `pfv`.
* `RZ_CMD_ARG_TYPE_PF_FORMAT_PATH` is the path-aware completer for
`pf.` and `pfw`. Both accept arbitrarily deep paths of the form
`name[.field[<idx>]?]*`, walking through nested struct fields
(e.g. `pf. troll.str[1].two` follows the same path syntax that
`pf_path_navigate` accepts at runtime). The completer:
- Finds the last `.` in the partial input; everything before
it (inclusive) is the committed path, what follows is the
segment being completed.
- Walks each committed `name[N]?` segment in turn, looking up
STRUCT fields' `type_name` in the typedb and re-parsing the
referenced format. Descent past a scalar or an inline struct
(no `type_name`) returns no options.
- Offers the resolved format's field names for the tail.
- Returns nothing when the tail contains `[` or `]` -- the
user is mid-index or mid-descent, where identifier
completion would produce a syntax error if accepted.
- Suppresses the trailing space after each successful
completion so `.` can be typed next without an inserted
space getting in the way of descent.
- Rewrites `res->start` past the last `.` so completion only
replaces the tail; the committed path stays put.
* `RZ_CMD_ARG_TYPE_PF_FDF_FILE` lists the basenames of FDFs found
in the user's home formats dir and the system formats dir,
mirroring the search order used by `pfo` itself; files present
in both locations are reported once via the same `HtSU` de-dup
used in `cmd_print_format_file_handler`. Used by `pfo`.
All three new types are added to `CD_ARG_LAST_TYPES` in
cmd_descs_util.py. The generator sets `RZ_CMD_ARG_FLAG_LAST` on the
final arg of a command whenever that arg's type is in this set;
that flag tells the runtime arg-preprocessor to merge any trailing
whitespace-separated tokens into a single argv slot. This is the
same implicit-FLAG_LAST treatment `RZ_CMD_ARG_TYPE_STRING` already
gets, and it is what keeps invocations like `pfc zd4x8 foo bar cow`
(five tokens, one logical format-with-names argument) working --
without it the cmd parser would reject the extra tokens with "Wrong
number of arguments". The implicit merge does not interfere with
autocompletion: the completer still receives the partial input up
to the cursor and prefix-matches against it, and the format-path
completer's dot/bracket scan is unaffected by whitespace.
The path completer uses a small helper, `pf_path_seg_consume`, that
parses one `name[N]?` segment with safe handling of unterminated `[`,
empty `[]`, non-numeric indices, and end-of-input. `pf_resolve_path_format`
walks all committed segments and returns the RzPfFormat the caller
should complete against; the resolver is the same shape as the
`pf_path_navigate` walker in librz/type/pf/pf_parser.c, except it
operates on RzPfFormat trees (typedb names) rather than RzPfValue
trees (decoded data), so it can run before any read has happened.
The completers live in `cautocmpl.c` next to the existing type-name
completers (`autocmplt_cmd_arg_struct_type` and friends) and follow
the same loop+strncmp pattern. The dispatcher entry for
`RZ_CMD_ARG_TYPE_FOLDER` was missing an explicit `break;` and would
fall through to `default`; harmless, but fixed in passing so the
three new cases sit cleanly above `default:`.
Note on dot-search direction: `rz_sub_str_rchr` is `start..end`
range search returning the FIRST hit, not a right-to-left "find
last" -- the `r` is for "range", not "right". The path completer
needs the last dot, so it walks the buffer backwards itself.
* test/integration: cover pf autocompletion
Thirteen new tests in test_autocmplt.c exercise the three new pf arg
types, including the multi-segment path resolver:
Format name completion:
* `pf_format_name` -- `pfn ut_<TAB>` after registering two formats
confirms both are offered.
Single-segment path completion:
* `pf_format_path` -- three-phase walk through `pf. ut_path<TAB>`,
`pf. ut_path.<TAB>`, `pf. ut_path.cou<TAB>`, covering name-only,
dot-only, and dot-with-prefix. Verifies that `res->end_string`
is empty in the name phase (so `.` can be typed next without an
inserted space) and that `res->start` advances past the dot in
the field phase (so the completion only replaces the field
portion).
* `pfw_format_path` -- the same `<format>.<field>` syntax must
work on the write side too; confirms the PATH completer fires
for `pfw` and is not pf.-specific.
* `pf_format_path_empty` -- bare `pf. <TAB>` lists every
registered format. Snapshots the baseline count first so the
assertion stays robust against any default formats the type DB
might seed.
* `pf_format_path_unknown_name` -- `pf. nonexistent.<TAB>`
returns an empty option list rather than crashing or leaking
diagnostics.
* `pf_format_path_anon_field` -- formats whose fields don't all
have names (e.g. a `.` skip slot) must be iterated safely; the
named fields are offered and the anonymous slot is silently
dropped.
Multi-segment / nested-struct path completion:
* `pf_format_path_nested` -- two-level descent through a STRUCT
field whose `type_name` references another registered format,
parsing the child format and offering its fields.
* `pf_format_path_three_levels` -- three-level descent narrows
correctly: A -> B -> C, then filter C's fields by a prefix.
* `pf_format_path_array_index` -- `pf. troll.str[1].<TAB>` mirrors
the existing cmd_pf2 runtime test; the array index in the
middle segment is parsed and skipped (it doesn't change the
target type).
* `pf_format_path_inside_brackets` -- cursor inside an
unclosed `[` returns no options (mid-index).
* `pf_format_path_after_close_bracket` -- cursor right after `]`
without a trailing `.` also returns no options (mid-descent).
* `pf_format_path_through_scalar` -- descent past a scalar field
is meaningless and returns no options.
The tests use plain `rz_core_new()` (the real cmd_descs already
registers all `pf*` commands), matching the pattern used by
`test_autocmplt_eco_themes`. Format strings use the parser's
"specifier-then-names" form (no internal whitespace in the spec
region) so `rz_pf_parse` produces the expected field count.
* doc,librz/core: align pf docs and `pf?` help with the parser
The standalone reference doc/pf.md and the in-tree `pf?` help (driven
by the details: block in librz/core/cmd_descs/cmd_print.yaml) had
drifted from each other and from what the parser actually accepts.
Both are now consistent with librz/type/pf/pf_parser.c.
Specific corrections:
* `n` family. doc/pf.md claimed `N1`/`N2`/`N4`/`N8` existed as BE
counterparts to `n1`-`n8`, and that bare `n`/`N` defaulted to
`ctx.bits/8`. The parser handles only `n{1,2,4,8}`; all four
forms are context-endian (follow `ctx->big_endian`), and bare
`n` produces "unknown specifier". Rewrite the section to match,
and explain why context-endian is the right choice for header
readers like ELF.
* Deprecation list. The `pf?` "deprecation" note listed `c, s, z`
among the deprecated bare-letter codes -- they are not deprecated
(`c` is the current 1-byte-as-char specifier, `s` is the current
pointer-to-zstring, `z` is the current inline zstring). It was
missing `C, i, Z, X, F, T` which the parser does warn on.
doc/pf.md had `c` in its table marked "unchanged" (so it was
visibly inconsistent with itself) and was missing the `x` row.
Both lists now mirror the parser's PF_DIAG(DEPRECATED) call
sites: b, C, d, f, F, i, o, q, t, T, w, x, X, Z.
* TLV `h=`. `pf?` said "h=v/a (header inclusion)" (two options);
the parser accepts `v` (value only, default), `l` (length covers
len+value), and `a` (length covers tag+len+value). doc/pf.md
already listed all three; help now matches.
* `v(N)` bitvector. doc/pf.md documented this in detail (the
1..4096-bit-wide field type used for things like ELF
`DT_FLAGS_1`, PE characteristics, page-allocation maps), but the
`pf?` help didn't mention it at all. Added to the DSL extensions
section.
* Pointer widths. doc/pf.md documented `p2`/`p4`/`p8` explicitly;
`pf?` only mentioned bare `p` with a "size from ctx.bits"
parenthetical. Help now lists the four forms in one entry.
* GUID layouts. Both docs claimed `G(le)` was "all little-endian",
but the renderer treats `G(le)` and `G(ms)` identically -- D4
(the trailing 8 bytes) is always in buffer order regardless of
layout. Document the actual behaviour rather than the implied
one; this is a description fix, not a code change. Anyone who
wants the byte-reversed-D4 reading can still file it as a
follow-up bug against the renderer in pf_render.c.
cmd_descs.[ch] is regenerated automatically by the custom_target rule
when cmd_print.yaml changes; the .c diff in this commit is the result
of that regeneration (5 lines of comment text inside the existing
detail entries).
The C grammar already exposes a struct/union member's bitfield width as a
"bitfield_clause" node and the parser already recognized "int a : 4;", but
it threw the width away (the member was stored with size 0 and a FIXME). As
a result "tc" rendered the member as a plain "int a;", "ts" produced a
full-int "pf" format ("pf d4d4d4 a b c") and "tp" mis-read every field as a
whole integer.
RzTypeStructMember and RzTypeUnionMember already carry a "size" field
documented as "in bits"; it is now used as the bitfield width, where 0 means
the member is not a bitfield:
- The struct and union member parsers store the parsed bit width on the
member instead of discarding it.
- The pretty printer emits " : <width>" before the member's trailing ';',
so "tc"/"tcd" round-trip "struct qwe { int a : 4; int b : 16; int c : 3; }".
- rz_base_type_as_format() (used by "ts"/"tp") now emits the pf packed-bits
spec ":N" for a bitfield member, with the bit order matching the target
endianness ("<" little-endian / ">" big-endian), instead of the member
type's full format. "ts qwe" becomes 'pf ":4<:16<:3< a b c"' and "tp"
unpacks the fields correctly (a=5, b=0x1234, c=0 for 0x00012345 on x86).
Serialization keeps the width in the 3rd comma-field of the existing
"struct.<name>.<member>" / "union.<name>.<member>" value, i.e.
"type,offset,bitsize". That field was already written (always as 0) and
ignored on load, so the on-disk layout is unchanged and only its meaning is
now honored. The project version is bumped to 24 with an additive no-op
migration: an absent or 0 bitsize deserializes as a regular, non-bitfield
member.
Tests: a unit test parses a bitfield struct and union and checks the member
widths, the rendered string and the "pf" format; a db/cmd/types test covers
the full "td"/"tcd"/"ts"/"tp" flow; a project-migration test with a new
v23-bitfield.rzdb fixture covers v23->v24, and the "migrate info" db test is
updated for the new version.
The PDB type parser (librz/arch/pdb_process.c) still drops bitfield members
because their LF_BITFIELD member type resolves to NULL; wiring LF_BITFIELD up
to the new member "size" field, with tests in test/unit/test_pdb.c, is left
as a follow-up.
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
The ELF section classifier in sections_obj() only marked a section as
data when its name contained "data". As a result .dynstr (type
SHT_STRTAB, SHF_ALLOC) was flagged neither as data nor as containing
strings, so is_data_section() rejected it. Two problems followed:
- the string search (iz, and the default AUTO scan used by -A) skipped
.dynstr, so its strings were never listed (only izz, which scans the
whole file regardless of section flags, showed them);
- because no RZ_META_TYPE_STRING metadata was applied over the region,
the disassembler rendered the NUL-terminated symbol names as code,
e.g. on MIPS "_GLOBAL_OFFSET_TABLE_" decoded to bgtzl/ldr/jalx/...
Mark a section as containing strings when it is mapped into memory
(SHF_ALLOC) and is either a string table (SHT_STRTAB) or carries the
explicit SHF_STRINGS flag. The SHF_ALLOC restriction keeps the loaded
string tables (.dynstr) while leaving the non-allocated .strtab and
.shstrtab to izz, matching the "iz lists the loaded image" semantics.
This is architecture independent; MIPS was simply where the bad
disassembly was first noticed.
Closes#5182
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
C23 lets an enum fix its underlying type, e.g. "enum E : long long { ... }".
The C grammar already exposes it as the "underlying_type" field of an
enum_specifier, and RzBaseType already has a "type" slot documented as
used by enums, but the parser ignored the field and always left it NULL.
- parse_enum_node() now reads the "underlying_type" field and stores the
parsed type on RzBaseType::type (reusing parse_type_node_single(), so
primitive, sized and typedef'd integer types are all handled). Classic
enums keep a NULL underlying type.
- The pretty printer emits " : <type>" between the enum name and its body
when an underlying type is present, so "tc"/"tcd"/"tec" round-trip it.
- enum_bitsize() now derives the width from the underlying type instead of
the hardcoded 32-bit default (resolving the long-standing FIXME); it
still falls back to 32 for a classic enum.
Single-token underlying types (int, uint64_t, char, ...) work end-to-end
with the bundled grammar revision: the existing "enhanced enum" db test is
updated to round-trip "enum v : int" and a unit test covers
"enum EU : uint64_t". Multi-word underlying types (long long, unsigned int)
are added as BROKEN db tests; the bundled grammar parses them to an ERROR
node and drops the underlying type, so these tests fail for now and will
pass once rizin-grammar-c accepts sized type specifiers as the enum
underlying type. No further rizin change is needed for that step.
Co-authored-by: Anton Kochkov <anton.kochkov@gmail.com>
The existing integration tests only reached the per-type graph
builders through the rz_core_graph() dispatcher. Add tests that call
rz_core_graph_callgraph/datarefs/coderefs/importxrefs directly and
assert they agree with the dispatcher, plus coverage for the
rz_core_graph_to_dot_str()/rz_core_graph_to_sdb_str() serializers.
Completes the work on rizinorg/rizin#992.
* Basic implement of list based and matrix based graph
Add comments
Basic support for list and matrix based graph refactor
Add rz_graph_*_new node and edge API
Modify calling for rz graph edge data
Add dfs and visitor mode
Add new get nth neighbours
Add unit test and wrapper for get edges
Bug fixed and unit test
Solve TODO about better semantic of get nodes
Doc RZ_API and other functions, split impl to new files
[cannot build] remove old graph impl and rename new graph API
* Rewrite and replace agrach, cgraph, drawable_graph, il_graph with new API
* Refactor graph API and fix unit test and multiple leaks
Update Wrapper for graph identifier
Ordered Nodes in graph drawable
Fix icfg and several agraph refactor issues
Use kahn to get topo sort in assign_layers
Introduce khan for assign_layer and find DAG cycle and backedges in algorithm
* Fix self loop and update db test
* Solve request changes
* Fix cbpf db
* Fix test graph warning in testing NULL deletion
* Fix mismatched rz_agraph_compute_layout nonnull
* Fix memleak
* Clean and fix code
* Fix memleak in graph free
* Fix memleak in matrix implement
* Add bindgen doc
* Fix memleak in early exit
* Fix rz-bindgen warning comments
* Make graph structures private.
* Fix Windows C2036
* Fix typo in type annotation.
* Add new type annotations for RzGraph and HtPP
---------
Co-authored-by: Heersin <teablearcher@gmail.com>
* Clarify docs of raw_alignment
* Set Go and Rust string encoding for binary string search.
* Add detailed doxygen about string search in bin plugins.
* Add the number string code points as detail to string search hits.
* Add helper to check if string encoding needs scanning.
* Add helper to generate a string wildcard pattern.
The pattern is supposed to match any human readable string.
Before the string search refactor there was no explicit definition
what characters mark the end of a string and which ones don't.
This pattern should match strings of the old interpretation.
* Enable the RzBin plugin to search strings with direct matching instead of scanning.
This significantly speeds up the string search when a binary is opened.
The prerequisite is that the plugin specifies the string encoding.
* Enforce UTF-8 for initial string search.
* Add custom string search for PE files.
* Clean up and document rz_utf8_encode
* Don't demote explicit UTF-8 encoding to ASCII.
* Increase min JIT stack size due to not reproducable search results.
* Simplify wildcard regex pattern.
* Remove unused declaration.
* Fix integration test to have a real scan example.
This also fixes a bug that the last NUL/invalid byte in a string
was not included in the UTF-8 bytes -> memory offset map.
* Add speed up search for unaligned buffers and string encodings.
The new search mode adjusts the buffer's data (swap endianness, fix alignment)
so the buffer can be searched with PCRE2 without scanning.
It also enables RzRegexMulti opbjects to have their own JIT stack.
This makes them usable in a thread safe manner.
* Fix regex compilation on big endian systems
* Update the scanning check (UTF needs no scanning anymore).
* Enable string search collection to search for multiple encodings.
This moves the encoding and alignment properties out of the collection
object into the RzDetectedStr object.
It essentially removes duplicates of this info.
Because of this move, rz_search_collection_string_add() will now
check for invalid collection configurations.
* Rename type -> encoding
* Add integration test for a simple string search.
* Clean up rz_str_is_false
* Doxygen
* Add rz_buf_type
* Sort search hits not just by address but also by size.
* Add a paranoid regex Hindi test.
* Add an integration test using RzCore and searching for UTF-16LE.
* Revert "Temporarily remove codecov workflow due to https://github.com/codecov/codecov-action/issues/1877 (#5366)"
This reverts commit dc5d50ce58.
* Add `rz_goto_if_reached()` to RZ_CHECKS_LEVEL 3
* Fix rz_return(_val)_if_reached() for RZ_CHECKS_LEVEL 0
* Add `rz_goto_if_reached()` to RZ_CHECKS_LEVEL 0
* Set default log level to WARN
This commit changes several settings. The main reason is to have them
contained in one search group, and not spread over the search and string group.
This becomes important with the search refactor, since the search is now also
more contained in a single module and can make use of the more settings.
- Remove str.search.max_uni_blocks - Effectively a metric the user should not know about; adds too much complexity. Also not documented.
- str.search.encoding -> str.encoding - Valid for all string interpretations.
- str.search.max_threads -> search.max_threads - This is a general setting for the search now.
- str.search.raw_alignment -> search.str.raw_alignment - Unify settings (only used for RzBin search.).
- str.search.min_length -> search.str.min_length - Unify settings.
- str.search.buffer_size -> search.str.max_length - Unify settings.
- str.search.max_region_size -> search.str.max_region_size - Unify settings.
- str.search.check_ascii_freq -> search.str.check_ascii_freq - Unify settings.
Part 3/9. Likely won't build in between parts.
Co-authored-by: wargio <deroad@kumo.xn--q9jyb4c>
* Added a CI-task to check if rizin could be used as subproject
* Fixed some new global-arguments in meson-build
Co-authored-by: amibranch <amibranch@users.noreply.github.com>
* Setting GNU99/C99 standard for subprojects
* Remove use of `add_global_arguments`
* Added extern "C" linkage specification to some public headers of rizin
* Changed the meson_git_wrapper.py-script to explicitly specify the output-path
* Made the RIZIN_BUILD_PATH conditional in the integration-test meson-build
Co-authored-by: amibranch <amibranch@users.noreply.github.com>
