* FunctionParser: Serialize Function._call_sites.
serialize stores the transition graph but not Function._call_sites, and
parse_from_cmsg never repopulated it, so functions loaded back from LMDB
(SpillingFunctionDict eviction, angrdb) returned nothing from
get_call_sites(), get_call_target(), and get_call_return().
_call_sites cannot be rebuilt from transition graph edges. Store the
mapping in a new repeated CallSite field on the Function message and
restore it in parse_from_cmsg.
* lint
* Fix pyright errors in FunctionParser.
* Fix pyright error in LMDB function manager test.
* Enable ruff isort rule
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* cfg_fast: iterate func.block_addrs for bad-func cleanup (works for spilled funcs)
CFGFast.drop_bad_functions's per-block cleanup iterated func.blocks,
which is empty for any bad function loaded via meta_only=True (e.g.
because it was spilled to LMDB by SpillingFunctionDict at drop
time). The loop was silently a no-op for those functions: the
function was removed from kb.functions but its CFG nodes survived
and its bytes stayed classified as 'code' in _seg_list, leaving the
model in a state that depended on LRU cache timing.
Fix: iterate func.block_addrs (populated for both cached and spilled
funcs) and look up block sizes from the CFG model via cfg_node.size.
No extra LMDB I/O; same logic for cached and spilled funcs.
Adds tests/knowledge_plugins/functions/test_function_meta_only_blocks_iterator.py
documenting the meta_only contract that this fix accounts for, plus
an end-to-end check that the cleanup completes on a meta-only-loaded
function.
Fixes#6418
* test_function_meta_only_blocks_iterator: cast cn.addr/size to int for pyright
* test_function_meta_only_blocks_iterator: assert isinstance cn.addr int for pyright
* cfg_fast: use block_addrs_set (not block_addrs) for drop_bad_functions cleanup
Function.block_addrs returns self._local_blocks.keys() which is empty
in meta-only mode. The accessor that returns the populated set on
spilled funcs is Function.block_addrs_set (returns _local_block_addrs).
* function_parser: derive fake_return outside flag from cmsg.blocks
parse_from_cmessage's call-edge handler called
obj._call_to(..., return_to_outside=fake_ret_edge is None)
which only checked whether a matching fake_return edge exists, not
whether its destination was external at save time. _call_to then
called _register_node(is_local=True, ret_node) and added the
destination to _local_block_addrs even when it was serialized as an
external_block.
This is a defense-in-depth follow-up to #6416. With that fix landed,
no newly-saved record should contain a fake_return edge with
is_outside=False whose dst is in cmsg.external_blocks. But:
* legacy LMDB records written before #6416 still have this shape
* other call sites (notably _add_fakeret_to(confirmed=None)) can
create the same in-memory inconsistency, which then makes the
parser inflate the block set on every roundtrip
Fix: derive return_to_outside / to_outside from cmsg.blocks
membership (the authoritative 'was local at save' signal) rather
than from the edge attribute. The same logic applies to the
subsequent obj._fakeret_to call, which previously used the edge
attribute directly.
Adds tests/knowledge_plugins/functions/test_function_parser_fakeret.py
which round-trips a function with the inconsistent edge shape and
asserts _local_block_addrs is preserved.
* test_function_parser_fakeret: assert func is not None for pyright
* test_function_parser_fakeret: drop __package__ override
* cfg_fast: route fake_return cleanup through Function._remove_fakeret
CFGFast._post_analysis's fake_return cleanup loop called
`f.transition_graph.remove_edge(*edge)` directly, bypassing the
@dirty_func-decorated Function._remove_fakeret API. The dirty flag was
never set after the mutation, so on SpillingFunctionDict the next
eviction was clean (no LMDB write) and the cleanup was silently
reverted on the next parse_from_cmessage rebuild.
Symptom: angr-serialize output on PE binaries large enough to hit
cache_limit=1000 functions is non-deterministic — function counts
depend on which functions happen to be cached vs spilled at
drop_bad_functions time, because the cleanup either does or doesn't
survive the LMDB round-trip per function.
Fix: replace the direct graph mutation with the existing
Function._remove_fakeret call. This is symmetric with the
`f._confirm_fakeret(src, dst)` call a few lines above, which already
goes through the Function API.
Adds tests/knowledge_plugins/functions/test_function_post_analysis_dirty.py
to lock in the post-mutation dirty flag.
Fixes#6414
* test_function_post_analysis_dirty: assert func is not None for pyright
* test_function_post_analysis_dirty: drop __package__ override
* test_function_post_analysis_dirty: drop serialize/parse round-trip, set _dirty=False directly
* FunctionManager: Spill to external storage.
* Remove atexit registration. Reduce map size.
* Introduce FuncNode in function graphs; Fix multiple issues with SpillingFunctionDict.
* Implement Function.dirty.
* A bit more optimization.
* FunctionManager loads only meta data for functions when graphs are not accessed; Save .info for functions.
* Only load meta data for Functions in more places.
* Remove FunctionManager.block_map because it's never really used.
* Retire blockaddr_to_function and replace it with blockaddr_to_funcaddr.
* More optimizations and fixes.
* More refactor and fixes.
* More optimizations.
* Fix the bug in lmdb spilling after raising MapFullError.
* Introduce RuntimeDb in KB. Migrate SpillingFunctionsDict to use RuntimeDb.
* Prioritize the basedir of the main executable for the runtime db path.
* Cache non-returning function addrs, unknown-returning function addrs, and function block count in FunctionManager.
* Bug fixes.
* Fix a bug in FunctionParser.
* CFGBase.make_functions: Copy over function metadata when creating functions in the first place.
* Update MockFunctionManager.
* Type check codenode.py and fix an RDA test case.
* Fix serialization tests; Introduce KnowledgeBasePlugin.set_kb(); SpillingFunctionDict now derives from UserDict; Fix KB.name stored in KB._plugins; Fix FunctionDict.__setstate__ swapping Function objects and function addresses.
* Update FactCollector to support FuncNode.
* FunctionInfo: Update Function.dirty and perform type checks on keys and values.
* Make function cache limit configurable.
* Adjust FunctionInfo type check.
* More fixes.
* More updates to account for FuncNode in function graphs.
* Update a test case.
* Fix another test case (do not use the size of FuncNodes).
* Fix Reassembler.
* CFGBase.make_functions: Add a missing insertion to _updated_nonreturning_functions.
* Update FunctionManager.rebuild_callgraph.
* FunctionParser: Call destinations must be FuncNodes.
* Serialize Function.is_default_name.
* Minor fixes.
* Fix CFunctionCall._is_target_ambiguous.
* Mark evicted Function instances as evicted.
* HashLookupAPIDeobfuscator: Take a list of function addresses instead of Function instances as arg.
* CC_NAMES: Fix the bug of missing SimCCCdecl.
* FunctionParser: Fix missing syscall function nodes.
* HookNode: Take a SimProcedure instance instead of the class as the sim_procedure argument.
* FunctionParser: Consider return-type edges when deserializing.
* SimTypeCppFunction: Fix to_json() serialization crash.
* FunctionParser: Fix missing return sites.
* Function.is_{syscall,simprocedure,alignment,plt} settings should mark the function dirty.
* RDA: Do not create blocks for FuncNodes or HookNodes.
* Update a test case.
* Fix issues with SimCppClass.to_json.
* HookNode: Fix HookNode.__eq__.
* Fix SootFunction.
* Lint function_manager.py.
* Lint and type check.
* More docs; Spill Function.ran_cca.
* RuntimeDb: Support specifying base dir using an environment variable.
* Lint and fix test cases.
* chore: Make SIM_LIBRARIES a dict of list of lib instead of a dict of lib
* lint
* Revert "Temporarily disabling a line in test_decompiling_4846..."
This reverts commit eafdbf857a.
* remove stray line
* support looking up sub_XXX names even when the function has been renamed.
* add ability to lookup functions by previous names after a renaming.
---------
Co-authored-by: Yan Shoshitaishvili <zardus@emotionlabs.io>
* SimType & friends: Fully typecheck, fix observed bugs
* fix small bugs
* lint
* adjust the semantics of SimType.size - it can return None now
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* whack
* whack
* whack
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* rename functions_callable to functions_reachable
rename functions_callable to functions_reachable
* fix operator for Constant.__sub__
* print hex address of Function
* support multiple blocks with the same address in StackCanarySimplifier. This happens when doing inlined decompilation
* support specifying initial register values in the StackPointerTracker for inlining functions onto stacks of other functions
* add a StackPointerTracker method to retrieve all offsets of a register
* move stack pointer analysis later, in preparation of pre-inlining refactor
* split decompilation process in preparation for inlining logic
* inlined decompilation!
* testcase for inlined decompiler
* support spilled (e.g., callee-saved) register simplification in the inlined callees
* fix variable reference issue due to duplicated blocks (triggered via inlining)
* Fix more type annotations in clinic.py.
Co-authored-by: Fish <fishw@asu.edu>
* Make CC and DEFAULT_CC platform-aware. Also enhance the input variable
discovery.
* Lint code.
* compatibility fixes
* SimProcedure.project can be None.
* Add a CC for CGC. Other fixes.
* Be more conservative when analyzing call sites.
* Initial pass, loosely following angr package layout
* Many updates to use the 'common' module for binaries repo
* Some unused imports and log handlers are cleaned up
* Fix functions_called
Infinite recursion bug in functions_called
* Add test for functions_called
Check cyclic dependencies of functions_called
* `Function.functions_called` split tests
* make the two cases explicitely separated
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
Co-authored-by: Pamplemousse <xav.maso@gmail.com>
