Also,
- Refactored variable_kb into kb.dec_variables.
- Spill decompilation cache into RuntimeDb.
- Save decompilation cache into angrDb. Decompilation results can be preserved across runs.
- No longer check in _pb2.py files; they are generated during build.
* FunctionParser: Serialize Function._call_sites.
serialize stores the transition graph but not Function._call_sites, and
parse_from_cmsg never repopulated it, so functions loaded back from LMDB
(SpillingFunctionDict eviction, angrdb) returned nothing from
get_call_sites(), get_call_target(), and get_call_return().
_call_sites cannot be rebuilt from transition graph edges. Store the
mapping in a new repeated CallSite field on the Function message and
restore it in parse_from_cmsg.
* lint
* Fix pyright errors in FunctionParser.
* Fix pyright error in LMDB function manager test.
CPython does not guarantee that __del__ runs at interpreter shutdown, so
RuntimeDb.cleanup() could be skipped, leaving *_angr_rtdb directories on disk.
This PR introducts an atexit hook that cleans up all live RuntimeDb instances.
* knowledge_plugins, code_location: recompute hashes after unpickling
Atom, Definition, and CodeLocation cache their hash in a `_hash` slot. The
default pickling persists that slot, but the hash folds in per-process-salted
hashes (e.g. of register-name strings), so a value pickled in one process is
stale when unpickled in another -- equal objects then hash differently, which
breaks sets/dicts (and assertCountEqual) rebuilt from a pickle.
Add `__getstate__`/`__setstate__` that drop `_hash` from the pickled state and
reset it to None on load, so it is recomputed lazily in-process. `__dict__`
and all slots are otherwise preserved, and the existing default-format
pickles still load.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Disable protected-member for hash checks
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Enable ruff isort rule
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* cfg_fast: iterate func.block_addrs for bad-func cleanup (works for spilled funcs)
CFGFast.drop_bad_functions's per-block cleanup iterated func.blocks,
which is empty for any bad function loaded via meta_only=True (e.g.
because it was spilled to LMDB by SpillingFunctionDict at drop
time). The loop was silently a no-op for those functions: the
function was removed from kb.functions but its CFG nodes survived
and its bytes stayed classified as 'code' in _seg_list, leaving the
model in a state that depended on LRU cache timing.
Fix: iterate func.block_addrs (populated for both cached and spilled
funcs) and look up block sizes from the CFG model via cfg_node.size.
No extra LMDB I/O; same logic for cached and spilled funcs.
Adds tests/knowledge_plugins/functions/test_function_meta_only_blocks_iterator.py
documenting the meta_only contract that this fix accounts for, plus
an end-to-end check that the cleanup completes on a meta-only-loaded
function.
Fixes#6418
* test_function_meta_only_blocks_iterator: cast cn.addr/size to int for pyright
* test_function_meta_only_blocks_iterator: assert isinstance cn.addr int for pyright
* cfg_fast: use block_addrs_set (not block_addrs) for drop_bad_functions cleanup
Function.block_addrs returns self._local_blocks.keys() which is empty
in meta-only mode. The accessor that returns the populated set on
spilled funcs is Function.block_addrs_set (returns _local_block_addrs).
* function_parser: derive fake_return outside flag from cmsg.blocks
parse_from_cmessage's call-edge handler called
obj._call_to(..., return_to_outside=fake_ret_edge is None)
which only checked whether a matching fake_return edge exists, not
whether its destination was external at save time. _call_to then
called _register_node(is_local=True, ret_node) and added the
destination to _local_block_addrs even when it was serialized as an
external_block.
This is a defense-in-depth follow-up to #6416. With that fix landed,
no newly-saved record should contain a fake_return edge with
is_outside=False whose dst is in cmsg.external_blocks. But:
* legacy LMDB records written before #6416 still have this shape
* other call sites (notably _add_fakeret_to(confirmed=None)) can
create the same in-memory inconsistency, which then makes the
parser inflate the block set on every roundtrip
Fix: derive return_to_outside / to_outside from cmsg.blocks
membership (the authoritative 'was local at save' signal) rather
than from the edge attribute. The same logic applies to the
subsequent obj._fakeret_to call, which previously used the edge
attribute directly.
Adds tests/knowledge_plugins/functions/test_function_parser_fakeret.py
which round-trips a function with the inconsistent edge shape and
asserts _local_block_addrs is preserved.
* test_function_parser_fakeret: assert func is not None for pyright
* test_function_parser_fakeret: drop __package__ override
* cfg_fast: route fake_return cleanup through Function._remove_fakeret
CFGFast._post_analysis's fake_return cleanup loop called
`f.transition_graph.remove_edge(*edge)` directly, bypassing the
@dirty_func-decorated Function._remove_fakeret API. The dirty flag was
never set after the mutation, so on SpillingFunctionDict the next
eviction was clean (no LMDB write) and the cleanup was silently
reverted on the next parse_from_cmessage rebuild.
Symptom: angr-serialize output on PE binaries large enough to hit
cache_limit=1000 functions is non-deterministic — function counts
depend on which functions happen to be cached vs spilled at
drop_bad_functions time, because the cleanup either does or doesn't
survive the LMDB round-trip per function.
Fix: replace the direct graph mutation with the existing
Function._remove_fakeret call. This is symmetric with the
`f._confirm_fakeret(src, dst)` call a few lines above, which already
goes through the Function API.
Adds tests/knowledge_plugins/functions/test_function_post_analysis_dirty.py
to lock in the post-mutation dirty flag.
Fixes#6414
* test_function_post_analysis_dirty: assert func is not None for pyright
* test_function_post_analysis_dirty: drop __package__ override
* test_function_post_analysis_dirty: drop serialize/parse round-trip, set _dirty=False directly
* Preliminary implementation of CFG edge spilling.
* Many tweaks.
* Use little endian; switch to native bytes and msgspec.
* More tweaking.
* Fix test cases.
* Get rid of _all_keys.
* Lint code.
* Type check.
* Preliminary implementation of a spilling CFG graph.
* Fix some test cases.
* Fix another test case.
* Fixes.
* Lint code.
* Implement InEdgeView and OutEdgeView.
* Implement InDegreeView and OutDegreeView.
* Fixes.
* Use (Node.addr, Node.size) as key.
* Fix test cases.
* More fixes.
* Fix CFGENode support.
* Fix the remaining issues.
* Get rid of SpillingCFGGraph.reverse().
* Some refactoring.
* Oops
* Fix perf issue.
* Fix set size change error during key iteration.
* Fix db_batch_size assignment. Fix some test cases.
* SpillingCFGNodeDict.__setstate__: Initialize self._all_keys.
* Getting rid of the nodes dictionary.
* Fix logic in cfg_emulated.py
* Multiple fixes.
* Type annotation fix.
* Fix missing edges after merging CFGNodes.
* Fix more logic.
* Add Soot CFGNodes to the CFG model.
* Lint and type check.
* Add USE_SPILLING_CFGNODE_DICT and CFGNode.dirty.
* Oops
* Lint code.
* No more pickling of CFGNodes.
* Fix no_ret being None.
* Lint code.
* Some refactor; Destroy the fallback mechanism.
* Fix node dirty bug after deserialization.
* FunctionManager: Spill to external storage.
* Remove atexit registration. Reduce map size.
* Introduce FuncNode in function graphs; Fix multiple issues with SpillingFunctionDict.
* Implement Function.dirty.
* A bit more optimization.
* FunctionManager loads only meta data for functions when graphs are not accessed; Save .info for functions.
* Only load meta data for Functions in more places.
* Remove FunctionManager.block_map because it's never really used.
* Retire blockaddr_to_function and replace it with blockaddr_to_funcaddr.
* More optimizations and fixes.
* More refactor and fixes.
* More optimizations.
* Fix the bug in lmdb spilling after raising MapFullError.
* Introduce RuntimeDb in KB. Migrate SpillingFunctionsDict to use RuntimeDb.
* Prioritize the basedir of the main executable for the runtime db path.
* Cache non-returning function addrs, unknown-returning function addrs, and function block count in FunctionManager.
* Bug fixes.
* Fix a bug in FunctionParser.
* CFGBase.make_functions: Copy over function metadata when creating functions in the first place.
* Update MockFunctionManager.
* Type check codenode.py and fix an RDA test case.
* Fix serialization tests; Introduce KnowledgeBasePlugin.set_kb(); SpillingFunctionDict now derives from UserDict; Fix KB.name stored in KB._plugins; Fix FunctionDict.__setstate__ swapping Function objects and function addresses.
* Update FactCollector to support FuncNode.
* FunctionInfo: Update Function.dirty and perform type checks on keys and values.
* Make function cache limit configurable.
* Adjust FunctionInfo type check.
* More fixes.
* More updates to account for FuncNode in function graphs.
* Update a test case.
* Fix another test case (do not use the size of FuncNodes).
* Fix Reassembler.
* CFGBase.make_functions: Add a missing insertion to _updated_nonreturning_functions.
* Update FunctionManager.rebuild_callgraph.
* FunctionParser: Call destinations must be FuncNodes.
* Serialize Function.is_default_name.
* Minor fixes.
* Fix CFunctionCall._is_target_ambiguous.
* Mark evicted Function instances as evicted.
* HashLookupAPIDeobfuscator: Take a list of function addresses instead of Function instances as arg.
* CC_NAMES: Fix the bug of missing SimCCCdecl.
* FunctionParser: Fix missing syscall function nodes.
* HookNode: Take a SimProcedure instance instead of the class as the sim_procedure argument.
* FunctionParser: Consider return-type edges when deserializing.
* SimTypeCppFunction: Fix to_json() serialization crash.
* FunctionParser: Fix missing return sites.
* Function.is_{syscall,simprocedure,alignment,plt} settings should mark the function dirty.
* RDA: Do not create blocks for FuncNodes or HookNodes.
* Update a test case.
* Fix issues with SimCppClass.to_json.
* HookNode: Fix HookNode.__eq__.
* Fix SootFunction.
* Lint function_manager.py.
* Lint and type check.
* More docs; Spill Function.ran_cca.
* RuntimeDb: Support specifying base dir using an environment variable.
* Lint and fix test cases.
* chore: Make SIM_LIBRARIES a dict of list of lib instead of a dict of lib
* lint
* Revert "Temporarily disabling a line in test_decompiling_4846..."
This reverts commit eafdbf857a.
* remove stray line
* support looking up sub_XXX names even when the function has been renamed.
* add ability to lookup functions by previous names after a renaming.
---------
Co-authored-by: Yan Shoshitaishvili <zardus@emotionlabs.io>
* fix typos
This mostly fixes comments, but in a handful of places fixes bugs due to typos.
One example:
```
- insn_op_idx=None if cmsg.operand_idx == -1 else cmsg.opearnd_idx,
+ insn_op_idx=None if cmsg.operand_idx == -1 else cmsg.operand_idx,
```
* more typo fixes
* address lint issues
* more import fixes?
* address feedback from PR
---------
Co-authored-by: Brian Caswell <bcaswell@microsoft.com>
* SimType & friends: Fully typecheck, fix observed bugs
* fix small bugs
* lint
* adjust the semantics of SimType.size - it can return None now
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* whack
* whack
* whack
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* rename functions_callable to functions_reachable
rename functions_callable to functions_reachable
* fix operator for Constant.__sub__
* print hex address of Function
* support multiple blocks with the same address in StackCanarySimplifier. This happens when doing inlined decompilation
* support specifying initial register values in the StackPointerTracker for inlining functions onto stacks of other functions
* add a StackPointerTracker method to retrieve all offsets of a register
* move stack pointer analysis later, in preparation of pre-inlining refactor
* split decompilation process in preparation for inlining logic
* inlined decompilation!
* testcase for inlined decompiler
* support spilled (e.g., callee-saved) register simplification in the inlined callees
* fix variable reference issue due to duplicated blocks (triggered via inlining)
* Fix more type annotations in clinic.py.
Co-authored-by: Fish <fishw@asu.edu>
* Add KB as parameter to KnowledgeBasePlugin for consistency
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Rattle the chains that bind us
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
---------
Co-authored-by: Florian Magin <fmagin@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Make CC and DEFAULT_CC platform-aware. Also enhance the input variable
discovery.
* Lint code.
* compatibility fixes
* SimProcedure.project can be None.
* Add a CC for CGC. Other fixes.
* Be more conservative when analyzing call sites.
* Initial pass, loosely following angr package layout
* Many updates to use the 'common' module for binaries repo
* Some unused imports and log handlers are cleaned up
* Attempt to provide an interface for writing RDA function handlers easily
* Aggressively touch up call and const atoms
* Totally rewrite function handlers
* Imports
* so many renames and fixes
* weh
* better errors
* Attempt to make decompiler work
* better behavior
* Fix a bunch of shit. thanks mypy
* Add DepGraph.find_path
* Use callsite insaddr for codeloc of function outputs
* Add LiveDefinitions: get_stack_values
* Don't require handle_impl_<func>
* wip: add logic for matching RDA definitions against SimVariables
* _narrow_exprs: Do not update call.ret_expr if ret_expr is None.
* Propagator: Fix an incorrect access to None in the AIL engine.
* RDA: Fix an incorrect CodeLocation definition in function handler.
* Clarify docs for CodeLocation
* fix handle_impl_ and clarify warning message
* Support pseudo function calls.
* Separate function effects applied at callsites and the ones applied inside callees.
* AILSimplifier._narrow_exprs: Filter away all definitions in callees.
* DepGraph no longer takes a reference to Project.
* Fix function argument inference for tail jumps.
* FunctionHandler: Use args. Support args_values.
* Tidy interfaces
* AILSimplifier._unify_local_variables: Ignore variables defined in callees.
* Fix CCA by enabling the function handler.
* Properly handle CallExprs.
* Fix call expression handling. Fix the test case.
* f
* DefinitionMatchPredicate: Fix default initializers
* Overhaul codelocs
* Fix context handling
* Fix values iter
* Sort some shit
* Add logic for handling incomplete prototypes in FunctionHandler
* FunctionHandler: apply effects from dependencies first.
* Properly handle Call.ret_expr in AIL. Fix a in VRA.
* Propagator: Be less strict about contatenation when the high bytes are concrete.
* oops
* Move uses to first pass of function handler effects
* Aggressive type hinting
* FunctionHandler: Try harder to resolve function address
* Type annotation
* Typing
* Adjust codegen to smooth over casts
* Tweak the function argument uses for the nth time
* ail simplifier: check that expr corresponding to use is not None
* Typing
* Objectively a correct bugfix
* Correctly generate args_atoms based on args_values
* in case of fire break glass
* Function handler: Add args uses even if there is no return value
* Add data.ret_values even if there is no ret atom (e.g. ccalls)
* Add the missing with_arch() call.
* function handler: hook prototypes don't have an arch
* Revert "Add the missing with_arch() call."
This reverts commit 319eb8f826f634c89106221e33359a2eb1009a48.
* Implement better is_testing detection.
* Uncomment an important fire extinguisher.
* RDA: AIL engine overwrites the whole register when the return value only occupies the partial register.
* FunctionHandler: Properly handle clobbered partial registers.
* Fix a type hint in atoms.Register.
* Do not add duplicated uses for values that are in args_values.
* copy guessed_prototype from hook to function data
* AILSimplifier: Ignore unused conflicting defs when unifying variables.
* c_args_as_atoms: Handle variadic functions.
* RDA: AILEngine: Do not redefine locals if call args are available.
* VRA: EngineAIL: Follow RDA's logic when handling call.ret_expr.
* Respect block_idx a few places; add caller-handled ret_defns to callsite info
* CallSiteMaker: Mark argument definitions in call stmts so RDA knows what to erase.
* RDA: AILEngine.handle_register: Use the optimal define location if possible.
* oops
* AILSimplifier._narrow_exprs: Fix the callee function check.
* Remove unused imports.
* docs
* Adapt StaticObjectFinder to the new function handler model.
* Make vex rda tests pass
* function handler: Apply bootstrap return value definition at callsite
* lint
* lint 2
---------
Co-authored-by: Audrey Dutcher <audrey@rhelmot.io>
Co-authored-by: Matt Borgerson <contact@mborgerson.com>
* SimMemoryObject: Use slots.
* Add LabeledMemory.
* CodeLocation: stmt_idx can be None.
* Propagator: Switch to LabeledMemory and claripy ASTs - Strike One.
* Implement SimLabeledMemoryObject.
* Propagator: Switch to LabeledMemory and claripy ASTs - Strike Two.
* Bug fixes. Migrate RDA live_definitions to LabeledMemory.
* Call the test case.
* Add MVListPage.
* some effort
* Update the test case.
* More progress
* More work done.
* Implement page merging.
* Implement add and sub for RDEngineVEX.
* More test cases passed
* More progress. Update test cases.
* RDA: `SimEngineRDVEX._handle_function_core` to use `MultiValues`
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
* Fix _process_block_end().
* Update _handle_CMPNE
* Update _handle_CMPLT
* Update _handle_CMPORD
* Update _handle_Sar
* Add none safety to some _handle_* functions
* Fix _handle_Not update
* Update _handle_Not1
* Update _handle_CCall
* Add exception checking during shifting
* Convert SimEngineRDAIL.
* bug fixes
* Fix the endianness bug.
* Several bug fixes.
* Do not resimplify a block if it is not changed.
* RDA: Do not reload register values.
* Fix DefinitionAnnotation.
* Fix MultiValues.__len__().
* Fix a test case.
* Propagator: Properly deal with stack addresses in Load.
* Migrate most of variable recovery analysis.
* Force AIL Store to use its own size.
* Fix incorrect propagation when bytes are extracted from an MO. Get rid of .processor_state in VRA.
* Get rid of ProcessorState.
* Minor bug fixes in _reference().
* bug fixes
* Propagator: Return 1-bit tops for CMPs.
* Cache TOPs.
* Fix variable indent for global variables.
* SimMemoryObject: Optimize __eq__().
* VRF: Fix four handlers in the AIL engine.
* Remove a breakpoint.
* MVListPage: Do not calculate changed bytes twice during merging.
* MVListPage: Reduce unnecessary scans of None in content.
* ListPage/MVListPage: Support custom SimMemoryObject comparators.
* MVListPage: Reduce calls to _contains().
* Propagator: Remove TOPs in replacements after analysis.
* Reduce max iterations of RDA and Propagator to 2.
* Bug fix in propagator.
* Fix an incorrect type annotation.
* Unify _post_analysis in Propagator.
* LiveDefinitions: Fix _mo_cmp.
* Revert "Reduce max iterations of RDA and Propagator to 2."
This reverts commit 536ea4d59f80095dc37ec9055ed25aab19ed7d83.
* _replace_and_build(): Returns if the replacement really happened.
* Propagator: Fix stack variable load/store.
* Compute less propagation and RDAs in AILSimplifier.
* RDA: Use cached and less-accurate dummy definitions for kill_definitions().
* Fix LiveDefinitions._mo_cmp.
* Propagator: Fix a minor issue in the Add handler in AIL engine.
* Propagator: Attempt to propagate everything coming out of tmps.
* Migrate variable recovery.
* RDA: Get rid of references to DataSet. Fix support for floating point values.
* LightEngineVEX: _handle_Const supports floating points.
* CallSiteMaker: Handle SimMemoryMissingError.
* extract_offset_to_sp: Handle __add__ with only one argument.
* Propagator: Do not store if the data is None.
* Implement get_all_definitions(). Fix bugs in SimEngineRDVEX.
* Fix more broken test cases.
* VariableRecovery: Fix a StopIteration bug.
* Fix more bugs.
* Lint the code.
* Fix test cases. Drop DataSet.
Co-authored-by: Pamplemousse <xav.maso@gmail.com>
Co-authored-by: Clasm <wfgibbs@asu.edu>
* RDA: Don't duplicate `get_sp` implementation
... accross `ReachingDefinitionsState` and `LiveDefinitions`
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
* RDA: `LiveDefinitions.get_sp()` not to fail
...when there are several SP definitions.
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
* RDA: Utilitary to get the size of several `DataSet`s
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
* RDA: More operations for `UnknownSize`
As the size of `DataSet`s represents a number of bits, the following
operation often happens in the code: `dataset._bits // 8` .
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
* RDA: Minor missing documention addition
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
* RDA: Favor `unittest` assertions to `nose`
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
* RDA: Basic heap memory modelisation
* have a `HeapAddress` dedicated type
* uber-basic simulation of an allocator behavior (addresses
generation)
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
* RDA: Add heap modelisation to the state
* content as part of the `LiveDefinitions`
* memory management using `HeapAllocator` and `HeapAddress`es
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
* RDA: Allow sum between `int` and `HeapAddress`
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
* RDA: Make `HeapAddress` hashable
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
* Fix functions_called
Infinite recursion bug in functions_called
* Add test for functions_called
Check cyclic dependencies of functions_called
* `Function.functions_called` split tests
* make the two cases explicitely separated
Signed-off-by: Pamplemousse <xav.maso@gmail.com>
Co-authored-by: Pamplemousse <xav.maso@gmail.com>
