angr/tests/knowledge_plugins/functions/test_function_manager_lmdb.py

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

282 lines
8.7 KiB
Python
Raw Permalink Normal View History

FunctionManager: Spill to external storage. (#5976) * FunctionManager: Spill to external storage. * Remove atexit registration. Reduce map size. * Introduce FuncNode in function graphs; Fix multiple issues with SpillingFunctionDict. * Implement Function.dirty. * A bit more optimization. * FunctionManager loads only meta data for functions when graphs are not accessed; Save .info for functions. * Only load meta data for Functions in more places. * Remove FunctionManager.block_map because it's never really used. * Retire blockaddr_to_function and replace it with blockaddr_to_funcaddr. * More optimizations and fixes. * More refactor and fixes. * More optimizations. * Fix the bug in lmdb spilling after raising MapFullError. * Introduce RuntimeDb in KB. Migrate SpillingFunctionsDict to use RuntimeDb. * Prioritize the basedir of the main executable for the runtime db path. * Cache non-returning function addrs, unknown-returning function addrs, and function block count in FunctionManager. * Bug fixes. * Fix a bug in FunctionParser. * CFGBase.make_functions: Copy over function metadata when creating functions in the first place. * Update MockFunctionManager. * Type check codenode.py and fix an RDA test case. * Fix serialization tests; Introduce KnowledgeBasePlugin.set_kb(); SpillingFunctionDict now derives from UserDict; Fix KB.name stored in KB._plugins; Fix FunctionDict.__setstate__ swapping Function objects and function addresses. * Update FactCollector to support FuncNode. * FunctionInfo: Update Function.dirty and perform type checks on keys and values. * Make function cache limit configurable. * Adjust FunctionInfo type check. * More fixes. * More updates to account for FuncNode in function graphs. * Update a test case. * Fix another test case (do not use the size of FuncNodes). * Fix Reassembler. * CFGBase.make_functions: Add a missing insertion to _updated_nonreturning_functions. * Update FunctionManager.rebuild_callgraph. * FunctionParser: Call destinations must be FuncNodes. * Serialize Function.is_default_name. * Minor fixes. * Fix CFunctionCall._is_target_ambiguous. * Mark evicted Function instances as evicted. * HashLookupAPIDeobfuscator: Take a list of function addresses instead of Function instances as arg. * CC_NAMES: Fix the bug of missing SimCCCdecl. * FunctionParser: Fix missing syscall function nodes. * HookNode: Take a SimProcedure instance instead of the class as the sim_procedure argument. * FunctionParser: Consider return-type edges when deserializing. * SimTypeCppFunction: Fix to_json() serialization crash. * FunctionParser: Fix missing return sites. * Function.is_{syscall,simprocedure,alignment,plt} settings should mark the function dirty. * RDA: Do not create blocks for FuncNodes or HookNodes. * Update a test case. * Fix issues with SimCppClass.to_json. * HookNode: Fix HookNode.__eq__. * Fix SootFunction. * Lint function_manager.py. * Lint and type check. * More docs; Spill Function.ran_cca. * RuntimeDb: Support specifying base dir using an environment variable. * Lint and fix test cases.
2026-01-12 19:58:18 -07:00
#!/usr/bin/env python3
# pylint:disable=missing-class-docstring,protected-access
FunctionManager: Spill to external storage. (#5976) * FunctionManager: Spill to external storage. * Remove atexit registration. Reduce map size. * Introduce FuncNode in function graphs; Fix multiple issues with SpillingFunctionDict. * Implement Function.dirty. * A bit more optimization. * FunctionManager loads only meta data for functions when graphs are not accessed; Save .info for functions. * Only load meta data for Functions in more places. * Remove FunctionManager.block_map because it's never really used. * Retire blockaddr_to_function and replace it with blockaddr_to_funcaddr. * More optimizations and fixes. * More refactor and fixes. * More optimizations. * Fix the bug in lmdb spilling after raising MapFullError. * Introduce RuntimeDb in KB. Migrate SpillingFunctionsDict to use RuntimeDb. * Prioritize the basedir of the main executable for the runtime db path. * Cache non-returning function addrs, unknown-returning function addrs, and function block count in FunctionManager. * Bug fixes. * Fix a bug in FunctionParser. * CFGBase.make_functions: Copy over function metadata when creating functions in the first place. * Update MockFunctionManager. * Type check codenode.py and fix an RDA test case. * Fix serialization tests; Introduce KnowledgeBasePlugin.set_kb(); SpillingFunctionDict now derives from UserDict; Fix KB.name stored in KB._plugins; Fix FunctionDict.__setstate__ swapping Function objects and function addresses. * Update FactCollector to support FuncNode. * FunctionInfo: Update Function.dirty and perform type checks on keys and values. * Make function cache limit configurable. * Adjust FunctionInfo type check. * More fixes. * More updates to account for FuncNode in function graphs. * Update a test case. * Fix another test case (do not use the size of FuncNodes). * Fix Reassembler. * CFGBase.make_functions: Add a missing insertion to _updated_nonreturning_functions. * Update FunctionManager.rebuild_callgraph. * FunctionParser: Call destinations must be FuncNodes. * Serialize Function.is_default_name. * Minor fixes. * Fix CFunctionCall._is_target_ambiguous. * Mark evicted Function instances as evicted. * HashLookupAPIDeobfuscator: Take a list of function addresses instead of Function instances as arg. * CC_NAMES: Fix the bug of missing SimCCCdecl. * FunctionParser: Fix missing syscall function nodes. * HookNode: Take a SimProcedure instance instead of the class as the sim_procedure argument. * FunctionParser: Consider return-type edges when deserializing. * SimTypeCppFunction: Fix to_json() serialization crash. * FunctionParser: Fix missing return sites. * Function.is_{syscall,simprocedure,alignment,plt} settings should mark the function dirty. * RDA: Do not create blocks for FuncNodes or HookNodes. * Update a test case. * Fix issues with SimCppClass.to_json. * HookNode: Fix HookNode.__eq__. * Fix SootFunction. * Lint function_manager.py. * Lint and type check. * More docs; Spill Function.ran_cca. * RuntimeDb: Support specifying base dir using an environment variable. * Lint and fix test cases.
2026-01-12 19:58:18 -07:00
"""Test cases for FunctionManager LMDB save/load and LRU cache functionality."""
FunctionManager: Spill to external storage. (#5976) * FunctionManager: Spill to external storage. * Remove atexit registration. Reduce map size. * Introduce FuncNode in function graphs; Fix multiple issues with SpillingFunctionDict. * Implement Function.dirty. * A bit more optimization. * FunctionManager loads only meta data for functions when graphs are not accessed; Save .info for functions. * Only load meta data for Functions in more places. * Remove FunctionManager.block_map because it's never really used. * Retire blockaddr_to_function and replace it with blockaddr_to_funcaddr. * More optimizations and fixes. * More refactor and fixes. * More optimizations. * Fix the bug in lmdb spilling after raising MapFullError. * Introduce RuntimeDb in KB. Migrate SpillingFunctionsDict to use RuntimeDb. * Prioritize the basedir of the main executable for the runtime db path. * Cache non-returning function addrs, unknown-returning function addrs, and function block count in FunctionManager. * Bug fixes. * Fix a bug in FunctionParser. * CFGBase.make_functions: Copy over function metadata when creating functions in the first place. * Update MockFunctionManager. * Type check codenode.py and fix an RDA test case. * Fix serialization tests; Introduce KnowledgeBasePlugin.set_kb(); SpillingFunctionDict now derives from UserDict; Fix KB.name stored in KB._plugins; Fix FunctionDict.__setstate__ swapping Function objects and function addresses. * Update FactCollector to support FuncNode. * FunctionInfo: Update Function.dirty and perform type checks on keys and values. * Make function cache limit configurable. * Adjust FunctionInfo type check. * More fixes. * More updates to account for FuncNode in function graphs. * Update a test case. * Fix another test case (do not use the size of FuncNodes). * Fix Reassembler. * CFGBase.make_functions: Add a missing insertion to _updated_nonreturning_functions. * Update FunctionManager.rebuild_callgraph. * FunctionParser: Call destinations must be FuncNodes. * Serialize Function.is_default_name. * Minor fixes. * Fix CFunctionCall._is_target_ambiguous. * Mark evicted Function instances as evicted. * HashLookupAPIDeobfuscator: Take a list of function addresses instead of Function instances as arg. * CC_NAMES: Fix the bug of missing SimCCCdecl. * FunctionParser: Fix missing syscall function nodes. * HookNode: Take a SimProcedure instance instead of the class as the sim_procedure argument. * FunctionParser: Consider return-type edges when deserializing. * SimTypeCppFunction: Fix to_json() serialization crash. * FunctionParser: Fix missing return sites. * Function.is_{syscall,simprocedure,alignment,plt} settings should mark the function dirty. * RDA: Do not create blocks for FuncNodes or HookNodes. * Update a test case. * Fix issues with SimCppClass.to_json. * HookNode: Fix HookNode.__eq__. * Fix SootFunction. * Lint function_manager.py. * Lint and type check. * More docs; Spill Function.ran_cca. * RuntimeDb: Support specifying base dir using an environment variable. * Lint and fix test cases.
2026-01-12 19:58:18 -07:00
from __future__ import annotations
__package__ = __package__ or "tests.knowledge_plugins.functions" # pylint:disable=redefined-builtin
import os
import unittest
import angr
from tests.common import bin_location
test_location = os.path.join(bin_location, "tests")
class TestFunctionManagerLMDB(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.bin_path = os.path.join(test_location, "x86_64", "fauxware")
def test_default_unlimited_cache(self):
"""Test that default cache is unlimited (no eviction)."""
proj = angr.Project(self.bin_path, auto_load_libs=False)
proj.analyses.CFGFast()
fm = proj.kb.functions
assert fm.cache_limit is None, "Default should be None (unlimited)"
assert fm.spilled_function_count == 0, "No functions should be spilled by default"
assert fm.cached_function_count == fm.total_function_count, "All functions should be in memory"
def test_set_cache_limit(self):
"""Test setting cache limit triggers eviction."""
proj = angr.Project(self.bin_path, auto_load_libs=False)
proj.analyses.CFGFast()
fm = proj.kb.functions
total_count = len(fm)
# Set cache limit
cache_limit = 5
fm.cache_limit = cache_limit
assert fm.cached_function_count <= cache_limit, (
f"Cache limit not respected: {fm.cached_function_count} > {cache_limit}"
)
FunctionManager: Spill to external storage. (#5976) * FunctionManager: Spill to external storage. * Remove atexit registration. Reduce map size. * Introduce FuncNode in function graphs; Fix multiple issues with SpillingFunctionDict. * Implement Function.dirty. * A bit more optimization. * FunctionManager loads only meta data for functions when graphs are not accessed; Save .info for functions. * Only load meta data for Functions in more places. * Remove FunctionManager.block_map because it's never really used. * Retire blockaddr_to_function and replace it with blockaddr_to_funcaddr. * More optimizations and fixes. * More refactor and fixes. * More optimizations. * Fix the bug in lmdb spilling after raising MapFullError. * Introduce RuntimeDb in KB. Migrate SpillingFunctionsDict to use RuntimeDb. * Prioritize the basedir of the main executable for the runtime db path. * Cache non-returning function addrs, unknown-returning function addrs, and function block count in FunctionManager. * Bug fixes. * Fix a bug in FunctionParser. * CFGBase.make_functions: Copy over function metadata when creating functions in the first place. * Update MockFunctionManager. * Type check codenode.py and fix an RDA test case. * Fix serialization tests; Introduce KnowledgeBasePlugin.set_kb(); SpillingFunctionDict now derives from UserDict; Fix KB.name stored in KB._plugins; Fix FunctionDict.__setstate__ swapping Function objects and function addresses. * Update FactCollector to support FuncNode. * FunctionInfo: Update Function.dirty and perform type checks on keys and values. * Make function cache limit configurable. * Adjust FunctionInfo type check. * More fixes. * More updates to account for FuncNode in function graphs. * Update a test case. * Fix another test case (do not use the size of FuncNodes). * Fix Reassembler. * CFGBase.make_functions: Add a missing insertion to _updated_nonreturning_functions. * Update FunctionManager.rebuild_callgraph. * FunctionParser: Call destinations must be FuncNodes. * Serialize Function.is_default_name. * Minor fixes. * Fix CFunctionCall._is_target_ambiguous. * Mark evicted Function instances as evicted. * HashLookupAPIDeobfuscator: Take a list of function addresses instead of Function instances as arg. * CC_NAMES: Fix the bug of missing SimCCCdecl. * FunctionParser: Fix missing syscall function nodes. * HookNode: Take a SimProcedure instance instead of the class as the sim_procedure argument. * FunctionParser: Consider return-type edges when deserializing. * SimTypeCppFunction: Fix to_json() serialization crash. * FunctionParser: Fix missing return sites. * Function.is_{syscall,simprocedure,alignment,plt} settings should mark the function dirty. * RDA: Do not create blocks for FuncNodes or HookNodes. * Update a test case. * Fix issues with SimCppClass.to_json. * HookNode: Fix HookNode.__eq__. * Fix SootFunction. * Lint function_manager.py. * Lint and type check. * More docs; Spill Function.ran_cca. * RuntimeDb: Support specifying base dir using an environment variable. * Lint and fix test cases.
2026-01-12 19:58:18 -07:00
assert fm.total_function_count == total_count, "Total function count should be preserved"
assert fm.spilled_function_count == total_count - fm.cached_function_count, "Spilled count incorrect"
def test_cache_properties(self):
"""Test cache monitoring properties."""
proj = angr.Project(self.bin_path, auto_load_libs=False)
proj.analyses.CFGFast()
fm = proj.kb.functions
total = len(fm)
# Set small cache limit
fm.cache_limit = 3
# Verify properties
assert fm.cached_function_count <= 3
assert fm.spilled_function_count >= 0
assert fm.total_function_count == total
assert fm.cached_function_count + fm.spilled_function_count == fm.total_function_count
def test_access_spilled_function(self):
"""Test that accessing a spilled function loads it from LMDB."""
proj = angr.Project(self.bin_path, auto_load_libs=False)
proj.analyses.CFGFast()
fm = proj.kb.functions
# Set small cache limit
fm.cache_limit = 3
# Find a spilled function
if fm.spilled_function_count == 0:
self.skipTest("No spilled functions to test")
spilled_addr = next(iter(fm._spilled_addrs))
# Access the spilled function
func = fm[spilled_addr]
# Verify it was loaded
assert func is not None, "Failed to load spilled function"
assert func.addr == spilled_addr, "Loaded function has wrong address"
def test_dynamic_cache_limit_decrease(self):
"""Test decreasing cache limit dynamically."""
proj = angr.Project(self.bin_path, auto_load_libs=False)
proj.analyses.CFGFast()
fm = proj.kb.functions
total = len(fm)
# Start with larger limit
fm.cache_limit = 10
assert fm.cached_function_count <= 10
# Decrease limit
fm.cache_limit = 5
assert fm.cached_function_count <= 5
# Further decrease
fm.cache_limit = 2
assert fm.cached_function_count <= 2
# Total should be preserved
assert fm.total_function_count == total
def test_dynamic_cache_limit_increase(self):
"""Test increasing cache limit dynamically."""
proj = angr.Project(self.bin_path, auto_load_libs=False)
proj.analyses.CFGFast()
fm = proj.kb.functions
total = len(fm)
# Start with small limit
fm.cache_limit = 3
cached_before = fm.cached_function_count
# Increase limit (shouldn't auto-load more)
fm.cache_limit = 10
assert fm.cached_function_count >= cached_before # May increase due to access
# Total preserved
assert fm.total_function_count == total
def test_contains_with_spilled(self):
"""Test __contains__ checks both in-memory and spilled functions."""
proj = angr.Project(self.bin_path, auto_load_libs=False)
proj.analyses.CFGFast()
fm = proj.kb.functions
all_addrs = set(fm)
# Set small cache limit
fm.cache_limit = 3
# All addresses should still be "in" the function manager
for addr in all_addrs:
assert addr in fm, f"Address {hex(addr)} should be in function manager"
def test_iter_with_spilled(self):
"""Test __iter__ includes both in-memory and spilled functions."""
proj = angr.Project(self.bin_path, auto_load_libs=False)
proj.analyses.CFGFast()
fm = proj.kb.functions
all_addrs_before = set(fm)
# Set small cache limit
fm.cache_limit = 3
# Iteration should still include all addresses
all_addrs_after = set(fm)
assert all_addrs_before == all_addrs_after, "Iteration should include all functions"
def test_len_with_spilled(self):
"""Test __len__ returns total count including spilled."""
proj = angr.Project(self.bin_path, auto_load_libs=False)
proj.analyses.CFGFast()
fm = proj.kb.functions
total_before = len(fm)
# Set small cache limit
fm.cache_limit = 3
# Length should still return total count
assert len(fm) == total_before, "len() should return total count"
def test_spilled_function_call_sites(self):
"""Test that call sites survive spilling to LMDB and reloading."""
proj = angr.Project(self.bin_path, auto_load_libs=False)
proj.analyses.CFGFast()
fm = proj.kb.functions
main_func = fm.function(name="main")
assert main_func is not None
main_addr = main_func.addr
fm[main_addr]._add_call_site(0x4007D3, None, None)
call_sites_before = dict(fm[main_addr]._call_sites)
assert len(call_sites_before) > 1
# Evict main from the cache
fm.cache_limit = 1
other_addr = next(addr for addr in fm if addr != main_addr)
_ = fm[other_addr]
assert main_addr in fm._spilled_addrs
loaded = fm[main_addr]
assert dict(loaded._call_sites) == call_sites_before
FunctionManager: Spill to external storage. (#5976) * FunctionManager: Spill to external storage. * Remove atexit registration. Reduce map size. * Introduce FuncNode in function graphs; Fix multiple issues with SpillingFunctionDict. * Implement Function.dirty. * A bit more optimization. * FunctionManager loads only meta data for functions when graphs are not accessed; Save .info for functions. * Only load meta data for Functions in more places. * Remove FunctionManager.block_map because it's never really used. * Retire blockaddr_to_function and replace it with blockaddr_to_funcaddr. * More optimizations and fixes. * More refactor and fixes. * More optimizations. * Fix the bug in lmdb spilling after raising MapFullError. * Introduce RuntimeDb in KB. Migrate SpillingFunctionsDict to use RuntimeDb. * Prioritize the basedir of the main executable for the runtime db path. * Cache non-returning function addrs, unknown-returning function addrs, and function block count in FunctionManager. * Bug fixes. * Fix a bug in FunctionParser. * CFGBase.make_functions: Copy over function metadata when creating functions in the first place. * Update MockFunctionManager. * Type check codenode.py and fix an RDA test case. * Fix serialization tests; Introduce KnowledgeBasePlugin.set_kb(); SpillingFunctionDict now derives from UserDict; Fix KB.name stored in KB._plugins; Fix FunctionDict.__setstate__ swapping Function objects and function addresses. * Update FactCollector to support FuncNode. * FunctionInfo: Update Function.dirty and perform type checks on keys and values. * Make function cache limit configurable. * Adjust FunctionInfo type check. * More fixes. * More updates to account for FuncNode in function graphs. * Update a test case. * Fix another test case (do not use the size of FuncNodes). * Fix Reassembler. * CFGBase.make_functions: Add a missing insertion to _updated_nonreturning_functions. * Update FunctionManager.rebuild_callgraph. * FunctionParser: Call destinations must be FuncNodes. * Serialize Function.is_default_name. * Minor fixes. * Fix CFunctionCall._is_target_ambiguous. * Mark evicted Function instances as evicted. * HashLookupAPIDeobfuscator: Take a list of function addresses instead of Function instances as arg. * CC_NAMES: Fix the bug of missing SimCCCdecl. * FunctionParser: Fix missing syscall function nodes. * HookNode: Take a SimProcedure instance instead of the class as the sim_procedure argument. * FunctionParser: Consider return-type edges when deserializing. * SimTypeCppFunction: Fix to_json() serialization crash. * FunctionParser: Fix missing return sites. * Function.is_{syscall,simprocedure,alignment,plt} settings should mark the function dirty. * RDA: Do not create blocks for FuncNodes or HookNodes. * Update a test case. * Fix issues with SimCppClass.to_json. * HookNode: Fix HookNode.__eq__. * Fix SootFunction. * Lint function_manager.py. * Lint and type check. * More docs; Spill Function.ran_cca. * RuntimeDb: Support specifying base dir using an environment variable. * Lint and fix test cases.
2026-01-12 19:58:18 -07:00
def test_delete_spilled_function(self):
"""Test deleting a spilled function."""
proj = angr.Project(self.bin_path, auto_load_libs=False)
proj.analyses.CFGFast()
fm = proj.kb.functions
total_before = len(fm)
# Set small cache limit
fm.cache_limit = 3
if fm.spilled_function_count == 0:
self.skipTest("No spilled functions to test")
# Get a spilled address
spilled_addr = next(iter(fm._spilled_addrs))
# Delete it
del fm[spilled_addr]
# Verify it's removed
assert spilled_addr not in fm
assert len(fm) == total_before - 1
def test_delete_cached_function(self):
"""Test deleting a cached (in-memory) function."""
proj = angr.Project(self.bin_path, auto_load_libs=False)
proj.analyses.CFGFast()
fm = proj.kb.functions
total_before = len(fm)
# Set cache limit
fm.cache_limit = 5
# Get a cached address
cached_addr = next(iter(fm._function_map.keys()))
# Delete it
del fm[cached_addr]
# Verify it's removed
assert cached_addr not in fm
assert len(fm) == total_before - 1
def test_clear_with_spilled(self):
"""Test clear() removes both in-memory and spilled functions."""
proj = angr.Project(self.bin_path, auto_load_libs=False)
proj.analyses.CFGFast()
fm = proj.kb.functions
# Set cache limit
fm.cache_limit = 3
# Clear
fm.clear()
assert len(fm) == 0
assert fm.cached_function_count == 0
assert fm.spilled_function_count == 0
def test_copy_with_spilled(self):
"""Test copy() works with spilled functions."""
proj = angr.Project(self.bin_path, auto_load_libs=False)
proj.analyses.CFGFast()
fm = proj.kb.functions
total = len(fm)
# Set cache limit
fm.cache_limit = 5
# Copy
fm_copy = fm.copy()
# Verify copy has all functions
assert fm_copy.total_function_count == total
if __name__ == "__main__":
unittest.main()