mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
2ad300aff8
## Problem
pyo3 marks `_native::Parser` as `#[pyclass(unsendable)]`, which causes a
hard thread-assertion panic when a parser created on one thread is
accessed from another:
```
thread '<unnamed>' panicked at pyo3-0.28.3/src/impl_/pyclass.rs:1055:9:
assertion `left == right` failed: _native::Parser is unsendable, but sent to another thread
left: ThreadId(2)
right: ThreadId(1)
```
The prior implementation stored parsers in a module-level `dict[str,
Any]` (`_tree_sitter_languages`). When `_run_compression_in_executor`
dispatched compression work to a `ThreadPoolExecutor`, pool workers
grabbed parsers from that shared dict that were originally created on
the main asyncio thread and panicked.
This produces a 500 on every request where code compression is attempted
via a pool thread.
## Fix
Replace the global dict with `threading.local()` so each thread creates
and owns its own parser instances. No cross-thread parser access is
possible.
```python
# before
_tree_sitter_languages: dict[str, Any] = {} # shared — crosses threads
# after
_tree_sitter_local = threading.local() # per-thread — isolated
```
`is_tree_sitter_loaded()` and `unload_tree_sitter()` updated to operate
on the current thread's local cache (semantics unchanged for
single-threaded callers).
## Tests
9 regression tests added in
`tests/test_transforms/test_tree_sitter_thread_safety.py`:
- Thread isolation: two threads get distinct parser instances
- Within-thread reuse: same thread gets the same cached instance
- Thread pool: parsers usable from `ThreadPoolExecutor` workers without
panic
- Concurrent workers: each distinct pool thread owns a unique parser
- `is_tree_sitter_loaded` / `unload_tree_sitter` lifecycle
Also adds a `filterwarnings` entry for
`PytestUnraisableExceptionWarning`: pyo3 emits this when short-lived
test threads drop parsers at teardown; it does not occur in production
where pool threads are long-lived.
## Relation to #564
PR #564 proposes the same `threading.local()` approach but was blocked
on missing tests (`CHANGES_REQUESTED`). This PR includes the full test
suite.
|
||
|---|---|---|
| .. | ||
| __init__.py | ||
| test_code_compressor.py | ||
| test_content_router.py | ||
| test_diff_compressor.py | ||
| test_diff_compressor_rust_parity.py | ||
| test_html_extractor.py | ||
| test_kompress_compressor.py | ||
| test_read_lifecycle.py | ||
| test_smart_crusher_bugs.py | ||
| test_smart_crusher_ccr_roundtrip.py | ||
| test_smart_crusher_lossless_default.py | ||
| test_smart_crusher_rust_parity.py | ||
| test_tag_protector.py | ||
| test_tree_sitter_thread_safety.py | ||