## Description
The #956 symptom — `compression_first_stage` always timing out at ~30s
with 0 tokens removed — is not a PyO3/event-loop issue (compression
itself runs ~120ms on Python 3.14). Root cause: `tiktoken` downloads its
BPE vocab via `requests.get(...)` with no timeout, loaded lazily inside
the compression worker (`TiktokenCounter.encoding`,
`AnthropicProvider.__init__`). On a firewalled network that blocks
indefinitely, so the worker hangs and `asyncio.wait_for` trips at 30s on
every request (the hung download never caches, so it repeats). Refs #956
(runtime half).
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- Load the encoding on a worker thread bounded by
`HEADROOM_TIKTOKEN_LOAD_TIMEOUT_SECONDS` (default 10s); on timeout raise
`TiktokenLoadError` and fall back to estimation (registry ->
EstimatingTokenCounter; Anthropic provider -> character estimate).
- Remember the first timed-out encoding so later requests fail fast
instead of re-blocking.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] New tests added for new functionality
### Test Output
```text
$ pytest tests/test_tokenizers/test_tiktoken_load_timeout.py -q
4 passed in 0.69s
```
## Real Behavior Proof
- Environment: Linux, Python 3.14 and 3.13
- Exact command / steps: timed the Rust compression call sync / via
run_in_executor / 2x concurrent on both interpreters; ran the
bounded-loader tests against a simulated stalled get_encoding
- Observed result: compression ~118-120ms on both 3.14 and 3.13 (no
event-loop block); the bounded loader raises/falls back within the
timeout instead of hanging
- Not tested: the real firewalled-network stall (could not reproduce on
an unfirewalled host); the no-timeout requests.get is confirmed in
tiktoken's source
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
