mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
17 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
JD Davis
|
f236ef2e31
|
test(ccr): cross SQLite max lifetime boundary (#2794)
## Description
Fixes the failing Rust test on `main` after #2669 made SQLite CCR
entries valid at the exact TTL boundary. The integration test waited
only 3.3 seconds for a three-second ceiling; unix-second truncation can
represent that as exactly three seconds, so the entry is correctly still
valid. The test now crosses a guaranteed four-second elapsed boundary.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Extend the max-lifetime test's access loop from four to five 700 ms
gaps.
- Document why four gaps can land on the valid equality boundary and why
five are deterministic.
- Leave production SQLite TTL behavior and defaults unchanged.
## Testing
- [x] Unit tests pass (`cargo test -p headroom-core --test
ccr_backends`)
- [x] Formatting passes (`cargo fmt --all -- --check`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
cargo test -p headroom-core --test ccr_backends
test result: ok. 12 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
5 consecutive repetitions of the previously failing test:
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 11 filtered out
```
## Real Behavior Proof
- Environment: macOS, Rust workspace at
`
|
||
|
Tejas Chopra
|
3e348f327f
|
fix(ccr): stop persisting retrieval markers as original content (#2694) (#2703)
## Description CCR entries could end up holding a `<<ccr:...>>` marker — or nothing at all — where the original bytes belonged, so `headroom_retrieve(hash)` answered with the very placeholder the caller was trying to resolve. For a base64/credential field that is permanent, silent data loss: the inner marker's hash is the only handle on the real payload, and it disappears from anywhere the model can see. Four sites, one root cause — **a compressed intermediate (or nothing) was stored in place of the source**, the same defect class as #1209 (tag placeholders persisted as originals). Closes #2694 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **`compaction/walker.rs`** — `walk_array` compacted through the store-LESS `compact()`. Opaque cells inside a compacted table got a marker whose payload was **never written**, so retrieval 404'd forever. Now uses `compact_with_store` so the emitted hash resolves. - **`compaction/classifier.rs`** — nothing stopped an already-marked string from being offloaded a second time, which stashed the MARKER as the new entry's "original". Marker-bearing text is our own output, not source content, so it is never classified opaque. One guard at the choke point both the walker and the table compactor share. - **`smart_crusher/crusher.rs`** — on the prose-hook path the row-drop marker hashed and stored rows whose leaves were **already** rewritten (prose compressed, blobs marker-substituted), so retrieving dropped rows returned compressed output. Now hashes and stashes the pre-processing array via `crush_array_with_source`. - **`content_router.py`** — compression pinning matched only `Retrieve more: hash=` / `Retrieve original: hash=`, **not** `<<ccr:`, so opaque-blob output was readmitted to the compressor on a later turn — the path that feeds the corruption above. Consolidated into `_is_already_compressed()` and applied at all three pinning sites. - **`cache/compression_store.py`** — store-level guard: refuse to persist a *bare* marker as `original_content` and log at ERROR, so a future producer regression surfaces loudly instead of silently converting "retrievable" into "gone". Deliberately narrow — originals may legally *contain* markers (nested offloads); only a bare marker is rejected. - **Regression tests** — `test_nested_table_markers_resolve_to_source_bytes` (asserts payloads are verbatim-retrievable, not merely that a marker was emitted) and `test_already_marked_content_is_not_re_offloaded`. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ cargo build -p headroom-core Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 37s $ cargo test -p headroom-core --lib smart_crusher test result: ok. 329 passed; 0 failed; 0 ignored; 0 measured; 583 filtered out; finished in 0.19s $ python -m pytest tests/test_transforms/test_smart_crusher_ccr_roundtrip.py -q 16 passed in 0.68s $ python -m pytest tests/test_ccr_row_drop_store_bridge.py tests/test_ccr_tool_injection.py -q 50 passed in 12.52s $ python -m pytest tests/test_compression_store.py tests/test_lossless_mode.py -q 100 passed in 13.14s $ ruff check headroom/transforms/content_router.py headroom/cache/compression_store.py \ tests/test_transforms/test_smart_crusher_ccr_roundtrip.py All checks passed! $ mypy headroom/transforms/content_router.py headroom/cache/compression_store.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - **Environment:** macOS (Darwin 25.4.0), Python 3.12.6, headroom-ai 0.33.0 editable, `HEADROOM_CCR_BACKEND=memory`, Rust extension rebuilt via `maturin develop --release`. - **Exact command / steps:** compact a nested document — 5 rows whose `detail` field is a stringified sub-array of 6 base64 blobs (1600 B each) — then, for every `<<ccr:HASH>>` marker in the output, call `ccr_get(HASH)` and check the payload is the verbatim source rather than a marker. ```python inner = [{"k": f"key{i}", "v": i, "tok": blob(1200)} for i in range(6)] doc = {"rows": [{"id": i, "detail": json.dumps(inner), "note": "x"} for i in range(5)]} out = SmartCrusher().compact_document_json(json.dumps(doc)) for h in re.findall(r"<<ccr:([0-9a-f]+)", out): payload = crusher.ccr_get(h) # must be real bytes, not a marker ``` - **Observed result — BEFORE (on `main`):** all six payloads collapsed into a single dead marker. The rendered sub-table was re-classified opaque (`html`, because `<<` reads as a tag), offloaded again, and its payload never stored — so the six inner hashes were erased from the visible text *and* the outer hash resolved to nothing. ```text {"rows":"[5]{detail:string,id:int,note:string} <<ccr:3fb1d44933da,html,289B>>,0,x <<ccr:3fb1d44933da,html,289B>>,1,x ..."} 3fb1d44933da -> RUST MISS # unrecoverable — 6 × 1600 B gone ``` - **Observed result — AFTER (this branch):** the sub-table stays inline, each blob keeps its own marker, and every marker resolves to verbatim source. ```text {"rows":"[5]{detail:string,id:int,note:string} \"[6]{k:string,tok:string,v:int} key0,\"\"<<ccr:955b1fed2ef7,base64,1.6KB>>\"\",0 ..."} 6ad5846997f4: resolves, len=1600, is-verbatim-source=True 78a0bd9364a7: resolves, len=1600, is-verbatim-source=True 955b1fed2ef7: resolves, len=1600, is-verbatim-source=True a0cef69da7f0: resolves, len=1600, is-verbatim-source=True dfcde5e940c0: resolves, len=1600, is-verbatim-source=True e57c4e0a3ce8: resolves, len=1600, is-verbatim-source=True RESULT: PASS — every marker resolves to real source bytes ``` ## Notes for reviewers - The issue also reports **function words dropped from retained prose** (`is`, `a`, `the`) and **interleaved log output corrupting `headroom doctor`'s table borders**. Those are separate defects on different paths (extractive prose compression and log-handler buffering respectively) and are **not** addressed here — this PR is scoped to the CCR store/retrieve corruption. They should be tracked separately; the prose one overlaps #2586. - The `crusher.rs` prose-hook fix is on the Rust pipeline (`json_offload`) rather than the Python proxy path, but it is the same store-the-intermediate bug and was cheap to close while in the file. |
||
|
Zhenjia ZHOU
|
e825588bfb
|
fix(ccr): sliding idle-window TTL with max-lifetime ceiling in the Rust core backends (#2604) (#2631)
## Description
Rust-core counterpart of the CCR mid-session expiry fix. #2604 (and its
duplicate #2616) report that the 30-minute wall-clock TTL kills entries
in the middle of a normal multi-agent burst: the clock starts at
compression time and never refreshes, so an entry the session keeps
touching still dies.
#2607 fixes this on the Python side by turning the TTL into an idle
window that restarts on every successful retrieval, bounded by an
absolute max lifetime (8x the idle TTL) — but it explicitly notes the
caveat that the Rust core still measures TTL from insertion. This PR
closes that gap: all three Rust CCR backends (`InMemoryCcrStore`,
`SqliteCcrStore`, `RedisCcrStore`) now use the same sliding idle-window
+ max-lifetime-ceiling semantics as the Python `CompressionStore`.
Scoped to the Rust core only; it deliberately does not touch
`DEFAULT_TTL`'s value (1800), which #2607 bumps to 3600 — happy to
rebase in lockstep whichever lands first.
Refs #2604, #2616. Complements #2607.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `crates/headroom-core/src/ccr/mod.rs`:
`DEFAULT_MAX_LIFETIME_MULTIPLIER = 8` + `max_lifetime_for()` helper;
documents the idle-window semantics.
- `in_memory.rs`: entries track `last_accessed`; a hit refreshes it
under the shard write lock (`get_mut`), expiry checks idle window OR max
lifetime, and the existing `remove_if` TOCTOU protection now uses the
same predicate. New `with_capacity_and_ttls` constructor for independent
control of window and ceiling.
- `sqlite.rs`: new `last_accessed` column (legacy DBs migrated in place
via `ALTER TABLE`, backfilled from `created_at` so old rows keep their
original expiry baseline); lazy purge and the lookup honour both bounds;
a hit touches the row under the same connection mutex as the read. New
`open_with_ttls` constructor.
- `redis.rs`: a hit re-arms the key's expiry, capped by a companion
`{prefix}:{hash}:born` key whose remaining TTL marks the absolute
ceiling; entries written by pre-sliding builds (no born key) are
backfilled rather than dropped.
- `tests/ccr_backends.rs`: 6 new tests — sliding-window survival and
max-lifetime cap for in-memory and SQLite, legacy-schema migration, and
a gated Redis sliding test.
No public API is broken: existing constructors keep their signatures and
derive the ceiling as 8x the idle TTL.
## Testing
- [x] Unit tests pass (`cargo test -p headroom-core`)
- [x] Linting passes (`cargo clippy -p headroom-core --all-features`,
`cargo fmt --check`)
- [ ] Type checking passes (`mypy headroom`) — N/A, no Python files
touched
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ cargo test -p headroom-core --test ccr_backends
test result: ok. 12 passed; 0 failed; 0 ignored (8.12s)
$ cargo test -p headroom-core ccr # all ccr-named tests across suites
38 passed, 949 filtered out (13 suites)
$ cargo test -p headroom-core --test ccr_roundtrip --test live_zone_ccr
18 passed (2 suites)
$ cargo check -p headroom-core --features redis # cfg-gated backend compiles
Finished `dev` profile in 25.09s
$ cargo clippy -p headroom-core --all-features
No issues found
```
## Real Behavior Proof
- Environment: macOS (Darwin 25.5), local checkout at upstream `main`
(
|
||
|
Ruben A.
|
e530de5ad2
|
feat(rust): port CodeCompressor AST compressor to Rust (parity-only) (#1154)
Adds crates/headroom-core/src/transforms/code_compressor.rs (1,882 lines): the AST-aware CodeCompressor ported to Rust on tree-sitter, with grammars for Python, JavaScript, TypeScript, Go, Rust, Java, C and C++. Parity-only, like #1153. Nothing calls it: the only references outside the module are the pub mod / pub use declarations in transforms/mod.rs, and live_zone.rs still routes SourceCode to a no-op. The pyo3 bridge is untouched and no Python source changes, so the engine is unreachable from the shipped package. #1155 wires it into live-zone dispatch. Every grammar is pinned with '=' to the exact version of the corresponding Python tree-sitter-<lang> PyPI wheel. Same version on crates.io and PyPI means the same grammar.js, hence the same generated parser.c, hence node-for-node identical ASTs — the precondition for byte-parity. A canary over 9 samples x 8 languages confirmed identical node-type and line-span trees at these pins; bumping any pin requires re-running it and re-recording the fixtures. Ships 30 recorded parity fixtures, a CodeCompressorComparator in headroom-parity, and scripts/record_code_compressor_fixtures.py. Verified byte-identical to the recorded Python output: [code_aware_compressor] total=30 matched=30 skipped=0 diffed=0 Full harness on the merge result: 227 fixtures, 182 matched, 45 skipped (cache_aligner + ccr stubs), 0 diffed, exit 0 — with kompress at 21/21 under ONNX Runtime 1.24.4 (see #2591). Also verified cargo check -p headroom-core --no-default-features passes, so the static-musl path stays intact. |
||
|
Ruben A.
|
83e27e5036
|
feat(rust): port Kompress ML prose compressor to Rust (parity-only) (#1153)
Adds crates/headroom-core/src/transforms/kompress.rs (672 lines): the Kompress ML prose compressor ported to Rust, running the ModernBERT tokenizer plus the kompress-v2-base ONNX model through ort with a cache-only loader that never touches the network. Parity-only. Nothing calls it: the only references outside the module are the pub mod / pub use declarations in transforms/mod.rs. live_zone.rs still carries TODO(PR-B4), so PlainText dispatch remains a no-op and Python continues to serve prose compression. The pyo3 bridge is untouched and no Python source changes, so the new engine is unreachable from the shipped package. #1155 wires it up. Ships 21 recorded parity fixtures, a KompressComparator in headroom-parity, and scripts/record_kompress_fixtures.py. Verified byte-identical to the recorded Python output: [kompress] total=21 matched=21 skipped=0 diffed=0 That required ONNX Runtime >= 1.24 (see #2591) — below it ort deadlocks instead of erroring, which is why these fixtures had never been run. In CI the model is absent from the HF cache, so the comparator errors and the fixtures report Skipped rather than hanging. Also gates the module behind the ml feature, matching magika_detector: kompress.rs uses ort, which is optional = true, so an unconditional pub mod broke cargo check --no-default-features (the static-musl path). CI does not catch that class of break because cargo test --workspace only builds default features. |
||
|
Rod Boev
|
9e0778553f
|
feat(rust): add structured prose offload plumbing (#334) (#2378)
## Description Structured payloads still leave long prose leaves without a dedicated prose compressor. The Rust pipeline already handles top-level log, diff, search, and JSON-array shapes, and the existing structured recursion rewrites stringified JSON and opaque blobs, but a plain prose string leaf inside structured content still falls back to generic opaque long-string handling instead of query-aware extractive compression. That wastes prompt budget on fields like `summary`, `description`, and `analysis` even though `headroom-core` already ships the deterministic, query-aware `TextCrusher`. This PR adds a bounded prose-field path for structured leaves. It introduces a reusable `ProseFieldOffload` backed by `TextCrusher`, then wires that offload into `JsonOffload`'s structured recursion with conservative byte and segment thresholds. Only detector-confirmed `PlainText` leaves are eligible. When a leaf clears those gates and the marker-inclusive output still saves bytes, the exact original leaf is written to CCR and the inline output carries a prose marker keyed to that store entry. Short prose, low-segment prose, diff-shaped strings, stringified JSON, and opaque base64 or HTML keep their existing behavior. This stays inside the Rust transform stack. It does not add a PyO3 shim, ONNX runtime, live-zone prose handling, or any new Python dependency. It also keeps the existing wrapper-level `JsonOffload` CCR entry, so the full structured payload remains recoverable as before. ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Add `ProseFieldOffload` as a `ContentType::PlainText` pipeline offload backed by `TextCrusher`, with conservative byte, segment, and target-ratio thresholds. - Thread the prose offload into the structured `JsonOffload` recursion so nested prose leaves can compress and recover through the orchestrator store. - Add a pipeline-aware `JsonOffload::from_pipeline` constructor so `offload.prose_field` overrides actually reach the live prose hook instead of falling back to embedded defaults. - Preserve current behavior for short prose, low-segment prose, diff-shaped leaves, stringified JSON containers, and opaque base64 or HTML leaves. - Add focused config, routing, determinism, and CCR roundtrip coverage for the new prose path. - Leave changelog generation to the repo's conventional-commit release flow rather than editing `CHANGELOG.md` directly. ## Testing - [x] Unit tests pass (`cargo test -p headroom-core --lib transforms::pipeline::offloads::prose_field::tests`) - [x] Linting passes (`cargo clippy -p headroom-core -- -D warnings`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [ ] Manual testing performed ### Test Output ```text cargo fmt --all -- --check cargo clippy -p headroom-core -- -D warnings cargo test -p headroom-core --lib transforms::pipeline::offloads::prose_field::tests test result: ok. 6 passed; 0 failed cargo test -p headroom-core --lib transforms::pipeline::offloads::json_offload::tests test result: ok. 17 passed; 0 failed cargo test -p headroom-core --lib transforms::smart_crusher::crusher::tests::default_crush_ignores_opt_in_prose_hook -- --exact test result: ok. 1 passed; 0 failed cargo test -p headroom-core --lib transforms::smart_crusher::crusher::tests::prose_hook_preserves_html_opaque_routing -- --exact test result: ok. 1 passed; 0 failed cargo test -p headroom-core --lib transforms::smart_crusher::crusher::tests::prose_hook_runs_for_dict_array_rows -- --exact test result: ok. 1 passed; 0 failed cargo test -p headroom-core --lib transforms::smart_crusher::crusher::tests::unchanged_stringified_json_container_skips_prose_hook -- --exact test result: ok. 1 passed; 0 failed cargo test -p headroom-core --test ccr_roundtrip nested_structured_prose_leaf_uses_ccr -- --exact test result: ok. 1 passed; 0 failed git diff --check ``` ## Real Behavior Proof - Environment: Windows 11, stable Rust toolchain, in-memory CCR store, no live provider - Exact command / steps: run the focused nested CCR roundtrip test through `CompressionPipeline::run` on a five-row structured payload containing a long prose leaf, then resolve the emitted prose marker key from the same orchestrator store - Observed result: the generic `CompressionPipeline` plus `JsonOffload` path applies, the nested prose leaf becomes shorter on the wire, and that prose key retrieves the byte-identical original leaf from the orchestrator store while HTML-shaped and diff-shaped leaves stay on their opaque marker routes - Not tested: live provider run ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - The upstream issue body originally parked PR3b behind a PyO3 shim or a later ONNX port. This PR takes the narrower Rust-native path instead by reusing the existing `TextCrusher` already in `headroom-core`. - This PR advances the pipeline-side PR3b slice from #334. It does not close #334, and it does not wire live-zone or PyO3 SmartCrusher callers to this path. - `CHANGELOG.md` is intentionally untouched because this repo's release pipeline generates changelog entries from conventional commits, and `repos/headroom/config.md` marks manual changelog edits as out of policy. - Python lint, type checking, and pytest are not part of the focused local proof for this slice because the change stays inside `crates/headroom-core`. |
||
|
Abhay Singh
|
8cddf9b58e
|
fix(proxy/auth): match real Anthropic OAuth token prefix (sk-ant-oat) (#1672)
## Description
`classify_auth_mode` (in `headroom/proxy/auth_mode.py`) checks for
Anthropic
OAuth tokens with:
```python
if token.startswith("sk-ant-oat-"):
return AuthMode.OAUTH
if token.startswith("sk-ant-api") or token.startswith("sk-"):
return AuthMode.PAYG
```
But real Anthropic OAuth access tokens are **`sk-ant-oat01-...`** — a
version
number right after `oat`, **no dash**. So the `sk-ant-oat-` check never
matches a
real token; it falls through to the broad `sk-` rule and gets classified
**`PAYG`**.
That's exactly the misclassification the module is built to prevent: a
subscription/OAuth-bound request tagged `PAYG` gets the
aggressive-compression
policy — lossy compression, auto `cache_control`, `prompt_cache_key`
injection —
instead of the passthrough-prefer path OAuth is meant to get.
The existing tests didn't catch it because they use a synthetic
`sk-ant-oat-01-`
fixture (dashed) that happens to match the buggy prefix. Corroboration
that the
real shape is dash-less:
- `.gitguardian.yaml` fixture: `sk-ant-oat01-oauth-fixture`
- `tests/test_oauth_bearer_routing.py`: `sk-ant-oat01-xxx`
- the sibling helper `headroom/proxy/helpers.py` matches on `sk-ant-`
(no `oat-`)
## Fix
Match the dash-less `sk-ant-oat` prefix. It still matches the legacy
dashed
shape, and ordering relative to `sk-ant-api` / `sk-` is unchanged (OAuth
is
still checked first).
```python
if token.startswith("sk-ant-oat"):
return AuthMode.OAUTH
```
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/proxy/auth_mode.py`: match OAuth tokens on the dash-less
`sk-ant-oat` prefix.
- `tests/test_auth_mode.py`: add a regression test using the real
`sk-ant-oat01-...` format (the existing test keeps the legacy dashed
fixture, which still classifies correctly).
- `CHANGELOG.md`: Bug Fixes entry under Unreleased.
## Testing
- [x] New regression test added (`tests/test_auth_mode.py`)
- [x] Linting passes (`ruff check`) and formatting is clean (`ruff
format --check`)
- [ ] Full `pytest` deferred to CI (local-OOM reason below).
```text
$ uv run ruff check headroom/proxy/auth_mode.py tests/test_auth_mode.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.12.11, headroom from this branch.
Importing `headroom` loads the torch/transformers stack and a full
`pytest` gets OOM-killed on this box, so I verified the classification
logic with a dependency-free script and left the full pytest to CI.
- Exact command / steps: replicated the Bearer-token branch of
`classify_auth_mode` in a standalone script (only stdlib, no `headroom`
import) and ran the real and legacy token shapes plus PAYG keys through
it.
- Observed result: the real `sk-ant-oat01-...` now classifies OAUTH (was
PAYG before the change); the legacy dashed fixture still classifies
OAUTH; `sk-ant-api*` / `sk-*` keys still classify PAYG:
```text
OK: sk-ant-oat01-... -> OAUTH (was PAYG before fix)
OK: sk-ant-oat-01-... -> OAUTH (legacy fixture still matches)
OK: sk-ant-api* / sk-* -> PAYG (unchanged)
AUTH LOGIC VERIFIED
```
- Not tested: a live proxied Anthropic OAuth request end-to-end (needs a
real subscription token); the classification is pure and covered by the
regression test. Full local `pytest` deferred to CI (OOM, per above).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes — ran
lint + a standalone logic check; full pytest deferred to CI (local OOM,
disclosed above)
- [x] I have updated the CHANGELOG.md if applicable
## Additional Notes
- `crates/headroom-core/src/auth_mode.rs` carries the identical dashed
prefix (its Rust test matrix uses the same synthetic dashed fixture). I
scoped this PR to the Python runtime classifier since that's the
request-time path; happy to mirror the one-line fix in Rust in the same
PR or a follow-up — I just couldn't `cargo build` locally to verify, so
I left it out rather than push an unverified Rust edit.
- @JerrettDavis tagging you since you've been triaging these — small,
contained fix with a regression test if you have a moment.
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
|
||
|
Tejas Chopra
|
17ffae0cd8 |
fix: clear CI mypy + rust test failures introduced in eaf5980
compression_units.py:
- Replace dict-unpacking pattern with dataclasses.replace() so mypy can
type-check fields. The `**base` form forced mypy to infer
`dict[str, object]`, which doesn't satisfy the per-field types of
UnitCompressionResult (46 arg-type errors).
- Use `isinstance(candidates, Iterable)` for the transform-iteration
guard. The previous `iter()` call had a `# type: ignore[arg-type]`
that was misclassified — mypy actually emits `call-overload` here.
live_zone_thresholds.rs:
- Update the JsonArray threshold assertion from 1024 to 512 to match
the new constant.
|
||
|
chopratejas
|
ca9de93cfc |
fix: PR-F1 classify_auth_mode helper (Phase F kickoff)
Add the classify_auth_mode helper that maps inbound request headers to one of three auth modes — Payg / OAuth / Subscription — at request entry. The mode is the first-class policy axis Phase F's remaining PRs (F2 cache+lossy gates, F3 TOIN per-tenant aggregation, F4 X-Forwarded-* skip) gate behavior on. Detection rules (most-specific signal wins): - Subscription UA prefix in user-agent → Subscription - Bearer sk-ant-oat-* → OAuth (Claude Pro/Max) - Bearer sk-ant-api* / Bearer sk-* → Payg - Bearer <jwt> (3 dot-segments) → OAuth (Codex/Cursor/Copilot) - Authorization present but not Bearer (AWS SigV4) → OAuth (Bedrock) - x-api-key / x-goog-api-key → Payg - Default → Payg Hard constraints met: pure function, no regex, no silent fallback (non-UTF-8 headers warn! and fall through), no hardcoded list (UA prefixes in module-scope const ready to swap for config in a follow-up). Files: - crates/headroom-core/src/auth_mode.rs (new) — Rust impl - crates/headroom-core/tests/auth_mode.rs (new) — 14 unit + 1 perf - crates/headroom-core/benches/auth_mode.rs (new) — Criterion bench - crates/headroom-core/Cargo.toml — add http dep + bench entry - crates/headroom-core/src/lib.rs — pub mod auth_mode - crates/headroom-proxy/src/proxy.rs — classify at request entry, store in extensions, log event=auth_mode_classified - headroom/proxy/auth_mode.py (new) — Python port (parity) - headroom/proxy/handlers/anthropic.py — wire into messages handler - headroom/proxy/handlers/openai.py — wire into chat + responses - tests/test_auth_mode.py (new) — 23 Python parity tests - docs/auth-modes.md (new) — detection rules + how-to-extend Tests: 15 Rust + 23 Python all green. cargo fmt + clippy + workspace tests + ci-precheck all green. Performance (criterion, M-series): - auth_mode/classify/empty: 68 ns - auth_mode/classify/payg_anthropic_api_key: 75 ns - auth_mode/classify/oauth_jwt: 182 ns - auth_mode/classify/subscription_claude_code: 81 ns All paths well under the <10us budget (~50-150x headroom). Refs: REALIGNMENT/08-phase-F-auth-mode.md PR-F1. |
||
|
chopratejas
|
00902b8fea |
fix: B7 — CCR hardening: persistent backends + always-on tool
P2-25, P2-26: CCR (Compress-Cache-Retrieve) used an in-memory store
that fragmented across uvicorn workers and was wiped on restart, and
the `headroom_retrieve` tool was registered/unregistered per-request
based on whether the latest body happened to contain compression
markers — every flip busted the prompt cache. Both are sticky
side-channels: once a session has done CCR, the tool list bytes and
the retrieval store must stay stable. This PR fixes both.
Rust:
* Split `ccr.rs` into `ccr/` with `backends/` submodule
(`in_memory.rs`, `sqlite.rs`, `redis.rs` cfg-gated).
* `SqliteCcrStore` (production default): WAL mode, prepared upsert,
lazy TTL purge on read, persistent across worker restarts and
shareable across workers on the same host via SQLite file locking.
* `RedisCcrStore` (cfg-gated behind `feature = "redis"`): SETEX with
startup PING smoke-test, no key-prefix collision risk, no sticky
session required at the LB.
* `CcrBackendConfig::{InMemory, Sqlite, Redis}` + `from_config(...)`
factory — every init failure surfaces (no silent fallback per
`feedback_no_silent_fallbacks.md`).
* `ccr::compute_key` (BLAKE3 → first 24 hex chars) and
`ccr::marker_for("HASH") -> "<<ccr:HASH>>"` centralize the hash +
marker format; one definition for the live-zone dispatcher and the
Python regex (`headroom/ccr/tool_injection.py:211`).
* `compress_anthropic_live_zone_with_ccr` accepts
`Option<&dyn CcrStore>`. When wired, every accepted compression
puts the original bytes into the backend and appends `<<ccr:HASH>>`
to the compressed string. The token-validation gate runs on the
marker-augmented string so the `compressed_tokens >=
original_tokens` rejection stays honest.
Python:
* `SessionCcrTracker` + `apply_session_sticky_ccr_tool` mirror the
PR-A7 `SessionToolTracker` / `apply_session_sticky_memory_tools`
pattern: once a session has done CCR, every subsequent request
injects the recorded golden tool-definition bytes. Tool list bytes
are byte-stable across turns (snapshot test pins them).
* `headroom/ccr/tool_injection.py::inject_tool_definition` accepts a
new `session_has_done_ccr` kwarg per the PR-B7 spec change at line
302-328. The legacy per-request path stays intact for callers that
don't yet thread a session id (e.g. Google handler).
* Anthropic + OpenAI handlers route their CCR tool-list updates
through `apply_session_sticky_ccr_tool`, keyed off the existing
`session_tracker_store.compute_session_id(...)` plumbing.
Backend selection model: `CcrBackendConfig::Sqlite { path }` is the
production default — single host, persistent, multi-worker safe with
sticky session. `CcrBackendConfig::Redis { url }` is the multi-host
scale-out option — no stickiness needed. `InMemory` is for tests
and single-worker dev only. RUST_DEV.md "Multi-worker deployment —
CCR fragmentation" rewritten around this matrix.
Tests:
* `crates/headroom-core/tests/ccr_backends.rs` — 7 tests covering
SQLite round-trip, TTL purge, proxy-restart survival, cross-backend
byte-equal keys, `from_config` paths, and the no-redis-feature
loud-failure check (+ 2 redis tests gated behind the feature).
* `crates/headroom-core/tests/live_zone_ccr.rs` — confirms
`<<ccr:HASH>>` marker injection, store population, and
no-marker-when-no-store invariants end-to-end.
* `tests/test_ccr_tool_always_on.py` — 12 tests pinning the
always-on behaviour, session/provider isolation, LRU bound, no-
session-id fallback, and (per-acceptance-criterion) the byte-stable
tool-definition snapshot.
Per-PR-B7 plan: REALIGNMENT/04-phase-B-live-zone.md.
|
||
|
chopratejas
|
6819b7e5e5 |
fix: B5 — TOIN observation-only refactor + per-tenant aggregation key
Retire the request-time hint API. PR-B5 splits TOIN into two phases:
1. Observation: TOIN keeps recording compressions/retrievals at runtime,
but `get_recommendation()` is deprecated and now returns None.
2. Publish-then-load: the new `headroom.cli.toin_publish` CLI walks the
on-disk store and emits `recommendations.toml`. The Rust proxy reads
that file once at startup via `transforms::recommendations` and
exposes `get(auth_mode, model, structure_hash) -> Option<&Rec>`.
PR-F3 will wire the loader into the live-zone dispatcher.
Per-tenant aggregation: `_patterns` is now keyed by
`(auth_mode, model_family, sig_hash)` so PAYG/OAuth/subscription tenants
no longer share buckets. Callers that don't supply auth/model land in the
`("unknown", "unknown", sig_hash)` slot. Added `_make_pattern_key` helper
+ updated tests that previously indexed by raw `structure_hash`.
AuthMode is canonical in `transforms::live_zone`; `transforms::recommendations`
re-exports it (no duplicate enum). Live-zone enum gained `Unknown`,
`as_str()`, and `Hash` derive to serve recommendations callers without a
second source of truth.
Why: per-request hint calls coupled output to mutable TOIN state, breaking
prompt-cache stability across runs (P2-27, P5-56). Pulling advice into a
startup-published TOML keeps per-request output deterministic and lets the
deploy pipeline gate publication independently of proxy uptime.
Per-PR-B5 plan: REALIGNMENT/04-phase-B-live-zone.md.
|
||
|
chopratejas
|
b3b3feff6f |
fix: B4 — token validation gate + per-content-type byte thresholds
Eliminate P3-33 / P3-34. Wraps every per-block compression in
the live-zone dispatcher with two new gates:
1. Per-content-type byte thresholds — pinned as `const` at the top
of `live_zone.rs` so the table is grep-able and reviewable in
one place. No magic numbers anywhere in the dispatch logic; a
`threshold_for(ContentType)` helper returns the value. Below
threshold → no compressor invoked, recorded as
`BlockAction::BelowByteThreshold { content_type, byte_count,
threshold_bytes }`. Thresholds:
- JSON-array tool_results: 1 KiB
- Build / log output: 512 B
- Search-result blocks: 1 KiB
- Git-diff blocks: 1 KiB
- Source code: 2 KiB (pinned for the future
Rust code-compressor port)
- Plain text: 5 KiB (pinned for Kompress wiring)
- HTML: 5 KiB (no compressor today)
2. Tokenizer-validated rejection — the byte-length proxy
(`compressed_bytes >= original_bytes`) is replaced with a
token-count check using `headroom_core::tokenizer::get_tokenizer`.
The dispatcher creates one tokenizer per request (model-aware
via the new `model: &str` parameter to
`compress_anthropic_live_zone`) and counts both the original
and compressed text. When `compressed_tokens >= original_tokens`
the candidate is rejected and the original bytes are kept.
`BlockAction::Compressed` and `BlockAction::RejectedNotSmaller`
gain `original_tokens` and `compressed_tokens` fields so the
proxy can log token-savings (the currency that actually matters
for prompt cache + provider billing) instead of bytes.
The proxy `live_zone_anthropic.rs` extracts `body["model"]` (or
falls back to `DEFAULT_MODEL = "claude-3-5-sonnet-20241022"` when
the field is missing — the chars-per-token estimator is calibrated
for the Claude family at 3.5 cpt) and threads it through. The
`Compressed` outcome now reports token counts from the manifest,
not byte counts, so the existing
`tokens_before / tokens_after` plumbing is suddenly accurate.
Tests added:
- `live_zone_thresholds.rs::below_threshold_no_compression_attempted`
— 200 B JSON array → `BelowByteThreshold` and `NoChange`.
- `live_zone_thresholds.rs::above_threshold_compression_attempted`
— 10 KB JSON array → byte-threshold gate clears and a compressor
runs (either `Compressed` or `RejectedNotSmaller`).
- `live_zone_token_validation.rs::compressed_more_tokens_falls_back`
— pathological input must not produce `Compressed` with
`compressed_tokens >= original_tokens`.
- `live_zone_token_validation.rs::compressed_fewer_tokens_accepted`
— well-formed JSON array of dicts → `Compressed` with strict
token shrinkage.
- Property test `live_zone_compression_token_count_non_increasing`
— for any well-formed body generated by `proptest`, the
dispatcher's emitted body has token-count <= input's token-count.
Pins the central PR-B4 invariant: the dispatcher never inflates
tokens.
Existing 12 unit tests in `live_zone.rs` and 6 integration tests
in `tests/live_zone_dispatch.rs` updated for the new field shape
and the `model` parameter; all pass. The diff-routing test's
fixture grew to 1.3 KiB so it clears the new GitDiff threshold
gate, exercising the dispatch path rather than short-circuiting.
Per-PR-B4 plan: REALIGNMENT/04-phase-B-live-zone.md.
|
||
|
chopratejas
|
2b55050a65 |
fix: B3 — wire type-aware compressors into live-zone dispatcher
Phase B step 3: replace PR-B2's no-op dispatcher with real per-block
compression. SmartCrusher / LogCompressor / SearchCompressor /
DiffCompressor are wired behind content-type detection. SourceCode
and PlainText remain no-op for now (Rust code-compressor port and
Kompress prose compressor land in follow-up work; they're explicit
TODOs in `dispatch_compressor`).
# What's wired
For each block in the latest user message (live zone):
| Detected type | Compressor | Strategy tag |
|---------------|------------------|--------------------|
| `JsonArray` | SmartCrusher | `smart_crusher` |
| `BuildOutput` | LogCompressor | `log_compressor` |
| `SearchResults` | SearchCompressor | `search_compressor` |
| `GitDiff` | DiffCompressor | `diff_compressor` |
| `SourceCode` | (no-op, Rust port pending) | |
| `PlainText` | (no-op, PR-B4 wires Kompress) | |
| `Html` | (no-op, no compressor) | |
Anthropic-specific block types (`tool_use`, `thinking`,
`redacted_thinking`, `compaction`) stay tagged `BlockAction::Excluded`
so they remain in the cache hot zone even when they appear in the
live-zone message.
# Cache-safety invariant — byte-range surgery
The PR replaces "deserialize → mutate → serialize" with byte-range
surgery: the dispatcher uses `serde_json::value::RawValue` borrowed
slices and pointer arithmetic to recover each block's exact byte
offset in the input buffer, then splices replacement bytes
in-place. Bytes outside any rewritten range are *literally copied*
from the input, never re-serialized.
The new integration test
`crates/headroom-core/tests/live_zone_dispatch.rs::byte_fidelity_outside_compressed_block`
pins this in CI: SHA-256 of `body[..block_start]` and
`body[block_end..]` must equal the input's, AND the block must
shrink by >2× on a 50 KB JSON-array tool_result.
# Provider scope (Phase B is Anthropic-only)
The entry point is renamed `compress_live_zone` →
`compress_anthropic_live_zone` to make scope explicit. OpenAI Chat
Completions, OpenAI Responses, and Google Gemini each need their
own dispatcher because the request shapes diverge: OpenAI puts
tool results in `role: "tool"` messages (not nested in user),
Responses uses `input` with `function_call_output` items, Gemini
uses `contents`/`parts`/`function_response`. Phase C
(`REALIGNMENT/05-phase-C-rust-proxy.md`) introduces those
dispatchers; they share `LiveZoneOutcome`, `BlockAction`,
`CompressionManifest` and the per-content-type compressor backend
from this module.
# BlockAction taxonomy (replacing PR-B2's `NoOpSkeleton`)
- `Compressed { strategy, original_bytes, compressed_bytes }` —
compressor ran and produced strictly smaller output; spliced in.
- `RejectedNotSmaller { strategy, original_bytes, compressed_bytes }`
— compressor ran but didn't shrink; original kept. PR-B4 swaps
this byte-length proxy for a tokenizer-validated count.
- `CompressorError { strategy, error }` — compressor failed loudly.
Per project memory `feedback_no_silent_fallbacks.md`, surfaced in
the manifest; proxy logs warn-level and forwards original bytes
for that block; other blocks in the same body still compress.
- `NoCompressionApplied { content_type }` — content type has no
applicable compressor (PlainText, SourceCode, Html, Image,
Unknown). Replaces PR-B2's `NoOpSkeleton` as the default.
- `Excluded { reason }` — block intentionally outside live zone
(HotZoneBlockType, BelowFrozenFloor, AboveLiveZone).
# Sequential per-block dispatch (parallelism deferred)
Per-block compression is sequential in B3. Most requests have 1-3
blocks in the latest user message; the rayon/spawn_blocking
overhead approaches the savings below ~4 blocks. PR-B4 will add
async coordination per block (since token validation needs an
async hop anyway) — that's the natural place to add parallelism
guarded by a benchmark-driven threshold.
# Observability
The proxy log line gains the new fields when bytes are rewritten:
- `decision="compressed"`, `reason="live_zone_blocks_rewritten"`
- `body_bytes_in`, `body_bytes_out`, `bytes_freed`
- `live_zone_strategies` (Vec of unique strategy tags)
- `live_zone_block_original_bytes`, `live_zone_block_compressed_bytes`
The PR-B2 `decision="no_change"` arm is preserved with
`reason="no_block_compressed"`.
# Files
- `crates/headroom-core/src/transforms/live_zone.rs` (≈1100 LOC,
+900 from B2): byte-range surgery; `dispatch_compressor` switch;
`OnceLock` singletons for SmartCrusher / Log / Search / Diff;
expanded `BlockAction` enum.
- `crates/headroom-proxy/src/compression/live_zone_anthropic.rs`:
translates `LiveZoneOutcome::Modified` → `Outcome::Compressed`
with aggregated manifest counters.
- `crates/headroom-core/tests/live_zone_dispatch.rs` (NEW):
routing tests + 50 KB byte-fidelity invariant test.
- `crates/headroom-proxy/tests/integration_compression.rs`: log
contract updated to `reason="no_block_compressed"`.
# Acceptance
- `cargo build --workspace` + `clippy` + `fmt` green.
- `cargo test --workspace --exclude headroom-py`: 881 passed.
- 6 new integration tests in `live_zone_dispatch.rs`:
json/log/diff routing, source-code no-op, unknown no-op,
byte-fidelity (50 KB → >2× reduction with byte-equal envelope).
- Existing 12 unit tests in `live_zone.rs` still pass.
Per-PR-B3 plan: REALIGNMENT/04-phase-B-live-zone.md.
|
||
|
chopratejas
|
3f99128236 |
fix(rust): A4 — honor cache_control markers; serde_json arbitrary_precision + raw_value
PR-A4 of the Realignment Phase A lockdown
(REALIGNMENT/03-phase-A-lockdown.md). Eliminates P0-3 (Rust proxy
ignores customer cache_control markers) and P0-5 (numeric precision
lost via serde_json::Value round-trip) at the library level; Phase B
PR-B2 wires the helper into the live-zone block dispatcher.
Cargo.toml — add `arbitrary_precision` and `raw_value` to
`serde_json` workspace features. `arbitrary_precision` keeps `1.0`
from collapsing to `1` and preserves >2^53 integers; `raw_value`
exposes `&RawValue` so PR-B2 can forward unmodified `messages[*]`
entries as exact byte copies.
crates/headroom-core/src/cache_control.rs (new) — `compute_frozen_count`
walks `messages[i].content[*].cache_control` via serde_json
accessors only (no regex) and returns the smallest N such that
`messages[i]` is frozen for every i < N. Markers in `system` or
`tools[*]` log at debug! but never bump the floor (those fields are
unconditionally cache-hot per invariant I2). TTL ordering violations
(5m before 1h, guide §2.19) emit `tracing::warn!` but the function
computes the correct count regardless — the customer's request, not
ours to reject.
crates/headroom-core/src/lib.rs — re-export `compute_frozen_count` at
crate root so the proxy crate has a stable import path.
crates/headroom-proxy/src/compression/anthropic.rs — add
`resolve_frozen_count` thin wrapper that consults the
`cache_control_auto_frozen` config flag. When `disabled`, returns 0
regardless of body content (operator opt-out for benchmarking).
crates/headroom-proxy/src/config.rs — add `CacheControlAutoFrozen`
enum and the matching CLI flag `--cache-control-auto-frozen` /
env var `HEADROOM_PROXY_CACHE_CONTROL_AUTO_FROZEN`. Default is
`enabled`. Documented in the doc comments.
Tests
- crates/headroom-core/src/cache_control.rs (inline): 11 unit tests
covering marker detection, system/tools negative cases, ordering
state machine, defensive (missing fields, non-array messages,
non-object content blocks).
- crates/headroom-core/tests/cache_control.rs: 11 unit + 3 property
tests (monotonic non-decrease as markers are added; system/tools
markers don't change count; empty messages → 0).
- crates/headroom-proxy/tests/integration_cache_control.rs: 8 tests
exercising the proxy wrapper (configurability gate; tracing
capture for the 5m-before-1h warn path).
Acceptance gates: `cargo build --workspace`, `cargo test --workspace`
(33 new tests green), `cargo clippy --workspace -- -D warnings`,
`cargo fmt --all --check` all clean. No new `regex::` imports;
`git grep -n 'regex::' crates/{headroom-core/src/cache_control.rs,
headroom-core/tests/cache_control.rs, headroom-proxy/tests/
integration_cache_control.rs}` empty.
Honors the realignment build constraints: configurable (CLI + env),
no hardcodes (TTL strings live as const), no regex (serde_json
accessor walk), no fallbacks (one impl), structured logging
(debug!/warn! with field/index/ttl/rule context), tests
comprehensive (unit + property + integration + tracing capture).
|
||
|
chopratejas
|
da7716a95a |
chore(rust): SmartCrusher CCR marker injection + walker unification
Closes four gaps in the Rust SmartCrusher pipeline that, together,
wire CCR storage end-to-end so the LLM can actually retrieve dropped
data:
1. CCR-Dropped marker is now injected into process_value's lossy-path
output as a sentinel object {"_ccr_dropped": "<<ccr:HASH N_rows_offloaded>>"}
appended to the kept-items array. Previously the store held the
original but no pointer reached the prompt -- the retrieval contract
was data-on-server, no-way-to-ask. Sentinel-as-object preserves the
array-of-dicts shape so downstream iteration with x.get(...) keeps
working.
2. Walker / process_value drift removed. process_value gains a
Value::String arm that handles stringified-JSON containers (parse,
recurse, re-encode) and opaque blobs (CCR marker + store) -- same
semantics walker.rs has always had, now reachable from the main
crush() pipeline.
3. Opaque-string CCR now stores originals. DocumentCompactor gains an
Option<Arc<dyn CcrStore>> field; emit_opaque_ccr_marker calls
store.put when one is configured. Same hash regardless of store
presence -- runtime contract is stable across configurations.
Same wiring is shared between walker.rs and process_value via the
extracted helper.
5. PyO3 surface adds SmartCrusher.compact_document_json(doc_json) ->
compacted-json string. Routes through the crusher's existing CCR
store, so ccr_get resolves both row-drop and opaque-string hashes.
Tests:
- 5 new Rust integration tests in ccr_roundtrip.rs (marker visibility,
nested-array marker, opaque-string roundtrip, stringified-JSON
recursion, walker-with-store)
- 4 new Python tests covering the marker visible-to-LLM contract via
both the native PyO3 surface and the Python shim
- 5 legacy parity fixtures re-recorded (dict_array_*, duplicate_dicts_40)
-- their lossy outputs now carry the sentinel; Rust + Python both
match the new bytes (parity-run smart_crusher: 17/17)
|
||
|
chopratejas
|
22c8fec4c1 |
chore(rust): SmartCrusher CCR storage layer + roundtrip verification
CcrStore trait + InMemoryCcrStore (1000 entries, 5-min TTL, FIFO eviction, idempotent re-store) live at the crate root. SmartCrusher's lossy crush_array path now actually stashes the full original [items] canonical-JSON into the configured store keyed by the same ccr_hash it embeds in the prompt marker -- closing the no-data-loss contract that was previously hash-only. PyO3 surface: - crusher.crush_array_json(items_json) -> dict with ccr_hash + kept items - crusher.ccr_get(hash) -> Optional[str] for retrieval - crusher.ccr_len() -> int for telemetry Python shim passes both through. Default constructors enable the store (matches Python's CCR-enabled default); without_compaction() also gets it because CCR is a contract, not an opt-in extra. Tests proving compress -> store -> retrieve -> reconstruct: - 7 unit tests in ccr.rs (put/get/eviction/expiry) - 9 Rust integration tests (crates/headroom-core/tests/ccr_roundtrip.rs) - 10 Python tests including 4 explicit before/after element-equality assertions through both the native PyO3 surface and the Python shim Plugin manifest versions auto-bumped by the sync-plugin-versions pre-commit hook (unrelated to CCR but co-resident in the working tree). |
||
|
chopratejas
|
9ce1c01b87 |
feat(rust): tokenizer crate with tiktoken-rs + HuggingFace + estimator
Stage 2 of the Rust port: a `headroom_core::tokenizer` module mirroring the Python `headroom.tokenizers` surface, with three backends behind a single `Tokenizer` trait. Backends, in dispatch order: 1. HuggingFace (`HfTokenizer`) — pure-Rust `tokenizers` crate loading any public `tokenizer.json`. Covers the gap between OpenAI (tiktoken) and the Anthropic/Gemini estimator: Cohere `command-*`, Llama-3.x, Mistral, Qwen, BERT, T5, etc. Construct from bytes or a file path; register against a model-name prefix via `register_hf` for automatic dispatch. No `hf-hub` auto-download yet — keeps networking, auth, and `~/.cache/huggingface` out of core. Longest-prefix wins; lookups are RwLock-protected. 2. Tiktoken (`TiktokenCounter`) — `tiktoken-rs` 0.11 BPE for OpenAI / o-series families. Byte-identical to Python `tiktoken` for ordinary text. Lazy shared `Arc<CoreBPE>` per encoding (o200k_base, cl100k_base, p50k_base, r50k_base). 3. Estimation (`EstimatingCounter`) — `chars / cpt` last-resort fallback. Matches Python's `max(1, int(len(text) / cpt + 0.5))` round-half-up formula (a self-review caught and fixed an earlier `ceil`-based version that diverged in the middle of the range, e.g. 5 chars at 4.0 cpt). Tests: 43 unit tests + 5 proptests; parity 40/40 byte-equal. Bench: criterion baseline on small/medium/large inputs. Workspace MSRV bumped 1.78 → 1.80 for `LazyLock`/`OnceLock`. No proxy wiring. Library-only; production behavior unchanged. |