mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
3 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Ehsan
|
f4070c44cb
|
fix(transforms/cross-turn-dedup): don't renumber-fold zero-padded line prefixes (#2369)
## Description
On an HTTP tool-output re-read, `cross_turn_dedup` folds a contiguous
span that
already appeared in an earlier block into a compact pointer, and when
the line
numbers shifted by a constant it carries the offset as a `delta` so the
original
bytes recover as `int(number) + delta`. The module states this renumber
path is
"strictly lossless" for UNPADDED numbers only.
`_LINENO_RE = ^(\d+)(:|\t)(.*)$` does not enforce the "unpadded"
restriction: `\d+`
also matches a LEADING-ZERO prefix. A timestamped log row such as
`08:00:01 ...`
is read as line number `8`, not as data, so a re-read shifted by a
constant (a
later window of the same hourly log) folds under a uniform delta.
Recovery then
renders `str(int("08") + 1)` = `"9"`, not `"09"`: the round-trip is not
byte-exact. This is a lossy (false-positive) fold in a module whose
stated
posture is to prefer false negatives (`CONTRIBUTING.md:129`,
`cross_turn_dedup.py:45-50`).
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `headroom/transforms/cross_turn_dedup.py`: restrict `_LINENO_RE` to
`[1-9]\d*`
so a leading-zero run stays non-numbered and can fold only on an EXACT
match
(delta 0), never under a lossy renumber. Real `grep -n` / `sed -n` / `rg
-n`
numbers never carry a leading zero, so the intended renumber-fold
feature is
unchanged. Added a comment stating why the character class is
load-bearing.
- `tests/test_cross_turn_dedup.py`: added a delta-aware reconstruction
helper and
three regression tests (the existing `_reconstruct` asserts delta is
absent, so
it never exercised the numbered path this bug lives on).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
Three named scenarios, one test each:
1. `test_zero_padded_prefix_not_folded_lossily`: a padded shifted
re-read is left
verbatim (`spans_folded == 0`). This fails on `main` (it folds under a
delta).
2. `test_unpadded_renumber_still_folds_and_recovers_exactly`: an
unpadded `grep -n`
read renumbered by `+5` still folds and reconstructs byte-exact (feature
guard).
3. `test_padded_content_exact_redisplay_still_folds`: the same padded
rows
re-displayed verbatim still fold with delta 0 (surgical-scope guard).
### Test Output
```text
--- ruff check ---
All checks passed!
--- ruff format --check ---
2 files already formatted
--- mypy ---
Success: no issues found in 1 source file
--- pytest: 3 new tests on the branch (fixed) ---
3 passed, 14 deselected
--- pytest: revert regex to \d+ (simulate main): the regression test must FAIL ---
1 failed
```
## Real Behavior Proof
- Environment: clean `python:3.12-slim` Docker, `PYTHONPATH` at the
source tree,
core deps installed by name (tiktoken, pydantic, litellm, click, rich,
opentelemetry-api, pyyaml, tomlkit), `ruff==0.15.17`, `mypy==1.20.2`.
- Exact command / steps: import the module and print provenance, then
run ruff,
ruff format, mypy on the two changed files, then `pytest` the three new
tests
on the branch, then revert only the regex to `\d+` and re-run the
regression
test.
- Observed result: module `cross_turn_dedup.py` (sha256
7ff204b65f40617d505895841aa1cfc1ee54bf63ffe6520198f0aa05a850aa8d), regex
now `^([1-9]\d*)(:|\t)(.*)$`; ruff, ruff format, mypy all green; branch
`3 passed`, reverted-regex main `1 failed`. Breakdown:
- `module: /src/headroom/transforms/cross_turn_dedup.py`
- `sha256:
7ff204b65f40617d505895841aa1cfc1ee54bf63ffe6520198f0aa05a850aa8d`
- `regex : ^([1-9]\d*)(:|\t)(.*)$`
- ruff, ruff format, mypy: all green (output above).
- Branch: `3 passed`. Reverted-regex main: the regression test `1
failed`.
- Not tested: the router-level and Rust-backed integration tests in this
file
(`test_apply_*`, `test_dedup_*`) need the compiled `headroom._core`
extension,
which is not built in this lightweight container; they are
`ModuleNotFoundError`
on both `main` and this branch here, so they were not exercised. The
change is a
pure-stdlib regex in a pure-stdlib function; the unit-level
`dedup_blocks` tests
above cover it directly. I also did not measure how often real-world
tool output
hits the leading-zero shifted shape; the argument is the module's own
strictly-lossless contract, not observed field frequency.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (N/A: no
doc surface)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
Per `CONTRIBUTING.md` ("Bug or small fix -> Open a PR with repro +
test"), this
goes straight to a PR rather than an issue. One concern only; no
dependency or
generated-file changes.
|
||
|
Tejas Chopra
|
10e4829201
|
fix(dedup): shorter fold pointer + MIN_LINES=3, dedup Anthropic list-… (#1932)
…content tool_results, line-number reanchoring
cross_turn_dedup:
- Compact the fold pointer (~35c vs ~100c) and lower MIN_LINES=3 /
MIN_CHARS=40 so the abundant short (2-4 line) re-read repeats pay off
(~4.3% -> ~6% lossless observation-byte reduction on Opus SWE-bench,
measured on 73 real trajectories).
- Add uniform-offset line-number reanchoring: a re-read whose line
numbers all shifted by a constant after an edit now still folds.
Matching keys on the number-stripped content; the offset is carried in
the pointer so recovery of the original numbered bytes stays exact
(unpadded grep/sed/rg -n form only). Cache-safe (prefix-monotonic) and
information-preserving (earliest copy kept verbatim). Small real-data
gain, but closes the renumbering blind spot.
content_router:
- Dedup Anthropic list-content tool_result blocks (the single
{"type":"text"} sub-block), which the string-only extraction path
previously skipped entirely, so cross-turn dedup never fired on
Anthropic-native tool results.
## Description
<!-- Briefly explain the change and why it is needed. -->
Closes #
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
-
## Testing
<!-- Check what you actually ran, then paste the real command output
below. -->
- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed
### Test Output
```text
# Paste relevant command output or artifact links here
```
## Real Behavior Proof
- Environment:
- Exact command / steps:
- Observed result:
- Not tested:
## Review Readiness
- [ ] I have performed a self-review
- [ ] This PR is ready for human review
## Checklist
- [ ] My code follows the project's style guidelines
- [ ] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
Add screenshots to help explain your changes.
## Additional Notes
<!-- Mention any N/A checklist items, tradeoffs, follow-ups, or
maintainer context. -->
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
Tejas Chopra
|
60af15f96f
|
feat(content-router): lossless-first dispatch, cross-turn dedup, and A7 lossy-after-fold (#1818)
## Description <!-- Briefly explain the change and why it is needed. --> Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # Paste relevant command output or artifact links here ``` ## Real Behavior Proof - Environment: - Exact command / steps: - Observed result: - Not tested: ## Review Readiness - [ ] I have performed a self-review - [ ] This PR is ready for human review ## Checklist - [ ] My code follows the project's style guidelines - [ ] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) Add screenshots to help explain your changes. ## Additional Notes <!-- Mention any N/A checklist items, tradeoffs, follow-ups, or maintainer context. --> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |