headroom

akadmin/headroom

Fork 0

mirror of https://github.com/headroomlabs-ai/headroom.git synced 2026-08-27 14:17:10 -04:00

Commit graph

Author	SHA1	Message	Date
Tejas Chopra	60af15f96f	feat(content-router): lossless-first dispatch, cross-turn dedup, and A7 lossy-after-fold (#1818 ) ## Description <!-- Briefly explain the change and why it is needed. --> Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - ## Testing <!-- Check what you actually ran, then paste the real command output below. --> - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [ ] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text # Paste relevant command output or artifact links here ``` ## Real Behavior Proof - Environment: - Exact command / steps: - Observed result: - Not tested: ## Review Readiness - [ ] I have performed a self-review - [ ] This PR is ready for human review ## Checklist - [ ] My code follows the project's style guidelines - [ ] I have performed a self-review of my code - [ ] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Screenshots (if applicable) Add screenshots to help explain your changes. ## Additional Notes <!-- Mention any N/A checklist items, tradeoffs, follow-ups, or maintainer context. --> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-06 08:32:06 -07:00
Tejas Chopra	eea667a720	feat(transforms): adaptive Otsu KEEP/DROP threshold (+ land relevance split on main) (#1726 ) ## Description Lands the prompt-conditioned relevance split on `main` and makes its KEEP/DROP threshold adaptive. Context: the Stage B work (#1722) was merged into the feature branch `tejas/proxy-lossless-mode` rather than `main`, so `relevance_split.py` never reached `main`. This PR cherry-picks that work onto `main` and adds the adaptive threshold on top, in three commits: 1. Prompt-conditioned KEEP/DROP tail split (Stage B) — segment LOG/SEARCH output into records, score each against the request's information need (user prompt + triggering tool-call args) via `headroom/relevance/`, keep relevant records verbatim, Kompress the low-relevance tail. Mode-agnostic (marker-free in lossless, retrieval-marker in CCR). 2. On by default with hot-path rails — background embedding-model pre-warm (BM25 until warm, never blocks a request) + optional `relevance_max_records` cap (default 0 = no cap). 3. Adaptive Otsu threshold (this PR's new work) — see below. ### Adaptive threshold The keep/drop cut is no longer a fixed constant. For each output we compute the natural relevant/irrelevant break in its own score distribution via Otsu's method (parameter-free — candidate cuts are the data's own values, no bins or magic numbers), floored by `relevance.relevance_threshold` so absolutely irrelevant records are never kept verbatim. The bar therefore moves with the content + prompt: a highly-relevant output keeps its top cluster and compresses the merely-moderate tail; a mostly-irrelevant output drops almost everything. All-equal scores fall back to the floor. Toggle via `relevance_adaptive_threshold` (default `True`). Closes # ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Changes Made - `relevance_split.py`: `adaptive_threshold()` + `_otsu_threshold()`; `plan_relevance_split(..., adaptive=True)` uses the adaptive cut, floored by `threshold`. - `content_router.py`: `relevance_adaptive_threshold` config (default `True`), threaded into the split. (Plus the Stage B split + default-on rails from the cherry-picked commits.) - `tests/test_relevance_split.py`: adaptive-threshold cases (bimodal split, floored, all-equal, moves-with-distribution) on top of the Stage B suite. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check`) - [x] Type checking passes (`mypy`) - [x] New tests added for new functionality ### Test Output ```text $ pytest tests/test_relevance_split.py tests/test_transforms_content_router.py tests/test_lossless_mode.py -q 80 passed, 1 warning in 3.41s $ ruff check headroom/transforms/relevance_split.py headroom/transforms/content_router.py tests/test_relevance_split.py All checks passed! $ ruff format --check <changed files> 3 files already formatted $ mypy headroom/transforms/relevance_split.py headroom/transforms/content_router.py Success: no issues found in 2 source files ``` ## Real Behavior Proof - Environment: local, Python 3.12.6. - Steps: `adaptive_threshold()` exercised directly on synthetic score distributions; `plan_relevance_split(adaptive=True)` and the real `ContentRouter._apply_strategy_to_content` path driven with a deterministic scorer + Kompress-tail stub (offline). - Observed: - Bimodal scores `[0.92, 0.88, 0.12, 0.05]` → cut lands in the valley (`0.12 < t < 0.88`), keeping the high cluster. - Mostly-irrelevant `[0.30, 0.28, 0.05, 0.03]` → cut floored at `0.25`. - All-equal scores → floor. - Higher-scoring distribution yields a higher cut than a lower one (bar adapts). - Router split still fires in both lossless and CCR mode; DIFF stays pure lossless; disabling the flag is byte-identical. - Not tested: live embedding model warm/latency at scale; end-to-end `/v1/retrieve` resolution of the CCR tail marker (marker plumbing itself is covered upstream). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes - Supersedes the orphaned #1722 merge (which landed on the feature branch, not `main`); this PR is the canonical path onto `main`. - Follow-ups discussed: TEXT-strategy extension (relevance split for plain prose, currently whole-block Kompress); batch multiple DROP runs into one Kompress call; eval of savings/fidelity on live traffic. - N/A: CHANGELOG (feature not yet released).	2026-07-02 22:25:18 -07:00

Author

SHA1

Message

Date

Tejas Chopra

60af15f96f

feat(content-router): lossless-first dispatch, cross-turn dedup, and A7 lossy-after-fold (#1818 )

## Description

<!-- Briefly explain the change and why it is needed. -->

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- 

## Testing

<!-- Check what you actually ran, then paste the real command output
below. -->

- [ ] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [ ] Manual testing performed

### Test Output

```text
# Paste relevant command output or artifact links here
```

## Real Behavior Proof

- Environment:
- Exact command / steps:
- Observed result:
- Not tested:

## Review Readiness

- [ ] I have performed a self-review
- [ ] This PR is ready for human review

## Checklist

- [ ] My code follows the project's style guidelines
- [ ] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Screenshots (if applicable)

Add screenshots to help explain your changes.

## Additional Notes

<!-- Mention any N/A checklist items, tradeoffs, follow-ups, or
maintainer context. -->

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

2026-07-06 08:32:06 -07:00

Tejas Chopra

eea667a720

feat(transforms): adaptive Otsu KEEP/DROP threshold (+ land relevance split on main) (#1726 )

## Description

Lands the prompt-conditioned relevance split **on `main`** and makes its
KEEP/DROP threshold **adaptive**.

Context: the Stage B work (#1722) was merged into the feature branch
`tejas/proxy-lossless-mode` rather than `main`, so `relevance_split.py`
never
reached `main`. This PR cherry-picks that work onto `main` and adds the
adaptive threshold on top, in three commits:

1. Prompt-conditioned KEEP/DROP tail split (Stage B) — segment
LOG/SEARCH output
into records, score each against the request's information need (user
prompt
+ triggering tool-call args) via `headroom/relevance/`, keep relevant
records
verbatim, Kompress the low-relevance tail. Mode-agnostic (marker-free in
   lossless, retrieval-marker in CCR).
2. On by default with hot-path rails — background embedding-model
pre-warm (BM25
until warm, never blocks a request) + optional `relevance_max_records`
cap
   (default 0 = no cap).
3. **Adaptive Otsu threshold** (this PR's new work) — see below.

### Adaptive threshold

The keep/drop cut is no longer a fixed constant. For each output we
compute the
natural relevant/irrelevant break in *its own* score distribution via
**Otsu's
method** (parameter-free — candidate cuts are the data's own values, no
bins or
magic numbers), floored by `relevance.relevance_threshold` so absolutely
irrelevant records are never kept verbatim. The bar therefore moves with
the
content + prompt: a highly-relevant output keeps its top cluster and
compresses
the merely-moderate tail; a mostly-irrelevant output drops almost
everything.
All-equal scores fall back to the floor. Toggle via
`relevance_adaptive_threshold` (default `True`).

Closes #

## Type of Change

- [x] New feature (non-breaking change that adds functionality)

## Changes Made

- `relevance_split.py`: `adaptive_threshold()` + `_otsu_threshold()`;
`plan_relevance_split(..., adaptive=True)` uses the adaptive cut,
floored by
  `threshold`.
- `content_router.py`: `relevance_adaptive_threshold` config (default
`True`),
threaded into the split. (Plus the Stage B split + default-on rails from
the
  cherry-picked commits.)
- `tests/test_relevance_split.py`: adaptive-threshold cases (bimodal
split,
floored, all-equal, moves-with-distribution) on top of the Stage B
suite.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy`)
- [x] New tests added for new functionality

### Test Output

```text
$ pytest tests/test_relevance_split.py tests/test_transforms_content_router.py tests/test_lossless_mode.py -q
80 passed, 1 warning in 3.41s

$ ruff check headroom/transforms/relevance_split.py headroom/transforms/content_router.py tests/test_relevance_split.py
All checks passed!

$ ruff format --check <changed files>
3 files already formatted

$ mypy headroom/transforms/relevance_split.py headroom/transforms/content_router.py
Success: no issues found in 2 source files
```

## Real Behavior Proof

- **Environment:** local, Python 3.12.6.
- **Steps:** `adaptive_threshold()` exercised directly on synthetic
score
  distributions; `plan_relevance_split(adaptive=True)` and the real
`ContentRouter._apply_strategy_to_content` path driven with a
deterministic
  scorer + Kompress-tail stub (offline).
- **Observed:**
  - Bimodal scores `[0.92, 0.88, 0.12, 0.05]` → cut lands in the valley
    (`0.12 < t < 0.88`), keeping the high cluster.
- Mostly-irrelevant `[0.30, 0.28, 0.05, 0.03]` → cut floored at `0.25`.
  - All-equal scores → floor.
- Higher-scoring distribution yields a higher cut than a lower one (bar
adapts).
- Router split still fires in both lossless and CCR mode; DIFF stays
pure
    lossless; disabling the flag is byte-identical.
- **Not tested:** live embedding model warm/latency at scale; end-to-end
`/v1/retrieve` resolution of the CCR tail marker (marker plumbing itself
is
  covered upstream).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Additional Notes

- Supersedes the orphaned #1722 merge (which landed on the feature
branch, not
  `main`); this PR is the canonical path onto `main`.
- **Follow-ups discussed:** TEXT-strategy extension (relevance split for
plain
prose, currently whole-block Kompress); batch multiple DROP runs into
one
  Kompress call; eval of savings/fidelity on live traffic.
- N/A: CHANGELOG (feature not yet released).

2026-07-02 22:25:18 -07:00

2 commits