Commit graph

1553 commits

Author SHA1 Message Date
JD Davis
fd73b88368
Merge pull request #554 from ashishpatel26/fix/481-dashboard-hero-tile-multi-worker-savings
fix(dashboard): stable 'Proxy $ Saved' hero tile under --workers > 1 (#481)
2026-06-08 18:11:26 -05:00
JD Davis
d10bd5f59c
Merge pull request #402 from chopratejas/realign-F4-trust-forwarded-only-gateway
fix(proxy): F4 — trust X-Forwarded-* only behind allow-listed gateway
2026-06-08 18:01:59 -05:00
Tejas Chopra
1c8c538780
fix: preserve Claude Code tool-search deferral through the proxy (#746) (#753)
Routing Claude Code through the proxy disabled its on-demand tool loading:
with a custom ANTHROPIC_BASE_URL and ENABLE_TOOL_SEARCH unset, Claude Code
stops deferring MCP/system tool schemas behind the server-side Tool Search
Tool and materializes them all into local context (~25K tokens) — the
opposite of what a context-optimization proxy should do.

Root cause is a client-side gate in Claude Code (isToolSearchEnabledOptimistic):
deferral is disabled when ENABLE_TOOL_SEARCH is unset AND provider is
first-party AND the base-URL host is not api.anthropic.com. It is a one-way
URL check, not a capability handshake, so no proxy/response header can flip
it. The only lever is the ENABLE_TOOL_SEARCH env var Claude Code reads at
startup.

Changes:
- wrap claude: inject ENABLE_TOOL_SEARCH into the launched Claude Code env
  (default "true"; --tool-search true|auto|auto:N|false; a pre-set env value
  is respected; blank is treated as unset). Keeps deferral on through the proxy.
- proxy: emit a one-time, actionable hint when a Claude Code request is
  detected loading tools eagerly (for users who run `claude` manually). Gated
  on a cheap one-shot flag and wrapped so it can never fail a request.
- docs: troubleshooting section with before/after verification.
- tests: 30 unit tests (value validation, injection precedence, detection,
  hint content, one-shot guard).
2026-06-08 11:20:48 -07:00
Vipin Singh
8dbe7ad41b
feat(proxy): show resolved upstream API targets in startup banner (#586)
Display the resolved upstream API targets (Anthropic, OpenAI, Gemini,
Cloud Code) in the proxy startup banner so users can verify their
custom endpoint configuration at a glance.

The new UPSTREAM TARGETS section appears between the Backend line and
the FEATURES section. URLs are resolved through the existing
resolve_api_targets pipeline, which normalizes trailing /v1 suffixes
and applies defaults for unconfigured providers.

Closes #583

Co-authored-by: vipin-si <vipin-si@users.noreply.github.com>
2026-06-08 00:19:20 -07:00
Tejas Chopra
827461c593 Merge remote-tracking branch 'origin/main' into realign-F4-trust-forwarded-only-gateway
# Conflicts:
#	headroom/proxy/handlers/anthropic.py
2026-06-07 22:17:15 -07:00
Kumario
9fe4886cf6
feat(perf): add --format {text,json,csv} to headroom perf (#648)
* feat(perf): add structured summary/record builders to analyzer

parse_log_files() already returns a fully-structured PerfReport, but
the only way to read it was the colored text report. Add reusable
machine-readable views so CI guards, dashboards, and agent harnesses
can consume perf data without scraping ANSI text:

- build_perf_summary(report) -> dict with the aggregated KPIs
  (savings_pct, cache_hit_pct, by_model, by_transform, ...), mirroring
  format_report() numbers exactly.
- perf_records_as_dicts(report) -> per-record list for --raw output.
- PERF_RECORD_FIELDS: shared column order for CSV/raw consumers.

Pure additions; no behaviour change to existing callers. Part of #595.

* feat(perf): add --format {text,json,csv} to headroom perf

Adds a machine-readable output path to the perf command (issue #595):

- --format json: aggregated summary (default) or, with --raw, a JSON
  array of per-record dicts.
- --format csv: per-model breakdown (default) or, with --raw, one row
  per PERF record using the shared PERF_RECORD_FIELDS column order.
- --format text (default): unchanged human-readable report.

Enables CI guards (jq '.savings_pct < 70'), dashboards, and agent
wrappers to consume perf data without scraping ANSI text.

Closes #595.

* test(perf): cover --format json/csv and structured builders

Unit tests for build_perf_summary (totals, savings/cache pct,
by_model/by_transform, empty-report zero-division guard) and
perf_records_as_dicts, plus CliRunner integration tests for
--format json, json --raw, csv, csv --raw, the unchanged text
default, and rejection of an unknown format. Part of #595.

* fix(perf): rename transform loop var to satisfy mypy

The structured-summary builder reused `recs` for both the per-model
(list[PerfRecord]) and per-transform (list[TransformRecord]) groupings, so
mypy flagged the second assignment as an incompatible-type reuse
(analyzer.py:704). Rename the transform loop variable to `t_recs` so each
loop keeps a single element type. No behavior change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Kumario1 <ramsakal.ipec@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-06 22:43:10 -07:00
Patrick A
45559011b1
fix(startup): suppress proxy startup log noise (#619)
* docs: add enterprise.md

* docs: add link to enterprisemd in README

* fix(copilot): restore generic endpoint for non-subscription OAuth (#610) (#612)

* fix(copilot): restore generic endpoint for non-subscription OAuth (#610)

0.23.0 re-pointed the shared Copilot OAuth branch from the generic
api.githubcopilot.com host to the account-specific endpoints.api host
returned by /copilot_internal/user, and made resolve_copilot_api_url
ignore the GITHUB_COPILOT_API_URL override whenever a token resolved.

That change was meant to add --subscription, but it also altered the
pre-existing non-subscription OAuth flow that worked on 0.22.4. The
account host does not serve newer models (e.g. gpt-5.4) on the responses
API, so wrapped requests began failing with unsupported-model errors
while plain Copilot and 0.22.4 kept working.

Restore 0.22.4 routing for non-subscription OAuth (generic host, still
overridable) and keep account resolution only for --subscription.
resolve_copilot_api_url now honors GITHUB_COPILOT_API_URL first, so the
override escape hatch works for every path. BYOK is unaffected.

Add a regression suite that mocks a successful user-info response, the
real-world path the prior test never exercised (it relied on the network
call failing in CI and falling back to the generic host).

* fix(copilot): route subscription + OAuth through the generic host (#610)

The 0.23.0 endpoint resolution derived the Copilot API host from
/copilot_internal/user (endpoints.api), which returns a segmented host
(e.g. api.individual.githubcopilot.com) that does not serve newer models
on the responses API and is not the host the official Copilot client
routes with (that comes from the token-exchange endpoint). --subscription
used the identical resolution, so it carried the same latent regression
as the non-subscription OAuth path.

Make Copilot host resolution override -> generic for BOTH --subscription
and the implicit OAuth path, and stop using user-info to route. Accounts
that require a dedicated host (enterprise / data residency) pin it via
GITHUB_COPILOT_API_URL. resolve_copilot_api_url no longer makes a network
call; _fetch_copilot_user_info is retained for token validation.

Update the subscription smoke tests that encoded the old account-host
assumption, and add wrap-level + unit coverage that --subscription routes
to the generic host even when user-info advertises an account host, and
that the GITHUB_COPILOT_API_URL override flows through both paths.

* docs(copilot): document generic-host routing + enterprise override (#610)

Spell out the routing contract introduced by the #610 fix so enterprise
users have a supported path. Headroom routes wrapped Copilot hosted
traffic (--subscription and OAuth) to the generic api.githubcopilot.com,
and accounts on a dedicated host (Enterprise Cloud data residency, egress
proxy) pin it via GITHUB_COPILOT_API_URL.

- copilot --help: note the generic host + GITHUB_COPILOT_API_URL override.
- TESTING-copilot-subscription.md: add "API host & Enterprise / data
  residency" section; correct the stale api.*.githubcopilot.com claim; and
  invite enterprise tenants who want token-exchange-based auto-detection to
  open an issue.
- integration-guide.md: short hosted-host + override note in the Copilot
  section.

* fix(wrap): report unbindable proxy ports (#602)

* fix(proxy): fail-open on corrupt golden bytes instead of RuntimeError (#603)

* fix(proxy): fail-open on corrupt golden bytes instead of RuntimeError

Permanent session corruption: once golden bytes become unreadable
(UnicodeDecodeError / JSONDecodeError), every subsequent request for
the session raised RuntimeError, returning 500 until proxy restart.

Fix: log at ERROR level and recover — skip the corrupt memory tool,
or regenerate a fresh CCR definition — rather than propagating
RuntimeError and permanently breaking the session.

Also change proxy_inbound_request_aborted from logger.info to
logger.error with exc_info=True so tracebacks appear in logs.

Closes: proxy silent-500 sessions in the wild (observed 2026-06-04)

* fix(tests): re-enable headroom log propagation in corrupt-bytes tests

configure_proxy_logging() sets headroom_logger.propagate = False to prevent
duplicate writes when the proxy redirects stderr to a log file. In CI the
proxy initialises its logging stack before the test suite, leaving propagation
disabled. pytest's caplog handler attaches to the root logger, so records that
stop at the headroom logger are never captured.

Added _enable_headroom_log_propagation autouse fixture that temporarily
re-enables propagation for the duration of each test, making caplog capture
work regardless of the surrounding logging configuration.

* fix(tests): remove unused imports from corrupt-bytes regression tests

Remove json, SessionCcrTracker, and SessionToolTracker imports that were
imported but never referenced in the test body. Fixes ruff F401 and I001
lint errors reported by CI.

---------

Co-authored-by: Patrick Ancillotti <patrick.ancillotti@people.inc>

* fix(startup): suppress log noise from litellm, trafilatura, HF hub, and tiktoken warning

* fix(startup): suppress httpx INFO logs from sentence_transformers HEAD checks

* docs(changelog): add entry for startup log noise suppression fixes

* refactor(startup): extract hf_hub_download_local_first into onnx_runtime

The three _hub_download/_hub_dl helpers in embedders.py, onnx_router.py, and
kompress_compressor.py are identical -- try local cache first, fall back to
network download. Extract into a single hf_hub_download_local_first() function
in onnx_runtime.py (the natural home for shared ORT/HF utilities) and update
all three callers to use it.

* fix(lint): sort imports and remove unused _FALLBACK_WARNING_SHOWN import

* fix(lint): cast hf_hub_download return to str for mypy no-any-return

---------

Co-authored-by: Patrick Ancillotti <patrick.ancillotti@people.inc>
2026-06-05 14:44:40 -08:00
Patrick A
53d2342291
ci: speed up GitHub Actions — path filters, caching, timeouts, version upgrades (#620)
* ci: speed up GitHub Actions - path filters, caching, timeouts, version upgrades

Performance improvements:
- init-e2e.yml, wrap-e2e.yml: add path filters so e2e Docker builds only run when
  e2e-related files change (saves ~10 min per irrelevant PR push)
- init-e2e.yml, wrap-e2e.yml: add concurrency groups to cancel superseded PR runs
- ci.yml: add pip caching to lint and build jobs
- ci.yml: cache actionlint + act binaries in workflow-validation (skip curl on hits)
- eval.yml: add pip caching to smoke-test and weekly-suite jobs
- docs.yml: add pip caching for mkdocs-material install
- rust.yml: replace cargo install --locked cargo-audit/deny with taiki-e/install-action
  (prebuilt binaries; saves 2-5 min per audit run)

Bug fixes:
- docker.yml: fix actions/checkout@v6 -> @v4 (v6 does not exist; would break all
  Docker builds on every release/PR touching docker paths)

Version upgrades:
- wagoid/commitlint-github-action: @v5 -> @v6
- devcontainers.yml: docker/setup-buildx-action@v3 -> @v4 (align with docker.yml)

Safety improvements:
- ci.yml: add timeout-minutes to all 13 jobs (changes, lint, build-wheel,
  prefetch-model, test x4, test-extras, test-agno, commitlint, build,
  workflow-validation, docker-native-e2e, windows-native-wrapper, macos-native-wrapper)
- docker.yml: add timeout-minutes to docker-build (75m), docker-manifest (20m),
  promote-latest (10m)
- eval.yml: add timeout-minutes to smoke-test (30m); bump weekly-suite 60->90m
- rust.yml: add timeout-minutes to test (30m), wheels (45m), audit (20m)

Observed wall-clock impact on recent PRs:
- Init E2E and Wrap E2E were running on every single PR push regardless of content
- CI workflow was taking 12-17 min; path filters reduce unnecessary e2e runs to 0

* fix(ci): bust actionlint+act cache when workflow file changes

Static cache key 'ci-tools-actionlint-act-v1' never invalidated on
tool version updates. Switched to hashFiles('.github/workflows/ci.yml')
so the cache busts automatically whenever the download scripts are
updated to point at a newer release.

Flagged by adversarial review (Architecture + Testing/Reliability personas).

* fix(ci): add missing Dockerfile COPY paths to e2e path filters

e2e/init/Dockerfile and e2e/wrap/Dockerfile COPY files not covered
by the initial path filter set:

  init-e2e: Cargo.toml, Cargo.lock, rust-toolchain.toml, uv.lock,
            .claude-plugin, .github/plugin/**, plugins/headroom-agent-hooks/**
  wrap-e2e: Cargo.toml, Cargo.lock, rust-toolchain.toml, uv.lock,
            sdk/typescript/**, plugins/openclaw/**

Without these, a Rust toolchain bump or SDK change on a PR would
skip the e2e gate entirely, only catching it on the merge to main.

Flagged by adversarial review (Domain/Correctness persona).

* fix(devcontainer): upgrade uv to >=0.7.0 to parse uv.lock revision=3

* fix(devcontainer): set UV_SKIP_WHEEL_FILENAME_CHECK=1 in post-create.sh for gitpython wheel

* ci: bump actions/checkout and actions/setup-node to v5 (Node.js 20 EOL Jun 16)

* fix(devcontainer): export UV_SKIP_WHEEL_FILENAME_CHECK so uv run also skips wheel check

* ci: bump all GitHub Actions to latest versions (Node.js 24)

* fix(test): accept release-please-action v4 or v5 in workflow assertion

* fix(format): ruff format test_release_workflows.py
2026-06-05 14:32:53 -08:00
Mansur Japarov
30c1ac8656
fix(proxy): route Claude Code model metadata to Anthropic (#627) 2026-06-05 14:28:17 -08:00
Leoy
88177bd7a6
feat(relevance): weight BM25 score_batch by corpus IDF (#646)
BM25Scorer.score_batch() ranks a real corpus of documents but weighted
every matched term with a constant idf=log(2.0), so a ubiquitous noise
word counted the same as a discriminative UUID. The _compute_idf() helper
needed to do this properly already existed (and was unit-tested) but was
never wired into scoring.

- Implement _compute_idf() with the standard floored BM25 IDF its docstring
  documents: log((N - n + 0.5) / (n + 0.5) + 1).
- Thread an optional per-term idf_map through _bm25_score(); single-document
  score() keeps the neutral log(2.0) weight (no corpus to estimate from).
- score_batch() now computes document frequency across the batch and builds
  the IDF map, so rare/discriminative terms outrank corpus-wide terms in the
  ranking that CompressionStore.search() and HybridScorer consume.

Adds tests covering the IDF formula and the batch ranking behaviour.
2026-06-05 14:15:44 -08:00
Patrick A
2170a1b4a0
fix(proxy): fail-open on corrupt golden bytes instead of RuntimeError (#603)
* fix(proxy): fail-open on corrupt golden bytes instead of RuntimeError

Permanent session corruption: once golden bytes become unreadable
(UnicodeDecodeError / JSONDecodeError), every subsequent request for
the session raised RuntimeError, returning 500 until proxy restart.

Fix: log at ERROR level and recover — skip the corrupt memory tool,
or regenerate a fresh CCR definition — rather than propagating
RuntimeError and permanently breaking the session.

Also change proxy_inbound_request_aborted from logger.info to
logger.error with exc_info=True so tracebacks appear in logs.

Closes: proxy silent-500 sessions in the wild (observed 2026-06-04)

* fix(tests): re-enable headroom log propagation in corrupt-bytes tests

configure_proxy_logging() sets headroom_logger.propagate = False to prevent
duplicate writes when the proxy redirects stderr to a log file. In CI the
proxy initialises its logging stack before the test suite, leaving propagation
disabled. pytest's caplog handler attaches to the root logger, so records that
stop at the headroom logger are never captured.

Added _enable_headroom_log_propagation autouse fixture that temporarily
re-enables propagation for the duration of each test, making caplog capture
work regardless of the surrounding logging configuration.

* fix(tests): remove unused imports from corrupt-bytes regression tests

Remove json, SessionCcrTracker, and SessionToolTracker imports that were
imported but never referenced in the test body. Fixes ruff F401 and I001
lint errors reported by CI.

---------

Co-authored-by: Patrick Ancillotti <patrick.ancillotti@people.inc>
2026-06-04 18:19:06 -07:00
oxura
6dfcaa839f
fix(wrap): report unbindable proxy ports (#602) 2026-06-04 18:19:00 -07:00
Tejas Chopra
18925b8c6e
fix(copilot): restore generic endpoint for non-subscription OAuth (#610) (#612)
* fix(copilot): restore generic endpoint for non-subscription OAuth (#610)

0.23.0 re-pointed the shared Copilot OAuth branch from the generic
api.githubcopilot.com host to the account-specific endpoints.api host
returned by /copilot_internal/user, and made resolve_copilot_api_url
ignore the GITHUB_COPILOT_API_URL override whenever a token resolved.

That change was meant to add --subscription, but it also altered the
pre-existing non-subscription OAuth flow that worked on 0.22.4. The
account host does not serve newer models (e.g. gpt-5.4) on the responses
API, so wrapped requests began failing with unsupported-model errors
while plain Copilot and 0.22.4 kept working.

Restore 0.22.4 routing for non-subscription OAuth (generic host, still
overridable) and keep account resolution only for --subscription.
resolve_copilot_api_url now honors GITHUB_COPILOT_API_URL first, so the
override escape hatch works for every path. BYOK is unaffected.

Add a regression suite that mocks a successful user-info response, the
real-world path the prior test never exercised (it relied on the network
call failing in CI and falling back to the generic host).

* fix(copilot): route subscription + OAuth through the generic host (#610)

The 0.23.0 endpoint resolution derived the Copilot API host from
/copilot_internal/user (endpoints.api), which returns a segmented host
(e.g. api.individual.githubcopilot.com) that does not serve newer models
on the responses API and is not the host the official Copilot client
routes with (that comes from the token-exchange endpoint). --subscription
used the identical resolution, so it carried the same latent regression
as the non-subscription OAuth path.

Make Copilot host resolution override -> generic for BOTH --subscription
and the implicit OAuth path, and stop using user-info to route. Accounts
that require a dedicated host (enterprise / data residency) pin it via
GITHUB_COPILOT_API_URL. resolve_copilot_api_url no longer makes a network
call; _fetch_copilot_user_info is retained for token validation.

Update the subscription smoke tests that encoded the old account-host
assumption, and add wrap-level + unit coverage that --subscription routes
to the generic host even when user-info advertises an account host, and
that the GITHUB_COPILOT_API_URL override flows through both paths.

* docs(copilot): document generic-host routing + enterprise override (#610)

Spell out the routing contract introduced by the #610 fix so enterprise
users have a supported path. Headroom routes wrapped Copilot hosted
traffic (--subscription and OAuth) to the generic api.githubcopilot.com,
and accounts on a dedicated host (Enterprise Cloud data residency, egress
proxy) pin it via GITHUB_COPILOT_API_URL.

- copilot --help: note the generic host + GITHUB_COPILOT_API_URL override.
- TESTING-copilot-subscription.md: add "API host & Enterprise / data
  residency" section; correct the stale api.*.githubcopilot.com claim; and
  invite enterprise tenants who want token-exchange-based auto-detection to
  open an issue.
- integration-guide.md: short hosted-host + override note in the Copilot
  section.
2026-06-04 16:27:54 -07:00
Tejas Chopra
fe50f9daed fix(ci): restore green lint gate on main
Merges of #564/#555 landed lint regressions that the old HF-429-masked CI hid:
- I001 unsorted imports in tests/repro_unsendable_panic.py (#564)
- ruff-format drift in helpers.py and code_compressor.py
- mypy arg-type on get_language() in code_compressor.py (#564 dropped the
  type: ignore the old get_parser() call had)

ruff check + ruff format --check + mypy headroom all green.
2026-06-04 14:15:49 -07:00
Tejas Chopra
5f5f261a03
Merge pull request #555 from ashishpatel26/fix/531-codex-wrap-fail-open-on-compression
fix(codex): auto-enable fail-open on compression timeout in headroom wrap codex (#531)
2026-06-04 13:54:02 -07:00
Tejas Chopra
e87cece9fd
Merge pull request #564 from mosatch/fix/thread-local-tree-sitter-parser
fix: use thread-local tree-sitter parsers to prevent unsendable panic
2026-06-04 13:52:58 -07:00
ashishpatel26
e8ecd08829 fix(codex): fail open for proxy compression timeout 2026-06-04 14:04:49 +05:30
Tejas Chopra
f4dff9b488
Merge pull request #576 from chopratejas/fix/copilot-subscription-auth
feat(copilot): GitHub Copilot subscription mode through Headroom
2026-06-04 00:28:29 -07:00
Tejas Chopra
6ed43027b7 style(copilot): ruff-format test_copilot_auth.py
Parenthesize a multi-line conditional lambda so 'ruff format --check .'
passes (the original commit added it unformatted).
2026-06-04 00:22:04 -07:00
Tejas Chopra
e94a36cb6d test(codex): de-flake semaphore-tail ratio check on fast runners
test_concurrent_compression_has_no_semaphore_tail computed
p99/max(p50,1). On a fast/quiet CI runner p50 rounds to 0ms, so the
ratio collapses to p99-in-ms and a few ms of ordinary scheduler jitter
(p50=0ms, p99=5ms) read as ~4.8x, tripping the <4x gate — noise, not the
semaphore-contention tail it targets (tens of ms, ~27x).

Only enforce the ratio once p99 clears a 25ms scheduler-noise floor
(assert ratio < 4.0 or p99 < 25ms). A real contention regression still
trips it (large absolute tail + high ratio); sub-ms jitter no longer
does. Verified locally: the test passes.
2026-06-03 23:57:31 -07:00
ashishpatel26
0d8de25bfc fix(proxy): remove invalid savings tracker flock 2026-06-04 11:55:14 +05:30
Tejas Chopra
f58340d11f
Merge pull request #573 from jamesx0416/codex-openai-responses-parallel-units
Speed up Headroom compression by 4x for large agent traffic
2026-06-03 23:23:46 -07:00
Tejas Chopra
72da461217 fix(copilot): deterministic subscription token handoff to the proxy
Pass the wrapper-resolved (and, for --subscription, GitHub-validated) Copilot
token to the proxy as an explicit launch argument instead of mutating the
parent process's global os.environ. The proxy pins it as
GITHUB_COPILOT_API_TOKEN, so upstream auth is deterministic rather than the
proxy re-running unvalidated token discovery (which could otherwise inject a
different token and 401). Removes the global-state mutation and the test
isolation it forced.

Add a hermetic cross-platform smoke suite (no Keychain/secret-tool/network)
proving the env-var token path resolves on any OS, each OS secret reader is
inert off-platform, and the proxy injects exactly the validated token.
2026-06-03 23:11:02 -07:00
jamesx0416
cfcaf0e364 Rename tool output compression parallelism env 2026-06-04 14:54:21 +10:00
jamesx0416
4c86826eab Cover OpenAI Responses unit cache edge cases 2026-06-04 14:20:25 +10:00
jamesx0416
d160f391d5 Speed up OpenAI Responses compression units 2026-06-04 12:52:00 +10:00
M Barry
bd0e84562c Merge branch 'chopratejas:main' into fix/thread-local-tree-sitter-parser 2026-06-03 21:45:10 -05:00
Devanshi Vyas
982d01b9c9
Merge pull request #541 from evanclan/fix/learn-gemini-flash-latest-default
fix(learn): finish gemini-flash-latest default model sweep
2026-06-03 16:23:00 -07:00
M Barry
6a70ea653c test(code_compressor): add unsendable-panic repro and thread-local parser tests 2026-06-03 17:02:07 -05:00
Mubashir R
0e551de9d8
fix: correct tiktoken encoding for unknown gpt-4 model snapshots (#552)
get_encoding_for_model() resolved an unknown model to an encoding by
scanning MODEL_TO_ENCODING for the first key that starts with the
matched prefix. Because the gpt-4o entries are defined before the
plain gpt-4 entries, the "gpt-4" prefix matched "gpt-4o" first and
returned o200k_base for any gpt-4 snapshot not already in the table
(e.g. a future dated build like gpt-4-2025-01-01). The gpt-4 family
uses cl100k_base, so token counts for those models were computed with
the wrong encoding, skewing every downstream budget/truncation
decision.

Map each prefix directly to its encoding (still ordered most-specific
first) so the result is deterministic and independent of dict
insertion order.

Regression test in tests/test_tokenizers.py asserts unknown gpt-4 /
gpt-4-turbo snapshots resolve to cl100k_base while gpt-4o snapshots
stay on o200k_base. It fails before this change and passes after.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-03 13:12:16 -07:00
Mubashir R
bdcfc322da
fix: ignore brackets inside JSON strings when splitting mixed content (#553)
_extract_json_block() counted raw [ ] { } per line via str.count() to
find where a JSON block ends. Any bracket/brace inside a JSON string
value (e.g. the "]" in {"path": "a]b"}) was counted as structural, so
the running balance hit zero early and the block was cut mid-array.

In ContentRouter._compress_mixed() this fragments one JSON array into
multiple sections: the array is truncated, a non-array fragment gets
mislabeled JSON_ARRAY, and the trailing "]" leaks into the next prose
section — so content is routed to the wrong compressor.

Walk the characters with a small in-string/escape state machine and
only count brackets/braces that are outside string literals. Behavior
is unchanged for JSON without brackets-in-strings.

Regression tests in tests/test_transforms_content_router.py cover both
the helper (_extract_json_block) and the end-to-end split
(split_into_sections). They fail before this change and pass after.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-03 12:32:56 -07:00
Tejas Chopra
92075b95af
Merge pull request #536 from SuperMarioYL/fix/utf8-encoding-owned-assets
fix: decode/encode owned config, state and template assets as UTF-8 (fixes #533)
2026-06-03 08:02:41 -07:00
Devdeep Sarkar
a0c54c7f22 style: fix linting and formatting issues 2026-06-03 20:08:26 +05:30
Devdeep Sarkar
896a093399 refactor: extract litellm model resolution to shared utility 2026-06-03 17:45:39 +05:30
Tejas Chopra
ff4a0c6bc6 fix(copilot): support subscription auth through Headroom
Route GitHub Copilot CLI subscription traffic through the Headroom
OpenAI-compatible proxy path and resolve the account-specific Copilot API
endpoint before launch.

Add source-aware Copilot token discovery for explicit Copilot env vars,
macOS Keychain, Windows Credential Manager, Linux Secret Service, credential
files, and generic GitHub fallbacks. Validate subscription candidates against
GitHub Copilot user metadata so generic GH_TOKEN/GITHUB_TOKEN values do not
shadow Copilot CLI auth.

Document the subscription command and platform status in README: macOS
Keychain auth reuse has been smoke-tested, while Windows, Linux, Docker, and
CI auth-discovery paths still need real OS validation.

Tests: .venv/bin/python -m pytest tests/test_copilot_auth.py
tests/test_copilot_macos_keychain.py tests/test_copilot_linux_secret.py
tests/test_cli/test_wrap_copilot.py tests/test_cli/test_wrap_persistent.py
tests/test_proxy_copilot_auth_hooks.py
2026-06-02 21:24:47 -07:00
Devanshi Vyas
55579445f8
fix(docs): mkdocs configuration to build with correct folder (#543)
* fix(docs): mkdocs configuration to build with correct folder

* fix(format): fix ruff format for test file
2026-06-02 18:07:11 -07:00
Evan Alferez
d7973665f4 fix(learn): finish gemini-flash-latest default model sweep (#532)
Google deprecated gemini/gemini-2.0-flash; headroom learn silently fails
when GEMINI_API_KEY is set. PR #532 updated the default in analyzer.py
but left stale references in the CLI help text and unit test assertion.
2026-06-03 08:32:43 +09:00
supermario_leo
2f1538a641 fix: decode/encode owned config, state and template assets as UTF-8
Headroom reads and writes its own dashboard template, JSON deployment/sync
state and provider config files using the platform default text codec. On
systems whose default encoding is not UTF-8 (e.g. Windows cp949/cp1252
locales) this raises UnicodeDecodeError when the file contains non-ASCII
bytes.

The dashboard template ships with non-ASCII UTF-8 content, so loading the
dashboard crashes on a Korean Windows locale at byte 20523 (fixes #533).
The same latent bug exists in the sibling JSON state/config I/O; since these
files are owned by Headroom and JSON is UTF-8 by spec (RFC 8259 §8.1), read
and write them with an explicit encoding="utf-8" so they round-trip on every
platform.

Add a regression test covering the dashboard load and a non-ASCII JSON
state round-trip.
2026-06-03 03:25:42 +08:00
ehendrix23
5f0c5f0fe1 Minimize Codex model metadata changes 2026-06-01 17:27:31 -06:00
ehendrix23
4be6717864 Remove Codex model registry cache 2026-06-01 17:20:12 -06:00
ehendrix23
8325c8a5c9 Remove temporary debug log level plumbing 2026-06-01 16:54:30 -06:00
ehendrix23
e0b863cb6d Add codex model discovery and logging fixes 2026-06-01 14:10:15 -06:00
Matt
849b46de59 fix(codex): keep init model_provider at config root (#260)
`headroom init codex` appended its provider block to the end of
~/.codex/config.toml via _replace_marker_block. When the file ended in a
table (e.g. [features]), TOML scoped the block's root keys (model_provider,
openai_base_url) under that table, so Codex refused to start with:
invalid type: string "headroom", expected a boolean in features.

Add an at_root option to _replace_marker_block that inserts the block before
the first table header (reusing the module's line-based header detection), and
have _ensure_codex_provider use it. _ensure_codex_provider also strips any prior
top-level model_provider/openai_base_url assignment first, so init replaces an
existing value (or one an older version mis-scoped under a table) instead of
emitting a duplicate top-level key. Files with no tables still append, so the
keys stay at the root.

Add regression tests that parse the result with tomllib and assert
model_provider lands at the document root, not under [features], and that a
pre-existing model_provider is replaced rather than duplicated.

Verified end-to-end against the real codex CLI across 7 config shapes
(trailing [features], fresh, no-tables, root-key+table, other trailing table,
double-init, pre-existing provider): `codex doctor` reports "config could not
be loaded" before the fix and "config loaded / parse ok" after.
2026-05-30 19:39:50 -04:00
MrAshRhodes
92d71b8866 test(learn): ruff-format scanner test and broaden dotted-home coverage
PR #506 merged with the test file left un-formatted, so 'ruff format --check' now fails on main (the 'test (3.12)' CI job). Apply 'ruff format' to tests/test_learn/test_scanner.py to restore a green format check.

Also replace the skip-only Unix username test with test_home_dir_username_stays_single_component, which roots a throwaway project at the real home and decodes its flattened name. It exercises the Users/home branch on both macOS (/Users) and Linux CI (/home/runner) instead of skipping off /Users, restoring patch coverage of the decode fix.
2026-05-30 13:49:23 +02:00
MrAshRhodes
491a8b3a1b fix(learn): decode Unix home dirs whose username contains '.', '-' or '_'
Claude Code escapes project paths by flattening '/', '.', '-' and '_' to
'-', so /Users/first.last/proj is stored as -Users-first-last-proj. The
decoder consumed only the first token after "Users"/"home" as the home
directory and walked from /Users/first, which does not exist, so it bailed
out. Callers then fell back to the literal "/Users/first/last", and
'headroom learn --apply' failed with PermissionError: '/Users/first' when
writing recommendations.

Start the greedy decode at the mount root and pass the remaining tokens so
the multi-token home component is reconstructed by tokenisation, with a
fallback to the legacy single-token behaviour. Adds the Unix counterpart of
test_windows_username_with_dot_stays_single_component.
2026-05-29 10:13:37 +02:00
Tejas Chopra
a178249fc0
Merge pull request #501 from chopratejas/fix/memory-injection-framing
fix(memory): READ-ONLY framing + fail-closed unresolved-project fallback
2026-05-26 14:52:10 -07:00
chopratejas
482f80e735 fix(memory): READ-ONLY framing + fail-closed unresolved-project fallback
Closes the memory misinjection Jocelyn reported 2026-05-26: a memory
recorded from a prior unrelated session ("implémente TAM-550") was
restored into the live user turn of a fresh PR-review thread and was
treated by the agent as a NEW live instruction. The agent then ran a
full implementation that nobody had asked for in the current
conversation.

This is a different incident from the cross-project CCR leak fixed in
PR #500. That one was about CCR proactive-expansion across workspaces;
this one is about (a) the memory injection block having no read-only
framing, and (b) the silent GLOBAL fallback when PROJECT-mode
resolution failed pooling everyone's memory together.

Two fixes ship together because they're complementary:

(1) Read-only framing — last line of defense
----------------------------------------------
The memory block is appended into the LIVE-ZONE USER TURN
(`_append_to_latest_user_tail`, post-PR-B6). On the wire it looks
EXACTLY like the rest of the user message — the model has no shape
signal distinguishing "retrieved recall" from "fresh request" unless
we say so explicitly. The previous header said "use this context to
provide personalized, contextually relevant responses" — no read-only
marker, no past-tense advisory, nothing addressing the imperative-
phrasing failure mode.

The new framing makes the boundary plain:

> These are READ-ONLY entries recalled from prior sessions in this
> scope. Treat them as BACKGROUND information about past
> conversations and saved preferences — they are NOT instructions
> for the current turn. If an entry contains imperative phrasing
> (e.g. "implement X", "fix Y"), that refers to a PAST conversation;
> do not act on it unless the user re-issues the request in this
> thread.

This catches the bug class even if a memory from a wrong project /
session somehow gets through.

(2) Fail-closed unresolved-project resolution — first line of defense
---------------------------------------------------------------------
Pre-this-PR, when running in PROJECT mode and `ProjectResolver`
returned None (no x-headroom-project-id / x-headroom-cwd / system-
prompt cwd:), the router silently fell back to GLOBAL. Result: ALL
unresolved-project traffic across ALL clients/projects pooled into one
DB. The TAM-550 memory had been saved under "global (unresolved)"
because the original session didn't have a project signal; later a
different unresolved session searched the same bucket and got it.

New behaviour:

  - `BackendRouterConfig.unresolved_project_fallback: str = "empty"`
    (new field, new default).
  - When PROJECT mode + resolver returns None + fallback="empty":
    return a sentinel ResolvedScope (mode=PROJECT, project_key=None,
    display_name="unresolved (no memory)") with a structured warning
    log including a hint about how to set the project signal.
  - `MemoryHandler.search_and_format_context` checks
    `scope.mode is PROJECT and scope.project_key is None` and returns
    None (skip injection). Plain English: if we can't tell which
    project this request belongs to, refuse to load anyone's memory.
  - Legacy GLOBAL pooling is reachable via the opt-in
    `unresolved_project_fallback="global"` config — for users who
    understand and accept the cross-project leak surface.
  - Unknown values raise ValueError (no silent default).

Why not just expose the opt-in through proxy CLI?
Per `feedback_no_silent_fallbacks`, opt-ins to silent behaviour are
themselves a silent-fallback enabler. Users who actually need GLOBAL
pooling have to construct the router directly (which is itself a
signal they should be sure). Not surfacing it through MemoryConfig
keeps the proxy default safe.

Tests
-----
- 2 new framing-regression tests in test_memory_auto_tail.py: pin the
  READ-ONLY/BACKGROUND/NOT-instructions/PAST-conversation strings, and
  verify the [id] → memory_update/memory_delete plumbing still works
  alongside the new read-only language.
- 1 new test in test_memory_handler_project_isolation.py: PROJECT mode
  + no resolution signal + seeded backend results → no memory
  injection (proves the gate is at scope resolution, not at empty
  store).
- test_memory_storage_router.py: the prior
  `test_router_project_mode_unresolved_falls_back_to_global` was
  asserting the OLD silent-GLOBAL behaviour — replaced with three
  tests: default fail-closed, opt-in GLOBAL via
  `unresolved_project_fallback="global"`, and unknown-value
  ValueError.
- Net: 24 (storage_router) + 5 (project_isolation) + 12 (auto_tail) =
  41 memory tests; 176/176 in the python test subset; ci-precheck
  fully green.

Trade-off
---------
Users who relied on the old silent GLOBAL pooling will see their
memories stop appearing until they (a) set x-headroom-cwd /
x-headroom-project-id, or (b) explicitly set
unresolved_project_fallback="global" in their router config. This is
intentional — the old behaviour was a cross-project leak vector and
the fix-forward path is the resolver signal, not the silent pool.
2026-05-26 14:32:08 -07:00
chopratejas
1bc163f5bc fix(ccr): scope proactive expansion by workspace (cross-project leak)
Closes the cross-project context leak Jocelyn reported 2026-05-26:
working on a Ruby/Rails project (daphni-rails), an unrelated Python
file (an Ollama inference provider from project `tamag0`) was being
injected into context as "Proactive Context Expansion - relevant to
your query". Two completely different projects, two different
languages, two different working directories — but the same proxy
process was serving both, and the in-memory ContextTracker had no
workspace identity to filter on.

Root cause
----------
`self.ccr_context_tracker` is one instance per proxy process. Every
session, every project, every user shared the same `_contexts` dict.
`track_compression()` stored sample content with no provenance key;
`analyze_query()` ran lexical keyword overlap across the full dict
without filtering. Within the 5-minute age window, surface-level
token matches ("provider", "session", "oauth", generic code/test
structure) scored above the 0.3 relevance threshold, recommendations
came back, and execute_expansions() injected the full original
content into a foreign session.

Refuted: this is NOT a race condition (joce's hypothesis). It
reproduces single-threaded, one-request-at-a-time. Plain shared
mutable state.

Fix
---
Add a required `workspace_key` to the tracker API and filter on it
inside `analyze_query`:

1. `CompressedContext` gets a `workspace_key: str` field.
2. `track_compression(..., workspace_key=...)` is now keyword-only,
   no default — fail-loud on missing.
3. `analyze_query(..., workspace_key=...)` is also keyword-only; an
   empty workspace_key short-circuits to `[]` (fail-closed per
   `feedback_no_silent_fallbacks`).
4. The loop at `analyze_query` skips any entry whose workspace_key
   differs from the request's.

In the Anthropic proxy handler:

5. New `_resolve_ccr_workspace(request, body)` static helper uses the
   memory subsystem's `ProjectResolver` so CCR and memory agree on
   project identity. Tier order: x-headroom-project-id →
   x-headroom-cwd → CLI override → cwd: line in system prompt.
6. Both track and analyze sites gate on `ccr_workspace_key` being
   non-empty — turning off proactive expansion entirely when project
   identity can't be resolved is the safest default (it's an
   optimization, not correctness).
7. `format_expansions_for_context(expansions, workspace_label=...)`
   was already wired (GH #462 Fix C); the call site now passes the
   label so the injected block declares its provenance, symmetric
   with the memory injection header.

Affected population
-------------------
- Default mode (no `--cache`): bug fixed.
- Cache mode: was never affected — proactive expansion short-
  circuits in cache mode to preserve prefix stability.

Tests
-----
- 6 new workspace-scoping tests in `test_ccr_context_tracker.py`:
  same-workspace match still works, cross-workspace silently
  filtered, empty workspace_key fail-closes, two workspaces each
  see only their own, workspace_label propagates to formatter, LRU
  cross-workspace doesn't leak even with full tracker.
- 6 new `_resolve_ccr_workspace` resolver tests in
  `test_proxy_handler_helpers.py`: explicit project-id wins, cwd
  header → key+label, two cwds get distinct keys, no-signal
  fail-closed, system-prompt cwd: fallback, malformed request
  fail-closed.
- 32 existing tracker tests updated to pass `workspace_key="ws-test"`.
- 55/55 tests pass; ci-precheck green.

Defense-in-depth follow-up
--------------------------
The compression_store itself (`headroom/cache/compression_store.py`)
also lacks workspace scoping — a CCR `headroom_retrieve` call from
Project B for a hash created by Project A would succeed. The
practical attack surface is closed by this PR (hashes only reach
Project B's model via proactive expansion, now gated), but
defense-in-depth hardening of the store is worth a separate PR.
Filed as task #44.
2026-05-26 13:23:51 -07:00
chopratejas
c74ad113a4 refactor(cli): factor shared wrap-subcommand scaffolding
Phase G's wrap-CLI breadth (PRs #492-#494) inherited a pre-existing
duplication pattern across the wrap subcommands and faithfully
extended it for cline/continue/goose/openhands. Each Pattern-B
subcommand (proxy-only watcher) inlined the same ~50 LOC of
proxy_holder + _make_cleanup + signal handlers + box-drawing banner
+ `while True: time.sleep(1)` watcher + try/except postlude. Each
Pattern-A subcommand (binary-launching) inlined the same ~15 LOC of
rtk-vs-lean-ctx fork + KeyboardInterrupt handler.

Replace with three focused helpers in wrap.py:

  _print_wrap_banner(agent)
    Centered 47-char unicode box. Adding a 9th agent no longer
    requires hand-padding the title to match the box width.

  _setup_context_tool_for_agent(...)
    rtk-or-lean-ctx fork + on_rtk_ready callback + rtk_required
    gate + KeyboardInterrupt -> SystemExit(130) with marker-path
    reporting. Used by cursor/cline/continue/goose/openhands.

  _run_proxy_only_watcher(...)
    Pattern-B scaffolding: signal handlers + banner + _ensure_proxy
    + setup callback + watcher loop + cleanup-on-finally. Used by
    cursor/cline/continue.

Production-code delta is small in raw LOC (+33 net on wrap.py)
because each subcommand still has a ~25-line `_print_X_setup`
callback closure. The win is architectural: adding wrap subcommand
#9 is now a ~25-line affair instead of ~150 lines, and behavior
(banner shape, Ctrl-C handling, cleanup ordering) is centralized
so a future fix lands in every subcommand at once.

Tests:
- New test_wrap_helpers.py (17 tests) directly pins each helper's
  contract — 5 branches of _setup_context_tool, 4 of
  _run_proxy_only_watcher, centering math of _print_wrap_banner.
- Merged the cline+goose hint-file tests into a single parametrized
  test_wrap_hintfile_agents.py (10 tests across [cline, goose]
  agents). test_wrap_cline.py is deleted; test_wrap_goose.py keeps
  only the goose-specific env-fan-out + binary-missing tests.
- Goose gained the "preserves existing hint-file content" test
  case that cline already had — net +1 coverage point.

Side benefit: cursor (pre-existing, not touched by G1) now gets
the SystemExit(130) on Ctrl-C-during-setup behavior the G1
subcommands had. Previously it would have surfaced a KeyboardInterrupt
traceback to the shell.

181 CLI tests pass; ci-precheck green.
2026-05-26 11:22:50 -07:00
chopratejas
0f3e3af6b2 fix(release): tag format vX.Y.Z (drop release-please component prefix)
release-please's default for manifest configs is
`include-component-in-tag: true`, which prepends the package name
to every tag: `headroom-ai-v0.22.4`. The repo's existing tags are
plain `vX.Y.Z` (v0.22.0, v0.22.1, v0.22.2, v0.22.3 — all created
by the prior release_version.py + create-release flow).

Without this override, the first release-please run on main
opened PR #496 trying to ship `headroom-ai-v0.23.0` and computed
its changelog from genesis because it couldn't find any tag
matching its expected prefix.

Set `include-component-in-tag: false` so the bot uses `vX.Y.Z`
and finds the existing baseline (v0.22.3) cleanly. Add a
regression test pinning this setting.
2026-05-25 20:40:12 -07:00