Merge cdc475f5ad into 27b4e2d147

CHANGELOG.md

@@ -252,6 +252,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 * **wrap (codex):** keep RTK guidance in the global Codex `AGENTS.md` instead of modifying the shared project `AGENTS.md` ([#1235](https://github.com/chopratejas/headroom/issues/1235)).
 * **subscription:** run the transcript token-window scan off the event loop (`asyncio.to_thread`). The subscription tracker's poll loop scanned every `~/.claude/projects/**/*.jsonl` transcript and `json.loads`'d each line inline on the proxy's single asyncio event loop; on large or long-running sessions this took seconds and froze `/health` and every in-flight proxied request — a periodic "wedge" recurring on the poll interval. The scan now runs in a worker thread so the loop stays responsive.
 * **gemini:** resolve future Gemini model capabilities through the shared model registry so token counting and context lookup no longer reject new Gemini families.
+* **parser:** fix the excessive-whitespace waste signal always reporting `0`. `detect_waste_signals` normalized matched whitespace runs with `" ".join(...)`, which kept each run intact and only inserted single spaces *between* runs, so the normalized form was never shorter than the original and `whitespace_tokens` was clamped to `0`. Each matched run (4+ spaces/tabs or 3+ newlines) now collapses to a single space, so the signal reports the real collapsible-whitespace savings it feeds into `WasteSignals.total()`.
 * **proxy:** enable SSO credential resolution in the native Bedrock route via the `aws-config` `sso` feature flag, making the credential chain match what `docs/bedrock.md` already documented ([#999](https://github.com/chopratejas/headroom/pull/999)).
 * **proxy:** route native Bedrock `/model/{id}/converse` requests to the upstream Converse endpoint instead of the hard-coded `/invoke` action — the non-streaming handler now resolves the action from the inbound path, matching the streaming handler ([#999](https://github.com/chopratejas/headroom/pull/999)).
 * **proxy:** preserve byte-faithful `/v1/messages` forwarding when Anthropic tool arrays are already canonical, and only canonicalize-and-mutate tool lists when sorting changes ordering ([#1042](https://github.com/chopratejas/headroom/issues/1042)).

headroom/parser.py

@@ -180,7 +180,9 @@ def detect_waste_signals(text: str, tokenizer: Tokenizer) -> WasteSignals:
     if ws_matches:
         # Count tokens that could be saved by normalizing whitespace to single spaces
         ws_text = "".join(ws_matches)
-        normalized_text = " ".join(ws_matches)
+        # Each matched run (4+ spaces/tabs or 3+ newlines) collapses to a
+        # single space, so the normalized form is one space per run.
+        normalized_text = " " * len(ws_matches)
         signals.whitespace_tokens = max(
             0, tokenizer.count_text(ws_text) - tokenizer.count_text(normalized_text)
         )

tests/test_parser.py

@@ -217,9 +217,28 @@ class TestDetectWasteSignals:
         assert signals.base64_tokens > 0
 
     def test_detect_excessive_whitespace(self, mock_tokenizer, whitespace_waste_text):
-        """Detects excessive whitespace as waste."""
+        """Detects excessive whitespace as waste.
+
+        The fixture has a 4-newline run and a 6-space run. Each run collapses
+        to a single space when normalized, so there are real savings to report:
+        ws_text "\\n\\n\\n\\n      " (10 chars -> 3 tokens) vs normalized "  "
+        (2 chars -> 1 token) = 2 tokens saved with the mock tokenizer.
+        """
         signals = detect_waste_signals(whitespace_waste_text, mock_tokenizer)
-        assert signals.whitespace_tokens >= 0  # May be 0 if normalized tokens <= matches
+        assert signals.whitespace_tokens == 2
+
+    def test_whitespace_savings_scale_with_run_length(self, mock_tokenizer):
+        """Long whitespace runs report proportionally large savings.
+
+        Regression guard: normalizing must collapse each matched run to a
+        single space, not just join the runs together (which left the runs
+        intact and made the signal always report 0).
+        """
+        text = "ERROR" + " " * 200 + "stack" + "\n" * 60 + "end"
+        signals = detect_waste_signals(text, mock_tokenizer)
+        # 260 whitespace chars collapse to 2 spaces; with ~1 token/4 chars the
+        # savings are large and clearly non-zero.
+        assert signals.whitespace_tokens > 50
 
     def test_detect_json_bloat(self, mock_tokenizer, json_bloat_text):
         """Detects large JSON blocks as bloat."""