Before this change, every persisted traffic_learner row in memory.db
landed with evidence_count=1, causing two user-visible problems:
1. The live flush gate (evidence_count >= 2) filtered out every row, so
CLAUDE.md / MEMORY.md never received the patterns the learner saw
repeatedly.
2. _saved_hashes is in-memory only and reset on each proxy restart, so
a pattern seen once in session A then twice in session B would insert
a *duplicate* DB row instead of bumping the existing one. Users
accumulated many rows stuck at 1 instead of a few rows with high
evidence.
Root cause chain:
- _accumulate tracks a running count in the _pattern_counts tuple but
enqueues the ExtractedPattern dataclass with its default
evidence_count=1 intact.
- _save_worker writes pattern.evidence_count into metadata verbatim.
- After save, the hash goes into _saved_hashes and further sightings
are early-returned — never bumped.
- Next process start has empty _saved_hashes, so the same content goes
through the accumulator as fresh and gets re-saved.
Fix:
- _accumulate now sets pattern.evidence_count = count before enqueuing,
so DB rows reflect the real number of sightings at save time.
- _save_worker captures the Memory.id returned by save_memory and
records content_hash → id in a new _persisted_ids map.
- _accumulate's saved-hash branch now awaits
_bump_persisted_evidence(memory_id), which runs an atomic
json_set('$.evidence_count', existing + 1) UPDATE via
asyncio.to_thread to keep the proxy hot path non-blocking.
- start() calls a new _hydrate_persisted_state() that reads existing
traffic_learner rows' (id, content) pairs from the DB and pre-seeds
_saved_hashes + _persisted_ids. Cross-session re-sightings bump the
seeded row instead of inserting a duplicate.
- _load_persisted_patterns_from_sqlite and _hydrate_persisted_state
query by json_extract(metadata, '$.source') = 'traffic_learner'
instead of the prior LIKE on raw JSON — the bump path uses json_set,
which rewrites the metadata string without the default ": " spacing,
which would otherwise make the LIKE blind to bumped rows.
Adds TestEvidencePersistence with three cases:
- save persists the actual accumulated count (not the default 1)
- re-sightings bump the persisted row instead of creating duplicates
- a fresh learner hydrates _saved_hashes from DB, so cross-session
re-sightings bump the pre-existing row
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the previous shutdown-only flush with a debounced, near-real-time
dirty-flag flush worker that writes patterns into the correct CLAUDE.md /
MEMORY.md bucket as traffic accumulates.
- New FLUSH_DEBOUNCE_SECONDS gate (10s) prevents context-file thrash on
bursty traffic while keeping updates "live" from the user's perspective.
- TrafficLearner.start() now spawns a _flush_worker alongside the save
worker; _accumulate() sets a dirty flag; _flush_worker() calls
flush_to_file() when dirty and past the debounce window.
- flush_to_file() now reads *both* persisted rows (memory.db) and the
in-memory accumulator via _load_persisted_patterns_from_sqlite and
_collect_all_patterns, so patterns survive proxy restarts and the
agent-native files converge toward the full learned set.
- Patterns are bucketed per-project via the learn plugin registry
(plugin.discover_projects()) and anchored to project roots through
longest-matching-path on content or entity_refs
(_project_for_pattern). Un-anchored patterns are dropped.
- Patterns are routed by PatternCategory to either CONTEXT_FILE
(CLAUDE.md) or MEMORY_FILE (MEMORY.md) via
_patterns_to_recommendations + _CATEGORY_TO_TARGET.
- Live flushes require evidence_count >= 2; shutdown flushes accept
single-evidence rows to avoid losing last-session signal.
Adds tests for project routing, persisted-pattern loading, category
routing, and the debounced flush worker.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`_finalize_stream_response` recorded metrics, cost, and prefix-cache stats
but never appended a `RequestLog` to the request logger. Because the
streaming Anthropic path is what Claude Code uses, this meant
`/stats.recent_requests` and `/transformations/feed` were permanently
empty for typical traffic — even when the proxy was started with
`--log-messages`. Only the non-streaming Anthropic path
(`anthropic.py:1167, 1599`) and the Bedrock streaming finalizer
(`streaming.py:_stream_response_bedrock`) were logged.
Wire the same `RequestLog` shape the other paths build, plumbing `tags`
through both `_finalize_stream_response` call sites in `_stream_response`
and respecting `config.log_full_messages` for `request_messages`.
Adds `tests/test_proxy_streaming_request_logger.py` covering the happy
path, both `log_full_messages` branches, the zero-original-tokens edge,
and the logger-disabled no-op.
Three independent pre-existing test-hygiene regressions on main, all
surfaced as cascading CI failures:
1. tests/test_cli/test_wrap_copilot.py (from #229) mutated
sys.modules["headroom.cli.main"] with a fake click.Group() at
module-import time and never restored it. Any later test that did
`from headroom.cli.main import main` got an empty group with no
version option and no registered subcommands, breaking ~20
test_cli/* and test_cli_proxy_env.py tests. Rewrite to import the
real `main` directly — the fake-group indirection served no
purpose.
2. tests/test_proxy_copilot_auth_hooks.py (from #229) installed fake
httpx / fastapi.responses / headroom.proxy.* modules into
sys.modules inside a helper called from test functions, never
cleaned up. Later tests that imported ASGITransport or JSONResponse
hit the fakes and failed with ImportError. Switch the helper to
monkeypatch.setitem so the fakes are scoped to the owning test.
3. tests/test_release_version.py hardcoded canonical=0.5.25 in the
subprocess-output assertion; the project version in pyproject.toml
has since bumped to 0.9.1. Compute the expected value dynamically
via get_canonical_version(ROOT) so the test tracks pyproject.
Drive-by: main is currently failing `ruff format --check .` because of
two missing blank lines between two top-level functions in this file
(introduced in 8bf11d2). Fixing it here so this PR's CI can go green —
no other way to unblock the format check without landing a separate PR
first.
`headroom learn` built the marker block from only the current run's
recommendations and wholesale-replaced any prior block via
`_MARKER_PATTERN.sub`. Sections learned weeks earlier that didn't
re-surface in a later run were silently dropped.
Fix: in `_merge_into_file`, parse recommendations out of the prior
block and union them with the new run's recommendations. Sections
re-surfaced by the new run take precedence (latest analysis wins);
sections not re-surfaced are carried forward so learnings accumulate
instead of getting clobbered.
To fully rebuild the block, delete it manually and re-run.
Tests: existing wholesale-replace test rewritten as a carry-forward
assertion. Added tests for same-section override, MEMORY.md
carry-forward, and round-trip of sections without a tokens annotation.
Closes#231
Move importorskip after playwright import so module-level
import error triggers skip rather than collection error.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
