mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
51a3b01174
## Description
Engineering hardening derived from the Box vendor security assessment.
Each change turns a "No/Partial" questionnaire answer into a genuine
"Yes" by making the product safer — not by editing the form. The
throughline is Headroom's core promise to enterprise pilots: **it runs
inside the customer's environment and never persists or leaks their
data.** These changes make that provable.
Three themes: (1) a complete **stateless write guarantee** (a stateless
proxy writes nothing to the workspace during serving), (2)
**data-at-rest** protection (no cleartext prompts written on errors),
and (3) **supply-chain integrity** (all model downloads pinned;
SCA/SAST/secret-scanning in CI).
Closes # (no tracking issue)
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
> Note: two deliberate, reversible default changes (not breaking): the
upstream-error debug dump is now off by default
(`HEADROOM_DEBUG_DUMP=1`/`=full` to opt in), and model downloads are
pinned (`HEADROOM_HF_PIN=off` to bypass). All stateless plumbing is a
pure no-op when not stateless.
## Changes Made
- **Stateless writes** — savings tracker + ledger, TOIN (`toin.json`),
and the output-savings recorder now honor stateless (in-memory only);
persistent memory is disabled under stateless with a warning. Added a
process-wide flag `headroom.paths.process_is_stateless()` (also honors
`HEADROOM_STATELESS`).
- **Debug dump** — the Anthropic *and* OpenAI handlers wrote full
requests (cleartext prompts/tools/system) to
`~/.headroom/logs/debug_400/` on every ≥400, even stateless. Now OFF by
default, stateless-aware, with a redacted middle tier; helpers extracted
to `handlers/_debug_dump.py`.
- **Model pinning** — all model downloads pin an immutable commit SHA:
our repos, kompress, image router/SigLIP, the third-party Qdrant memory
embedder (centralized in `onnx_runtime`), and the fastembed relevance
model (via the `revision` kwarg fastembed forwards to
`snapshot_download`). `HEADROOM_HF_PIN=off` bypasses.
- **CI security gate** — new `security.yml`: dependency audit
(pip-audit, scoped to the CVE-free `[all]` set), CodeQL (Python +
JS/TS), and gitleaks secret scanning (binary, MIT-licensed; PR-diff
scoped). `.gitleaks.toml` allowlists SBOM/lockfiles.
- **Dependabot** — extended to Rust (cargo) and npm (TS SDK, plugins,
docs).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ ruff check <10 changed source files>
All checks passed!
$ mypy <changed source files>
Success: no issues found in 7 source files # + handlers/server: no issues (annotation-unchecked notes only)
$ pytest tests/test_stateless_writers.py tests/test_stateless_toin.py \
tests/test_debug_dump_gating.py tests/test_hf_revision_pinning.py \
tests/test_proxy_savings_history.py tests/test_observability_metrics.py \
tests/test_toin.py tests/test_paths.py -q
================= 176 passed, 6 skipped, 2 warnings in 13.92s ==================
```
New tests (18): `tests/test_stateless_writers.py`,
`tests/test_stateless_toin.py`, `tests/test_debug_dump_gating.py`,
`tests/test_hf_revision_pinning.py`, plus stateless control assertions
in `tests/test_proxy_savings_history.py`. They include the non-stateless
control cases (savings/TOIN still persist) and a regression guard that
fails if any handler writes a debug dump without gating it.
## Real Behavior Proof
- **Environment:** macOS, Python 3.12 (`.venv`); CI on `ubuntu-latest`.
- **Exact command / steps:**
- Stateless guarantee: `SavingsRecorder(tmp/"output_savings.json")` +
`set_process_stateless(True)` → `flush()`; TOIN
`ToolIntelligenceNetwork(TOINConfig(storage_path=""))`;
`create_app(ProxyConfig(memory_enabled=True, stateless=True))`.
- Debug-dump gating: `_debug_dump_mode(SimpleNamespace(stateless=...))`
across env values.
- Model pinning: model SHAs fetched/verified against the live
HuggingFace API; `_resolve_revision` / `_pinned_revision` resolvers
tested.
- **Observed result:** under stateless, no `proxy_savings.json` /
`savings_events.jsonl` / `toin.json` / `output_savings.json` /
`memory.db` is created; `proxy.memory_handler is None`. With
`stateless=False` the control tests confirm each still persists. Debug
dump resolves to `off` by default and is forced off in stateless. CI:
dependency-audit, CodeQL (python + js/ts) pass; secret-scan now runs the
gitleaks binary.
- **Not tested:** `pip-audit` was not run on the local machine (broken
`ensurepip`); CI is the first real run (the committed all-extras grype
scan is clean). The fastembed download path is exercised by CI/runtime,
not in unit tests (the revision resolver is unit-tested).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
- **Concurrency:** `stateless` is a per-process config flag, never
per-request/per-session. Many sessions share one proxy's setting; a
stateless and a stateful proxy are separate OS processes with isolated
state. The one in-process edge (two proxies, different settings —
essentially tests) fails closed to in-memory, so a stateless proxy can
never leak.
- **Memory under stateless** is *disabled* (not in-RAM): the memory
subsystem is multi-component (SQLite + vector + markdown bridge) and a
partial in-RAM mode would be risky; ephemeral containers and
cross-session learning are contradictory. An ephemeral in-RAM memory
mode is a possible follow-up.
- **Docs/CHANGELOG** left unchecked: the two new env vars
(`HEADROOM_DEBUG_DUMP`, `HEADROOM_HF_PIN`) and the stateless behavior
changes are documented in code comments; happy to add user docs + a
CHANGELOG entry if preferred.
- CI deprecation warnings (Node 20, CodeQL Action v3) are GitHub-side
and out of scope here.
53 lines
2 KiB
Python
53 lines
2 KiB
Python
"""Tests for HuggingFace model-revision pinning (supply-chain integrity).
|
|
|
|
Model artifacts are pinned to immutable commit SHAs so a changed or compromised
|
|
upstream repo cannot be pulled silently. Pinning is centralized in the download
|
|
helper so every call site (kompress, memory embedder, image router) inherits it.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from headroom.onnx_runtime import _PINNED_REVISIONS, _resolve_revision
|
|
|
|
|
|
def test_known_repos_are_pinned_to_sha():
|
|
# Every pinned revision must be a full 40-char git SHA, not a branch/tag.
|
|
assert _PINNED_REVISIONS, "expected at least one pinned model repo"
|
|
for repo, sha in _PINNED_REVISIONS.items():
|
|
assert len(sha) == 40 and all(c in "0123456789abcdef" for c in sha), (
|
|
f"{repo} is not pinned to a full commit SHA: {sha!r}"
|
|
)
|
|
|
|
|
|
def test_default_kompress_model_is_pinned():
|
|
# The shipping model must be pinned.
|
|
assert "chopratejas/kompress-v2-base" in _PINNED_REVISIONS
|
|
|
|
|
|
def test_resolve_uses_pin_for_known_repo(monkeypatch):
|
|
monkeypatch.delenv("HEADROOM_HF_PIN", raising=False)
|
|
repo = "chopratejas/kompress-v2-base"
|
|
assert _resolve_revision(repo, None) == _PINNED_REVISIONS[repo]
|
|
|
|
|
|
def test_explicit_revision_overrides_pin(monkeypatch):
|
|
monkeypatch.delenv("HEADROOM_HF_PIN", raising=False)
|
|
assert _resolve_revision("chopratejas/kompress-v2-base", "deadbeef") == "deadbeef"
|
|
|
|
|
|
def test_unknown_repo_is_not_pinned(monkeypatch):
|
|
monkeypatch.delenv("HEADROOM_HF_PIN", raising=False)
|
|
assert _resolve_revision("some/unknown-model", None) is None
|
|
|
|
|
|
@pytest.mark.parametrize("value", ["off", "0", "false", "no", "OFF"])
|
|
def test_pin_can_be_disabled_via_env(monkeypatch, value):
|
|
monkeypatch.setenv("HEADROOM_HF_PIN", value)
|
|
assert _resolve_revision("chopratejas/kompress-v2-base", None) is None
|
|
|
|
|
|
def test_pin_disabled_still_respects_explicit_revision(monkeypatch):
|
|
monkeypatch.setenv("HEADROOM_HF_PIN", "off")
|
|
assert _resolve_revision("chopratejas/kompress-v2-base", "abc123") == "abc123"
|