mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-10 14:27:00 -04:00
## Description
Engineering hardening derived from the Box vendor security assessment.
Each change turns a "No/Partial" questionnaire answer into a genuine
"Yes" by making the product safer — not by editing the form. The
throughline is Headroom's core promise to enterprise pilots: **it runs
inside the customer's environment and never persists or leaks their
data.** These changes make that provable.
Three themes: (1) a complete **stateless write guarantee** (a stateless
proxy writes nothing to the workspace during serving), (2)
**data-at-rest** protection (no cleartext prompts written on errors),
and (3) **supply-chain integrity** (all model downloads pinned;
SCA/SAST/secret-scanning in CI).
Closes # (no tracking issue)
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
> Note: two deliberate, reversible default changes (not breaking): the
upstream-error debug dump is now off by default
(`HEADROOM_DEBUG_DUMP=1`/`=full` to opt in), and model downloads are
pinned (`HEADROOM_HF_PIN=off` to bypass). All stateless plumbing is a
pure no-op when not stateless.
## Changes Made
- **Stateless writes** — savings tracker + ledger, TOIN (`toin.json`),
and the output-savings recorder now honor stateless (in-memory only);
persistent memory is disabled under stateless with a warning. Added a
process-wide flag `headroom.paths.process_is_stateless()` (also honors
`HEADROOM_STATELESS`).
- **Debug dump** — the Anthropic *and* OpenAI handlers wrote full
requests (cleartext prompts/tools/system) to
`~/.headroom/logs/debug_400/` on every ≥400, even stateless. Now OFF by
default, stateless-aware, with a redacted middle tier; helpers extracted
to `handlers/_debug_dump.py`.
- **Model pinning** — all model downloads pin an immutable commit SHA:
our repos, kompress, image router/SigLIP, the third-party Qdrant memory
embedder (centralized in `onnx_runtime`), and the fastembed relevance
model (via the `revision` kwarg fastembed forwards to
`snapshot_download`). `HEADROOM_HF_PIN=off` bypasses.
- **CI security gate** — new `security.yml`: dependency audit
(pip-audit, scoped to the CVE-free `[all]` set), CodeQL (Python +
JS/TS), and gitleaks secret scanning (binary, MIT-licensed; PR-diff
scoped). `.gitleaks.toml` allowlists SBOM/lockfiles.
- **Dependabot** — extended to Rust (cargo) and npm (TS SDK, plugins,
docs).
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ ruff check <10 changed source files>
All checks passed!
$ mypy <changed source files>
Success: no issues found in 7 source files # + handlers/server: no issues (annotation-unchecked notes only)
$ pytest tests/test_stateless_writers.py tests/test_stateless_toin.py \
tests/test_debug_dump_gating.py tests/test_hf_revision_pinning.py \
tests/test_proxy_savings_history.py tests/test_observability_metrics.py \
tests/test_toin.py tests/test_paths.py -q
================= 176 passed, 6 skipped, 2 warnings in 13.92s ==================
```
New tests (18): `tests/test_stateless_writers.py`,
`tests/test_stateless_toin.py`, `tests/test_debug_dump_gating.py`,
`tests/test_hf_revision_pinning.py`, plus stateless control assertions
in `tests/test_proxy_savings_history.py`. They include the non-stateless
control cases (savings/TOIN still persist) and a regression guard that
fails if any handler writes a debug dump without gating it.
## Real Behavior Proof
- **Environment:** macOS, Python 3.12 (`.venv`); CI on `ubuntu-latest`.
- **Exact command / steps:**
- Stateless guarantee: `SavingsRecorder(tmp/"output_savings.json")` +
`set_process_stateless(True)` → `flush()`; TOIN
`ToolIntelligenceNetwork(TOINConfig(storage_path=""))`;
`create_app(ProxyConfig(memory_enabled=True, stateless=True))`.
- Debug-dump gating: `_debug_dump_mode(SimpleNamespace(stateless=...))`
across env values.
- Model pinning: model SHAs fetched/verified against the live
HuggingFace API; `_resolve_revision` / `_pinned_revision` resolvers
tested.
- **Observed result:** under stateless, no `proxy_savings.json` /
`savings_events.jsonl` / `toin.json` / `output_savings.json` /
`memory.db` is created; `proxy.memory_handler is None`. With
`stateless=False` the control tests confirm each still persists. Debug
dump resolves to `off` by default and is forced off in stateless. CI:
dependency-audit, CodeQL (python + js/ts) pass; secret-scan now runs the
gitleaks binary.
- **Not tested:** `pip-audit` was not run on the local machine (broken
`ensurepip`); CI is the first real run (the committed all-extras grype
scan is clean). The fastembed download path is exercised by CI/runtime,
not in unit tests (the revision resolver is unit-tested).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
- **Concurrency:** `stateless` is a per-process config flag, never
per-request/per-session. Many sessions share one proxy's setting; a
stateless and a stateful proxy are separate OS processes with isolated
state. The one in-process edge (two proxies, different settings —
essentially tests) fails closed to in-memory, so a stateless proxy can
never leak.
- **Memory under stateless** is *disabled* (not in-RAM): the memory
subsystem is multi-component (SQLite + vector + markdown bridge) and a
partial in-RAM mode would be risky; ephemeral containers and
cross-session learning are contradictory. An ephemeral in-RAM memory
mode is a possible follow-up.
- **Docs/CHANGELOG** left unchecked: the two new env vars
(`HEADROOM_DEBUG_DUMP`, `HEADROOM_HF_PIN`) and the stateless behavior
changes are documented in code comments; happy to add user docs + a
CHANGELOG entry if preferred.
- CI deprecation warnings (Node 20, CodeQL Action v3) are GitHub-side
and out of scope here.
120 lines
3.9 KiB
YAML
120 lines
3.9 KiB
YAML
name: Security
|
|
|
|
# Security gate: dependency vulnerability scanning (SCA), static analysis
|
|
# (CodeQL/SAST), and secret scanning. Runs on every PR to main, on push to
|
|
# main, weekly (to catch newly-disclosed CVEs without a code change), and on
|
|
# demand. Each job is an independent required check.
|
|
|
|
on:
|
|
push:
|
|
branches: [main]
|
|
pull_request:
|
|
branches: [main]
|
|
schedule:
|
|
# Mondays 06:00 UTC — surface CVEs disclosed since the last commit.
|
|
- cron: "0 6 * * 1"
|
|
workflow_dispatch:
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
concurrency:
|
|
group: security-${{ github.workflow }}-${{ github.ref }}
|
|
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
|
|
|
|
jobs:
|
|
# ---- SCA: dependency vulnerability scan -------------------------------
|
|
# Audits the PRODUCTION dependency set ([all]) exported from uv.lock. The
|
|
# `benchmark` extra is intentionally excluded from [all] (it pulls lm-eval's
|
|
# sqlitedict/nltk, which carry unpatchable upstream High CVEs and are never
|
|
# installed in production), so this gate fails only on actionable findings.
|
|
dependency-audit:
|
|
name: Dependency audit (pip-audit)
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 15
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
|
|
- name: Set up Python
|
|
uses: actions/setup-python@v5
|
|
with:
|
|
python-version: "3.12"
|
|
|
|
- name: Install uv
|
|
uses: astral-sh/setup-uv@v5
|
|
|
|
- name: Export production dependency set from uv.lock
|
|
run: |
|
|
uv export --frozen --no-dev --no-emit-project --no-hashes \
|
|
--extra all --format requirements-txt > requirements-prod.txt
|
|
echo "Production dependencies audited:"
|
|
wc -l requirements-prod.txt
|
|
|
|
- name: Audit dependencies (pip-audit)
|
|
uses: pypa/gh-action-pip-audit@v1.1.0
|
|
with:
|
|
inputs: requirements-prod.txt
|
|
|
|
# ---- SAST: CodeQL static analysis ------------------------------------
|
|
codeql:
|
|
name: CodeQL (${{ matrix.language }})
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 30
|
|
permissions:
|
|
contents: read
|
|
security-events: write
|
|
actions: read
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
language: [python, javascript-typescript]
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
|
|
- name: Initialize CodeQL
|
|
uses: github/codeql-action/init@v3
|
|
with:
|
|
languages: ${{ matrix.language }}
|
|
queries: security-extended
|
|
|
|
- name: Perform CodeQL analysis
|
|
uses: github/codeql-action/analyze@v3
|
|
with:
|
|
category: "/language:${{ matrix.language }}"
|
|
|
|
# ---- Secret scanning -------------------------------------------------
|
|
# Uses the gitleaks BINARY (MIT-licensed, no key) instead of
|
|
# gitleaks-action, which requires a paid GITLEAKS_LICENSE for organization
|
|
# repos. On PRs we scan only the PR's commits so pre-existing history can't
|
|
# block a PR; on push/schedule we scan the working tree. Config + allowlist
|
|
# live in .gitleaks.toml at the repo root (auto-loaded).
|
|
secret-scan:
|
|
name: Secret scan (gitleaks)
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 10
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
with:
|
|
fetch-depth: 0
|
|
|
|
- name: Install gitleaks
|
|
run: |
|
|
version=8.18.4
|
|
curl -sSfL \
|
|
"https://github.com/gitleaks/gitleaks/releases/download/v${version}/gitleaks_${version}_linux_x64.tar.gz" \
|
|
-o /tmp/gitleaks.tar.gz
|
|
tar -xzf /tmp/gitleaks.tar.gz -C /tmp gitleaks
|
|
sudo install /tmp/gitleaks /usr/local/bin/gitleaks
|
|
gitleaks version
|
|
|
|
- name: Scan for secrets
|
|
env:
|
|
BASE_SHA: ${{ github.event.pull_request.base.sha }}
|
|
run: |
|
|
if [ -n "$BASE_SHA" ]; then
|
|
echo "Scanning PR commits ${BASE_SHA}..HEAD"
|
|
gitleaks detect --source . --log-opts="${BASE_SHA}..HEAD" --redact --no-banner
|
|
else
|
|
echo "Scanning working tree"
|
|
gitleaks detect --source . --no-git --redact --no-banner
|
|
fi
|