headroom/.github/workflows/security.yml
Tejas Chopra 51a3b01174
feat(security): pilot hardening — stateless guarantee, model pinning, CI security gate (#1515)
## Description

Engineering hardening derived from the Box vendor security assessment.
Each change turns a "No/Partial" questionnaire answer into a genuine
"Yes" by making the product safer — not by editing the form. The
throughline is Headroom's core promise to enterprise pilots: **it runs
inside the customer's environment and never persists or leaks their
data.** These changes make that provable.

Three themes: (1) a complete **stateless write guarantee** (a stateless
proxy writes nothing to the workspace during serving), (2)
**data-at-rest** protection (no cleartext prompts written on errors),
and (3) **supply-chain integrity** (all model downloads pinned;
SCA/SAST/secret-scanning in CI).

Closes # (no tracking issue)

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

> Note: two deliberate, reversible default changes (not breaking): the
upstream-error debug dump is now off by default
(`HEADROOM_DEBUG_DUMP=1`/`=full` to opt in), and model downloads are
pinned (`HEADROOM_HF_PIN=off` to bypass). All stateless plumbing is a
pure no-op when not stateless.

## Changes Made

- **Stateless writes** — savings tracker + ledger, TOIN (`toin.json`),
and the output-savings recorder now honor stateless (in-memory only);
persistent memory is disabled under stateless with a warning. Added a
process-wide flag `headroom.paths.process_is_stateless()` (also honors
`HEADROOM_STATELESS`).
- **Debug dump** — the Anthropic *and* OpenAI handlers wrote full
requests (cleartext prompts/tools/system) to
`~/.headroom/logs/debug_400/` on every ≥400, even stateless. Now OFF by
default, stateless-aware, with a redacted middle tier; helpers extracted
to `handlers/_debug_dump.py`.
- **Model pinning** — all model downloads pin an immutable commit SHA:
our repos, kompress, image router/SigLIP, the third-party Qdrant memory
embedder (centralized in `onnx_runtime`), and the fastembed relevance
model (via the `revision` kwarg fastembed forwards to
`snapshot_download`). `HEADROOM_HF_PIN=off` bypasses.
- **CI security gate** — new `security.yml`: dependency audit
(pip-audit, scoped to the CVE-free `[all]` set), CodeQL (Python +
JS/TS), and gitleaks secret scanning (binary, MIT-licensed; PR-diff
scoped). `.gitleaks.toml` allowlists SBOM/lockfiles.
- **Dependabot** — extended to Rust (cargo) and npm (TS SDK, plugins,
docs).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ ruff check <10 changed source files>
All checks passed!

$ mypy <changed source files>
Success: no issues found in 7 source files   # + handlers/server: no issues (annotation-unchecked notes only)

$ pytest tests/test_stateless_writers.py tests/test_stateless_toin.py \
    tests/test_debug_dump_gating.py tests/test_hf_revision_pinning.py \
    tests/test_proxy_savings_history.py tests/test_observability_metrics.py \
    tests/test_toin.py tests/test_paths.py -q
================= 176 passed, 6 skipped, 2 warnings in 13.92s ==================
```

New tests (18): `tests/test_stateless_writers.py`,
`tests/test_stateless_toin.py`, `tests/test_debug_dump_gating.py`,
`tests/test_hf_revision_pinning.py`, plus stateless control assertions
in `tests/test_proxy_savings_history.py`. They include the non-stateless
control cases (savings/TOIN still persist) and a regression guard that
fails if any handler writes a debug dump without gating it.

## Real Behavior Proof

- **Environment:** macOS, Python 3.12 (`.venv`); CI on `ubuntu-latest`.
- **Exact command / steps:**
- Stateless guarantee: `SavingsRecorder(tmp/"output_savings.json")` +
`set_process_stateless(True)` → `flush()`; TOIN
`ToolIntelligenceNetwork(TOINConfig(storage_path=""))`;
`create_app(ProxyConfig(memory_enabled=True, stateless=True))`.
- Debug-dump gating: `_debug_dump_mode(SimpleNamespace(stateless=...))`
across env values.
- Model pinning: model SHAs fetched/verified against the live
HuggingFace API; `_resolve_revision` / `_pinned_revision` resolvers
tested.
- **Observed result:** under stateless, no `proxy_savings.json` /
`savings_events.jsonl` / `toin.json` / `output_savings.json` /
`memory.db` is created; `proxy.memory_handler is None`. With
`stateless=False` the control tests confirm each still persists. Debug
dump resolves to `off` by default and is forced off in stateless. CI:
dependency-audit, CodeQL (python + js/ts) pass; secret-scan now runs the
gitleaks binary.
- **Not tested:** `pip-audit` was not run on the local machine (broken
`ensurepip`); CI is the first real run (the committed all-extras grype
scan is clean). The fastembed download path is exercised by CI/runtime,
not in unit tests (the revision resolver is unit-tested).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

- **Concurrency:** `stateless` is a per-process config flag, never
per-request/per-session. Many sessions share one proxy's setting; a
stateless and a stateful proxy are separate OS processes with isolated
state. The one in-process edge (two proxies, different settings —
essentially tests) fails closed to in-memory, so a stateless proxy can
never leak.
- **Memory under stateless** is *disabled* (not in-RAM): the memory
subsystem is multi-component (SQLite + vector + markdown bridge) and a
partial in-RAM mode would be risky; ephemeral containers and
cross-session learning are contradictory. An ephemeral in-RAM memory
mode is a possible follow-up.
- **Docs/CHANGELOG** left unchecked: the two new env vars
(`HEADROOM_DEBUG_DUMP`, `HEADROOM_HF_PIN`) and the stateless behavior
changes are documented in code comments; happy to add user docs + a
CHANGELOG entry if preferred.
- CI deprecation warnings (Node 20, CodeQL Action v3) are GitHub-side
and out of scope here.
2026-06-27 17:44:10 -07:00

120 lines
3.9 KiB
YAML

name: Security
# Security gate: dependency vulnerability scanning (SCA), static analysis
# (CodeQL/SAST), and secret scanning. Runs on every PR to main, on push to
# main, weekly (to catch newly-disclosed CVEs without a code change), and on
# demand. Each job is an independent required check.
on:
push:
branches: [main]
pull_request:
branches: [main]
schedule:
# Mondays 06:00 UTC — surface CVEs disclosed since the last commit.
- cron: "0 6 * * 1"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: security-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
jobs:
# ---- SCA: dependency vulnerability scan -------------------------------
# Audits the PRODUCTION dependency set ([all]) exported from uv.lock. The
# `benchmark` extra is intentionally excluded from [all] (it pulls lm-eval's
# sqlitedict/nltk, which carry unpatchable upstream High CVEs and are never
# installed in production), so this gate fails only on actionable findings.
dependency-audit:
name: Dependency audit (pip-audit)
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Install uv
uses: astral-sh/setup-uv@v5
- name: Export production dependency set from uv.lock
run: |
uv export --frozen --no-dev --no-emit-project --no-hashes \
--extra all --format requirements-txt > requirements-prod.txt
echo "Production dependencies audited:"
wc -l requirements-prod.txt
- name: Audit dependencies (pip-audit)
uses: pypa/gh-action-pip-audit@v1.1.0
with:
inputs: requirements-prod.txt
# ---- SAST: CodeQL static analysis ------------------------------------
codeql:
name: CodeQL (${{ matrix.language }})
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
security-events: write
actions: read
strategy:
fail-fast: false
matrix:
language: [python, javascript-typescript]
steps:
- uses: actions/checkout@v4
- name: Initialize CodeQL
uses: github/codeql-action/init@v3
with:
languages: ${{ matrix.language }}
queries: security-extended
- name: Perform CodeQL analysis
uses: github/codeql-action/analyze@v3
with:
category: "/language:${{ matrix.language }}"
# ---- Secret scanning -------------------------------------------------
# Uses the gitleaks BINARY (MIT-licensed, no key) instead of
# gitleaks-action, which requires a paid GITLEAKS_LICENSE for organization
# repos. On PRs we scan only the PR's commits so pre-existing history can't
# block a PR; on push/schedule we scan the working tree. Config + allowlist
# live in .gitleaks.toml at the repo root (auto-loaded).
secret-scan:
name: Secret scan (gitleaks)
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Install gitleaks
run: |
version=8.18.4
curl -sSfL \
"https://github.com/gitleaks/gitleaks/releases/download/v${version}/gitleaks_${version}_linux_x64.tar.gz" \
-o /tmp/gitleaks.tar.gz
tar -xzf /tmp/gitleaks.tar.gz -C /tmp gitleaks
sudo install /tmp/gitleaks /usr/local/bin/gitleaks
gitleaks version
- name: Scan for secrets
env:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
run: |
if [ -n "$BASE_SHA" ]; then
echo "Scanning PR commits ${BASE_SHA}..HEAD"
gitleaks detect --source . --log-opts="${BASE_SHA}..HEAD" --redact --no-banner
else
echo "Scanning working tree"
gitleaks detect --source . --no-git --redact --no-banner
fi