headroom/scripts
Tejas Chopra 5383c6bf2f
fix(release): sync generated version metadata on the release branch (#2659)
## Description

The 0.33.0 release PR (#2339) has sat in `changes-requested` since
2026-07-17. Root cause: **release-please only rewrites `pyproject.toml`
and its configured `extra-files`**, but other tracked files also carry
the version — and `server.json` is asserted byte-for-byte against
`render_server_json()`, which derives its version from `pyproject.toml`.
So the bump alone fails
`tests/test_mcp_registry/test_server_json.py::test_root_server_json_matches_builder`
(the `test (2)` shard) on every regenerated release PR.

Nothing in the repo regenerated `server.json` at all, so it fell behind
every release.

Unblocks #2339.

### Why the release *build* passes but the release PR does not

`release.yml` already runs `scripts/version-sync.py` immediately before
its own `verify-versions.py` gate (lines 145 and 278). That is why
`build` and `build-wheels` are green on #2339 despite the drift — it
syncs in the workspace, uncommitted. The regular CI test job does
**not** sync, so the fix has to be committed to the branch.

This also explains why reviewers kept seeing `verify-versions.py` fail
locally while CI's build jobs passed: the verifier is never run
un-synced inside `release.yml`.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **`scripts/version-sync.py`**: also write `server.json`. It was the
one version-carrying file with no writer anywhere. Values are rewritten
in place so key order and formatting keep matching the builder's
byte-for-byte output (verified: the file is pure ASCII and round-trips
exactly through `json.dumps(..., indent=2) + "\n"`).
- **`.github/workflows/release-metadata-sync.yml`** (new): on a push to
`release-please--branches--**`, run version-sync → gate on
verify-versions → commit if changed.
- **Keyed off the branch push** because release-please force-regenerates
that branch on every merge to main. That is precisely what wiped the
hand-pushed metadata fixes on #2339 (`2a86c8ff`, `d5ea4dc5`) — a push
trigger re-heals after every regeneration instead of being lost.
- **Uses the same PAT as `release-please.yml`**: a `GITHUB_TOKEN` push
does not trigger workflows, so the release PR's checks would never
re-run against the synced commit and would stay red.
- **Idempotent**: the self-triggered rerun finds no diff and exits
before pushing, so the loop terminates after one no-op run.
- **Corrected pre-existing drift on `main`**: the agent-hooks plugin
manifests, both marketplace manifests, and `.releasemetadata` were
stranded at **0.31.0** — never bumped for 0.32.0 either.
`verify-versions.py` now passes on `main`.

### Why not more `extra-files` entries

That would need ~13 jsonpath entries restating what `version-sync.py`
already knows, and a jsonpath that fails to match **fails silently** —
the same class of failure this PR removes, discoverable only after a
real release PR regenerates. There is also no precedent for nested
jsonpath (`$.packages[0].version`, `$.metadata.version`) in the config
today; both existing entries are plain `$.version`. Running the script
keeps one source of truth, and files added to it later are covered with
no change here.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`) — no `headroom/` sources
touched
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ python -m pytest scripts/tests/ tests/test_release_workflows.py tests/test_mcp_registry/ -q
207 passed in 2.69s

$ ruff check scripts/version-sync.py scripts/tests/test_version_sync.py tests/test_release_workflows.py
All checks passed!
$ ruff format --check <same>
3 files already formatted

$ actionlint .github/workflows/release-metadata-sync.yml
(clean)
```

New tests:
- `test_server_json_version_is_synchronized` — version-sync moves both
`server.json` version fields and preserves the other keys.
- `test_release_metadata_sync_runs_on_release_please_branch` — asserts
the trigger, the sync→verify→commit ordering, the no-op guard, and the
PAT.
- `test_version_sync_covers_every_file_the_verifier_gates` — guards
`version-sync.py` and `verify-versions.py` against drifting apart again,
which is the root cause here.

## Real Behavior Proof

- **Environment:** macOS (Darwin arm64), Python 3.12, repo venv.
- **Exact command / steps:** reproduced the CI failure locally by
simulating release-please's partial bump, then applying the fix.

**Reproducing the exact `test (2)` failure** — set `pyproject` to 0.33.0
while `server.json` stays at 0.32.0, as release-please leaves it:

```text
$ python -m pytest tests/test_mcp_registry/test_server_json.py -q
FAILED tests/test_mcp_registry/test_server_json.py::test_root_server_json_matches_builder
1 failed, 3 passed
```

**After `version-sync.py`:**

```text
$ python scripts/version-sync.py && python -m pytest tests/test_mcp_registry/test_server_json.py -q
4 passed
```

**Both gates green on a simulated 0.33.0 bump:**

```text
$ python scripts/version-sync.py --version 0.33.0
Version synchronized to 0.33.0
$ python scripts/verify-versions.py
All versions aligned at 0.33.0
$ python -m pytest tests/test_mcp_registry/test_server_json.py -q
4 passed
```

**Idempotency** (the property the workflow's loop-termination relies
on): re-running against an already-synced tree leaves `pyproject.toml`,
`server.json`, `openclaw`, and `sdk/typescript` untouched.

- **Not tested:** the workflow has not executed on a real release-please
branch regeneration — that can only be exercised once this is on `main`
and release-please next updates #2339. The PAT push path and the
self-trigger no-op are reasoned from `release-please.yml`'s existing
token comment and from local idempotency, not observed in CI.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

**Context on the v0.32.0 release failure, since it is easy to misread as
"images never build".** Every artifact built for v0.32.0 — all 5 wheel
platforms including Windows, all 16 Docker builds + 8 manifests +
`promote-latest`, npm, and GitHub Packages. Only `publish-pypi` failed
(PyPI attestations, already fixed by `f9cbdd6e` / #2405), and
`create-release` was skipped because it depends on it. That is why the
release looked like it produced nothing.

**Separate, approaching blocker — not addressed here.** PyPI is at
**9.69 GB of its 10 GB project cap (96.9%)**, leaving ~305 MB against
~68 MB per release, so roughly 4 more releases fit. The `0.21.x` series
alone holds **6.58 GB across 31 releases**, from the old
every-push-is-a-release era; pruning it would reclaim two thirds of the
quota. Worth a separate issue.

**`.releasemetadata` is written but never read** by anything outside
`version-sync.py` and its test. It is kept in sync here for internal
consistency, but it may be a deletion candidate.
2026-07-29 15:12:04 -07:00
..
ci ci: harden PR governance and model cache checks (#1401) 2026-06-26 21:34:34 -07:00
fixtures feat(scripts): add Codex proxy reconnect-storm repro harness 2026-04-20 22:02:02 +07:00
tests fix(release): sync generated version metadata on the release branch (#2659) 2026-07-29 15:12:04 -07:00
audit_wheel_glibc_symbols.py fix(crusher): shim __libc_single_threaded for glibc < 2.32 + extend audit 2026-05-05 13:59:21 -07:00
bootstrap-windows-dev.ps1 chore(release): harden local artifact smokes (#1824) 2026-07-14 16:07:34 -04:00
build_npm_release_assets.mjs chore(release): harden local artifact smokes (#1824) 2026-07-14 16:07:34 -04:00
build_python_release_smoke.py chore(release): harden local artifact smokes (#1824) 2026-07-14 16:07:34 -04:00
build_rust_extension.sh refactor: single-wheel maturin build backend (fixes #355) 2026-05-03 13:16:41 -07:00
changelog-gen.py chore: renormalize line endings to LF 2026-04-24 15:33:30 +02:00
eval_output_shaper.py feat: output-token reduction — verbosity shaper, per-user learning, counterfactual savings (#965) 2026-06-16 21:06:43 -07:00
export_kompress_v2_onnx.py feat: switch Kompress default to kompress-v2-base with weight-only int8 ONNX (#799) 2026-06-09 23:28:40 -07:00
install-git-hooks.sh Fix CI lint failure by formatting PR governance scripts (#933) 2026-06-12 17:11:39 -05:00
install.ps1 fix(install): default docker image to headroomlabs-ai GHCR registry (#1867) (#2039) 2026-07-13 14:01:28 -04:00
install.sh fix(install): default docker image to headroomlabs-ai GHCR registry (#1867) (#2039) 2026-07-13 14:01:28 -04:00
pr-governance.py ci: harden PR governance and model cache checks (#1401) 2026-06-26 21:34:34 -07:00
README.md chore(release): harden local artifact smokes (#1824) 2026-07-14 16:07:34 -04:00
record_code_compressor_fixtures.py feat(rust): port CodeCompressor AST compressor to Rust (parity-only) (#1154) 2026-07-27 09:21:57 -07:00
record_fixtures.py feat(rust): scaffold workspace + parity harness (phase-0) 2026-04-24 13:39:48 -07:00
record_kompress_fixtures.py feat(rust): port Kompress ML prose compressor to Rust (parity-only) (#1153) 2026-07-27 08:17:53 -07:00
refresh_model_limits.sh fix(rust): wire ICM compressor into Rust proxy on /v1/messages 2026-05-01 16:44:44 -07:00
release_smoke_all.py chore(release): harden local artifact smokes (#1824) 2026-07-14 16:07:34 -04:00
replay_codex_ws_load.py fix(tests): ship scripts/replay_codex_ws_load.py so CI can import it 2026-05-14 13:44:41 -07:00
repro_codex_replay.py fix: replace asyncio.timeout with 3.10-compat shim in repro harness 2026-04-20 13:41:02 -05:00
smoke_issue_327.py fix(proxy): remove content-keyed TTL walker that conflated content with positional cache (#327) 2026-05-01 12:04:28 -07:00
sync-plugin-versions.py fix(proxy): lazy-import server to avoid fastapi crash (#442) 2026-06-10 12:44:23 -05:00
validate-workflows.sh ci: scope PR workflow runs by changed paths (#1067) 2026-06-16 19:11:45 -07:00
verify-ruff-version.py fix(ci): align Ruff tooling versions (#2406) 2026-07-18 20:55:20 -07:00
verify-versions.py fix: make proxy upgrades version-aware 2026-05-09 15:58:27 -07:00
verify_npm_release_assets.mjs chore(release): harden local artifact smokes (#1824) 2026-07-14 16:07:34 -04:00
version-sync.py fix(release): sync generated version metadata on the release branch (#2659) 2026-07-29 15:12:04 -07:00

scripts/

Utility scripts bundled with the Headroom repo. Most are one-off operator tools; a few are runnable as part of development workflows.

Reproducing the reconnect storm

repro_codex_replay.py reproduces the multi-agent Codex reconnect/retry storm against a local Headroom proxy (default http://127.0.0.1:8787). Use it to:

  • Regression-check that /livez stays responsive under a cold-start storm.
  • Empirically tune the Unit 4 pre-upstream semaphore default (HEADROOM_ANTHROPIC_PRE_UPSTREAM_CONCURRENCY).
  • Exercise the Codex WS lifecycle + Anthropic HTTP path simultaneously without needing to replay captured production traffic.

Run

# Default: 8 WS + 4 HTTP clients, 30s storm, p99 /livez must stay <= 500ms.
python scripts/repro_codex_replay.py

# Tighter budget, shorter run:
python scripts/repro_codex_replay.py \
    --url http://127.0.0.1:8787 \
    --ws-clients 16 \
    --anthropic-clients 8 \
    --duration 60 \
    --livez-threshold-ms 100

# Dump the full summary as JSON for downstream tooling:
python scripts/repro_codex_replay.py --json

Exit code:

  • 0 — warmup succeeded (or was skipped), storm ran for the requested duration, and /livez p99 stayed under --livez-threshold-ms.
  • 1 — soft assertion failed, proxy unreachable, or unhandled exception. Proxy-unreachable is detected and reported within ~5 seconds.

Fixtures

The script loads two hand-crafted, fully synthetic JSON fixtures:

  • scripts/fixtures/anthropic_replay_body.json — shape of a large agent reconnect replay /v1/messages?beta=true POST body.
  • scripts/fixtures/codex_response_create_frame.json — first Codex WS frame with the {"type": "response.create", "response": {...}} envelope.

Override via --ws-frame-fixture / --anthropic-body-fixture if you have captured traffic to replay instead.

Interpretation

  • /livez p99 under threshold means the event loop is not starved during the storm. If it rises with the semaphore unbounded (HEADROOM_ANTHROPIC_PRE_UPSTREAM_CONCURRENCY=10000) and drops back under the default, Unit 4's backpressure is working.
  • Codex WS: opened should equal --ws-clients. response.completed typically stays low when upstream auth isn't configured locally — the goal is handshake + relay wiring, not real upstream traffic.
  • Anthropic HTTP: ok_2xx + non_2xx + timed_out + errors should roughly equal attempted. Sustained non-zero timed_out during the storm is the failure signal the plan targets.

A smoke test at tests/test_scripts/test_repro_codex_replay_smoke.py exercises the script against a mock FastAPI server on every PR.

Install scripts

  • install.sh — POSIX installer.
  • install.ps1 — Windows PowerShell installer.

These are generated by the release pipeline; edit with care.

Windows development bootstrap

bootstrap-windows-dev.ps1 prepares a Windows development checkout. It resolves or creates a repo-local Python virtual environment, checks for Rust, installs Python build/test tooling, installs npm dependencies for the TypeScript SDK and OpenClaw plugin, and runs a small smoke set.

powershell -ExecutionPolicy Bypass -File scripts/bootstrap-windows-dev.ps1

Use -CheckOnly to print detected tool versions without installing packages. Use -SkipSmoke, -SkipDocs, -SkipNode, or -SkipRust when intentionally debugging one part of the environment.

npm release asset smoke

build_npm_release_assets.mjs locally reproduces the release workflow's npm asset build. It builds the TypeScript SDK tarball, installs that tarball into OpenClaw, rewrites OpenClaw's release dependency to the same version, regenerates dist/package.json, packs OpenClaw, and then runs verify_npm_release_assets.mjs.

node scripts/build_npm_release_assets.mjs <version>

By default, output goes into a timestamped release-assets-local/<version>-* directory. Pass an explicit empty directory when you want a predictable path:

node scripts/build_npm_release_assets.mjs <version> release-assets-local/smoke

Expected tarballs:

  • headroom-ai-<version>.tgz
  • headroom-openclaw-<version>.tgz

The script restores package metadata after it finishes so the source tree keeps the registry-installable development dependency range.

Python release artifact smoke

build_python_release_smoke.py locally reproduces the Python artifact smoke: it builds a wheel with maturin, builds an sdist, verifies the sdist License-File metadata against tarball contents, installs the wheel into a fresh python -m venv environment, and imports the native headroom._core extension from that installed wheel.

python scripts/build_python_release_smoke.py

By default, the wheel uses the faster Cargo ci profile and output goes into a timestamped release-assets-local/python-<version>-* directory. Use --release when you want the slower shipped-wheel profile:

python scripts/build_python_release_smoke.py --release --out release-assets-local/python-release-smoke

Expected artifacts:

  • headroom_ai-<version>-*.whl
  • headroom_ai-<version>.tar.gz

Full local release smoke

release_smoke_all.py is the one-command local release gate. It first runs scripts/verify-versions.py, then runs the npm release asset smoke and the Python wheel/sdist smoke into sibling output directories.

python scripts/release_smoke_all.py

By default, output goes into release-assets-local/all-<version>-*/npm and release-assets-local/all-<version>-*/python. Pass an explicit empty output directory for a predictable evidence path:

python scripts/release_smoke_all.py --out release-assets-local/full-release-smoke

Use --python-release when the Python smoke should build with maturin's slower release profile. Use --skip-npm or --skip-python only when intentionally debugging one side of the artifact pipeline.