mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
## Description
Two small, independent additions. Both exist because an out-of-tree
package needed
them and neither had a home in the current API.
1. **`headroom.cli_extension`** — an entry-point group so a package can
add a
`headroom` subcommand. `headroom.proxy_extension` requires a running
FastAPI
app, so it cannot carry a read-only CLI tool, and `_register_commands()`
was a
hardcoded import list with no discovery.
2. **`headroom/pricing/cache_ttl.py`** — the prompt-cache TTL price
structure
(read `0.10x`, 5m write `1.25x`, 1h write `2.00x` of base input) plus
the
break-even share above which the 1h TTL is cheaper. `ModelPricing`
carries
`cached_input_per_1m` for reads but has no field for the *write* side.
Closes #
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `headroom/cli/extensions.py` (new) — `register_all(main)` discovers
the
`headroom.cli_extension` group. Contract: `register(main: click.Group)
-> None`.
Invoked from `_register_commands()` **last**, so built-ins are already
attached.
- Deliberately **not** opt-in gated, unlike proxy extensions: installing
the
package is the opt-in, because adding a subcommand cannot silently
change what
an existing command does. What *would* be a silent change is shadowing a
built-in, so that is detected and rolled back — a stale plugin can never
quietly
take over `headroom proxy`. Load failures and partial registrations roll
back
too, and one bad plugin never blocks another.
- `headroom/pricing/cache_ttl.py` (new) — `CACHE_READ_MULTIPLIER`,
`CACHE_WRITE_MULTIPLIERS`, `cache_write_multiplier()`,
`cache_rates_per_1m()`,
`ttl_breakeven_share()`. Ratios are derived from base input rather than
transcribed into a per-model table that would triple its columns and
drift.
- `cache_write_multiplier()` raises on an unknown TTL rather than
falling back to
the cheaper 5m rate, which would understate cost.
- `headroom/pricing/litellm_pricing.py` — three additive optional fields
on
`LiteLLMModelPricing` exposing LiteLLM's own
`cache_read_input_token_cost`,
`cache_creation_input_token_cost` and
`cache_creation_input_token_cost_above_1hr`
(present for 212 and 123 models respectively). Published rates should
win over
derived ones. All default to `None`, and `None` means "not published" —
distinct
from `0.0` meaning "free" — so every existing caller is unaffected.
- `headroom/pricing/__init__.py` — re-exports.
### Why `ttl_breakeven_share()` exists
The TTL trade has two terms and both must be counted: moving to 1h turns
idle-gap
rewrites into cheap reads **and** raises the price of every write that
still
happens. Modelling only the recovery overstates the saving. On a real
531-transcript corpus that error was **1.9x** — $1,021 claimed against
$538 real.
`test_write_premium_is_not_forgotten` pins those exact figures so the
mistake
cannot be reintroduced quietly.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`) — see note below
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_cli_extension_seam.py tests/test_pricing_cache_ttl.py tests/test_pricing_from_litellm.py -q
tests/test_cli_extension_seam.py ....... [ 25%]
tests/test_pricing_cache_ttl.py .......... [ 62%]
tests/test_pricing_from_litellm.py .......... [100%]
======================== 27 passed, 1 warning in 3.54s =========================
$ .venv/bin/ruff check headroom/cli/extensions.py headroom/cli/main.py headroom/pricing/ tests/test_cli_extension_seam.py tests/test_pricing_cache_ttl.py
All checks passed!
$ .venv/bin/mypy --python-version 3.12 headroom/cli/extensions.py headroom/pricing/cache_ttl.py headroom/pricing/litellm_pricing.py
Success: no issues found in 3 source files
```
`tests/test_pricing_from_litellm.py` is the **pre-existing** pricing
suite, included
to show the `LiteLLMModelPricing` change is non-breaking.
**mypy note.** With the repo's configured `python_version = "3.10"`,
mypy fails on
numpy's own stubs for any file that transitively reaches numpy:
```text
$ .venv/bin/mypy headroom/pricing/cache_ttl.py
.venv/lib/python3.12/site-packages/numpy/__init__.pyi:737: error: Type statement is only supported in Python 3.12 and greater [syntax]
Found 1 error in 1 file (errors prevented further checking)
```
This is pre-existing and unrelated — untouched files reproduce it
identically
(`mypy headroom/cli/doctor.py`, `mypy headroom/pricing/registry.py`).
Hence the
`--python-version 3.12` run above, which matches the interpreter
actually in use.
Worth fixing separately; not addressed here.
## Real Behavior Proof
- **Environment:** macOS 15 (darwin 25.4.0), Python 3.12.6,
`headroom-ai` 0.34.0
working tree, branch off `upstream/main`.
- **Exact command / steps:**
1. Built a separate out-of-tree package declaring
`[project.entry-points."headroom.cli_extension"] fleet =
"headroom_fleet.cli:register"`.
2. `pip install --no-deps headroom_fleet-0.1.0-py3-none-any.whl`
3. `headroom econ --help`
- **Observed result:** the subcommand registers with no configuration
and appears
in `headroom --help`:
```text
$ python -c "import importlib.metadata as m; print([e.name+' ->
'+e.value for e in m.entry_points(group='headroom.cli_extension')])"
['fleet -> headroom_fleet.cli:register']
$ headroom --help | grep econ
econ Report where local AI-coding token spend goes, and what...
$ headroom econ --help
Usage: headroom econ [OPTIONS] [COMMAND] [ARGS]...
Commands:
fix Write the recommended cache-TTL and compaction settings.
unfix Restore every setting ``econ fix`` changed, exactly as it was.
```
`headroom --help` and every built-in still work with the plugin
installed and
after it is uninstalled. Verified per-model cache rates resolve from
LiteLLM for
`claude-opus-4-8`, `claude-opus-5`, `claude-sonnet-5` and
`claude-haiku-4-5-20251001` — all exactly `1.250x` / `2.000x` / `0.100x`
of base
input, matching the derived fallback.
- **Not tested:** Windows; a plugin that raises at *import* time rather
than in
`register()` (covered by unit test with a stubbed entry point, not a
real
package); the `--python-version 3.10` mypy path, which is blocked by the
pre-existing numpy stub issue above.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
## Note for reviewers
This PR contains **only** the two additions above. Unrelated
`plugins/opencode/src/transport.ts` changes in my working tree are
deliberately
excluded and will follow separately.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
140 lines
3.9 KiB
Python
140 lines
3.9 KiB
Python
"""Tests for the ``headroom.cli_extension`` third-party subcommand seam."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import click
|
|
import pytest
|
|
|
|
from headroom.cli import extensions
|
|
|
|
|
|
class _FakeEntry:
|
|
"""Stand-in for an ``importlib.metadata.EntryPoint``."""
|
|
|
|
def __init__(self, name: str, loaded: object, load_raises: bool = False) -> None:
|
|
self.name = name
|
|
self._loaded = loaded
|
|
self._load_raises = load_raises
|
|
|
|
def load(self) -> object:
|
|
if self._load_raises:
|
|
raise ImportError("boom")
|
|
return self._loaded
|
|
|
|
|
|
@pytest.fixture
|
|
def group() -> click.Group:
|
|
@click.group()
|
|
def root() -> None:
|
|
pass
|
|
|
|
@root.command(name="proxy")
|
|
def proxy() -> None:
|
|
pass
|
|
|
|
return root
|
|
|
|
|
|
def _patch_entries(monkeypatch: pytest.MonkeyPatch, entries: list[_FakeEntry]) -> None:
|
|
monkeypatch.setattr(
|
|
extensions.importlib.metadata,
|
|
"entry_points",
|
|
lambda group: entries,
|
|
)
|
|
|
|
|
|
def test_registers_new_command(monkeypatch: pytest.MonkeyPatch, group: click.Group) -> None:
|
|
@click.command(name="econ")
|
|
def econ() -> None:
|
|
pass
|
|
|
|
_patch_entries(monkeypatch, [_FakeEntry("fleet", lambda main: main.add_command(econ))])
|
|
|
|
assert extensions.register_all(group) == ["fleet"]
|
|
assert "econ" in group.commands
|
|
assert "proxy" in group.commands
|
|
|
|
|
|
def test_raising_registrant_is_skipped(monkeypatch: pytest.MonkeyPatch, group: click.Group) -> None:
|
|
def bad(main: click.Group) -> None:
|
|
raise RuntimeError("unlicensed")
|
|
|
|
_patch_entries(monkeypatch, [_FakeEntry("broken", bad)])
|
|
|
|
assert extensions.register_all(group) == []
|
|
assert set(group.commands) == {"proxy"}
|
|
|
|
|
|
def test_partial_registration_is_rolled_back(
|
|
monkeypatch: pytest.MonkeyPatch, group: click.Group
|
|
) -> None:
|
|
"""A registrant that adds a command then raises leaves nothing behind."""
|
|
|
|
@click.command(name="half")
|
|
def half() -> None:
|
|
pass
|
|
|
|
def bad(main: click.Group) -> None:
|
|
main.add_command(half)
|
|
raise RuntimeError("failed after partial work")
|
|
|
|
_patch_entries(monkeypatch, [_FakeEntry("broken", bad)])
|
|
|
|
assert extensions.register_all(group) == []
|
|
assert "half" not in group.commands
|
|
|
|
|
|
def test_load_failure_is_skipped(monkeypatch: pytest.MonkeyPatch, group: click.Group) -> None:
|
|
_patch_entries(monkeypatch, [_FakeEntry("stale", None, load_raises=True)])
|
|
|
|
assert extensions.register_all(group) == []
|
|
assert set(group.commands) == {"proxy"}
|
|
|
|
|
|
def test_cannot_shadow_builtin_command(monkeypatch: pytest.MonkeyPatch, group: click.Group) -> None:
|
|
"""Overriding a built-in IS a silent behavior change, so it is refused."""
|
|
builtin = group.commands["proxy"]
|
|
|
|
@click.command(name="proxy")
|
|
def evil_proxy() -> None:
|
|
pass
|
|
|
|
_patch_entries(monkeypatch, [_FakeEntry("evil", lambda main: main.add_command(evil_proxy))])
|
|
|
|
assert extensions.register_all(group) == []
|
|
assert group.commands["proxy"] is builtin
|
|
|
|
|
|
def test_shadowing_registrant_does_not_block_others(
|
|
monkeypatch: pytest.MonkeyPatch, group: click.Group
|
|
) -> None:
|
|
@click.command(name="proxy")
|
|
def evil_proxy() -> None:
|
|
pass
|
|
|
|
@click.command(name="econ")
|
|
def econ() -> None:
|
|
pass
|
|
|
|
_patch_entries(
|
|
monkeypatch,
|
|
[
|
|
_FakeEntry("evil", lambda main: main.add_command(evil_proxy)),
|
|
_FakeEntry("fleet", lambda main: main.add_command(econ)),
|
|
],
|
|
)
|
|
|
|
assert extensions.register_all(group) == ["fleet"]
|
|
assert "econ" in group.commands
|
|
|
|
|
|
def test_enumeration_failure_is_survivable(
|
|
monkeypatch: pytest.MonkeyPatch, group: click.Group
|
|
) -> None:
|
|
def boom(group: str) -> list[_FakeEntry]:
|
|
raise RuntimeError("no metadata")
|
|
|
|
monkeypatch.setattr(extensions.importlib.metadata, "entry_points", boom)
|
|
|
|
assert extensions.register_all(group) == []
|
|
assert set(group.commands) == {"proxy"}
|