headroom/tests/test_cli_extension_seam.py
Tejas Chopra 6ec3e3478a
feat(cli,pricing): add CLI extension seam and prompt-cache TTL pricing (#2802)
## Description

Two small, independent additions. Both exist because an out-of-tree
package needed
them and neither had a home in the current API.

1. **`headroom.cli_extension`** — an entry-point group so a package can
add a
`headroom` subcommand. `headroom.proxy_extension` requires a running
FastAPI
app, so it cannot carry a read-only CLI tool, and `_register_commands()`
was a
   hardcoded import list with no discovery.
2. **`headroom/pricing/cache_ttl.py`** — the prompt-cache TTL price
structure
(read `0.10x`, 5m write `1.25x`, 1h write `2.00x` of base input) plus
the
break-even share above which the 1h TTL is cheaper. `ModelPricing`
carries
`cached_input_per_1m` for reads but has no field for the *write* side.

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/cli/extensions.py` (new) — `register_all(main)` discovers
the
`headroom.cli_extension` group. Contract: `register(main: click.Group)
-> None`.
Invoked from `_register_commands()` **last**, so built-ins are already
attached.
- Deliberately **not** opt-in gated, unlike proxy extensions: installing
the
package is the opt-in, because adding a subcommand cannot silently
change what
an existing command does. What *would* be a silent change is shadowing a
built-in, so that is detected and rolled back — a stale plugin can never
quietly
take over `headroom proxy`. Load failures and partial registrations roll
back
  too, and one bad plugin never blocks another.
- `headroom/pricing/cache_ttl.py` (new) — `CACHE_READ_MULTIPLIER`,
`CACHE_WRITE_MULTIPLIERS`, `cache_write_multiplier()`,
`cache_rates_per_1m()`,
`ttl_breakeven_share()`. Ratios are derived from base input rather than
transcribed into a per-model table that would triple its columns and
drift.
- `cache_write_multiplier()` raises on an unknown TTL rather than
falling back to
  the cheaper 5m rate, which would understate cost.
- `headroom/pricing/litellm_pricing.py` — three additive optional fields
on
`LiteLLMModelPricing` exposing LiteLLM's own
`cache_read_input_token_cost`,
`cache_creation_input_token_cost` and
`cache_creation_input_token_cost_above_1hr`
(present for 212 and 123 models respectively). Published rates should
win over
derived ones. All default to `None`, and `None` means "not published" —
distinct
  from `0.0` meaning "free" — so every existing caller is unaffected.
- `headroom/pricing/__init__.py` — re-exports.

### Why `ttl_breakeven_share()` exists

The TTL trade has two terms and both must be counted: moving to 1h turns
idle-gap
rewrites into cheap reads **and** raises the price of every write that
still
happens. Modelling only the recovery overstates the saving. On a real
531-transcript corpus that error was **1.9x** — $1,021 claimed against
$538 real.
`test_write_premium_is_not_forgotten` pins those exact figures so the
mistake
cannot be reintroduced quietly.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`) — see note below
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_cli_extension_seam.py tests/test_pricing_cache_ttl.py tests/test_pricing_from_litellm.py -q
tests/test_cli_extension_seam.py .......                                 [ 25%]
tests/test_pricing_cache_ttl.py ..........                               [ 62%]
tests/test_pricing_from_litellm.py ..........                            [100%]
======================== 27 passed, 1 warning in 3.54s =========================

$ .venv/bin/ruff check headroom/cli/extensions.py headroom/cli/main.py headroom/pricing/ tests/test_cli_extension_seam.py tests/test_pricing_cache_ttl.py
All checks passed!

$ .venv/bin/mypy --python-version 3.12 headroom/cli/extensions.py headroom/pricing/cache_ttl.py headroom/pricing/litellm_pricing.py
Success: no issues found in 3 source files
```

`tests/test_pricing_from_litellm.py` is the **pre-existing** pricing
suite, included
to show the `LiteLLMModelPricing` change is non-breaking.

**mypy note.** With the repo's configured `python_version = "3.10"`,
mypy fails on
numpy's own stubs for any file that transitively reaches numpy:

```text
$ .venv/bin/mypy headroom/pricing/cache_ttl.py
.venv/lib/python3.12/site-packages/numpy/__init__.pyi:737: error: Type statement is only supported in Python 3.12 and greater  [syntax]
Found 1 error in 1 file (errors prevented further checking)
```

This is pre-existing and unrelated — untouched files reproduce it
identically
(`mypy headroom/cli/doctor.py`, `mypy headroom/pricing/registry.py`).
Hence the
`--python-version 3.12` run above, which matches the interpreter
actually in use.
Worth fixing separately; not addressed here.

## Real Behavior Proof

- **Environment:** macOS 15 (darwin 25.4.0), Python 3.12.6,
`headroom-ai` 0.34.0
  working tree, branch off `upstream/main`.
- **Exact command / steps:**
  1. Built a separate out-of-tree package declaring
`[project.entry-points."headroom.cli_extension"] fleet =
"headroom_fleet.cli:register"`.
  2. `pip install --no-deps headroom_fleet-0.1.0-py3-none-any.whl`
  3. `headroom econ --help`
- **Observed result:** the subcommand registers with no configuration
and appears
  in `headroom --help`:

  ```text
$ python -c "import importlib.metadata as m; print([e.name+' ->
'+e.value for e in m.entry_points(group='headroom.cli_extension')])"
  ['fleet -> headroom_fleet.cli:register']

  $ headroom --help | grep econ
econ Report where local AI-coding token spend goes, and what...

  $ headroom econ --help
  Usage: headroom econ [OPTIONS] [COMMAND] [ARGS]...
  Commands:
    fix    Write the recommended cache-TTL and compaction settings.
unfix Restore every setting ``econ fix`` changed, exactly as it was.
  ```

`headroom --help` and every built-in still work with the plugin
installed and
after it is uninstalled. Verified per-model cache rates resolve from
LiteLLM for
  `claude-opus-4-8`, `claude-opus-5`, `claude-sonnet-5` and
`claude-haiku-4-5-20251001` — all exactly `1.250x` / `2.000x` / `0.100x`
of base
  input, matching the derived fallback.

- **Not tested:** Windows; a plugin that raises at *import* time rather
than in
`register()` (covered by unit test with a stubbed entry point, not a
real
package); the `--python-version 3.10` mypy path, which is blocked by the
  pre-existing numpy stub issue above.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines

## Note for reviewers

This PR contains **only** the two additions above. Unrelated
`plugins/opencode/src/transport.ts` changes in my working tree are
deliberately
excluded and will follow separately.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-08-05 12:24:49 -07:00

140 lines
3.9 KiB
Python

"""Tests for the ``headroom.cli_extension`` third-party subcommand seam."""
from __future__ import annotations
import click
import pytest
from headroom.cli import extensions
class _FakeEntry:
"""Stand-in for an ``importlib.metadata.EntryPoint``."""
def __init__(self, name: str, loaded: object, load_raises: bool = False) -> None:
self.name = name
self._loaded = loaded
self._load_raises = load_raises
def load(self) -> object:
if self._load_raises:
raise ImportError("boom")
return self._loaded
@pytest.fixture
def group() -> click.Group:
@click.group()
def root() -> None:
pass
@root.command(name="proxy")
def proxy() -> None:
pass
return root
def _patch_entries(monkeypatch: pytest.MonkeyPatch, entries: list[_FakeEntry]) -> None:
monkeypatch.setattr(
extensions.importlib.metadata,
"entry_points",
lambda group: entries,
)
def test_registers_new_command(monkeypatch: pytest.MonkeyPatch, group: click.Group) -> None:
@click.command(name="econ")
def econ() -> None:
pass
_patch_entries(monkeypatch, [_FakeEntry("fleet", lambda main: main.add_command(econ))])
assert extensions.register_all(group) == ["fleet"]
assert "econ" in group.commands
assert "proxy" in group.commands
def test_raising_registrant_is_skipped(monkeypatch: pytest.MonkeyPatch, group: click.Group) -> None:
def bad(main: click.Group) -> None:
raise RuntimeError("unlicensed")
_patch_entries(monkeypatch, [_FakeEntry("broken", bad)])
assert extensions.register_all(group) == []
assert set(group.commands) == {"proxy"}
def test_partial_registration_is_rolled_back(
monkeypatch: pytest.MonkeyPatch, group: click.Group
) -> None:
"""A registrant that adds a command then raises leaves nothing behind."""
@click.command(name="half")
def half() -> None:
pass
def bad(main: click.Group) -> None:
main.add_command(half)
raise RuntimeError("failed after partial work")
_patch_entries(monkeypatch, [_FakeEntry("broken", bad)])
assert extensions.register_all(group) == []
assert "half" not in group.commands
def test_load_failure_is_skipped(monkeypatch: pytest.MonkeyPatch, group: click.Group) -> None:
_patch_entries(monkeypatch, [_FakeEntry("stale", None, load_raises=True)])
assert extensions.register_all(group) == []
assert set(group.commands) == {"proxy"}
def test_cannot_shadow_builtin_command(monkeypatch: pytest.MonkeyPatch, group: click.Group) -> None:
"""Overriding a built-in IS a silent behavior change, so it is refused."""
builtin = group.commands["proxy"]
@click.command(name="proxy")
def evil_proxy() -> None:
pass
_patch_entries(monkeypatch, [_FakeEntry("evil", lambda main: main.add_command(evil_proxy))])
assert extensions.register_all(group) == []
assert group.commands["proxy"] is builtin
def test_shadowing_registrant_does_not_block_others(
monkeypatch: pytest.MonkeyPatch, group: click.Group
) -> None:
@click.command(name="proxy")
def evil_proxy() -> None:
pass
@click.command(name="econ")
def econ() -> None:
pass
_patch_entries(
monkeypatch,
[
_FakeEntry("evil", lambda main: main.add_command(evil_proxy)),
_FakeEntry("fleet", lambda main: main.add_command(econ)),
],
)
assert extensions.register_all(group) == ["fleet"]
assert "econ" in group.commands
def test_enumeration_failure_is_survivable(
monkeypatch: pytest.MonkeyPatch, group: click.Group
) -> None:
def boom(group: str) -> list[_FakeEntry]:
raise RuntimeError("no metadata")
monkeypatch.setattr(extensions.importlib.metadata, "entry_points", boom)
assert extensions.register_all(group) == []
assert set(group.commands) == {"proxy"}