refactor: remove the dead headroom/prediction module (#2692)

## Description

Deletes `headroom/prediction/` — 2,614 LOC of LLM output-length
prediction feature extraction that was never wired into anything and
shipped in every platform wheel regardless.

It was added on 2026-01-26 in `da743418` ("Add hierarchical memory
system with graph + vector storage"), where it appears as a single
bullet: *"`headroom/prediction/feature_extractor.py`: Content analysis
features"*. Nothing ever consumed it.

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- Delete `headroom/prediction/__init__.py` (85 LOC) and
`headroom/prediction/feature_extractor.py` (2,529 LOC).
- Drop the now-dangling reference to `prediction/feature_extractor.py`
from the `SemanticDetector` comment in
`headroom/cache/dynamic_detector.py:751`. The surviving sibling it cites
(`memory/adapters/embedders.py`) is unchanged, and no behavior changes.

### Why this is dead code, not dormant code

1. **Zero importers.** Nothing in `headroom/`, `tests/`, `docs/`,
`benchmarks/`, `plugins/`, or the lazy `_LAZY_EXPORTS` map in
`headroom/__init__.py` references `headroom.prediction`,
`PromptFeatureExtractor`, or `feature_extractor`.
2. **Never installable as documented.** Both module docstrings instruct
`pip install headroom[prediction]`. **No `[prediction]` extra has ever
existed** in `pyproject.toml` (28 extras are defined; that is not one of
them).
3. **Superseded.** The output-length concern was reimplemented five
months later in seven `headroom/proxy/output_*.py` modules —
`output_savings.py` and `output_shaper.py` (2026-06-16),
`output_steering.py` (2026-07-10), plus `output_effort_policy.py`,
`output_savings_policy.py`, `output_turn_policy.py`,
`output_verbosity_policy.py`. None import `prediction`.
4. **Abandoned.** 6 commits total; last substantive change 2026-02-01
(`f2014808`, `MLModelRegistry`). The only later touch is `2ae71fe4`
(2026-04-07), a repo-wide `nosec B324` chore sweep by another
contributor.

### One judgement call for the reviewer

`headroom/prediction/` was a non-underscore package with a populated
`__all__` that shipped in every wheel, so an external consumer *could*
have imported it directly. I titled this `refactor:` rather than
`refactor!:` because the documented install path never existed, but if
you consider the bare import path a public contract, retitle to
`refactor!:` before merge.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [ ] New tests added for new functionality
- [x] Manual testing performed

N/A: no new tests — this PR only removes unreferenced code and one stale
comment line.

### Test Output

```text
$ .venv/bin/ruff check headroom/
All checks passed!

$ .venv/bin/ruff format --check headroom/cache/dynamic_detector.py
1 file already formatted

$ .venv/bin/mypy headroom/
Success: no issues found in 506 source files

$ python -m pytest tests/test_cache/test_dynamic_detector.py tests/test_package_init_lazy.py tests/test_release_workflows.py -q
110 passed, 2 skipped, 2 warnings in 25.70s
```

The 2 warnings are the pre-existing third-party `SwigPyObject has no
__module__ attribute` DeprecationWarnings, unrelated to this change.

## Real Behavior Proof

- **Environment:** macOS 25.4.0 (darwin arm64), Python 3.12.6, branch
off `upstream/main` @ `f2c48e26`.
- **Exact command / steps:**
  ```
$ grep -rn
"headroom\.prediction\|PromptFeatureExtractor\|feature_extractor" . \
--exclude-dir=.git --exclude-dir=.venv --exclude-dir=onnx | grep -v
'^./headroom/prediction/'
# -> only 2 hits, both textual: CHANGELOG.md:172 (historical entry, left
untouched)
# and headroom/cache/dynamic_detector.py:752 (the comment fixed in this
PR)

  $ git rm -r headroom/prediction/

  $ .venv/bin/python -c "
  import headroom, importlib
  for name in headroom.__all__: getattr(headroom, name)
  print('lazy exports OK:', len(headroom.__all__))
try: importlib.import_module('headroom.prediction'); print('STILL
PRESENT')
  except ModuleNotFoundError: print('headroom.prediction gone')
  "
  ```
- **Observed result:**
  ```
  import headroom OK, version 0.34.0-dev
  lazy exports checked: 84 failures: []
  headroom.prediction correctly gone
  ```
All 84 lazily-exported names on the top-level `headroom` façade still
resolve after the deletion — this is the check that matters, because
`headroom/__init__.py` resolves exports through a string map that static
tooling cannot follow.
- **Not tested:** the full `pytest tests/` suite (ran the 3 relevant
files: the detector whose comment changed, the lazy-export surface, and
the release-workflow gates). No wheel was built, so the packaging change
is verified by `[tool.maturin] python-source = "."` including
`headroom/` wholesale rather than by inspecting a built artifact.
`CHANGELOG.md:172` still mentions `prediction/feature_extractor.py` in a
historical entry; left alone deliberately, since the changelog guard
rejects hand edits.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

N/A on tests: a deletion of unreferenced code has nothing to add a test
for. Documentation needed no change because the module was absent from
all docs — the only place it was ever "documented" was its own
docstring, which pointed at a non-existent extra.

## Additional Notes

Found while mapping module coupling for a possible package split. Two
related items deliberately **not** in this PR:

- `headroom/engine/` and `headroom/diagnostics/` exist as empty
untracked directories locally. They are leftovers from branch checkouts,
not tracked content — `engine/` lives on the still-open #606, and
`diagnostics/` on an unmerged local branch. Nothing to delete on `main`.
- `headroom/exceptions.py` (192 LOC) has an in-degree of 0 for direct
imports; it is reached only through the `__init__.py` string map. That
is working as intended, not dead — no change proposed.

headroom/cache/dynamic_detector.py

@@ -748,8 +748,8 @@ class SemanticDetector:
             # normalization sentence_transformers returns raw vectors (norm
             # ~5-15), so the dot product is an unbounded inner product, not a
             # cosine similarity — nearly every sentence would clear the 0.7
-            # threshold and be misflagged as dynamic. Matches the siblings in
-            # prediction/feature_extractor.py and memory/adapters/embedders.py.
+            # threshold and be misflagged as dynamic. Matches the sibling in
+            # memory/adapters/embedders.py.
             self._exemplar_embeddings = self._model.encode(
                 self.DYNAMIC_EXEMPLARS,
                 convert_to_numpy=True,

headroom/prediction/__init__.py

@@ -1,85 +0,0 @@
-"""LLM Output Length Prediction Module.
-
-This module provides comprehensive feature extraction and prediction
-capabilities for estimating LLM response lengths from input prompts.
-
-Features are organized into 5 categories:
-1. Text Statistics - Length, vocabulary, compression metrics
-2. Structural - Questions, lists, code blocks, formatting
-3. Semantic - Task type, domain, complexity indicators
-4. Embedding - Neural embeddings and similarity scores
-5. Meta - Model settings, historical patterns
-
-Example:
-    from headroom.prediction import PromptFeatureExtractor, extract_features
-
-    # Full extractor (with embeddings)
-    extractor = PromptFeatureExtractor(use_embeddings=True)
-    features = extractor.extract("What is machine learning?", model="gpt-4o")
-
-    # Quick extraction (no embeddings)
-    features = extract_features("Explain quantum computing")
-
-    # Get ML-ready vector
-    vector = features.to_vector()
-    names = features.feature_names()
-
-Install full dependencies:
-    pip install headroom[prediction]
-
-This installs:
-    - sentence-transformers (for embedding features)
-    - spacy (for NER, optional)
-"""
-
-from .feature_extractor import (
-    ComplexityLevel,
-    DomainType,
-    EmbeddingExtractor,
-    EmbeddingFeatures,
-    MetaExtractor,
-    MetaFeatures,
-    # Main extractor
-    PromptFeatureExtractor,
-    # Feature dataclasses
-    PromptFeatures,
-    PromptFormat,
-    SemanticExtractor,
-    SemanticFeatures,
-    StructuralExtractor,
-    StructuralFeatures,
-    # Enums
-    TaskType,
-    # Individual extractors
-    TextStatisticsExtractor,
-    TextStatisticsFeatures,
-    # Utility functions
-    extract_features,
-    get_feature_vector,
-)
-
-__all__ = [
-    # Main extractor
-    "PromptFeatureExtractor",
-    # Individual extractors
-    "TextStatisticsExtractor",
-    "StructuralExtractor",
-    "SemanticExtractor",
-    "EmbeddingExtractor",
-    "MetaExtractor",
-    # Feature dataclasses
-    "PromptFeatures",
-    "TextStatisticsFeatures",
-    "StructuralFeatures",
-    "SemanticFeatures",
-    "EmbeddingFeatures",
-    "MetaFeatures",
-    # Enums
-    "TaskType",
-    "DomainType",
-    "ComplexityLevel",
-    "PromptFormat",
-    # Utility functions
-    "extract_features",
-    "get_feature_vector",
-]