## Description
Adds a content-type-aware path for **tabular data** — CSV/TSV, markdown
tables, fixed-width text, and binary `.xlsx`/`.xls` spreadsheets — by
routing them through the existing, battle-tested `SmartCrusher` instead
of letting them fall through to `PLAIN_TEXT → Kompress`.
The pipeline already compressed tables losslessly when handed a JSON
array of records. This wires up the missing front door: detect tabular
text (and ingest binary spreadsheets), convert to JSON records, and
reuse `SmartCrusher.crush()`. No new compression algorithm.
Closes #
## Type of Change
- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- **Detection** (`content_detector.py`): new `ContentType.TABULAR` +
`_try_detect_tabular()` for CSV/TSV, markdown tables, and fixed-width
columns. Ordered after search/log (which also look "delimited") and
before code, with a prose-rejection guard so it never steals
`file:line:content` search output, `key: value` logs, or sentences with
incidental commas. Rust backend returns `plain_text` for unknown types
and the router already falls back to the Python detector, so **no Rust
change**.
- **Bridge** (`tabular_ingest.py`): stdlib parsers + `to_records()` + a
`TabularCompressor` that parses → JSON records → `SmartCrusher`
(lossless `csv-schema` first; lossy row-drop with reversible
`<<ccr:HASH>>` markers stays SmartCrusher's built-in fallback). Only
adopts a result when it actually saves bytes.
- **Spreadsheets** (`spreadsheet_ingest.py`): `.xlsx`/`.xls` → per-sheet
CSV text at the SDK boundary. Optional deps (`pip install
headroom-ai[spreadsheet]`) fail loudly with an install hint, never
silently degrade.
- **Routing** (`content_router.py`): `CompressionStrategy.TABULAR`,
`enable_tabular_compressor` flag, lazy getter, apply branch, strategy
maps, Kompress fallback eligibility.
- **SDK** (`compress.py`): `compress_spreadsheet(path, ...)` helper (one
message per sheet).
- **Packaging** (`pyproject.toml`): new `[spreadsheet]` extra;
`openpyxl` added to `[dev]` so the xlsx path is exercised in CI.
- **Docs/demo**: `examples/tabular_compression_demo.py` + README entry.
### Design note: lossless-only
Compact, all-unique tables with no query yield ~0 savings — this is
correct, not a bug. SmartCrusher returns
`skip:unique_entities_no_signal` and won't drop unique rows without a
duplicate/relevance signal. Real wins come from verbose/redundant tables
and query-driven selection. A pressure-driven lossy row sampler was
considered and intentionally not added.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ python -m pytest tests/test_transforms_tabular.py -q
collected 20 items
tests/test_transforms_tabular.py .................... [100%]
============================== 20 passed in 7.15s ==============================
$ ruff check headroom/transforms/tabular_ingest.py headroom/transforms/spreadsheet_ingest.py
All checks passed!
$ mypy headroom/transforms/tabular_ingest.py headroom/transforms/spreadsheet_ingest.py
Success: no issues found in 2 source files
```
`tests/test_transforms_tabular.py` (20 tests): detection true positives
+ no-misroute negatives (search/log/JSON/prose), parser units (incl.
fixed-width), the CSV→SmartCrusher bridge, router routing + disable
flag, and `.xlsx` ingestion (skipif openpyxl missing) + error paths.
`spreadsheet_ingest` 100% / `tabular_ingest` 90% line coverage.
## Real Behavior Proof
- **Environment:** local checkout of `feat/tabular-compression`, Python
3.x, `pip install -e ".[dev]"`.
- **Exact command / steps:** `python
examples/tabular_compression_demo.py` (no API key required).
- **Observed result:**
```text
=== Raw tabular text (ContentRouter, char-level) ===
compact unique CSV strat=tabular chars 1306 -> 1072 ( 17.9% saved)
redundant CSV strat=tabular chars 2661 -> 1350 ( 49.3% saved)
verbose markdown strat=tabular chars 2019 -> 1580 ( 21.7% saved)
=== Full pipeline (real tokenizer) ===
redundant CSV tokens 768 -> 394 ( 48.7% saved)
=== Binary spreadsheet (.xlsx) ===
2-sheet workbook tokens 1092 -> 683 ( 37.5% saved)
```
- **Not tested:** legacy `.xls` binary path (needs optional `xlrd` +
binary fixture; `# pragma: no cover`); base64-embedded `.xlsx` inside
multimodal blocks (out of scope, noted as a follow-up).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
- CHANGELOG/version are intentionally untouched: this repo uses
**release-please**, which bumps the version and CHANGELOG via automated
`chore: release main` PRs, not per-feature PRs.
- The `.xls` path is `# pragma: no cover` (legacy, needs optional `xlrd`
+ a binary fixture).
- Follow-up (out of scope): base64-embedded `.xlsx` inside
tool-result/multimodal blocks; porting tabular parsers into the Rust
core for parity.
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* ci: speed up GitHub Actions - path filters, caching, timeouts, version upgrades
Performance improvements:
- init-e2e.yml, wrap-e2e.yml: add path filters so e2e Docker builds only run when
e2e-related files change (saves ~10 min per irrelevant PR push)
- init-e2e.yml, wrap-e2e.yml: add concurrency groups to cancel superseded PR runs
- ci.yml: add pip caching to lint and build jobs
- ci.yml: cache actionlint + act binaries in workflow-validation (skip curl on hits)
- eval.yml: add pip caching to smoke-test and weekly-suite jobs
- docs.yml: add pip caching for mkdocs-material install
- rust.yml: replace cargo install --locked cargo-audit/deny with taiki-e/install-action
(prebuilt binaries; saves 2-5 min per audit run)
Bug fixes:
- docker.yml: fix actions/checkout@v6 -> @v4 (v6 does not exist; would break all
Docker builds on every release/PR touching docker paths)
Version upgrades:
- wagoid/commitlint-github-action: @v5 -> @v6
- devcontainers.yml: docker/setup-buildx-action@v3 -> @v4 (align with docker.yml)
Safety improvements:
- ci.yml: add timeout-minutes to all 13 jobs (changes, lint, build-wheel,
prefetch-model, test x4, test-extras, test-agno, commitlint, build,
workflow-validation, docker-native-e2e, windows-native-wrapper, macos-native-wrapper)
- docker.yml: add timeout-minutes to docker-build (75m), docker-manifest (20m),
promote-latest (10m)
- eval.yml: add timeout-minutes to smoke-test (30m); bump weekly-suite 60->90m
- rust.yml: add timeout-minutes to test (30m), wheels (45m), audit (20m)
Observed wall-clock impact on recent PRs:
- Init E2E and Wrap E2E were running on every single PR push regardless of content
- CI workflow was taking 12-17 min; path filters reduce unnecessary e2e runs to 0
* fix(ci): bust actionlint+act cache when workflow file changes
Static cache key 'ci-tools-actionlint-act-v1' never invalidated on
tool version updates. Switched to hashFiles('.github/workflows/ci.yml')
so the cache busts automatically whenever the download scripts are
updated to point at a newer release.
Flagged by adversarial review (Architecture + Testing/Reliability personas).
* fix(ci): add missing Dockerfile COPY paths to e2e path filters
e2e/init/Dockerfile and e2e/wrap/Dockerfile COPY files not covered
by the initial path filter set:
init-e2e: Cargo.toml, Cargo.lock, rust-toolchain.toml, uv.lock,
.claude-plugin, .github/plugin/**, plugins/headroom-agent-hooks/**
wrap-e2e: Cargo.toml, Cargo.lock, rust-toolchain.toml, uv.lock,
sdk/typescript/**, plugins/openclaw/**
Without these, a Rust toolchain bump or SDK change on a PR would
skip the e2e gate entirely, only catching it on the merge to main.
Flagged by adversarial review (Domain/Correctness persona).
* fix(devcontainer): upgrade uv to >=0.7.0 to parse uv.lock revision=3
* fix(devcontainer): set UV_SKIP_WHEEL_FILENAME_CHECK=1 in post-create.sh for gitpython wheel
* ci: bump actions/checkout and actions/setup-node to v5 (Node.js 20 EOL Jun 16)
* fix(devcontainer): export UV_SKIP_WHEEL_FILENAME_CHECK so uv run also skips wheel check
* ci: bump all GitHub Actions to latest versions (Node.js 24)
* fix(test): accept release-please-action v4 or v5 in workflow assertion
* fix(format): ruff format test_release_workflows.py
The memory-stack devcontainer (Neo4j + Postgres + Redis + Qdrant
on top of the Docker base) is too heavy for the current
GitHub-hosted runner image: this PR's two CI runs both failed
with "No space left on device" before the smoke test could
finish. The default devcontainer passes on the same image —
memory-stack is the only path that exceeds the runner's disk.
devcontainers.yml only triggers when one of these changes:
.devcontainer/**
.github/workflows/devcontainers.yml
pyproject.toml
uv.lock
PR #492/#493/#494 didn't touch any of those, so the failure was
latent. PR #495 bumps pyproject.toml (0.9.1 -> 0.22.3) which
surfaced it.
Fix: add `jlumbroso/free-disk-space@v1.3.1` ahead of "Start
memory-stack" to reclaim ~14 GB from preinstalled Android SDK +
.NET + Haskell tool caches the devcontainer doesn't need. Scoped
to `if: matrix.name == 'memory-stack'` — the default validate
still benefits from the toolcache.
Pinned to v1.3.1 (the latest release at time of writing) to match
the project's tag-pinning style for third-party actions.
Make the devcontainer bootstrap and CI smoke tests work from linked worktrees, including git metadata translation, cache ownership fixes, workspace path handling, and config-scoped smoke checks.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a default devcontainer and a compose-backed memory-stack profile, validate them in CI, and document the contributor workflow.
Also lock the memory-stack dependencies, pin related container tooling, and sync the latest healthcheck shutdown fix for stubbed memory handlers.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
