mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-08-27 14:17:10 -04:00
## Description Adds a content-type-aware path for **tabular data** — CSV/TSV, markdown tables, fixed-width text, and binary `.xlsx`/`.xls` spreadsheets — by routing them through the existing, battle-tested `SmartCrusher` instead of letting them fall through to `PLAIN_TEXT → Kompress`. The pipeline already compressed tables losslessly when handed a JSON array of records. This wires up the missing front door: detect tabular text (and ingest binary spreadsheets), convert to JSON records, and reuse `SmartCrusher.crush()`. No new compression algorithm. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **Detection** (`content_detector.py`): new `ContentType.TABULAR` + `_try_detect_tabular()` for CSV/TSV, markdown tables, and fixed-width columns. Ordered after search/log (which also look "delimited") and before code, with a prose-rejection guard so it never steals `file:line:content` search output, `key: value` logs, or sentences with incidental commas. Rust backend returns `plain_text` for unknown types and the router already falls back to the Python detector, so **no Rust change**. - **Bridge** (`tabular_ingest.py`): stdlib parsers + `to_records()` + a `TabularCompressor` that parses → JSON records → `SmartCrusher` (lossless `csv-schema` first; lossy row-drop with reversible `<<ccr:HASH>>` markers stays SmartCrusher's built-in fallback). Only adopts a result when it actually saves bytes. - **Spreadsheets** (`spreadsheet_ingest.py`): `.xlsx`/`.xls` → per-sheet CSV text at the SDK boundary. Optional deps (`pip install headroom-ai[spreadsheet]`) fail loudly with an install hint, never silently degrade. - **Routing** (`content_router.py`): `CompressionStrategy.TABULAR`, `enable_tabular_compressor` flag, lazy getter, apply branch, strategy maps, Kompress fallback eligibility. - **SDK** (`compress.py`): `compress_spreadsheet(path, ...)` helper (one message per sheet). - **Packaging** (`pyproject.toml`): new `[spreadsheet]` extra; `openpyxl` added to `[dev]` so the xlsx path is exercised in CI. - **Docs/demo**: `examples/tabular_compression_demo.py` + README entry. ### Design note: lossless-only Compact, all-unique tables with no query yield ~0 savings — this is correct, not a bug. SmartCrusher returns `skip:unique_entities_no_signal` and won't drop unique rows without a duplicate/relevance signal. Real wins come from verbose/redundant tables and query-driven selection. A pressure-driven lossy row sampler was considered and intentionally not added. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_transforms_tabular.py -q collected 20 items tests/test_transforms_tabular.py .................... [100%] ============================== 20 passed in 7.15s ============================== $ ruff check headroom/transforms/tabular_ingest.py headroom/transforms/spreadsheet_ingest.py All checks passed! $ mypy headroom/transforms/tabular_ingest.py headroom/transforms/spreadsheet_ingest.py Success: no issues found in 2 source files ``` `tests/test_transforms_tabular.py` (20 tests): detection true positives + no-misroute negatives (search/log/JSON/prose), parser units (incl. fixed-width), the CSV→SmartCrusher bridge, router routing + disable flag, and `.xlsx` ingestion (skipif openpyxl missing) + error paths. `spreadsheet_ingest` 100% / `tabular_ingest` 90% line coverage. ## Real Behavior Proof - **Environment:** local checkout of `feat/tabular-compression`, Python 3.x, `pip install -e ".[dev]"`. - **Exact command / steps:** `python examples/tabular_compression_demo.py` (no API key required). - **Observed result:** ```text === Raw tabular text (ContentRouter, char-level) === compact unique CSV strat=tabular chars 1306 -> 1072 ( 17.9% saved) redundant CSV strat=tabular chars 2661 -> 1350 ( 49.3% saved) verbose markdown strat=tabular chars 2019 -> 1580 ( 21.7% saved) === Full pipeline (real tokenizer) === redundant CSV tokens 768 -> 394 ( 48.7% saved) === Binary spreadsheet (.xlsx) === 2-sheet workbook tokens 1092 -> 683 ( 37.5% saved) ``` - **Not tested:** legacy `.xls` binary path (needs optional `xlrd` + binary fixture; `# pragma: no cover`); base64-embedded `.xlsx` inside multimodal blocks (out of scope, noted as a follow-up). ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes - CHANGELOG/version are intentionally untouched: this repo uses **release-please**, which bumps the version and CHANGELOG via automated `chore: release main` PRs, not per-feature PRs. - The `.xls` path is `# pragma: no cover` (legacy, needs optional `xlrd` + a binary fixture). - Follow-up (out of scope): base64-embedded `.xlsx` inside tool-result/multimodal blocks; porting tabular parsers into the Rust core for parity. --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
122 lines
4.7 KiB
YAML
122 lines
4.7 KiB
YAML
name: Dev Containers
|
|
|
|
on:
|
|
push:
|
|
branches: [main]
|
|
paths:
|
|
- ".devcontainer/**"
|
|
- ".github/workflows/devcontainers.yml"
|
|
- "pyproject.toml"
|
|
- "uv.lock"
|
|
pull_request:
|
|
branches: [main]
|
|
paths:
|
|
- ".devcontainer/**"
|
|
- ".github/workflows/devcontainers.yml"
|
|
- "pyproject.toml"
|
|
- "uv.lock"
|
|
workflow_dispatch:
|
|
|
|
jobs:
|
|
validate:
|
|
runs-on: ubuntu-latest
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include:
|
|
- name: default
|
|
config: .devcontainer/devcontainer.json
|
|
- name: memory-stack
|
|
config: .devcontainer/memory-stack/devcontainer.json
|
|
|
|
steps:
|
|
- uses: actions/checkout@v6
|
|
|
|
# The memory-stack devcontainer brings up Neo4j + Postgres +
|
|
# Redis + Qdrant on top of the Docker base. PR #495 surfaced
|
|
# that the GitHub-hosted runner now ships with too little
|
|
# headroom for that stack: the runner's diagnostic log writer
|
|
# hit "No space left on device" mid-smoke-test, killing the
|
|
# job before `devcontainer exec` could finish. The default
|
|
# devcontainer is light enough to pass on the same image.
|
|
#
|
|
# Reclaim ~14 GB by stripping preinstalled tools none of the
|
|
# devcontainer paths use (Android SDK, .NET, Haskell). Run
|
|
# ONLY on memory-stack — the default validate is fast and
|
|
# benefits from the preinstalled toolcache.
|
|
- name: Free runner disk (memory-stack only)
|
|
if: matrix.name == 'memory-stack'
|
|
uses: jlumbroso/free-disk-space@v1.3.1
|
|
with:
|
|
tool-cache: true
|
|
android: true
|
|
dotnet: true
|
|
haskell: true
|
|
large-packages: false
|
|
docker-images: false
|
|
swap-storage: false
|
|
|
|
- name: Set up Node
|
|
uses: actions/setup-node@v6
|
|
with:
|
|
node-version: "20"
|
|
|
|
- name: Set up Docker Buildx
|
|
uses: docker/setup-buildx-action@v4
|
|
|
|
- name: Install Dev Container CLI
|
|
run: npm install -g @devcontainers/cli@0.85.0
|
|
|
|
- name: Start ${{ matrix.name }}
|
|
run: devcontainer up --workspace-folder . --config ${{ matrix.config }} --remove-existing-container
|
|
|
|
- name: Smoke test ${{ matrix.name }}
|
|
shell: bash
|
|
run: |
|
|
if [[ "${{ matrix.name }}" == "memory-stack" ]]; then
|
|
devcontainer exec --workspace-folder . --config ${{ matrix.config }} bash -lc 'git rev-parse --show-toplevel >/dev/null && uv --version && node --version && gh --version >/dev/null && uv run python -c "import socket; socket.create_connection((\"qdrant\", 6333), 5).close(); socket.create_connection((\"neo4j\", 7687), 5).close(); from mem0 import Memory; from qdrant_client import QdrantClient; import neo4j; import headroom; print(\"memory-stack smoke test passed\")"'
|
|
else
|
|
devcontainer exec --workspace-folder . --config ${{ matrix.config }} bash -lc 'git rev-parse --show-toplevel >/dev/null && uv --version && node --version && gh --version >/dev/null && uv run python -c "import headroom; print(\"default smoke test passed\")"'
|
|
fi
|
|
|
|
validate-worktree:
|
|
runs-on: ubuntu-latest
|
|
|
|
steps:
|
|
- uses: actions/checkout@v6
|
|
|
|
# The worktree devcontainer runs the same `uv sync --extra dev` build as
|
|
# the default validate job, which now pulls transformers 5.x. Copying that
|
|
# into the venv volume exhausts the GitHub runner's disk ("No space left on
|
|
# device", see PR #495). Reclaim ~14 GB by stripping preinstalled tools the
|
|
# build never touches — mirrors the memory-stack job's existing remedy.
|
|
- name: Free runner disk
|
|
uses: jlumbroso/free-disk-space@v1.3.1
|
|
with:
|
|
tool-cache: true
|
|
android: true
|
|
dotnet: true
|
|
haskell: true
|
|
large-packages: false
|
|
docker-images: false
|
|
swap-storage: false
|
|
|
|
- name: Create linked worktree
|
|
run: git worktree add "$RUNNER_TEMP/headroom-worktree" HEAD
|
|
|
|
- name: Set up Node
|
|
uses: actions/setup-node@v6
|
|
with:
|
|
node-version: "20"
|
|
|
|
- name: Set up Docker Buildx
|
|
uses: docker/setup-buildx-action@v4
|
|
|
|
- name: Install Dev Container CLI
|
|
run: npm install -g @devcontainers/cli@0.85.0
|
|
|
|
- name: Start linked worktree devcontainer
|
|
run: devcontainer up --workspace-folder "$RUNNER_TEMP/headroom-worktree" --config "$RUNNER_TEMP/headroom-worktree/.devcontainer/devcontainer.json" --remove-existing-container
|
|
|
|
- name: Smoke test linked worktree
|
|
run: devcontainer exec --workspace-folder "$RUNNER_TEMP/headroom-worktree" --config "$RUNNER_TEMP/headroom-worktree/.devcontainer/devcontainer.json" bash -lc 'git rev-parse --show-toplevel && uv run python -c "import headroom; print(\"worktree smoke test passed\")"'
|