akadmin/headroom
1
0
Fork 0
mirror of https://github.com/headroomlabs-ai/headroom.git synced 2026-08-27 14:17:10 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
2696 commits 410 branches 169 tags 142 MiB
Commit graph

3 commits

Author SHA1 Message Date
Ashish
d789a7c528
feat(transforms): tabular + spreadsheet (.xlsx/.xls) compression (#1128) 
## Description

Adds a content-type-aware path for **tabular data** — CSV/TSV, markdown
tables, fixed-width text, and binary `.xlsx`/`.xls` spreadsheets — by
routing them through the existing, battle-tested `SmartCrusher` instead
of letting them fall through to `PLAIN_TEXT → Kompress`.

The pipeline already compressed tables losslessly when handed a JSON
array of records. This wires up the missing front door: detect tabular
text (and ingest binary spreadsheets), convert to JSON records, and
reuse `SmartCrusher.crush()`. No new compression algorithm.

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **Detection** (`content_detector.py`): new `ContentType.TABULAR` +
`_try_detect_tabular()` for CSV/TSV, markdown tables, and fixed-width
columns. Ordered after search/log (which also look "delimited") and
before code, with a prose-rejection guard so it never steals
`file:line:content` search output, `key: value` logs, or sentences with
incidental commas. Rust backend returns `plain_text` for unknown types
and the router already falls back to the Python detector, so **no Rust
change**.
- **Bridge** (`tabular_ingest.py`): stdlib parsers + `to_records()` + a
`TabularCompressor` that parses → JSON records → `SmartCrusher`
(lossless `csv-schema` first; lossy row-drop with reversible
`<<ccr:HASH>>` markers stays SmartCrusher's built-in fallback). Only
adopts a result when it actually saves bytes.
- **Spreadsheets** (`spreadsheet_ingest.py`): `.xlsx`/`.xls` → per-sheet
CSV text at the SDK boundary. Optional deps (`pip install
headroom-ai[spreadsheet]`) fail loudly with an install hint, never
silently degrade.
- **Routing** (`content_router.py`): `CompressionStrategy.TABULAR`,
`enable_tabular_compressor` flag, lazy getter, apply branch, strategy
maps, Kompress fallback eligibility.
- **SDK** (`compress.py`): `compress_spreadsheet(path, ...)` helper (one
message per sheet).
- **Packaging** (`pyproject.toml`): new `[spreadsheet]` extra;
`openpyxl` added to `[dev]` so the xlsx path is exercised in CI.
- **Docs/demo**: `examples/tabular_compression_demo.py` + README entry.

### Design note: lossless-only

Compact, all-unique tables with no query yield ~0 savings — this is
correct, not a bug. SmartCrusher returns
`skip:unique_entities_no_signal` and won't drop unique rows without a
duplicate/relevance signal. Real wins come from verbose/redundant tables
and query-driven selection. A pressure-driven lossy row sampler was
considered and intentionally not added.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ python -m pytest tests/test_transforms_tabular.py -q
collected 20 items
tests/test_transforms_tabular.py ....................                    [100%]
============================== 20 passed in 7.15s ==============================

$ ruff check headroom/transforms/tabular_ingest.py headroom/transforms/spreadsheet_ingest.py
All checks passed!

$ mypy headroom/transforms/tabular_ingest.py headroom/transforms/spreadsheet_ingest.py
Success: no issues found in 2 source files
```

`tests/test_transforms_tabular.py` (20 tests): detection true positives
+ no-misroute negatives (search/log/JSON/prose), parser units (incl.
fixed-width), the CSV→SmartCrusher bridge, router routing + disable
flag, and `.xlsx` ingestion (skipif openpyxl missing) + error paths.
`spreadsheet_ingest` 100% / `tabular_ingest` 90% line coverage.

## Real Behavior Proof

- **Environment:** local checkout of `feat/tabular-compression`, Python
3.x, `pip install -e ".[dev]"`.
- **Exact command / steps:** `python
examples/tabular_compression_demo.py` (no API key required).
- **Observed result:**
  ```text
  === Raw tabular text (ContentRouter, char-level) ===
compact unique CSV strat=tabular chars 1306 -> 1072 ( 17.9% saved)
redundant CSV strat=tabular chars 2661 -> 1350 ( 49.3% saved)
verbose markdown strat=tabular chars 2019 -> 1580 ( 21.7% saved)

  === Full pipeline (real tokenizer) ===
  redundant CSV            tokens       768 ->    394  ( 48.7% saved)

  === Binary spreadsheet (.xlsx) ===
  2-sheet workbook         tokens      1092 ->    683  ( 37.5% saved)
  ```
- **Not tested:** legacy `.xls` binary path (needs optional `xlrd` +
binary fixture; `# pragma: no cover`); base64-embedded `.xlsx` inside
multimodal blocks (out of scope, noted as a follow-up).

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable

## Additional Notes

- CHANGELOG/version are intentionally untouched: this repo uses
**release-please**, which bumps the version and CHANGELOG via automated
`chore: release main` PRs, not per-feature PRs.
- The `.xls` path is `# pragma: no cover` (legacy, needs optional `xlrd`
+ a binary fixture).
- Follow-up (out of scope): base64-embedded `.xlsx` inside
tool-result/multimodal blocks; porting tabular parsers into the Rust
core for parity.

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-19 11:30:20 -05:00
chopratejas
95e9b39b6d feat: Add AWS Strands Agents SDK integration 
## Description

Add Headroom integration with AWS Strands Agents SDK, enabling automatic
context optimization and tool output compression for Strands-based agents.

Fixes #14

## Type of Change

- [x] New feature (non-breaking change that adds functionality)
- [x] Documentation update

## Changes Made

### Core Integration (`headroom/integrations/strands/`)

- **HeadroomHookProvider** - Implements Strands `HookProvider` interface for
  automatic tool output compression via `AfterToolCallEvent`. Compresses
  verbose tool outputs before they enter conversation context.

- **HeadroomStrandsModel** - Model wrapper that extends Strands `Model` base
  class for message-level optimization. Implements all required abstract
  methods: `stream()`, `get_config()`, `update_config()`, `structured_output()`.

- **Provider auto-detection** - Automatically detects appropriate Headroom
  provider (Anthropic, OpenAI, Google) based on wrapped Strands model type.

- **`strands-agents` as optional dependency** - Install with
  `pip install headroom-ai[strands]`

### Testing (`tests/integrations/test_strands/`)

- **Real integration tests (25 tests)** - Use actual AWS Bedrock API calls
  with Claude 3 Haiku. Skip automatically when credentials unavailable.

- **Unit tests (57 tests)** - Mock-based tests for internal logic, edge cases,
  and error handling. No credentials required.

### Demo (`examples/strands_bedrock_demo.py`)

- Interactive demo showcasing both integration patterns
- Visual before/after compression comparison with token savings
- 4 verbose tools (search, logs, database, metrics) demonstrating real savings
- Supports `--hook` and `--model` flags for individual demos

## Testing

All tests verified:

- [x] Unit tests pass (57 tests)
- [x] Integration tests pass (25 tests with real Bedrock API)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom/integrations/strands/`)
- [x] Formatting passes (`ruff format --check`)
- [x] Demo runs successfully with ~50% token savings

## Test Output

```
$ pytest tests/integrations/test_strands/ -v
=================== 82 passed in 90.09s ===================

$ ruff check headroom/integrations/strands/ --ignore E402
All checks passed!

$ mypy headroom/integrations/strands/ --ignore-missing-imports
Success: no issues found
```

## Demo Results

```
╭────────────────────────────────────────────────────────────╮
│              HeadroomHookProvider Results                  │
│────────────────────────────────────────────────────────────│
│ Tokens BEFORE compression: 51,961                          │
│ Tokens AFTER compression:  25,658                          │
│ Tokens SAVED:              26,303 (50.6%)                  │
╰────────────────────────────────────────────────────────────╯
```
 2026-01-31 00:31:37 -08:00
chopratejas
175746cc26 Prepare for OSS release v0.2.0 
This commit prepares Headroom for public open source release with
comprehensive documentation, licensing, and community infrastructure.

License & Legal:
- Add Apache 2.0 LICENSE file
- Add NOTICE file with third-party attributions
- Add SECURITY.md for vulnerability reporting

Community:
- Add CONTRIBUTING.md with contribution guidelines
- Add CODE_OF_CONDUCT.md (Contributor Covenant)
- Add GitHub issue templates (bug report, feature request)
- Add pull request template

Documentation:
- Update README.md with compelling value proposition
- Add docs/getting-started.md
- Add docs/proxy.md for proxy server documentation
- Add docs/transforms.md for transform reference
- Add docs/api.md for API reference
- Add examples/README.md

Package Infrastructure:
- Add headroom/py.typed for PEP 561 compliance
- Add headroom/cli.py for CLI entry point
- Add .github/workflows/ci.yml for CI pipeline
- Add .github/workflows/publish.yml for PyPI publishing
- Update pyproject.toml with proper metadata

New Features:
- Add multi-provider support (Google, Cohere, LiteLLM, OpenAI-compatible)
- Add universal tokenizer registry with multiple backends
- Add model registry with pricing and context limits
- Add production proxy server with caching and rate limiting

Code Quality:
- Fix 83 lint issues via ruff auto-fix
- Fix version consistency (benchmarks 0.1.0 → 0.2.0)
- Add skip decorators for optional dependency tests
 2026-01-07 11:36:44 -08:00