Add pytest.importorskip("trafilatura") to HTML extractor test modules
to skip tests gracefully when the optional trafilatura dependency is
not installed. This fixes CI failures in the base test matrix that
doesn't include the html extras.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
HTMLExtractor uses trafilatura to extract main content from HTML pages,
removing scripts, styles, navigation, and ads. This achieves 94.9%
compression while preserving 98.2% recall on the Scrapinghub benchmark.
Key features:
- Automatic HTML detection in content router
- Configurable output format (markdown or text)
- Metadata extraction (title, author, date, description)
- Batch extraction support
Evaluation framework:
- OSS benchmark integration (Scrapinghub Article Extraction Benchmark)
- LLM-as-judge evaluation for QA accuracy preservation
- F1 score: 0.919 on 181-sample benchmark (baseline: 0.958)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
