Design proposal only. No behaviour changes; nothing here is compiled,
built or deployed.
Compress chunks once at version import into a derived, content-addressed
sidecar store, rather than compressing at serve time or as a separate
manual pass. Codec is zstd, declared as a versioned per-chunk property so
it can be replaced later without a migration.
Three structural decisions carry the design:
- The compress/skip decision is per chunk, not per version. Measured
zstd-19 ratios across real 64MB samples span 0.057 (.cam) to 0.997
(.bdt); content-type variance dwarfs codec choice by roughly 17x
against 1.10x.
- Raw is not a stored representation, it is the absence of one. A chunk
that does not clear the ratio threshold is simply not written, and is
served by the existing offset-slicing path unchanged. Incompressible
content therefore costs zero extra storage and zero new serve-path code.
- Compression is opt-in at the wire level via X-Drop-Accept-Codecs. A
client that declares nothing gets exactly what it gets today, so old
clients keep working with no change at all.
Also covers: the ManifestWriterFactory seam (already writes chunk bytes,
currently passed None by torrential, so compression adds no read I/O);
a distinct IV for the compressed representation to avoid AES-CTR
keystream reuse; bounded and niced encoder pool held apart from
READER_SEMAPHORE so .88 is not starved; failure paths that all degrade to
raw; and an opportunistic backfill that rides the existing 42-game
metadata reimport backlog instead of a 130 CPU-hour batch.
Deliberately not promising a headline savings figure: the measured 35%
mean is per-file, not byte-weighted, and large archives compress worst.
The pipeline reports the real byte-weighted ratio per import instead.
Open questions for review are in section 15.
1c894a3685