wdunn001 1c894a3685 docs(design): 0001 import-time chunk compression pipeline

Design proposal only. No behaviour changes; nothing here is compiled,
built or deployed.

Compress chunks once at version import into a derived, content-addressed
sidecar store, rather than compressing at serve time or as a separate
manual pass. Codec is zstd, declared as a versioned per-chunk property so
it can be replaced later without a migration.

Three structural decisions carry the design:

- The compress/skip decision is per chunk, not per version. Measured
  zstd-19 ratios across real 64MB samples span 0.057 (.cam) to 0.997
  (.bdt); content-type variance dwarfs codec choice by roughly 17x
  against 1.10x.
- Raw is not a stored representation, it is the absence of one. A chunk
  that does not clear the ratio threshold is simply not written, and is
  served by the existing offset-slicing path unchanged. Incompressible
  content therefore costs zero extra storage and zero new serve-path code.
- Compression is opt-in at the wire level via X-Drop-Accept-Codecs. A
  client that declares nothing gets exactly what it gets today, so old
  clients keep working with no change at all.

Also covers: the ManifestWriterFactory seam (already writes chunk bytes,
currently passed None by torrential, so compression adds no read I/O);
a distinct IV for the compressed representation to avoid AES-CTR
keystream reuse; bounded and niced encoder pool held apart from
READER_SEMAPHORE so .88 is not starved; failure paths that all degrade to
raw; and an opportunistic backfill that rides the existing 42-game
metadata reimport backlog instead of a 130 CPU-hour batch.

Deliberately not promising a headline savings figure: the measured 35%
mean is per-file, not byte-weighted, and large archives compress worst.
The pipeline reports the real byte-weighted ratio per import instead.

Open questions for review are in section 15.

2026-08-03 15:17:31 -04:00

1.9 KiB

Raw Permalink Blame History

Engineering design documents

This directory holds engineering design documents: proposals, architecture decisions, and interface sketches that are argued and agreed before code is written.

It is not the user-facing documentation site. That lives in sites/docs (Astro Starlight, published). Nothing here is built, deployed, compiled, or type-checked.

Layout

Path	Contents
`docs/design/NNNN-<slug>.md`	One design document per proposal, numbered in the order they were opened.
`docs/design/sketches/`	Non-compiling interface sketches referenced by a design document.

Sketch files

Files under docs/design/sketches/ carry a trailing .sketch extension (foo.rs.sketch, foo.ts.sketch, foo.proto.sketch) so that no build tool picks them up:

there is no root Cargo.toml workspace, and no crate's src/ includes this path, so .rs.sketch is invisible to cargo;
pnpm-workspace.yaml globs only server/, libraries/base/, sites/* and desktop/, so .ts.sketch is invisible to pnpm, nuxt and tsc;
torrential/build.rs compiles only torrential/proto/*.proto, so .proto.sketch is invisible to protoc.

Sketches are illustrative. They name the types, traits and field layouts a design implies. They are deliberately incomplete and will not compile.

Status values

Each design document declares a status in its frontmatter:

Draft: written, not yet reviewed.
Accepted: agreed. Implementation may begin.
Implemented: shipped. The document is now history; the code is the truth.
Superseded by NNNN: replaced.

A document at Draft or Accepted describes something that does not exist yet. Do not read it as a description of current behaviour.

Index

#	Title	Status
0001	Import-time chunk compression for game delivery	Draft

1.9 KiB Raw Permalink Blame History