drop/docs/README.md

51 lines
1.9 KiB
Markdown
Raw Permalink Normal View History

docs(design): 0001 import-time chunk compression pipeline Design proposal only. No behaviour changes; nothing here is compiled, built or deployed. Compress chunks once at version import into a derived, content-addressed sidecar store, rather than compressing at serve time or as a separate manual pass. Codec is zstd, declared as a versioned per-chunk property so it can be replaced later without a migration. Three structural decisions carry the design: - The compress/skip decision is per chunk, not per version. Measured zstd-19 ratios across real 64MB samples span 0.057 (.cam) to 0.997 (.bdt); content-type variance dwarfs codec choice by roughly 17x against 1.10x. - Raw is not a stored representation, it is the absence of one. A chunk that does not clear the ratio threshold is simply not written, and is served by the existing offset-slicing path unchanged. Incompressible content therefore costs zero extra storage and zero new serve-path code. - Compression is opt-in at the wire level via X-Drop-Accept-Codecs. A client that declares nothing gets exactly what it gets today, so old clients keep working with no change at all. Also covers: the ManifestWriterFactory seam (already writes chunk bytes, currently passed None by torrential, so compression adds no read I/O); a distinct IV for the compressed representation to avoid AES-CTR keystream reuse; bounded and niced encoder pool held apart from READER_SEMAPHORE so .88 is not starved; failure paths that all degrade to raw; and an opportunistic backfill that rides the existing 42-game metadata reimport backlog instead of a 130 CPU-hour batch. Deliberately not promising a headline savings figure: the measured 35% mean is per-file, not byte-weighted, and large archives compress worst. The pipeline reports the real byte-weighted ratio per import instead. Open questions for review are in section 15.
2026-08-03 15:17:31 -04:00
# Engineering design documents
This directory holds **engineering design documents**: proposals, architecture
decisions, and interface sketches that are argued and agreed *before* code is
written.
It is not the user-facing documentation site. That lives in `sites/docs`
(Astro Starlight, published). Nothing here is built, deployed, compiled, or
type-checked.
## Layout
| Path | Contents |
|---|---|
| `docs/design/NNNN-<slug>.md` | One design document per proposal, numbered in the order they were opened. |
| `docs/design/sketches/` | Non-compiling interface sketches referenced by a design document. |
## Sketch files
Files under `docs/design/sketches/` carry a trailing `.sketch` extension
(`foo.rs.sketch`, `foo.ts.sketch`, `foo.proto.sketch`) so that no build tool
picks them up:
* there is no root `Cargo.toml` workspace, and no crate's `src/` includes this
path, so `.rs.sketch` is invisible to cargo;
* `pnpm-workspace.yaml` globs only `server/`, `libraries/base/`, `sites/*` and
`desktop/`, so `.ts.sketch` is invisible to pnpm, nuxt and tsc;
* `torrential/build.rs` compiles only `torrential/proto/*.proto`, so
`.proto.sketch` is invisible to protoc.
Sketches are illustrative. They name the types, traits and field layouts a
design implies. They are deliberately incomplete and will not compile.
## Status values
Each design document declares a status in its frontmatter:
* **Draft**: written, not yet reviewed.
* **Accepted**: agreed. Implementation may begin.
* **Implemented**: shipped. The document is now history; the code is the truth.
* **Superseded by NNNN**: replaced.
A document at `Draft` or `Accepted` describes something that **does not exist
yet**. Do not read it as a description of current behaviour.
## Index
| # | Title | Status |
|---|---|---|
| [0001](design/0001-import-time-chunk-compression.md) | Import-time chunk compression for game delivery | Draft |