1c894a3685
Design proposal only. No behaviour changes; nothing here is compiled, built or deployed. Compress chunks once at version import into a derived, content-addressed sidecar store, rather than compressing at serve time or as a separate manual pass. Codec is zstd, declared as a versioned per-chunk property so it can be replaced later without a migration. Three structural decisions carry the design: - The compress/skip decision is per chunk, not per version. Measured zstd-19 ratios across real 64MB samples span 0.057 (.cam) to 0.997 (.bdt); content-type variance dwarfs codec choice by roughly 17x against 1.10x. - Raw is not a stored representation, it is the absence of one. A chunk that does not clear the ratio threshold is simply not written, and is served by the existing offset-slicing path unchanged. Incompressible content therefore costs zero extra storage and zero new serve-path code. - Compression is opt-in at the wire level via X-Drop-Accept-Codecs. A client that declares nothing gets exactly what it gets today, so old clients keep working with no change at all. Also covers: the ManifestWriterFactory seam (already writes chunk bytes, currently passed None by torrential, so compression adds no read I/O); a distinct IV for the compressed representation to avoid AES-CTR keystream reuse; bounded and niced encoder pool held apart from READER_SEMAPHORE so .88 is not starved; failure paths that all degrade to raw; and an opportunistic backfill that rides the existing 42-game metadata reimport backlog instead of a 130 CPU-hour batch. Deliberately not promising a headline savings figure: the measured 35% mean is per-file, not byte-weighted, and large archives compress worst. The pipeline reports the real byte-weighted ratio per import instead. Open questions for review are in section 15.
50 lines
1.9 KiB
Markdown
50 lines
1.9 KiB
Markdown
# Engineering design documents
|
|
|
|
This directory holds **engineering design documents**: proposals, architecture
|
|
decisions, and interface sketches that are argued and agreed *before* code is
|
|
written.
|
|
|
|
It is not the user-facing documentation site. That lives in `sites/docs`
|
|
(Astro Starlight, published). Nothing here is built, deployed, compiled, or
|
|
type-checked.
|
|
|
|
## Layout
|
|
|
|
| Path | Contents |
|
|
|---|---|
|
|
| `docs/design/NNNN-<slug>.md` | One design document per proposal, numbered in the order they were opened. |
|
|
| `docs/design/sketches/` | Non-compiling interface sketches referenced by a design document. |
|
|
|
|
## Sketch files
|
|
|
|
Files under `docs/design/sketches/` carry a trailing `.sketch` extension
|
|
(`foo.rs.sketch`, `foo.ts.sketch`, `foo.proto.sketch`) so that no build tool
|
|
picks them up:
|
|
|
|
* there is no root `Cargo.toml` workspace, and no crate's `src/` includes this
|
|
path, so `.rs.sketch` is invisible to cargo;
|
|
* `pnpm-workspace.yaml` globs only `server/`, `libraries/base/`, `sites/*` and
|
|
`desktop/`, so `.ts.sketch` is invisible to pnpm, nuxt and tsc;
|
|
* `torrential/build.rs` compiles only `torrential/proto/*.proto`, so
|
|
`.proto.sketch` is invisible to protoc.
|
|
|
|
Sketches are illustrative. They name the types, traits and field layouts a
|
|
design implies. They are deliberately incomplete and will not compile.
|
|
|
|
## Status values
|
|
|
|
Each design document declares a status in its frontmatter:
|
|
|
|
* **Draft**: written, not yet reviewed.
|
|
* **Accepted**: agreed. Implementation may begin.
|
|
* **Implemented**: shipped. The document is now history; the code is the truth.
|
|
* **Superseded by NNNN**: replaced.
|
|
|
|
A document at `Draft` or `Accepted` describes something that **does not exist
|
|
yet**. Do not read it as a description of current behaviour.
|
|
|
|
## Index
|
|
|
|
| # | Title | Status |
|
|
|---|---|---|
|
|
| [0001](design/0001-import-time-chunk-compression.md) | Import-time chunk compression for game delivery | Draft |
|