mirror of
https://github.com/diangogav/EDOpro-server-ts
synced 2026-08-24 00:23:05 -04:00
aa7c96bd46
* feat(resources): add declarative resources.manifest.json
Introduces the normative manifest (11 sources, 22 assembly rules)
replacing all hardcoded URLs and mappings in the bash pipeline.
Covers RSM-001 through RSM-012 source + assembly schema.
* feat(resources): add resources-lib.sh shared bash library
Implements all shared functions sourced by the two resource scripts:
- fail() logger, jq preflight (fail-fast on missing jq)
- validate_manifest (RSM-005 a-e: JSON parse, type check, dangling from,
file+dir combo, target collision without overwrite)
- http_integrity_check (RSM-006: non-empty + SQLite magic for .cdb)
- apply_rule (RSM-002: whole-source/dir/file/only-glob via find -name,
cwd-invariant, spaces+parens safe)
- apply_rule_with_fallback (RSM-003: ordered from[] fallback chain)
- sync_repo (RSM-007: moved from clone_repositories.sh, id-keyed dirs)
Manifest path resolved via BASH_SOURCE[0] (D1 cwd invariant, D8 id-keying).
* test(resources): add bats fixture tree and resources-lib test suite
Fixture tree (test/fixtures/):
- repositories/source-a, source-b: git-less fixture dirs for assembly tests
- repositories/source-with-dir: dir-only copy shape fixture
- repositories/http-source: crafted .cdb files (valid SQLite magic, empty,
corrupt) and a non-cdb file for integrity check tests
- manifests/: 6 JSON fixtures covering each RSM-005 failure scenario plus a
valid minimal manifest
Bats suite (test/resources-lib.bats, 24 tests):
- RSM-005 a-e: each bad-manifest fixture causes validate_manifest exit 1
- RSM-006: empty file, non-SQLite .cdb, valid .cdb, non-cdb integrity checks
- RSM-002: all rule shapes including only-glob with spaces+parens in filenames
- RSM-003: first-source-wins, fallback-to-second, chain exhausted
- RSM-004: collision detection + intentional overwrite pass
- RSM-012: unused evolution-assets files absent; manifest counts (11/22)
All 24 tests pass: tools/bats-core/bin/bats test/resources-lib.bats
biome.json: exclude test/fixtures/** (fixture files are intentionally invalid
JSON in some cases and are not production code).
* test(resources): make bats suite self-contained (generate fixtures in setup)
Previously, test/fixtures/repositories/ was generated on disk but gitignored
by *repositories/ and *.cdb patterns in .gitignore, so fixture source files
and crafted .cdb binaries were never committed.
Fix: move all fixture source generation into setup() — each test creates its
own tree under mktemp -d. Binary fixture data (SQLite magic bytes, empty file,
corrupt bytes) is produced inline via printf. The test/fixtures/manifests/ JSON
files remain in git as before.
All 24 tests pass with the self-contained approach.
* fix(resources): harden integrity check and manifest validation per review
- http_integrity_check: binary cmp of first 16 bytes (no command substitution) so the
trailing SQLite magic NUL is compared; a 15-byte file is rejected
- validate_manifest: RSM-001 checks for non-empty/unique id, non-empty url, http
filename, and array-from requiring file
- resources.manifest.json: order ocg lflist rule after the ygopro/formats whole-dir
rule (D2 prefix-nested invariant)
- tests: dir+only shape, apply_rule failure paths, executor overwrite, specific
offending-id/target assertions
* fix(resources): harden validate_manifest and http_integrity_check
- Add early structural guard: .sources and .assembly must be arrays;
abort naming the violation if either is a non-array type.
- Capture jq exit status in the unknown-type check (b); a jq crash
(e.g. non-string type field) now fails hard instead of fail-open.
- Use (.type|tostring) so integer or boolean type values produce a
readable error instead of crashing jq.
- Case-insensitive .cdb match in http_integrity_check via ${filepath,,}
so uppercase .CDB extensions also trigger the SQLite header check.
- Remove vestigial 6th "" argument from all apply_rule calls in the
bats suite (function signature is 5 args: src_dir dir file only target).
- Add 6 new bats tests covering all four hardening scenarios (41 total).
Fixes JD backlog items 1-4 from the PR1a adversarial review.
* feat(resources): rewrite clone_repositories.sh as manifest interpreter
Replaces all hardcoded git/wget calls with a loop over sources[] from
resources.manifest.json. Sources are keyed by id (repositories/<id>).
- Source resources-lib.sh (validate_manifest, sync_repo,
http_integrity_check, fail) so no logic is duplicated.
- Run validate_manifest fail-fast before any network operation (RSM-005).
- git sources: call sync_repo <id> <url> <branch> (D8 id-keyed dirs).
Missing branch field passes "" so sync_repo uses remote default (RSM-007).
- http sources: wget -qO repositories/<filename> <url>, then
http_integrity_check for non-empty + SQLite magic on .cdb files (RSM-006).
- Remove cd repositories line; all paths are repo-root-relative (D1).
- No hardcoded URLs, branches, or filenames remain (RSM-010).
Bats coverage: 6 new tests (47 total) covering validation abort,
git/http stub flows, empty-download fail, RSM-010 lint, D1 cwd lint.
* feat(resources): rewrite setup_resources.sh as assembly interpreter
Replaces all hardcoded cp/MAP blocks with a loop over assembly[] from
resources.manifest.json. Atomic swap and GC skeleton survive byte-compatible.
- Source resources-lib.sh (apply_rule, apply_rule_with_fallback, fail).
- Iterate assembly[] rules: dispatch to apply_rule for single-source
rules (whole-source / dir / file / only-glob) and to
apply_rule_with_fallback for from-array (RSM-003 fallback chain).
- Resolve source dir per source type: git -> repositories/<id>;
http -> repositories/ (flat file, keyed by filename from clone stage).
- .git directories are stripped from staging before the atomic publish
(JD backlog item 6; matches design D1/PUBLISH stage).
- Atomic symlink swap and GC are preserved verbatim from the original
script; RESOURCES_KEEP_RELEASES env var behaviour is unchanged.
- REPOS_ROOT / RELEASES_ROOT env var overrides for test isolation.
- No hardcoded source paths, directory names, or copy mappings remain
(RSM-010).
Bats coverage: 5 new tests (52 total) — assembly, .git strip,
abort-before-swap, GC keep limit, RSM-010 lint.
* test(resources): verify content parity between old and new layout
Main scripts (from main branch) cloned sources and assembled old layout.
New scripts (this branch) cloned sources and assembled new layout.
Old layout path-translated in temp dir:
ygopro/alternatives -> ygopro/formats
ygopro/ocg -> ygopro/formats/ocg
ygopro/prereleases-cdb -> ygopro/extensions/prereleases
ygopro/cards-art -> ygopro/extensions/cards-art
Result: diff -rq exit code 0 — zero content differences.
The 4 unused evolution-assets files are absent from the new layout.
* feat(docker): install jq and copy resources-lib + manifest into build stages
The resource provisioning scripts (clone_repositories.sh, setup_resources.sh)
source resources-lib.sh and read resources.manifest.json via jq. Both build
stages that run or ship these scripts must therefore have jq installed and the
library + manifest present, or the Docker build and the runtime updater sidecar
break. Add jq to the apt-get lists of the resources-builder and final stages,
and add resources-lib.sh + resources.manifest.json to both COPY lines.
* fix(resources): harden symlink target, REPOS_ROOT honoring, and http-rule validation
- setup_resources.sh: derive the current symlink target from basename $RELEASES
instead of hardcoding 'releases/', so a non-default RELEASES_ROOT no longer
produces a dangling symlink.
- setup_resources.sh: define _resolve_src_dir() once before the assembly loop
(not redefined per iteration) and pass the source id via jq --arg instead of
interpolating it into the filter string.
- clone_repositories.sh + resources-lib.sh (sync_repo): honor REPOS_ROOT
(default ./repositories) so both fetch and assembly stages agree on the
checkout root; defaults are unchanged.
- resources-lib.sh (validate_manifest): reject any assembly rule that references
an http source without a file key (http sources resolve to a flat file under
repositories/, so a dir/only/whole-source rule would copy the whole tree);
the error names the offending rule target.
* test(resources): make git-clone and .git-strip assertions load-bearing
- clone git-source test: assert the recorded CALL_LOG contains the expected
'git clone --depth 1 --branch main <url>' invocation instead of only checking
exit 0, and isolate the checkout via REPOS_ROOT so it never touches the real
repositories/ tree (which also guarantees the clean-clone path).
- .git-strip test: the fixture only used file: rules, so .git never reached
staging and the assertion passed vacuously even with the strip line removed;
additionally find on the current symlink without -L never descended into the
release. Add a whole-source rule over the git fixture so .git lands in staging
and use 'find -L' so the check is load-bearing (fails when the strip line is
removed, passes when present).
- add http-source-no-file.json fixture + test for the new validate_manifest
rule rejecting http sources referenced without a file key; give valid-minimal
the http rule a file key so it stays valid under the new check.
- isolate the http clone tests via REPOS_ROOT to stop real-tree pollution.
* chore(config): align YGOPRO folder env to new resource layout
Update YGOPRO_FOLDERS and YGOPRO_EXTRA_FOLDERS in .env.example and
docker-compose.prod.yaml to the enumerated leaf-dir shape under the new
formats/ and extensions/ layout. Leaf dirs are enumerated because the
loader's readdir is non-recursive (goat/rush/speed/gx/mdc omitted).
* fix(resources): harden manifest validation and .git strip
- validate_manifest: reject assembly rules with empty/missing target
(previously published a literal null/ dir), naming the rule index.
- validate_manifest: for rules resolving to an http source, require the
rule file to equal the source filename, naming target + source id.
- setup_resources.sh: remove '|| true' (and 2>/dev/null) from the .git
strip so a failed strip aborts before the atomic publish.
* test(resources): cover target/filename validation and fix stub bashism
- Add fixtures + tests for missing-target and http file/filename mismatch.
- Fix git stub shebang to bash (${@: -1} is a bashism under /bin/sh).
- Add resources-lib.sh to the RSM-010 hardcoded-URL lint (lib is clean).
* fix(resources): guard against non-object assembly elements in validate_manifest
* ci(resources): add bash-resources-tests job to pipeline
* docs(resources): update README env blocks to new layout and add migration + jq notes
* feat(resources): add runtime.ygopro pool section to manifest (RFD-001)
* test(resources): bats regression for validate_manifest tolerates runtime section (RFD-002)
* test(resolver): add failing unit tests for ResourcePoolResolver (RFD-003/004/005)
* feat(resolver): implement ResourcePoolResolver (RFD-003/004/005)
* feat(config): add manifestPath (MANIFEST_PATH env, default ./resources.manifest.json)
* feat(loader): wire YGOProResourceLoader to use ResourcePoolResolver for pool derivation
* feat(search): wire YGOProCardSearchRepository to use resolved extended pool
* test(resolver): add whole-chain integration test with in-memory fixtures (RFD-007)
* fix(config): remove enumerated YGOPRO_FOLDERS/EXTRA values; derivation is now the default (RFD-006)
* docs(resources): update README env section for manifest-driven pool derivation (RFD-008)
- Replace YGOPRO_FOLDERS/EXTRA_FOLDERS as required env vars with RESOURCES_DIR + MANIFEST_PATH
- Document standard/extended pool derivation from resources.manifest.json
- Add pre-deploy smoke check procedure with pool inspection snippet
- Mark YGOPRO_FOLDERS/EXTRA_FOLDERS as optional override (deprecated)
* feat(resources): adopt production source taxonomy and add classic pre-errata pool
Rename manifest source ids to edopro-*/ygopro-* scheme, add custom-cards
and classic (pre-errata alt-coded variants for edison/hat) with its own
scripts, and expose classic as a distinct cdb via classic.cdb filename.
Point edopro banlist bootstrap at the new evolution-lflists/lflists dirs.
* feat(resources): warn on missing pool dirs and duplicate cdb basenames
* fix(resources): make pool diagnostics one-shot and fix doc/test drift
* refactor(resources)!: remove deprecated YGOPRO_FOLDERS env override; manifest is sole pool source
YGOPRO_FOLDERS and YGOPRO_EXTRA_FOLDERS are removed entirely. The manifest
runtime.ygopro.standard/.extended section is now the only source of pool
membership. Removed env field from ResourcePoolResolverOptions, env-override
branches from resolvePools, and 5 associated tests. All callers updated.
MANIFEST_PATH, RESOURCES_DIR, and YGOPRO_EXTRA_SCRIPTS remain untouched.
* refactor(scripts): move resource pipeline scripts into scripts/ dir
Moves clone_repositories.sh, setup_resources.sh, resources-lib.sh,
resources-updater.sh, entrypoint.sh, build_core_integrator.sh, and
install_dependencies.sh from repo root into scripts/. resources.manifest.json
stays at root as user-facing config.
Landmines handled:
- .dockerignore: scripts/ excluded but !scripts/*.sh re-included so the
build context delivers the .sh files to Docker COPY
- resources-lib.sh: MANIFEST_PATH default changed from BASH_SOURCE-relative
to CWD-relative (resources.manifest.json) since the manifest stays at root
- entrypoint.sh: bash resources-updater.sh -> bash scripts/resources-updater.sh
- resources-updater.sh: bare clone/setup calls -> scripts/ prefixed
Dockerfile: two COPYs per stage (scripts/ + manifest), RUN and CMD updated.
test/resources-lib.bats + test/manifest-runtime-tolerance.bats: LIB and all
bash "$REPO_ROOT/..." paths updated to scripts/ prefix. README: all command
examples updated. .github/ docs move included (was staged, not yet committed).
* test(resources): drop brittle manifest-snapshot assertions
The RSM-012 count/reference tests hardcoded the migration-era manifest
(11 sources, 22 rules, no classic.cdb). The production manifest is a
living config: it now has 10 sources, 21 rules, and legitimately
references classic.cdb. These snapshot tests assert content that is
meant to evolve; schema/behavior tests provide the real coverage.
* chore(deps): bump koishipro-core.js to ^1.5.2 and ygopro-msg-encode to ^1.3.0
The resource folder-discovery work was developed and tested against
koishipro-core.js 1.5.2 (non-recursive readdir pool scanning). Pinning
the lockfile to the tested versions keeps CI consistent with local.
35 lines
850 B
Text
35 lines
850 B
Text
# ── Server Ports ──
|
|
HOST_PORT=7911
|
|
YGOPRO_PORT=7711
|
|
HTTP_PORT=7922
|
|
WEBSOCKET_PORT=4000
|
|
|
|
# ── YGOPro Card Resources ──
|
|
# Resources live under resources/current (a symlink to the active release,
|
|
# maintained by setup_resources.sh / the in-container refresh loop).
|
|
RESOURCES_DIR=./resources/current
|
|
|
|
# ── Database ──
|
|
POSTGRES_HOST=localhost
|
|
POSTGRES_PORT=5432
|
|
POSTGRES_DB=evolution
|
|
POSTGRES_USER=evolution
|
|
POSTGRES_PASSWORD=your-postgres-password
|
|
|
|
# ── Redis / Valkey ──
|
|
USE_REDIS=false
|
|
REDIS_URI=redis://localhost:6379
|
|
|
|
# ── Ranking ──
|
|
RANK_ENABLED=false
|
|
SEASON=5
|
|
|
|
# ── Rate Limiting ──
|
|
RATE_LIMIT_ENABLED=true
|
|
RATE_LIMIT=3
|
|
RATE_LIMIT_WINDOW=300
|
|
|
|
# ── Misc ──
|
|
NODE_ENV=development
|
|
ADMIN_API_KEY=your-admin-api-key
|
|
SIDE_TIMEOUT_MINUTES=3
|