Commit graph

4 commits

Author SHA1 Message Date
Patrick A
2ad300aff8
fix(transforms): use thread-local tree-sitter parsers to prevent pyo3 Unsendable panic (#604)
## Problem

pyo3 marks `_native::Parser` as `#[pyclass(unsendable)]`, which causes a
hard thread-assertion panic when a parser created on one thread is
accessed from another:

```
thread '<unnamed>' panicked at pyo3-0.28.3/src/impl_/pyclass.rs:1055:9:
assertion `left == right` failed: _native::Parser is unsendable, but sent to another thread
  left: ThreadId(2)
 right: ThreadId(1)
```

The prior implementation stored parsers in a module-level `dict[str,
Any]` (`_tree_sitter_languages`). When `_run_compression_in_executor`
dispatched compression work to a `ThreadPoolExecutor`, pool workers
grabbed parsers from that shared dict that were originally created on
the main asyncio thread and panicked.

This produces a 500 on every request where code compression is attempted
via a pool thread.

## Fix

Replace the global dict with `threading.local()` so each thread creates
and owns its own parser instances. No cross-thread parser access is
possible.

```python
# before
_tree_sitter_languages: dict[str, Any] = {}  # shared — crosses threads

# after
_tree_sitter_local = threading.local()  # per-thread — isolated
```

`is_tree_sitter_loaded()` and `unload_tree_sitter()` updated to operate
on the current thread's local cache (semantics unchanged for
single-threaded callers).

## Tests

9 regression tests added in
`tests/test_transforms/test_tree_sitter_thread_safety.py`:

- Thread isolation: two threads get distinct parser instances
- Within-thread reuse: same thread gets the same cached instance
- Thread pool: parsers usable from `ThreadPoolExecutor` workers without
panic
- Concurrent workers: each distinct pool thread owns a unique parser
- `is_tree_sitter_loaded` / `unload_tree_sitter` lifecycle

Also adds a `filterwarnings` entry for
`PytestUnraisableExceptionWarning`: pyo3 emits this when short-lived
test threads drop parsers at teardown; it does not occur in production
where pool threads are long-lived.

## Relation to #564

PR #564 proposes the same `threading.local()` approach but was blocked
on missing tests (`CHANGES_REQUESTED`). This PR includes the full test
suite.
2026-06-10 18:30:00 -05:00
Patrick A
53d2342291
ci: speed up GitHub Actions — path filters, caching, timeouts, version upgrades (#620)
* ci: speed up GitHub Actions - path filters, caching, timeouts, version upgrades

Performance improvements:
- init-e2e.yml, wrap-e2e.yml: add path filters so e2e Docker builds only run when
  e2e-related files change (saves ~10 min per irrelevant PR push)
- init-e2e.yml, wrap-e2e.yml: add concurrency groups to cancel superseded PR runs
- ci.yml: add pip caching to lint and build jobs
- ci.yml: cache actionlint + act binaries in workflow-validation (skip curl on hits)
- eval.yml: add pip caching to smoke-test and weekly-suite jobs
- docs.yml: add pip caching for mkdocs-material install
- rust.yml: replace cargo install --locked cargo-audit/deny with taiki-e/install-action
  (prebuilt binaries; saves 2-5 min per audit run)

Bug fixes:
- docker.yml: fix actions/checkout@v6 -> @v4 (v6 does not exist; would break all
  Docker builds on every release/PR touching docker paths)

Version upgrades:
- wagoid/commitlint-github-action: @v5 -> @v6
- devcontainers.yml: docker/setup-buildx-action@v3 -> @v4 (align with docker.yml)

Safety improvements:
- ci.yml: add timeout-minutes to all 13 jobs (changes, lint, build-wheel,
  prefetch-model, test x4, test-extras, test-agno, commitlint, build,
  workflow-validation, docker-native-e2e, windows-native-wrapper, macos-native-wrapper)
- docker.yml: add timeout-minutes to docker-build (75m), docker-manifest (20m),
  promote-latest (10m)
- eval.yml: add timeout-minutes to smoke-test (30m); bump weekly-suite 60->90m
- rust.yml: add timeout-minutes to test (30m), wheels (45m), audit (20m)

Observed wall-clock impact on recent PRs:
- Init E2E and Wrap E2E were running on every single PR push regardless of content
- CI workflow was taking 12-17 min; path filters reduce unnecessary e2e runs to 0

* fix(ci): bust actionlint+act cache when workflow file changes

Static cache key 'ci-tools-actionlint-act-v1' never invalidated on
tool version updates. Switched to hashFiles('.github/workflows/ci.yml')
so the cache busts automatically whenever the download scripts are
updated to point at a newer release.

Flagged by adversarial review (Architecture + Testing/Reliability personas).

* fix(ci): add missing Dockerfile COPY paths to e2e path filters

e2e/init/Dockerfile and e2e/wrap/Dockerfile COPY files not covered
by the initial path filter set:

  init-e2e: Cargo.toml, Cargo.lock, rust-toolchain.toml, uv.lock,
            .claude-plugin, .github/plugin/**, plugins/headroom-agent-hooks/**
  wrap-e2e: Cargo.toml, Cargo.lock, rust-toolchain.toml, uv.lock,
            sdk/typescript/**, plugins/openclaw/**

Without these, a Rust toolchain bump or SDK change on a PR would
skip the e2e gate entirely, only catching it on the merge to main.

Flagged by adversarial review (Domain/Correctness persona).

* fix(devcontainer): upgrade uv to >=0.7.0 to parse uv.lock revision=3

* fix(devcontainer): set UV_SKIP_WHEEL_FILENAME_CHECK=1 in post-create.sh for gitpython wheel

* ci: bump actions/checkout and actions/setup-node to v5 (Node.js 20 EOL Jun 16)

* fix(devcontainer): export UV_SKIP_WHEEL_FILENAME_CHECK so uv run also skips wheel check

* ci: bump all GitHub Actions to latest versions (Node.js 24)

* fix(test): accept release-please-action v4 or v5 in workflow assertion

* fix(format): ruff format test_release_workflows.py
2026-06-05 14:32:53 -08:00
JerrettDavis
fdf7d1e138 fix: harden devcontainer worktree startup
Make the devcontainer bootstrap and CI smoke tests work from linked worktrees, including git metadata translation, cache ownership fixes, workspace path handling, and config-scoped smoke checks.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-10 13:49:29 -05:00
JerrettDavis
a1beb08d53 feat: add reproducible devcontainers
Add a default devcontainer and a compose-backed memory-stack profile, validate them in CI, and document the contributor workflow.

Also lock the memory-stack dependencies, pin related container tooling, and sync the latest healthcheck shutdown fix for stubbed memory handlers.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-10 12:58:35 -05:00