opencode-pine2pyne 0.1.1__tar.gz → 0.2.0__tar.gz

opencode_pine2pyne-0.2.0/PKG-INFO

@@ -0,0 +1,301 @@
+Metadata-Version: 2.4
+Name: opencode-pine2pyne
+Version: 0.2.0
+Summary: Pine Script v6 to PyneCore Python transpiler
+Author: rubycell
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/rubycell/pine2pyne
+Project-URL: Repository, https://github.com/rubycell/pine2pyne
+Keywords: pinescript,tradingview,transpiler,pynecore,backtesting
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Office/Business :: Financial :: Investment
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: runtime
+Requires-Dist: opencode-pyneruntime[cli]>=6.9.1; extra == "runtime"
+Dynamic: license-file
+
+# PyneCore - Pine Script to Python Transpiler
+
+> **Backlog & known bugs:** [Projects board #10](https://github.com/users/rubycell/projects/10)
+> — open work is in **Backlog** (prioritised P0/P1/P2), shipped work in **Done**.
+> Confirmed unfixed transpiler bugs live there too; several are pinned by strict
+> `xfail` tests (see [Running Tests](#running-tests)).
+
+## Installation (New Computer)
+
+### Fastest: the sandbox-setup skill
+
+From the repo root, run the bundled setup script (idempotent — safe to re-run):
+
+```bash
+bash ~/.claude/skills/opencode-pine2pyne-setup/setup.sh   # from inside this clone; the skill lives in ~/.claude/skills
+```
+
+It builds `.venv`, installs everything below, detects `PINE_CPU_COUNT` /
+`PINE_MAX_WORKERS`, seeds a demo dataset, and runs a smoke test that ends in
+`smoke test : PASS` / `READY`. In Claude Code you can also invoke it as
+`/sandbox-setup`.
+
+### Manual (equivalent steps)
+
+```bash
+# From the repo root:
+python3 -m venv .venv                          # 1. virtualenv
+.venv/bin/pip install -e .                      # 2. this transpiler (editable)
+.venv/bin/pip install 'opencode-pyneruntime[cli]>=6.9.1'  # 3. PyneCore runtime + `pyne` CLI
+.venv/bin/pip install optuna numpy              # 4. optimizer deps (not pulled in by the runtime)
+source .venv/bin/activate                       # 5. activate (Linux/macOS)
+# .venv\Scripts\activate                        #    (Windows)
+```
+
+Verify the install:
+
+```bash
+.venv/bin/python -c "import pynecore, pine2pyne, optuna, numpy; print('all ok')"
+```
+
+**Note**: All commands below assume you are at the repo root with the venv activated.
+
+## How to Convert Pine Script to Python
+
+### Single File Conversion
+
+```bash
+python -m pine2pyne path/to/script.pine -o path/to/output.py
+```
+
+### Batch Conversion
+
+```bash
+python -m pine2pyne "sample/pinescript/*.pine" -o workdir/scripts/
+```
+
+## Running Tests
+
+There are **three** independent test systems. They answer different questions,
+so a green run of one says nothing about the others.
+
+| System | Question it answers | Runtime |
+|---|---|---|
+| **1. Unit tests** (`tests/`, pytest) | Are the transpiler's internals correct? | ~5 s |
+| **2. Sample suites** (`tools/test_all_*.py`) | Does every corpus sample transpile and run? | ~35 s |
+| **3. Ground truth** (`tools/*_ground_truth*`) | Does it produce the *same numbers* as before? | minutes |
+
+### 1. Unit tests — pytest
+
+2,000+ tests over `pine2pyne/` (93% line / 86% branch coverage).
+
+```bash
+.venv/bin/python -m pytest tests/ --ignore=tests/test_screener_regression.py -q
+
+# with coverage
+.venv/bin/python -m pytest tests/ --ignore=tests/test_screener_regression.py \
+    --cov=pine2pyne --cov-report=term-missing
+```
+
+> `--ignore` is required: `tests/test_screener_regression.py` imports a
+> `screener` module from a **sibling repo** that is absent here, and fails at
+> collection — which aborts the whole run. Tracked as a P0 card on
+> [board #10](https://github.com/users/rubycell/projects/10).
+
+Some tests are `@pytest.mark.xfail(strict=True)` and pin **confirmed unfixed
+bugs**, each with a card on [board #10](https://github.com/users/rubycell/projects/10).
+Strict means fixing a bug turns its marker into a failure until the marker is
+removed, so nothing regresses silently in either direction.
+
+The former P0 — codegen dropping Pine grouping, so `10 - (5 - 3)` became
+`10 - 5 - 3` — is fixed (card #6) and guarded by
+`tests/test_codegen_operator_grouping.py`. See [docs/test_plan.md](docs/test_plan.md) for the full bug inventory and the
+measured coverage/mutation results.
+
+### 2. Sample suites — end-to-end transpile + run
+
+Transpiles every `.pine` in the corpus and runs it through `pyne`.
+
+```bash
+python tools/test_all_samples.py          # sample/pinescript  (~411 files)
+python tools/test_all_sample_pds.py       # sample_pds         (~328 files)
+
+python tools/test_all_samples.py "ex_001*"      # filter by pattern
+python tools/test_all_samples.py --timeout 30   # default 15s
+python tools/test_all_samples.py --verbose      # stderr on failure
+python tools/test_all_samples.py -j 0           # all CPU cores
+python tools/test_all_samples.py --run-dir /tmp/run1   # private scripts/, output/ and report files (card #70)
+```
+
+- Source: `sample/pinescript/`, `sample_pds/` · Output: `workdir/scripts/`
+- `--run-dir DIR` keeps a run's transpiled scripts (`DIR/scripts/`), pyne outputs (`DIR/output/`, passed to `pyne`
+  as explicit `--plot/--strat/--trade` paths) and `test_results*.{json,txt}` reports in `DIR`, so two runs, or a run
+  and an old report, cannot read each other's files. Without it nothing changes: the shared `workdir/` paths and
+  the repo-root reports. `--retry-failed` with `--run-dir` reads `DIR`'s own report.
+- Data: `workdir/data/demo.ohlcv` · Results: `test_results{,_pds}.{json,txt}`
+
+**Per-sample expectations** live in `tools/sample_expectations.toml`:
+
+```toml
+[expds_156_Out_of_bounds_index]
+expect_error = "index_error"                    # MUST raise; running clean = FAIL
+
+[ex_300_ticker_modify]
+requires_feeds = { D = "demo", "2D" = "demo_2D" }   # -> --security D=demo …
+```
+
+`expect_error` marks Pine documentation samples that exist to *demonstrate* an
+error — they are reported as `XFAIL` and count as passes. `requires_feeds`
+supplies extra OHLCV feeds for samples calling `request.security()` at a
+timeframe other than the chart's; pynecore deliberately will not resample the
+chart feed, so the feed must be explicit. Generate one with:
+
+```bash
+cd workdir && pyne data aggregate data/demo.ohlcv -tf 2D
+```
+
+Shared logic for both suites lives in `tools/_suite_common.py` — edit it there,
+not in each harness.
+
+### 3. Ground truth — numeric regression
+
+Checks every freshly generated output file (~1,650 across ~550 samples) against
+its recorded digest. This is the only system that catches *silently wrong
+numbers*: output that runs fine but computes something different.
+
+```bash
+python tools/generate_ground_truth.py     # (re)build baselines
+python tools/compare_ground_truth.py      # check current output vs baselines
+python tools/compare_ground_truth.py --suite sample_pds -j 12
+```
+
+**The baseline is `ground_truth/digests.json`** (card #62). It holds a sha256 and a
+line count for each output file, plus the sha256 of the data file. Git tracks
+only it and `ground_truth/manifest.json` (under 1 MB together). The output
+CSVs themselves (~920 MB) are gitignored: the generator still copies them to
+`ground_truth/{sample,sample_pds}/` on your machine for diffing. Before writing a
+new baseline, it moves the previous CSVs to `backup/deleteable/ground_truth_csv_<time>/`
+(nothing is deleted; clear old ones out by hand). `manifest.json` lists its samples sorted by
+suite, then filename (card #68), so a one-suite rebuild (`--suite sample`) and a full rebuild on the same data
+give the same samples in the same order, and a rebuild never reorders the tracked file; only `generated_at`
+changes by design.
+
+- **Exit codes.** The compare exits 1 when `digests.json` is missing or empty, and
+  2 with `DATA CHANGED` when the data file is not the one the baseline was made
+  on (restore it, see *Test data* below, or rebaseline).
+- **Every sample runs in its own directory** (card #70): the generator and the compare make one fresh run
+  root, `backup/deleteable/ground_truth_runs/<tool>-<time>/` (or an empty `--run-dir DIR`; a directory that already
+  holds files is refused), with `<suite>/<sample>/<data file>/` inside it holding the script, its 10B settings and the three
+  CSVs, and collect only from there. An earlier run's file of the same name cannot be read as this run's; a run
+  that exits 0 and writes nothing is `no_output` (a failure), never `ok` or `NEW PASS`. Run roots are never
+  deleted (see the paragraph below on where they go).
+- **Where run roots go, and the disk they use.** The default run root of the generator, the compare and the
+  checker (`ground_truth_check.py capture` and `compare` each make one) is
+  `backup/deleteable/ground_truth_runs/<tool>-<time>/`, next to the other disposable output, and each run leaves
+  one tree of roughly 0.9 GB there (script, settings and three CSVs per sample). A generate run also copies the
+  collected files into `ground_truth/<suite>/` (gitignored). Nothing here is deleted by a tool: when you want the
+  space back, move old run roots (and the `gt_compare_<time>/` scratch copies) to the USB backup disk, never `rm`
+  them. An explicit `--run-dir DIR` is used as given (it must be empty).
+- **A `REGRESSION` line** names the sample and file whose digest changed. To see
+  *what* changed, diff that sample's CSV in the run root's `<suite>/<sample>/<data file>/`
+  against a baseline copy. That copy is in `ground_truth/<suite>/<sample>/` if you
+  generated on the old commit, or, after a later generate, in the newest
+  `backup/deleteable/ground_truth_csv_<time>/`.
+- **`--suite sample` or `--suite sample_pds`** rebuilds one suite and keeps the
+  other suite's digests and manifest entries. It refuses (exit 1) unless a
+  baseline made on the same data already exists.
+- **Expected noise:** 11 `sample_pds` samples call unseeded `math.random()` and
+  differ on every run (expds_022, 037, 039, 043, 045, 081, 091, 092, 104, 114,
+  115).
+
+The old end-to-end shell wrapper is retired (card #66): it read `ground_truth/samples/`,
+`ground_truth/samples_pds/` and `ground_truth/strategies/`, which no tool writes, so its
+`--quick` mode passed with zero checks. Its default and `--samples` modes are now
+`python tools/compare_ground_truth.py` (`--suite sample`, `sample_pds` or `both`; it reads
+`workdir/data/VN30F1M_15m.ohlcv` by default, see *Test data*); its `--quick` and
+`--strategies` modes are `python tools/ground_truth_check.py capture`, then `compare`.
+
+`tools/ground_truth_check.py` (`capture` / `compare`) is a separate, local check of the
+bigtest strategies and the sample suites. It writes only to `ground_truth_check/` at the
+repo root (gitignored), never to `ground_truth/`, so it cannot change the tracked baseline
+above. `capture --force` moves the old `ground_truth_check/` to
+`backup/deleteable/ground_truth_check_<time>/` (nothing is deleted) and touches nothing else. Both actions run the
+bigtest strategies and the two suite harnesses (`--run-dir`) in one fresh run root under
+`backup/deleteable/ground_truth_runs/` (`--run-dir DIR` names an empty one), read only that run's files, and fail the capture when a suite wrote no report
+or a bigtest file did not transpile. `compare` writes its
+scratch copy of the current outputs to `backup/deleteable/gt_compare_<time>/` and never deletes it (card #67); each
+run leaves one such folder, which you move to the backup disk when you want the space back. A capture made
+before card #63 sits in `ground_truth/` (`transpiled/`, `strategy_runs/`, `test_suite_*`);
+`compare` no longer reads it and says so. Run `capture` again.
+
+> Regenerate baselines only when you have *confirmed* the new output is correct
+> — otherwise a bug gets frozen in as the expected result.
+
+Both tools run every sample on `workdir/data/VN30F1M_15m.ohlcv`, with a
+`<script>.toml` that sets `initial_capital` to 10,000,000,000, and with one
+per-sample timeout (120 s). Both values live in `tools/_ground_truth_common.py`.
+At pynecore's default capital of 1,000,000, one VN30F1M contract costs more than
+equity, so nearly every strategy would record zero trades (card #57).
+
+### Test data
+
+`workdir/data/` is gitignored, so a fresh clone has none and the sample suites
+abort with `Data file not found`. Restore the exact dataset the recorded results
+were measured against:
+
+```bash
+mkdir -p workdir/data
+# Sample suites: CCXT BTCUSDT 1D, plus the 2D/5D feeds `requires_feeds` names
+git show 227f3d2^:workdir/data/demo.ohlcv > workdir/data/demo.ohlcv
+cp tests/fixtures/data/demo.toml workdir/data/demo.toml
+(cd workdir && pyne data aggregate data/demo.ohlcv -tf 2D && pyne data aggregate data/demo.ohlcv -tf 5D)
+# Ground truth: VN30F1M 15m
+git show 5d63d21:workdir/data/VN30F1M_15m.ohlcv > workdir/data/VN30F1M_15m.ohlcv
+cp tests/fixtures/data/VN30F1M_15m.toml workdir/data/VN30F1M_15m.toml
+```
+
+The `.ohlcv` files come from those commits (sha256 `67e827cc1a5f71ed…` for
+demo, `e9ef984b09eaa6d2…` for VN30F1M). Take the `.toml` files from
+`tests/fixtures/data/`, not from the commits. The committed ones number weekdays
+1..7, and pynecore rejects day 7 (it numbers 0 = Monday … 6 = Sunday). They also
+declare one session per week, which leaves session-anchored values such as
+`ta.vwap` NaN or weekly (card #57).
+
+(`/sandbox-setup` instead seeds a small *synthetic* fixture — fine for a smoke
+test, but results will not match the recorded rates.)
+
+## For Claude Code / LLM Development
+
+See `CLAUDE.md` in this directory for transpiler architecture, output format specs, optimizer documentation, and CSV rounding rules. See `pine2pyne/README.md` for transpiler internals and transformation rules.
+
+## Optimizing Strategies
+
+Quick start: `pyne optimize script.py data.ohlcv params.json -n 20`
+
+See [Optimizer.md](./Optimizer.md) for full documentation (parameter JSON format, parallel execution, output files).
+
+### Distributed Optimization (multi-machine)
+
+Two tools distribute `pyne optimize` across SSH clusters:
+
+| Tool | Model | Best for |
+|------|-------|----------|
+| `pyne-dynamic.sh` | Flat-queue (on-demand dispatch) | Multi-variant runs, heterogeneous clusters, long jobs |
+| `pyne-parallel.sh` | Static pre-assignment | Quick jobs on similar-speed machines |
+
+```bash
+# Dynamic (recommended): flat-queue, pre-syncs once, auto-adapts to machine speed
+# Run from workdir/ directory:
+../tools/pyne-dynamic.sh scripts/strategy.py data/data.ohlcv optimize_variants/ \
+  -H ../tools/machines.txt -C 24 --name my_run --output-dir runs/output/
+
+# Static: pre-assigns chunks by core count
+./tools/pyne-parallel.sh scripts/strategy.py data/data.ohlcv optimize.json \
+  -H tools/machines.txt --sync
+
+# Check progress / collect results
+../tools/pyne-dynamic.sh --status
+../tools/pyne-dynamic.sh --collect
+```
+
+Both use the same `machines.txt` format. See `CLAUDE.md` for cluster setup, machine file format, and troubleshooting.

opencode_pine2pyne-0.2.0/README.md

@@ -0,0 +1,281 @@
+# PyneCore - Pine Script to Python Transpiler
+
+> **Backlog & known bugs:** [Projects board #10](https://github.com/users/rubycell/projects/10)
+> — open work is in **Backlog** (prioritised P0/P1/P2), shipped work in **Done**.
+> Confirmed unfixed transpiler bugs live there too; several are pinned by strict
+> `xfail` tests (see [Running Tests](#running-tests)).
+
+## Installation (New Computer)
+
+### Fastest: the sandbox-setup skill
+
+From the repo root, run the bundled setup script (idempotent — safe to re-run):
+
+```bash
+bash ~/.claude/skills/opencode-pine2pyne-setup/setup.sh   # from inside this clone; the skill lives in ~/.claude/skills
+```
+
+It builds `.venv`, installs everything below, detects `PINE_CPU_COUNT` /
+`PINE_MAX_WORKERS`, seeds a demo dataset, and runs a smoke test that ends in
+`smoke test : PASS` / `READY`. In Claude Code you can also invoke it as
+`/sandbox-setup`.
+
+### Manual (equivalent steps)
+
+```bash
+# From the repo root:
+python3 -m venv .venv                          # 1. virtualenv
+.venv/bin/pip install -e .                      # 2. this transpiler (editable)
+.venv/bin/pip install 'opencode-pyneruntime[cli]>=6.9.1'  # 3. PyneCore runtime + `pyne` CLI
+.venv/bin/pip install optuna numpy              # 4. optimizer deps (not pulled in by the runtime)
+source .venv/bin/activate                       # 5. activate (Linux/macOS)
+# .venv\Scripts\activate                        #    (Windows)
+```
+
+Verify the install:
+
+```bash
+.venv/bin/python -c "import pynecore, pine2pyne, optuna, numpy; print('all ok')"
+```
+
+**Note**: All commands below assume you are at the repo root with the venv activated.
+
+## How to Convert Pine Script to Python
+
+### Single File Conversion
+
+```bash
+python -m pine2pyne path/to/script.pine -o path/to/output.py
+```
+
+### Batch Conversion
+
+```bash
+python -m pine2pyne "sample/pinescript/*.pine" -o workdir/scripts/
+```
+
+## Running Tests
+
+There are **three** independent test systems. They answer different questions,
+so a green run of one says nothing about the others.
+
+| System | Question it answers | Runtime |
+|---|---|---|
+| **1. Unit tests** (`tests/`, pytest) | Are the transpiler's internals correct? | ~5 s |
+| **2. Sample suites** (`tools/test_all_*.py`) | Does every corpus sample transpile and run? | ~35 s |
+| **3. Ground truth** (`tools/*_ground_truth*`) | Does it produce the *same numbers* as before? | minutes |
+
+### 1. Unit tests — pytest
+
+2,000+ tests over `pine2pyne/` (93% line / 86% branch coverage).
+
+```bash
+.venv/bin/python -m pytest tests/ --ignore=tests/test_screener_regression.py -q
+
+# with coverage
+.venv/bin/python -m pytest tests/ --ignore=tests/test_screener_regression.py \
+    --cov=pine2pyne --cov-report=term-missing
+```
+
+> `--ignore` is required: `tests/test_screener_regression.py` imports a
+> `screener` module from a **sibling repo** that is absent here, and fails at
+> collection — which aborts the whole run. Tracked as a P0 card on
+> [board #10](https://github.com/users/rubycell/projects/10).
+
+Some tests are `@pytest.mark.xfail(strict=True)` and pin **confirmed unfixed
+bugs**, each with a card on [board #10](https://github.com/users/rubycell/projects/10).
+Strict means fixing a bug turns its marker into a failure until the marker is
+removed, so nothing regresses silently in either direction.
+
+The former P0 — codegen dropping Pine grouping, so `10 - (5 - 3)` became
+`10 - 5 - 3` — is fixed (card #6) and guarded by
+`tests/test_codegen_operator_grouping.py`. See [docs/test_plan.md](docs/test_plan.md) for the full bug inventory and the
+measured coverage/mutation results.
+
+### 2. Sample suites — end-to-end transpile + run
+
+Transpiles every `.pine` in the corpus and runs it through `pyne`.
+
+```bash
+python tools/test_all_samples.py          # sample/pinescript  (~411 files)
+python tools/test_all_sample_pds.py       # sample_pds         (~328 files)
+
+python tools/test_all_samples.py "ex_001*"      # filter by pattern
+python tools/test_all_samples.py --timeout 30   # default 15s
+python tools/test_all_samples.py --verbose      # stderr on failure
+python tools/test_all_samples.py -j 0           # all CPU cores
+python tools/test_all_samples.py --run-dir /tmp/run1   # private scripts/, output/ and report files (card #70)
+```
+
+- Source: `sample/pinescript/`, `sample_pds/` · Output: `workdir/scripts/`
+- `--run-dir DIR` keeps a run's transpiled scripts (`DIR/scripts/`), pyne outputs (`DIR/output/`, passed to `pyne`
+  as explicit `--plot/--strat/--trade` paths) and `test_results*.{json,txt}` reports in `DIR`, so two runs, or a run
+  and an old report, cannot read each other's files. Without it nothing changes: the shared `workdir/` paths and
+  the repo-root reports. `--retry-failed` with `--run-dir` reads `DIR`'s own report.
+- Data: `workdir/data/demo.ohlcv` · Results: `test_results{,_pds}.{json,txt}`
+
+**Per-sample expectations** live in `tools/sample_expectations.toml`:
+
+```toml
+[expds_156_Out_of_bounds_index]
+expect_error = "index_error"                    # MUST raise; running clean = FAIL
+
+[ex_300_ticker_modify]
+requires_feeds = { D = "demo", "2D" = "demo_2D" }   # -> --security D=demo …
+```
+
+`expect_error` marks Pine documentation samples that exist to *demonstrate* an
+error — they are reported as `XFAIL` and count as passes. `requires_feeds`
+supplies extra OHLCV feeds for samples calling `request.security()` at a
+timeframe other than the chart's; pynecore deliberately will not resample the
+chart feed, so the feed must be explicit. Generate one with:
+
+```bash
+cd workdir && pyne data aggregate data/demo.ohlcv -tf 2D
+```
+
+Shared logic for both suites lives in `tools/_suite_common.py` — edit it there,
+not in each harness.
+
+### 3. Ground truth — numeric regression
+
+Checks every freshly generated output file (~1,650 across ~550 samples) against
+its recorded digest. This is the only system that catches *silently wrong
+numbers*: output that runs fine but computes something different.
+
+```bash
+python tools/generate_ground_truth.py     # (re)build baselines
+python tools/compare_ground_truth.py      # check current output vs baselines
+python tools/compare_ground_truth.py --suite sample_pds -j 12
+```
+
+**The baseline is `ground_truth/digests.json`** (card #62). It holds a sha256 and a
+line count for each output file, plus the sha256 of the data file. Git tracks
+only it and `ground_truth/manifest.json` (under 1 MB together). The output
+CSVs themselves (~920 MB) are gitignored: the generator still copies them to
+`ground_truth/{sample,sample_pds}/` on your machine for diffing. Before writing a
+new baseline, it moves the previous CSVs to `backup/deleteable/ground_truth_csv_<time>/`
+(nothing is deleted; clear old ones out by hand). `manifest.json` lists its samples sorted by
+suite, then filename (card #68), so a one-suite rebuild (`--suite sample`) and a full rebuild on the same data
+give the same samples in the same order, and a rebuild never reorders the tracked file; only `generated_at`
+changes by design.
+
+- **Exit codes.** The compare exits 1 when `digests.json` is missing or empty, and
+  2 with `DATA CHANGED` when the data file is not the one the baseline was made
+  on (restore it, see *Test data* below, or rebaseline).
+- **Every sample runs in its own directory** (card #70): the generator and the compare make one fresh run
+  root, `backup/deleteable/ground_truth_runs/<tool>-<time>/` (or an empty `--run-dir DIR`; a directory that already
+  holds files is refused), with `<suite>/<sample>/<data file>/` inside it holding the script, its 10B settings and the three
+  CSVs, and collect only from there. An earlier run's file of the same name cannot be read as this run's; a run
+  that exits 0 and writes nothing is `no_output` (a failure), never `ok` or `NEW PASS`. Run roots are never
+  deleted (see the paragraph below on where they go).
+- **Where run roots go, and the disk they use.** The default run root of the generator, the compare and the
+  checker (`ground_truth_check.py capture` and `compare` each make one) is
+  `backup/deleteable/ground_truth_runs/<tool>-<time>/`, next to the other disposable output, and each run leaves
+  one tree of roughly 0.9 GB there (script, settings and three CSVs per sample). A generate run also copies the
+  collected files into `ground_truth/<suite>/` (gitignored). Nothing here is deleted by a tool: when you want the
+  space back, move old run roots (and the `gt_compare_<time>/` scratch copies) to the USB backup disk, never `rm`
+  them. An explicit `--run-dir DIR` is used as given (it must be empty).
+- **A `REGRESSION` line** names the sample and file whose digest changed. To see
+  *what* changed, diff that sample's CSV in the run root's `<suite>/<sample>/<data file>/`
+  against a baseline copy. That copy is in `ground_truth/<suite>/<sample>/` if you
+  generated on the old commit, or, after a later generate, in the newest
+  `backup/deleteable/ground_truth_csv_<time>/`.
+- **`--suite sample` or `--suite sample_pds`** rebuilds one suite and keeps the
+  other suite's digests and manifest entries. It refuses (exit 1) unless a
+  baseline made on the same data already exists.
+- **Expected noise:** 11 `sample_pds` samples call unseeded `math.random()` and
+  differ on every run (expds_022, 037, 039, 043, 045, 081, 091, 092, 104, 114,
+  115).
+
+The old end-to-end shell wrapper is retired (card #66): it read `ground_truth/samples/`,
+`ground_truth/samples_pds/` and `ground_truth/strategies/`, which no tool writes, so its
+`--quick` mode passed with zero checks. Its default and `--samples` modes are now
+`python tools/compare_ground_truth.py` (`--suite sample`, `sample_pds` or `both`; it reads
+`workdir/data/VN30F1M_15m.ohlcv` by default, see *Test data*); its `--quick` and
+`--strategies` modes are `python tools/ground_truth_check.py capture`, then `compare`.
+
+`tools/ground_truth_check.py` (`capture` / `compare`) is a separate, local check of the
+bigtest strategies and the sample suites. It writes only to `ground_truth_check/` at the
+repo root (gitignored), never to `ground_truth/`, so it cannot change the tracked baseline
+above. `capture --force` moves the old `ground_truth_check/` to
+`backup/deleteable/ground_truth_check_<time>/` (nothing is deleted) and touches nothing else. Both actions run the
+bigtest strategies and the two suite harnesses (`--run-dir`) in one fresh run root under
+`backup/deleteable/ground_truth_runs/` (`--run-dir DIR` names an empty one), read only that run's files, and fail the capture when a suite wrote no report
+or a bigtest file did not transpile. `compare` writes its
+scratch copy of the current outputs to `backup/deleteable/gt_compare_<time>/` and never deletes it (card #67); each
+run leaves one such folder, which you move to the backup disk when you want the space back. A capture made
+before card #63 sits in `ground_truth/` (`transpiled/`, `strategy_runs/`, `test_suite_*`);
+`compare` no longer reads it and says so. Run `capture` again.
+
+> Regenerate baselines only when you have *confirmed* the new output is correct
+> — otherwise a bug gets frozen in as the expected result.
+
+Both tools run every sample on `workdir/data/VN30F1M_15m.ohlcv`, with a
+`<script>.toml` that sets `initial_capital` to 10,000,000,000, and with one
+per-sample timeout (120 s). Both values live in `tools/_ground_truth_common.py`.
+At pynecore's default capital of 1,000,000, one VN30F1M contract costs more than
+equity, so nearly every strategy would record zero trades (card #57).
+
+### Test data
+
+`workdir/data/` is gitignored, so a fresh clone has none and the sample suites
+abort with `Data file not found`. Restore the exact dataset the recorded results
+were measured against:
+
+```bash
+mkdir -p workdir/data
+# Sample suites: CCXT BTCUSDT 1D, plus the 2D/5D feeds `requires_feeds` names
+git show 227f3d2^:workdir/data/demo.ohlcv > workdir/data/demo.ohlcv
+cp tests/fixtures/data/demo.toml workdir/data/demo.toml
+(cd workdir && pyne data aggregate data/demo.ohlcv -tf 2D && pyne data aggregate data/demo.ohlcv -tf 5D)
+# Ground truth: VN30F1M 15m
+git show 5d63d21:workdir/data/VN30F1M_15m.ohlcv > workdir/data/VN30F1M_15m.ohlcv
+cp tests/fixtures/data/VN30F1M_15m.toml workdir/data/VN30F1M_15m.toml
+```
+
+The `.ohlcv` files come from those commits (sha256 `67e827cc1a5f71ed…` for
+demo, `e9ef984b09eaa6d2…` for VN30F1M). Take the `.toml` files from
+`tests/fixtures/data/`, not from the commits. The committed ones number weekdays
+1..7, and pynecore rejects day 7 (it numbers 0 = Monday … 6 = Sunday). They also
+declare one session per week, which leaves session-anchored values such as
+`ta.vwap` NaN or weekly (card #57).
+
+(`/sandbox-setup` instead seeds a small *synthetic* fixture — fine for a smoke
+test, but results will not match the recorded rates.)
+
+## For Claude Code / LLM Development
+
+See `CLAUDE.md` in this directory for transpiler architecture, output format specs, optimizer documentation, and CSV rounding rules. See `pine2pyne/README.md` for transpiler internals and transformation rules.
+
+## Optimizing Strategies
+
+Quick start: `pyne optimize script.py data.ohlcv params.json -n 20`
+
+See [Optimizer.md](./Optimizer.md) for full documentation (parameter JSON format, parallel execution, output files).
+
+### Distributed Optimization (multi-machine)
+
+Two tools distribute `pyne optimize` across SSH clusters:
+
+| Tool | Model | Best for |
+|------|-------|----------|
+| `pyne-dynamic.sh` | Flat-queue (on-demand dispatch) | Multi-variant runs, heterogeneous clusters, long jobs |
+| `pyne-parallel.sh` | Static pre-assignment | Quick jobs on similar-speed machines |
+
+```bash
+# Dynamic (recommended): flat-queue, pre-syncs once, auto-adapts to machine speed
+# Run from workdir/ directory:
+../tools/pyne-dynamic.sh scripts/strategy.py data/data.ohlcv optimize_variants/ \
+  -H ../tools/machines.txt -C 24 --name my_run --output-dir runs/output/
+
+# Static: pre-assigns chunks by core count
+./tools/pyne-parallel.sh scripts/strategy.py data/data.ohlcv optimize.json \
+  -H tools/machines.txt --sync
+
+# Check progress / collect results
+../tools/pyne-dynamic.sh --status
+../tools/pyne-dynamic.sh --collect
+```
+
+Both use the same `machines.txt` format. See `CLAUDE.md` for cluster setup, machine file format, and troubleshooting.