opencode-pine2pyne 0.2.2__py3-none-any.whl → 0.2.3__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- opencode_pine2pyne-0.2.3.dist-info/METADATA +343 -0
- {opencode_pine2pyne-0.2.2.dist-info → opencode_pine2pyne-0.2.3.dist-info}/RECORD +13 -11
- {opencode_pine2pyne-0.2.2.dist-info → opencode_pine2pyne-0.2.3.dist-info}/WHEEL +1 -1
- pine2pyne/__init__.py +5 -1
- pine2pyne/legacy_function_names.py +66 -0
- pine2pyne/lexer.py +26 -0
- pine2pyne/parser.py +39 -3
- pine2pyne/pine_builtins.py +8 -0
- pine2pyne/scope_renamer.py +8 -2
- pine2pyne/strategy_when_lowering.py +101 -0
- opencode_pine2pyne-0.2.2.dist-info/METADATA +0 -301
- {opencode_pine2pyne-0.2.2.dist-info → opencode_pine2pyne-0.2.3.dist-info}/entry_points.txt +0 -0
- {opencode_pine2pyne-0.2.2.dist-info → opencode_pine2pyne-0.2.3.dist-info}/licenses/LICENSE +0 -0
- {opencode_pine2pyne-0.2.2.dist-info → opencode_pine2pyne-0.2.3.dist-info}/top_level.txt +0 -0
|
@@ -0,0 +1,343 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: opencode-pine2pyne
|
|
3
|
+
Version: 0.2.3
|
|
4
|
+
Summary: Pine Script v6 to PyneCore Python transpiler
|
|
5
|
+
Author: rubycell
|
|
6
|
+
License-Expression: Apache-2.0
|
|
7
|
+
Project-URL: Homepage, https://github.com/rubycell/pine2pyne
|
|
8
|
+
Project-URL: Repository, https://github.com/rubycell/pine2pyne
|
|
9
|
+
Keywords: pinescript,tradingview,transpiler,pynecore,backtesting
|
|
10
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
11
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
12
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
13
|
+
Classifier: Topic :: Office/Business :: Financial :: Investment
|
|
14
|
+
Requires-Python: >=3.11
|
|
15
|
+
Description-Content-Type: text/markdown
|
|
16
|
+
License-File: LICENSE
|
|
17
|
+
Provides-Extra: runtime
|
|
18
|
+
Requires-Dist: opencode-pyneruntime[cli]>=6.9.1; extra == "runtime"
|
|
19
|
+
Dynamic: license-file
|
|
20
|
+
|
|
21
|
+
# pine2pyne - Pine Script to PyneCore transpiler
|
|
22
|
+
|
|
23
|
+
Snapshot written 2026-10-19 at origin/main `dd8b5d3` (branch `card-118-local-named-like-method`, clean apart from this README.md).
|
|
24
|
+
The old README is kept at `/home/mike/workspace/github/pine2pyne/backup/README.before-2026-10-19.md` (git-ignored).
|
|
25
|
+
Each fact below names its source file. A fact marked UNVERIFIED was not checked in this snapshot.
|
|
26
|
+
|
|
27
|
+
Backlog and known bugs: [Projects board #10](https://github.com/users/rubycell/projects/10)
|
|
28
|
+
(Backlog = open, P0/P1/P2; Done = shipped). The board page itself was not opened in this snapshot: UNVERIFIED.
|
|
29
|
+
|
|
30
|
+
## 1. What this repo is
|
|
31
|
+
|
|
32
|
+
- A transpiler. It reads a TradingView Pine Script file (`.pine`) and writes a PyneCore Python file (`.py`).
|
|
33
|
+
Source: `pine2pyne/__init__.py` (`transpile()`), `pyproject.toml` (description "Pine Script v6 to PyneCore Python transpiler").
|
|
34
|
+
- Users: the strategy developers of this workspace. They transpile a strategy, then run or optimize it with the `pyne` CLI
|
|
35
|
+
of the PyneCore runtime (repo `/home/mike/workspace/github/pynecore`).
|
|
36
|
+
- Where it runs: a command-line tool and a Python library. It has no service, no port and no host of its own.
|
|
37
|
+
Development happens on mike-PC. Full test suites run on Minh_pc (see section 7).
|
|
38
|
+
- Package: `opencode-pine2pyne` on PyPI, version `0.2.3` in `pine2pyne/__init__.py` and `CHANGELOG.md`.
|
|
39
|
+
Live PyPI state is UNVERIFIED: 0.2.3 is prepared (commit and local tag `v0.2.3`), not uploaded.
|
|
40
|
+
`CHANGELOG.md` lists #115 to #118 under 0.2.3; the tag `v0.2.2` is the release before them.
|
|
41
|
+
- The transpiler imports only the standard library (`pyproject.toml`: `dependencies = []`).
|
|
42
|
+
The generated script needs the runtime `opencode-pyneruntime` (`runtime` extra, `>=6.9.1`).
|
|
43
|
+
- License: Apache-2.0 (`pyproject.toml`, `LICENSE`).
|
|
44
|
+
|
|
45
|
+
## 2. Architecture
|
|
46
|
+
|
|
47
|
+
The pipeline is in `transpile()` in `/home/mike/workspace/github/pine2pyne/pine2pyne/__init__.py`:
|
|
48
|
+
|
|
49
|
+
```mermaid
|
|
50
|
+
flowchart LR
|
|
51
|
+
A[.pine source] --> B[parse_source<br/>lexer + parser + parse_repair]
|
|
52
|
+
B --> C[qualify_legacy_function_names<br/>v1-v4 names to v5 names]
|
|
53
|
+
C --> D[enforce_pine_semantics<br/>refuse what TradingView refuses]
|
|
54
|
+
D --> E[Transformer<br/>Pine AST to PyneCore AST]
|
|
55
|
+
E --> F[CodeGenerator]
|
|
56
|
+
F --> G[.py script]
|
|
57
|
+
G --> H[pyne run / pyne optimize<br/>PyneCore runtime]
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
- Lowering passes run in the parser and in the transformer. They rewrite a Pine construct into plain
|
|
61
|
+
statements. Examples: `once`, `when=` of a strategy call (v1-v5), grouped history `(a + b)[1]`,
|
|
62
|
+
block values of `if`/`switch`, strict `and`/`or` for Pine v5 and earlier.
|
|
63
|
+
Sources: imports in `pine2pyne/parser.py` and `pine2pyne/transformer.py`.
|
|
64
|
+
- Two tables are generated from an installed runtime: `pine2pyne/namespace_members.py` and `pine2pyne/bool_builtins.py`
|
|
65
|
+
(both say "GENERATED ... from opencode-pyneruntime 6.9.4; do not edit by hand").
|
|
66
|
+
Regenerate them after a runtime upgrade with `tools/collect_namespace_members.py` and `tools/collect_bool_builtins.py`.
|
|
67
|
+
- The transpiler refuses a script when TradingView refuses it (`SemanticError`, card #28) and warns with
|
|
68
|
+
`SemanticWarning` when it only advises. `pyproject.toml` turns an unexpected `SemanticWarning` into a test failure.
|
|
69
|
+
- Runtime code placement rule: group a new runtime member beside its kin. Source: `CLAUDE.md`, "Runtime Code Placement".
|
|
70
|
+
- Three separate test systems check it (section 7). A green run of one says nothing about the others.
|
|
71
|
+
Source: `CLAUDE.md`.
|
|
72
|
+
|
|
73
|
+
## 3. Modules
|
|
74
|
+
|
|
75
|
+
All in `/home/mike/workspace/github/pine2pyne/pine2pyne/` (33 `.py` files, 13,176 lines in total by `wc -l`).
|
|
76
|
+
|
|
77
|
+
Core pipeline:
|
|
78
|
+
- `lexer.py`, `tokens.py` - tokens, with indentation tracking.
|
|
79
|
+
- `parser.py`, `ast_nodes.py`, `ast_walk.py` - recursive descent parser, AST node classes, shared child walk.
|
|
80
|
+
- `parse_repair.py` - repairs a rejected known spelling of valid Pine (card #55).
|
|
81
|
+
- `legacy_function_names.py` - Pine v1-v4 call names to v5 names (card #116).
|
|
82
|
+
- `pine_semantics.py` - checks TradingView's compiler makes that the grammar does not (card #28).
|
|
83
|
+
- `transformer.py` - Pine AST to PyneCore AST. The largest file.
|
|
84
|
+
- `codegen.py` - emits Python source.
|
|
85
|
+
- `cli.py`, `__main__.py` - command line (`pine2pyne` command and `python -m pine2pyne`).
|
|
86
|
+
- `errors.py` - error types with source locations.
|
|
87
|
+
|
|
88
|
+
Support tables and typing:
|
|
89
|
+
- `pine_builtins.py` - maps Pine built-ins to PyneCore.
|
|
90
|
+
- `namespace_members.py`, `bool_builtins.py` - generated runtime tables.
|
|
91
|
+
- `symbol_table.py`, `type_inference.py`, `bool_value_typing.py` - scopes, Series or Persistent typing, bool typing.
|
|
92
|
+
- `import_resolver.py`, `emitted_names.py` - import lines; emitted names that must not collide.
|
|
93
|
+
|
|
94
|
+
Lowering and refusal passes:
|
|
95
|
+
- `scope_renamer.py` - gives a shadowing local, or a binding named like a runtime name, a name of its own.
|
|
96
|
+
- `library_collision.py` - a user binding named like a `pynecore.lib` name.
|
|
97
|
+
- `overload_resolution.py` - same-name functions and methods.
|
|
98
|
+
- `once_lowering.py`, `strategy_when_lowering.py`, `block_value_lowering.py`, `grouped_history_lowering.py`.
|
|
99
|
+
- `for_loop_end_bound.py`, `strict_boolean.py`, `udt_field_defaults.py`, `conditional_history_calls.py`.
|
|
100
|
+
|
|
101
|
+
Other top-level folders:
|
|
102
|
+
- `tests/` - 86 `test_*.py` files, fixtures in `tests/fixtures/`, old strategy outputs in `tests/ground_truth/`.
|
|
103
|
+
- `tools/` - corpus harnesses, ground-truth tools, table generators (section 7).
|
|
104
|
+
- `sample/pinescript/` (413 `.pine` files) and `sample_pds/` (331 `.pine` files) - the corpus. Counts by `ls | wc -l`.
|
|
105
|
+
- `sandbox/` - Docker image (section 6).
|
|
106
|
+
- `docs/` - `test_plan.md`, `instructions/` (optimizer guides), `misc/`, `code_review/`.
|
|
107
|
+
- `ground_truth/` - tracked baseline: `digests.json` and `manifest.json` only.
|
|
108
|
+
- `other_transpiler/`, `runs/`, `scripts/`, `strategy/`, `upstream-run/`, `agents/`, `.claude/agentandskill/` - strategy
|
|
109
|
+
experiments and agent prompts. Their use today is UNVERIFIED.
|
|
110
|
+
- `workdir/` - git-ignored scratch folder for runs (`.gitignore`). `backup/` and `dist/` are git-ignored too.
|
|
111
|
+
- `pine2pyne/README.md` is the package's own README (set as package data in `pyproject.toml`).
|
|
112
|
+
It may be stale (it says "transformer.py (1626 lines)"): UNVERIFIED.
|
|
113
|
+
|
|
114
|
+
## 4. Services
|
|
115
|
+
|
|
116
|
+
None. The repo starts no long-running process: no systemd unit, no cron job, no listening port.
|
|
117
|
+
Source: no unit, crontab installer or server entry point found in the repo; `pyproject.toml` has one console script,
|
|
118
|
+
`pine2pyne = "pine2pyne.cli:main"`. Distributed optimizer runs start remote `pyne optimize` processes over SSH
|
|
119
|
+
from tools that live in another repo (section 7).
|
|
120
|
+
|
|
121
|
+
## 5. Dashboard and its features
|
|
122
|
+
|
|
123
|
+
None. This repo has no web UI. Source: no web framework or HTML page in the repo.
|
|
124
|
+
|
|
125
|
+
## 6. Docker
|
|
126
|
+
|
|
127
|
+
- One Dockerfile: `/home/mike/workspace/github/pine2pyne/sandbox/Dockerfile`. Base image `python:3.13-slim`.
|
|
128
|
+
Build arg `SOURCE=pypi` (default) installs `opencode-pine2pyne` and `opencode-pyneruntime[cli]>=6.9.1` from PyPI.
|
|
129
|
+
`SOURCE=local` installs the wheels in `sandbox/wheels/`.
|
|
130
|
+
The user is `pyne`, the work folder is `/work`, and the default command prints `pine2pyne --version`, then opens a shell.
|
|
131
|
+
- Suggested image name in `sandbox/README.md`: `pyne-sandbox`. Run with `-v "$PWD/workdir:/work"`.
|
|
132
|
+
`sandbox/smoke.sh <data.ohlcv>` transpiles a small strategy and runs `pyne run` on it.
|
|
133
|
+
- `sandbox/wheels/` is stale: it holds `opencode_pine2pyne-0.1.0` and `opencode_pyneruntime-6.6.4` (file names, dated 2026-07-31).
|
|
134
|
+
`git ls-files sandbox/wheels` lists nothing: the wheels are not tracked.
|
|
135
|
+
- No compose file, no ports, no volumes other than the `/work` mount in `sandbox/README.md`.
|
|
136
|
+
- Live state, read only, 2026-10-19. `ssh -o BatchMode=yes minh@minhpc 'docker ps -a; docker images'` and
|
|
137
|
+
`docker ps -a; docker images` on mike-PC show no `pyne-sandbox` image and no pine2pyne container.
|
|
138
|
+
The containers seen there belong to other projects (`vnstock-openclaw-trial-*`, `botrehearsal-trial-*`, `svb-sandbox:*` images).
|
|
139
|
+
Result: the sandbox image is not built on either host.
|
|
140
|
+
|
|
141
|
+
## 7. Commands
|
|
142
|
+
|
|
143
|
+
Run all commands from the repo root with the venv active. Paths and options come from `tools/*.py` and `pine2pyne/cli.py`.
|
|
144
|
+
|
|
145
|
+
Set up:
|
|
146
|
+
```bash
|
|
147
|
+
bash /home/mike/.claude/skills/opencode-pine2pyne-setup/setup.sh # skill script: builds .venv, seeds demo data, smoke test
|
|
148
|
+
# or by hand:
|
|
149
|
+
python3 -m venv .venv
|
|
150
|
+
.venv/bin/pip install -e .
|
|
151
|
+
.venv/bin/pip install 'opencode-pyneruntime[cli]>=6.9.1' # same floor as pyproject.toml [runtime]
|
|
152
|
+
.venv/bin/pip install optuna numpy # optimizer deps, not pulled in by the runtime
|
|
153
|
+
source .venv/bin/activate # Linux and macOS; on Windows: .venv\Scripts\activate
|
|
154
|
+
.venv/bin/python -c "import pynecore, pine2pyne, optuna, numpy; print('all ok')"
|
|
155
|
+
```
|
|
156
|
+
The setup script ends with the lines `smoke test : PASS` and `READY`. A failure prints `READY : NO` and exits 1
|
|
157
|
+
(`setup.sh` lines 22, 31 and 111-112). It writes `.sandbox-env.sh` with `PINE_CPU_COUNT` and `PINE_MAX_WORKERS`
|
|
158
|
+
(source: `/home/mike/.claude/skills/opencode-pine2pyne-setup/setup.sh`; the file is git-ignored).
|
|
159
|
+
The `runtime` extra also works: `pip install "opencode-pine2pyne[runtime]"` (`pyproject.toml`).
|
|
160
|
+
Test tools (pytest, hypothesis) are in no manifest: a venv built from this README cannot run the unit suite
|
|
161
|
+
(project memory "Test deps undeclared"; not re-checked, UNVERIFIED).
|
|
162
|
+
|
|
163
|
+
Transpile (`pine2pyne/cli.py`, options `-o`, `--validate`, `-v`, `--version`):
|
|
164
|
+
```bash
|
|
165
|
+
python -m pine2pyne input.pine # to stdout
|
|
166
|
+
python -m pine2pyne input.pine -o output.py
|
|
167
|
+
python -m pine2pyne "sample/pinescript/*.pine" -o workdir/scripts/ # batch into a folder
|
|
168
|
+
python -m pine2pyne strategy.pine --validate # parse and check only
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
Run and optimize (the `pyne` CLI of the runtime; from `workdir/`):
|
|
172
|
+
```bash
|
|
173
|
+
pyne run strategy.py data/data.ohlcv
|
|
174
|
+
pyne optimize strategy.py data/data.ohlcv optimize.json -n 20
|
|
175
|
+
```
|
|
176
|
+
Optimizer guides: `docs/instructions/Optimizer.md` (the root `Optimizer.md` was deleted in commit `cde04c9`;
|
|
177
|
+
old links to it are dead).
|
|
178
|
+
|
|
179
|
+
Test. Where each run goes: `/home/mike/.claude/where-tests-run.md`. Every full suite runs on Minh_pc through the
|
|
180
|
+
`qa-remote-suite` skill. A batch that checks this tool across many strategies is a test, so it runs on Minh_pc too.
|
|
181
|
+
On mike-PC only a targeted run is allowed, through `mike-pc-test-run`.
|
|
182
|
+
|
|
183
|
+
There are three test systems. A green run of one says nothing about the others.
|
|
184
|
+
|
|
185
|
+
1. Unit tests, pytest, `tests/` (`pyproject.toml`: `testpaths = ["tests"]`):
|
|
186
|
+
```bash
|
|
187
|
+
.venv/bin/python -m pytest tests/ -q
|
|
188
|
+
.venv/bin/python -m pytest tests/ --cov=pine2pyne --cov-report=term-missing
|
|
189
|
+
.venv/bin/python -m pytest tests/ -m 'not corpus_runtime' -q # skip tests that run the runtime
|
|
190
|
+
.venv/bin/python -m pytest tests/ -m 'not release_artifact' -q # skip the wheel build test (needs uv)
|
|
191
|
+
```
|
|
192
|
+
- No `--ignore` is needed (`CLAUDE.md`; the screener test was retired in card #106).
|
|
193
|
+
- Last measured numbers: 6,870 tests, 6,848 passed, 1 skipped, 21 xfailed, 93% line and 86% branch coverage,
|
|
194
|
+
at `a16621b` (old README). Not re-run in this snapshot: UNVERIFIED. `docs/test_plan.md` holds older numbers
|
|
195
|
+
(2062 passed, 93.3% line, 86.6% branch) and the mutation results.
|
|
196
|
+
- Strict xfail marks a pinned, unfixed bug. Do not delete the marker to get green. When the bug is fixed the
|
|
197
|
+
test XPASSes, strict turns that into a failure, and you then convert the test to a regression assertion.
|
|
198
|
+
Source: `CLAUDE.md` ("6 sites (21 test cases)" there; not re-counted, UNVERIFIED).
|
|
199
|
+
- Guard for the operator-grouping bug (card #6): `tests/test_codegen_operator_grouping.py`.
|
|
200
|
+
|
|
201
|
+
2. Sample suites, end-to-end transpile and `pyne run` over the corpus (`tools/test_all_samples.py`,
|
|
202
|
+
`tools/test_all_sample_pds.py`, shared code in `tools/_suite_common.py`):
|
|
203
|
+
```bash
|
|
204
|
+
python tools/test_all_samples.py # sample/pinescript
|
|
205
|
+
python tools/test_all_sample_pds.py # sample_pds
|
|
206
|
+
python tools/test_all_samples.py "ex_001*" # filter by pattern
|
|
207
|
+
python tools/test_all_samples.py --timeout 30 --verbose -j 0
|
|
208
|
+
python tools/test_all_samples.py --run-dir /tmp/run1 # private scripts/, output/ and reports (card #70)
|
|
209
|
+
```
|
|
210
|
+
- Defaults: data `workdir/data/demo.ohlcv`, timeout 15 s, `-j` 24 (`tools/test_all_samples.py`).
|
|
211
|
+
Reports go to `test_results{,_pds}.{json,txt}` at the repo root (git-ignored), or into `--run-dir`.
|
|
212
|
+
- Latest local reports, 2026-10-02 08:45 (`test_results.json`, `test_results_pds.json`; local files, not tracked):
|
|
213
|
+
- `sample/pinescript`: 411 run, 2 skipped, 411 transpiled, 393 passed, 18 failed
|
|
214
|
+
(11 `pynecore_api`, 5 `missing_security_data`, 2 `type_error`).
|
|
215
|
+
- `sample_pds`: 328 run, 3 skipped, 319 transpiled, 322 passed, 6 failed (all `missing_security_data`).
|
|
216
|
+
Of the 322 passes, 304 ran clean and 18 are expected-error samples (`expect_error` in
|
|
217
|
+
`tools/sample_expectations.toml`): 9 `pine_refusal` (transpile refused, so not in the 319),
|
|
218
|
+
5 `index_error` and 4 `value_error`. The report's `runtime_failed` is 15 = 6 failures + 5 + 4 expected-error passes.
|
|
219
|
+
(As of 2026-10-20 `expds_160` is no longer declared: TradingView does not halt on it, measured. The declared
|
|
220
|
+
`index_error` samples are 4 now; the report above was made before that.)
|
|
221
|
+
Counted from the `results` list of `test_results_pds.json` (passed, transpile, runtime, error_type per sample).
|
|
222
|
+
- `CLAUDE.md` holds the same numbers (table "Converting Pine Script to PyneCore Python"). Its old 2026-08-17
|
|
223
|
+
figures (391/411, 304/328) were replaced on 2026-10-19.
|
|
224
|
+
- Per-sample rules are in `tools/sample_expectations.toml` (25 sections). Keys: `expect_error` (must fail with that
|
|
225
|
+
category; running clean is a FAIL), `requires_feeds` (extra feeds, passed as `--security TF=name`), `runs_clean`,
|
|
226
|
+
`card`, `refusal`, `why`. Pytest validates the file (`tests/test_error_demo_claims.py`).
|
|
227
|
+
- A feed for `requires_feeds` is made with `cd workdir && pyne data aggregate data/demo.ohlcv -tf 2D`.
|
|
228
|
+
|
|
229
|
+
3. Ground truth, numeric regression (`tools/generate_ground_truth.py`, `tools/compare_ground_truth.py`,
|
|
230
|
+
`tools/ground_truth_check.py`, shared code in `tools/_ground_truth_common.py`):
|
|
231
|
+
```bash
|
|
232
|
+
python tools/generate_ground_truth.py # (re)build the baseline; only when the new output is confirmed correct
|
|
233
|
+
python tools/compare_ground_truth.py # compare current output with the baseline
|
|
234
|
+
python tools/compare_ground_truth.py --suite sample_pds -j 12
|
|
235
|
+
python tools/ground_truth_check.py capture # then: compare. Local check, writes only ground_truth_check/ (git-ignored)
|
|
236
|
+
```
|
|
237
|
+
- `capture --force` moves the old `ground_truth_check/` to `backup/deleteable/ground_truth_check_<time>/`;
|
|
238
|
+
it deletes nothing (`tools/ground_truth_check.py` lines 27-28, 201-206).
|
|
239
|
+
- Baseline: `ground_truth/digests.json` (sha256 and line count per output file, plus the data sha256) and
|
|
240
|
+
`ground_truth/manifest.json`. The output CSVs (about 920 MB) are git-ignored (`.gitignore`, card #62).
|
|
241
|
+
- Current baseline (`manifest.json`, generated 2026-10-20 on pynecore `ae6f03172d6127e88c6ae4f2ced61f0c93a6a331`):
|
|
242
|
+
562 tested, 551 passed, 9 transpile failures, 2 runtime failures, 0 timeouts, 1,637 output files.
|
|
243
|
+
`digests.json` has 551 samples. The 2026-10-03 baseline (550 passed, 1,634 files) matched no current runtime;
|
|
244
|
+
it was made on a pynecore side branch (`88741ade6092`, checked on `ex_001` only).
|
|
245
|
+
- `pynecore_commit` (in `digests.json` and `manifest.json`): the full sha of the pynecore checkout that the `pyne`
|
|
246
|
+
in use imports, written by `generate_ground_truth.py`. `-dirty` is added when a tracked file under that
|
|
247
|
+
checkout's `src/` differs; the value is `null` when the runtime is not a pynecore checkout (`src/pynecore` of a
|
|
248
|
+
git work tree). `compare_ground_truth.py` prints the baseline and the runtime commit and WARNS when they differ,
|
|
249
|
+
or when either is `-dirty` or unknown; it never refuses for this reason.
|
|
250
|
+
- Data: `workdir/data/VN30F1M_15m.ohlcv`, data sha256 `e9ef984b09eaa6d2...` (`digests.json`).
|
|
251
|
+
Capital 10,000,000,000 and timeout 120 s per sample (`tools/_ground_truth_common.py`).
|
|
252
|
+
Why this capital: at 1,000,000 one VN30F1M contract costs more than the equity, so nearly every strategy
|
|
253
|
+
records zero trades (card #57; `tools/_ground_truth_common.py` lines 14-18).
|
|
254
|
+
- Exit codes of compare: 0 = every recorded sample verified (skipped nondeterministic files are listed, not judged);
|
|
255
|
+
1 = digests missing or empty, a regression, a new failure, an unverified recorded sample or a stale mark;
|
|
256
|
+
2 = `DATA CHANGED` (the data file is not the one of the baseline).
|
|
257
|
+
- A `REGRESSION` line names the sample and the file (`tools/compare_ground_truth.py` line 271). To see what changed,
|
|
258
|
+
diff that sample's CSV in the run root against the baseline copy in `ground_truth/<suite>/<sample>/`.
|
|
259
|
+
- `--suite sample` or `--suite sample_pds` rebuilds one suite and keeps the other suite's digests. It refuses
|
|
260
|
+
(exit 1) unless a baseline made on the same data AND the same known `pynecore_commit` exists
|
|
261
|
+
(`tools/generate_ground_truth.py` lines 250-258): one commit must describe the whole file.
|
|
262
|
+
- Before a new baseline is written, the old CSV folder moves to `backup/deleteable/ground_truth_csv_<time>/`;
|
|
263
|
+
nothing is deleted (`tools/generate_ground_truth.py` lines 260-267).
|
|
264
|
+
- Each sample runs in its own folder under `backup/deleteable/ground_truth_runs/<tool>-<time>/` (about 0.9 GB each).
|
|
265
|
+
No tool deletes them. Move old ones to the backup disk by hand, never `rm` (card #70, old README).
|
|
266
|
+
- Nondeterministic plot files: 11 `sample_pds` samples call an unseeded `math.random()` in a plot, so the digest of
|
|
267
|
+
their plot csv `<name>.csv` changes on every run (expds_022, 037, 039, 043, 045, 081, 091, 092, 104, 114, 115).
|
|
268
|
+
`manifest.json` marks exactly those files with `nondeterministic_files` on the sample entry. The list of the 11
|
|
269
|
+
samples is one tuple, `NONDETERMINISTIC_PLOT_SAMPLES` in `tools/_ground_truth_common.py`, and
|
|
270
|
+
`generate_ground_truth.py` writes the marks (a full run, or `--mark-nondeterministic-only`, which changes
|
|
271
|
+
only the manifest and runs no sample). Do not edit the marks by hand. `compare_ground_truth.py` skips only the
|
|
272
|
+
marked files, prints each one under `SKIPPED NONDETERMINISTIC FILES (N)`, and still checks every other file of
|
|
273
|
+
those samples (`_strat.csv`, `_trade.csv`, `_plot_meta.json`). A mark that no longer holds (the file matches
|
|
274
|
+
its digest, or the baseline records no such file) is listed under `STALE NONDETERMINISTIC MARKS` and fails the
|
|
275
|
+
run: regenerate the marks. A new sample with an unseeded `math.random()` needs its key in the tuple.
|
|
276
|
+
Checked 2026-10-20 on pynecore `ae6f03172`: two compares in a row each exited 0 (551 OK, 0 REGRESSION, 11 files skipped).
|
|
277
|
+
- The old shell wrapper was retired in card #66. `tools/ground_truth_check.py capture` then `compare` replace it.
|
|
278
|
+
|
|
279
|
+
Other tools in `tools/`: `corpus_transpile_ab.py --base <git ref>` (A/B of the transpiled corpus against another revision),
|
|
280
|
+
`_run_layout.py`, `_suite_report_compare.py`, `probes/`.
|
|
281
|
+
|
|
282
|
+
Test data. `workdir/` is git-ignored, so a fresh clone has no data. Both data files were checked on 2026-10-19:
|
|
283
|
+
```bash
|
|
284
|
+
mkdir -p workdir/data
|
|
285
|
+
git show 227f3d2^:workdir/data/demo.ohlcv > workdir/data/demo.ohlcv # sha256 starts 67e827cc1a5f71ed
|
|
286
|
+
cp tests/fixtures/data/demo.toml workdir/data/demo.toml
|
|
287
|
+
(cd workdir && pyne data aggregate data/demo.ohlcv -tf 2D && pyne data aggregate data/demo.ohlcv -tf 5D)
|
|
288
|
+
git show 5d63d21:workdir/data/VN30F1M_15m.ohlcv > workdir/data/VN30F1M_15m.ohlcv # sha256 starts e9ef984b09eaa6d2
|
|
289
|
+
cp tests/fixtures/data/VN30F1M_15m.toml workdir/data/VN30F1M_15m.toml
|
|
290
|
+
```
|
|
291
|
+
Take the `.toml` files from `tests/fixtures/data/`, not from the old commits: the old ones number weekdays 1..7,
|
|
292
|
+
and pynecore rejects day 7 (0 = Monday ... 6 = Sunday). Source: old README, project memory "pyne runtime gotchas".
|
|
293
|
+
The `/opencode-pine2pyne-setup` skill seeds a small synthetic fixture instead. That is fine for a smoke test, but the
|
|
294
|
+
results do not match the recorded numbers.
|
|
295
|
+
|
|
296
|
+
Lint: no linter config found in `pyproject.toml`. UNVERIFIED for any other file.
|
|
297
|
+
|
|
298
|
+
Deploy (release to PyPI): use the skill `/home/mike/.claude/skills/pypi-publish/SKILL.md`. The repo has no CI file
|
|
299
|
+
and no Makefile. `CHANGELOG.md` is the release log; add the entry for #115 to #118 at the next release.
|
|
300
|
+
|
|
301
|
+
Distributed optimization. `CLAUDE.md` and the old README name `tools/pyne-dynamic.sh`, `tools/pyne-parallel.sh` and
|
|
302
|
+
`tools/machines.txt`. **These files are not in this repo.** They moved out in commit `227f3d2`
|
|
303
|
+
("move workdir/ and tools/ out to the pinescript workbench"). They exist at
|
|
304
|
+
`/home/mike/workspace/github/pinescript/tools/pyne-dynamic.sh`, `.../pyne-parallel.sh` and `.../machines.txt`.
|
|
305
|
+
Use those paths. That they run unchanged against this repo is UNVERIFIED. The tool options are in `CLAUDE.md`
|
|
306
|
+
("Distributed Optimization") and `docs/misc/DynamicJobQueue.md`. `CLAUDE.md` also holds the cluster setup, the
|
|
307
|
+
machine file format and the troubleshooting table (headings "Shared: Machines file format", "Adding a new node",
|
|
308
|
+
"Troubleshooting").
|
|
309
|
+
|
|
310
|
+
## 8. Data and keys
|
|
311
|
+
|
|
312
|
+
- Corpus: `sample/pinescript/` and `sample_pds/` (tracked `.pine` files).
|
|
313
|
+
- Test data: `workdir/data/` (git-ignored; restore with the commands in section 7).
|
|
314
|
+
- Baseline: `ground_truth/digests.json`, `ground_truth/manifest.json` (tracked).
|
|
315
|
+
- Run output: `workdir/scripts/`, `workdir/output/`, `test_results*.json|txt` (git-ignored).
|
|
316
|
+
- Disposable output: `backup/deleteable/` (git-ignored). Disk rule: `~/.claude/CLAUDE.md`, "Never Delete on the root disk".
|
|
317
|
+
- Key names only. `/home/mike/workspace/github/pine2pyne/.env` holds `PYPI_TOKEN`. `.gitignore` excludes it
|
|
318
|
+
(`.env`, `.env.*`, `*.pem`, `*.key`, `.pypirc`, `.netrc`). `git ls-files .env` shows it is not tracked.
|
|
319
|
+
- Environment names: `PINE_CPU_COUNT` and `PINE_MAX_WORKERS` (written by the setup skill into `.sandbox-env.sh`).
|
|
320
|
+
`PYNE_OPTIMIZE_MODE=1` is set by the optimizer in
|
|
321
|
+
`/home/mike/workspace/github/pynecore/plugins/pyne_optimize_and_stats_tools/pynecore_optimize_and_stats_tools/optimize.py`
|
|
322
|
+
(lines 312, 832 and 887).
|
|
323
|
+
- No other key or secret file was found in the repo files that were read.
|
|
324
|
+
|
|
325
|
+
## 9. Related repos and skills
|
|
326
|
+
|
|
327
|
+
Calls this repo:
|
|
328
|
+
- `/home/mike/workspace/github/pinescript` (the strategy workbench). Its `CLAUDE.md` says the transpiler and runtime
|
|
329
|
+
moved out to `pine2pyne` and `pynecore`, and lists two ways to get the toolchain (editable installs, or PyPI).
|
|
330
|
+
The workbench holds `workdir/` and the cluster scripts of section 7.
|
|
331
|
+
- The skills below call `pine2pyne`. They are listed in the session skill list; their code was not read here.
|
|
332
|
+
`opencode-pyne-run`, `opencode-pine-test-run`, `sandbox-e2e`, `opencode-strategy-optimize`.
|
|
333
|
+
|
|
334
|
+
This repo calls:
|
|
335
|
+
- `/home/mike/workspace/github/pynecore` (runtime `opencode-pyneruntime`, the `pyne` CLI). The `.venv` of this repo
|
|
336
|
+
imports it from `/home/mike/workspace/github/pynecore/src/pynecore/` (checked with `import pynecore`).
|
|
337
|
+
The runtime is a fork of PyneSys/pynecore. Never change original pynecore files (see that repo's rules).
|
|
338
|
+
- Skills it uses: `opencode-pine2pyne-setup`, `pypi-publish`, `qa-remote-suite`, `mike-pc-test-run`.
|
|
339
|
+
Skills live in `/home/mike/.claude/skills/`.
|
|
340
|
+
|
|
341
|
+
Rules for every repo are in `/home/mike/.claude/CLAUDE.md`. This README does not copy them. Repo-specific
|
|
342
|
+
instructions (runtime code placement, pinned xfail bugs, pipeline notes) are in
|
|
343
|
+
`/home/mike/workspace/github/pine2pyne/CLAUDE.md`.
|
|
@@ -1,8 +1,8 @@
|
|
|
1
|
-
opencode_pine2pyne-0.2.
|
|
1
|
+
opencode_pine2pyne-0.2.3.dist-info/licenses/LICENSE,sha256=l6ZEJ0BCsllGRX5Pg-ygs6mQD03fLXJggIyKnSMwC8E,11342
|
|
2
2
|
pine2pyne/README.md,sha256=7O2y_Jn9EQ6yNXZio7qAvEqdfVnuiJE_C6zyTdWkUgM,6021
|
|
3
3
|
pine2pyne/TRANSPILER_BEST_PRACTICES.md,sha256=I9UQYI-KlObGGODfItU0kIt-Bo6MVh9XgPfXPRvjh_4,18197
|
|
4
4
|
pine2pyne/TRANSPILER_USAGE.md,sha256=7iT-dnig8t-gMxozVYHgKoxr7W--SITLoyJk41PYlrs,5569
|
|
5
|
-
pine2pyne/__init__.py,sha256=
|
|
5
|
+
pine2pyne/__init__.py,sha256=BV4O9H7xipptM5ZmeW0wBotZosKe50QrdXUaILQNuj8,2436
|
|
6
6
|
pine2pyne/__main__.py,sha256=w6UY1yZijkjI0YRgq03guOunEYTp9S4DjwsAips2GOk,228
|
|
7
7
|
pine2pyne/ast_nodes.py,sha256=xk0Zrjdip1sZWE6WINKFK4AK3ikxcOnZkirC9RwDl0E,11354
|
|
8
8
|
pine2pyne/ast_walk.py,sha256=Qt0HUyF7nR3G9kErTB22TuarUJ0g4xQV8QBQ3YeuuIg,4037
|
|
@@ -17,24 +17,26 @@ pine2pyne/errors.py,sha256=6T2OoC0u2inr1XWMIeC3glI6aplL8dcgET7maA4jDjY,2622
|
|
|
17
17
|
pine2pyne/for_loop_end_bound.py,sha256=y3lTjEvL2jZOVXMZ7zHG8X683mz9IQeySouhNw8uAcI,6350
|
|
18
18
|
pine2pyne/grouped_history_lowering.py,sha256=hxoQcJy0HyvzA7jXQLkcYeJUJ0onDJgKs3Jm5auf5FM,16963
|
|
19
19
|
pine2pyne/import_resolver.py,sha256=mWperlEZomLZ44GTFx6Cez9Wz-AexCPioboAwOJfGXE,22272
|
|
20
|
-
pine2pyne/
|
|
20
|
+
pine2pyne/legacy_function_names.py,sha256=zdIF4fa40A-fkFiwECa3GeGmMT7lZjvVHACs1FbjQ6g,3499
|
|
21
|
+
pine2pyne/lexer.py,sha256=wT-F7dptRTqaWXVduVP8xHVjZH8KfUH9pplZvR9RdE0,22810
|
|
21
22
|
pine2pyne/library_collision.py,sha256=0c-t-lgNlAmmhhNO286eCp99rW0FhC9wafcGkIWkXI8,22323
|
|
22
23
|
pine2pyne/namespace_members.py,sha256=CO-oAesSy6LWthD2qItUrL3cTm8ZZ2Yulg3q_mD69BA,12859
|
|
23
24
|
pine2pyne/once_lowering.py,sha256=frr0g9nX612MKsPiLyeFEDvyoRcQDQTsBiUZ-qMUiq8,3481
|
|
24
25
|
pine2pyne/overload_resolution.py,sha256=gPELykcmixn0LQT--nEaS85Uwn8FmrOa_fkuytoC8ag,24904
|
|
25
26
|
pine2pyne/parse_repair.py,sha256=Uec959EDCvnqeHdVD1ZpSizH150rUAk55BnYF9EbNBM,18882
|
|
26
|
-
pine2pyne/parser.py,sha256=
|
|
27
|
-
pine2pyne/pine_builtins.py,sha256=
|
|
27
|
+
pine2pyne/parser.py,sha256=c-JQu6LEls_EacpeSGfOHZJSJo_k25wJ9CnojZI3CqE,77634
|
|
28
|
+
pine2pyne/pine_builtins.py,sha256=XRds4zd7t2Twdgi7dK06iEvpw73glo863CaSz0wKG5s,16733
|
|
28
29
|
pine2pyne/pine_semantics.py,sha256=e0CSoUCFp6BYzb2XNIi5_cQVgHfm--lMTKGvaTuraJw,34765
|
|
29
|
-
pine2pyne/scope_renamer.py,sha256=
|
|
30
|
+
pine2pyne/scope_renamer.py,sha256=uM935RXVdQb6QRHd3-mTZN02343EnZE8pFibqr30H5c,44446
|
|
31
|
+
pine2pyne/strategy_when_lowering.py,sha256=bN5rAEKGy6bkp3Rm093GnAfpwtswExYsBbBGOdMeMbo,5334
|
|
30
32
|
pine2pyne/strict_boolean.py,sha256=7Z-q2tdVk0rFgu4SecX9GVaWgBof_IFtBKjw1tBolB0,5471
|
|
31
33
|
pine2pyne/symbol_table.py,sha256=SO40do-faSu-n29-5GOrkKAqEK6Ef9BGtw4sC0w9Jbo,6146
|
|
32
34
|
pine2pyne/tokens.py,sha256=diG6z5_9DsDCdSN-sapdDAbdIwvklQPuC-05tkhQzDk,3792
|
|
33
35
|
pine2pyne/transformer.py,sha256=mwgIMx1GRapzPsZfDWNY16NTxDPVsYVI3Y4svWlud3w,115673
|
|
34
36
|
pine2pyne/type_inference.py,sha256=UFg7y9Q5Vkd_NnjQ1DD9LrqJ3ywpHSMHv28D_lQlb08,19895
|
|
35
37
|
pine2pyne/udt_field_defaults.py,sha256=DpOwtiCxKdQjAK5senw2wAzT7JNfgRl3NxCrlLTfk3c,7746
|
|
36
|
-
opencode_pine2pyne-0.2.
|
|
37
|
-
opencode_pine2pyne-0.2.
|
|
38
|
-
opencode_pine2pyne-0.2.
|
|
39
|
-
opencode_pine2pyne-0.2.
|
|
40
|
-
opencode_pine2pyne-0.2.
|
|
38
|
+
opencode_pine2pyne-0.2.3.dist-info/METADATA,sha256=PFEJPwfv86uUgLoSrrUQ2Iw6hB87d1b0haOCVzZqNoI,24337
|
|
39
|
+
opencode_pine2pyne-0.2.3.dist-info/WHEEL,sha256=a-udL2m4V_6Yn8b-uKzWb12RjRxwwMo6ohYFM24tpaY,91
|
|
40
|
+
opencode_pine2pyne-0.2.3.dist-info/entry_points.txt,sha256=OQmVtMqXUdYsmXqUEe09w30h54Gmn3JsPjGgVnTO4sA,49
|
|
41
|
+
opencode_pine2pyne-0.2.3.dist-info/top_level.txt,sha256=ge_qvuxsfM_9zOv_hIwyLCMe1S8RAr5nUR6n3aBO6J0,10
|
|
42
|
+
opencode_pine2pyne-0.2.3.dist-info/RECORD,,
|
pine2pyne/__init__.py
CHANGED
|
@@ -5,12 +5,13 @@ A tool to convert TradingView Pine Script indicators and strategies
|
|
|
5
5
|
to PyneCore Python code for local backtesting and analysis.
|
|
6
6
|
"""
|
|
7
7
|
|
|
8
|
-
__version__ = '0.2.
|
|
8
|
+
__version__ = '0.2.3'
|
|
9
9
|
__author__ = 'Pine2Pyne Transpiler'
|
|
10
10
|
|
|
11
11
|
from .lexer import Lexer, tokenize
|
|
12
12
|
from .parser import Parser, parse
|
|
13
13
|
from .parse_repair import parse_source
|
|
14
|
+
from .legacy_function_names import qualify_legacy_function_names
|
|
14
15
|
from .pine_semantics import enforce as enforce_pine_semantics
|
|
15
16
|
from .transformer import Transformer, PyneTransformedScript
|
|
16
17
|
from .codegen import CodeGenerator, generate_code
|
|
@@ -55,6 +56,9 @@ def transpile(source: str) -> str:
|
|
|
55
56
|
# Tokenize and parse; a rejected known spelling of valid Pine is repaired (card #55)
|
|
56
57
|
ast, _repairs = parse_source(source)
|
|
57
58
|
|
|
59
|
+
# Pine v1-v4 call names are spelled the way Pine v5 spells them (card #116)
|
|
60
|
+
ast = qualify_legacy_function_names(ast)
|
|
61
|
+
|
|
58
62
|
# Refuse what TradingView's compiler refuses, at the Pine line (card #28)
|
|
59
63
|
enforce_pine_semantics(ast)
|
|
60
64
|
|
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
"""Pine v1-v4 call names, spelled the way Pine v5 spells them (card #116).
|
|
2
|
+
|
|
3
|
+
Before Pine v5 the technical-analysis, math, request and ticker functions were global: `sma(close, 5)`,
|
|
4
|
+
`mom(close, 2)`, `security(...)`, `heikenashi(...)`. v5 moved them into the `ta.`, `math.`, `str.`, `request.`
|
|
5
|
+
and `ticker.` namespaces. The transformer and the runtime know only the namespaced names, so a bare `sma(...)`
|
|
6
|
+
reached the generated code as a bare `sma(...)` that nothing imports: `NameError` at run.
|
|
7
|
+
|
|
8
|
+
`qualify_legacy_function_names` renames those calls in a script that declares //@version 1-4. Only a call of a
|
|
9
|
+
name from the table is renamed, and only when the script declares no FUNCTION of that name itself: a user's own
|
|
10
|
+
`sma(x, n) => ...` stays. A variable, a parameter or a loop counter of that name does not stop the rename: a call
|
|
11
|
+
`sma(...)` looks up a function, a variable lives apart from it (`rsi = rsi(close, 14)` is a common idiom) or is out of
|
|
12
|
+
scope at the call, so the builtin is the callee. A source with no //@version line, or one at v5 or later, is not
|
|
13
|
+
touched.
|
|
14
|
+
|
|
15
|
+
Left out on purpose (card #116 follow-ups): names a Python builtin of the same spelling already answers
|
|
16
|
+
(`abs`, `max`, `min`, `round`, `pow`, `range`), which run today with Python semantics; and the legacy names
|
|
17
|
+
that are series, not calls (a bare `tr`, `vwap`, `obv`, `tickerid`, ...).
|
|
18
|
+
"""
|
|
19
|
+
from __future__ import annotations
|
|
20
|
+
|
|
21
|
+
from typing import Dict, Set
|
|
22
|
+
|
|
23
|
+
from .ast_nodes import FuncDecl, FunctionCall, Script
|
|
24
|
+
from .ast_walk import walk
|
|
25
|
+
|
|
26
|
+
TECHNICAL_ANALYSIS_NAMES = (
|
|
27
|
+
'alma atr barssince bb bbw cci change cmo cog correlation cross crossover crossunder cum dev dmi ema falling '
|
|
28
|
+
'hma highest highestbars kc kcw linreg lowest lowestbars macd mfi mom percentile_linear_interpolation '
|
|
29
|
+
'percentile_nearest_rank percentrank pivothigh pivotlow rising rma roc rsi sar sma stdev stoch supertrend '
|
|
30
|
+
'swma tr tsi valuewhen variance vwma wma wpr'
|
|
31
|
+
).split()
|
|
32
|
+
MATH_NAMES = ('acos asin atan avg ceil cos exp floor log log10 random round_to_mintick sign sin sqrt sum tan '
|
|
33
|
+
'todegrees toradians').split()
|
|
34
|
+
|
|
35
|
+
# call name before Pine v5 -> the name Pine v5 gives it
|
|
36
|
+
LEGACY_FUNCTION_NAMES: Dict[str, str] = {
|
|
37
|
+
**{name: f'ta.{name}' for name in TECHNICAL_ANALYSIS_NAMES},
|
|
38
|
+
**{name: f'math.{name}' for name in MATH_NAMES},
|
|
39
|
+
'tostring': 'str.tostring',
|
|
40
|
+
'tonumber': 'str.tonumber',
|
|
41
|
+
'security': 'request.security',
|
|
42
|
+
'heikenashi': 'ticker.heikinashi',
|
|
43
|
+
}
|
|
44
|
+
|
|
45
|
+
|
|
46
|
+
def _declared_function_names(script: Script) -> Set[str]:
|
|
47
|
+
"""The names of the functions the script itself declares, anywhere in it."""
|
|
48
|
+
return {node.name for node in walk(script) if isinstance(node, FuncDecl) and isinstance(node.name, str)}
|
|
49
|
+
|
|
50
|
+
|
|
51
|
+
def _declared_version(script: Script) -> int:
|
|
52
|
+
"""The //@version number, or 0 when absent or not a number."""
|
|
53
|
+
text = script.version.version if script.version else ''
|
|
54
|
+
return int(text) if str(text).isdigit() else 0
|
|
55
|
+
|
|
56
|
+
|
|
57
|
+
def qualify_legacy_function_names(script: Script) -> Script:
|
|
58
|
+
"""The script with the calls of a Pine v1-v4 script spelled as Pine v5 spells them."""
|
|
59
|
+
if not 1 <= _declared_version(script) <= 4:
|
|
60
|
+
return script
|
|
61
|
+
own_names = _declared_function_names(script)
|
|
62
|
+
renames = {legacy: current for legacy, current in LEGACY_FUNCTION_NAMES.items() if legacy not in own_names}
|
|
63
|
+
for node in walk(script):
|
|
64
|
+
if isinstance(node, FunctionCall) and isinstance(node.func, str) and node.func in renames:
|
|
65
|
+
node.func = renames[node.func]
|
|
66
|
+
return script
|
pine2pyne/lexer.py
CHANGED
|
@@ -225,6 +225,10 @@ class Lexer:
|
|
|
225
225
|
while self.current_char() and (self.current_char().isalnum() or self.current_char() in ('_', '.')):
|
|
226
226
|
identifier += self.advance()
|
|
227
227
|
|
|
228
|
+
# `type` is a keyword only at the head of a user-defined-type declaration (card #115)
|
|
229
|
+
if identifier == 'type' and not self._heads_type_declaration():
|
|
230
|
+
return Token(TokenType.IDENTIFIER, identifier, start_line, start_col, identifier)
|
|
231
|
+
|
|
228
232
|
# Check if it's a keyword
|
|
229
233
|
if identifier in KEYWORDS:
|
|
230
234
|
token_type = KEYWORDS[identifier]
|
|
@@ -236,6 +240,28 @@ class Lexer:
|
|
|
236
240
|
|
|
237
241
|
return Token(TokenType.IDENTIFIER, identifier, start_line, start_col, identifier)
|
|
238
242
|
|
|
243
|
+
def _heads_type_declaration(self) -> bool:
|
|
244
|
+
"""True for `type Name`: first on its statement (or after `export`), a name following on the same line.
|
|
245
|
+
|
|
246
|
+
Anywhere else `type` is an ordinary name (the Pine v2-v4 `type=` argument of input(), a variable,
|
|
247
|
+
a parameter, a field). The parser accepted a TYPE token only at the head of a declaration, so a
|
|
248
|
+
source that parsed before is not changed by this.
|
|
249
|
+
"""
|
|
250
|
+
previous = self.tokens[-1].type if self.tokens else None
|
|
251
|
+
if previous not in (None, TokenType.NEWLINE, TokenType.INDENT, TokenType.DEDENT, TokenType.EXPORT):
|
|
252
|
+
return False
|
|
253
|
+
offset = 0
|
|
254
|
+
while self.peek_char(offset) in (' ', '\t'):
|
|
255
|
+
offset += 1
|
|
256
|
+
following = self.peek_char(offset)
|
|
257
|
+
if following is None or not (following.isalpha() or following == '_'):
|
|
258
|
+
return False
|
|
259
|
+
name_end = offset
|
|
260
|
+
while (self.peek_char(name_end) or ' ').isalnum() or self.peek_char(name_end) == '_':
|
|
261
|
+
name_end += 1
|
|
262
|
+
following_word = self.source[self.pos + offset:self.pos + name_end]
|
|
263
|
+
return following_word not in ('and', 'or', 'in', 'to', 'by')
|
|
264
|
+
|
|
239
265
|
def _is_continuation_line(self) -> bool:
|
|
240
266
|
"""Check if this line is a continuation of the previous expression.
|
|
241
267
|
|
pine2pyne/parser.py
CHANGED
|
@@ -10,6 +10,7 @@ from .tokens import Token, TokenType
|
|
|
10
10
|
from .ast_nodes import *
|
|
11
11
|
from .errors import ParserError, SemanticError, UnsupportedFeatureError
|
|
12
12
|
from .once_lowering import lower_once_structures
|
|
13
|
+
from .strategy_when_lowering import lower_strategy_when_arguments
|
|
13
14
|
from .scope_renamer import fresh_name
|
|
14
15
|
|
|
15
16
|
|
|
@@ -254,7 +255,8 @@ class Parser:
|
|
|
254
255
|
|
|
255
256
|
self.skip_newlines()
|
|
256
257
|
|
|
257
|
-
|
|
258
|
+
script = lower_once_structures(script)
|
|
259
|
+
return lower_strategy_when_arguments(script, _pine_version(script.version))
|
|
258
260
|
|
|
259
261
|
def _at_script_declaration(self, version: Optional[VersionAnnotation]) -> bool:
|
|
260
262
|
"""At an indicator/strategy/library call; before v5 a study() call is the indicator (card #55)."""
|
|
@@ -788,12 +790,38 @@ class Parser:
|
|
|
788
790
|
|
|
789
791
|
# Check for = input.
|
|
790
792
|
result = (self.match(TokenType.ASSIGN) and
|
|
791
|
-
self.peek_token().type == TokenType.IDENTIFIER and
|
|
792
|
-
|
|
793
|
+
((self.peek_token().type == TokenType.IDENTIFIER and
|
|
794
|
+
self.peek_token().value.startswith('input.')) or
|
|
795
|
+
self._legacy_typed_input_call_ahead()))
|
|
793
796
|
|
|
794
797
|
self.pos = saved_pos
|
|
795
798
|
return result
|
|
796
799
|
|
|
800
|
+
def _legacy_typed_input_call_ahead(self) -> bool:
|
|
801
|
+
"""At `= input(` ... `type=<kind>` ... `)` with a kind parse_function_call rewrites to an input.* call
|
|
802
|
+
(card #115): such a call is an input declaration, as its v5 spelling is."""
|
|
803
|
+
from .pine_builtins import LEGACY_INPUT_TYPE_ARGUMENT_FUNCTIONS
|
|
804
|
+
if self.peek_token().value != 'input' or self.peek_token(2).type != TokenType.LPAREN:
|
|
805
|
+
return False
|
|
806
|
+
depth, offset = 0, 2
|
|
807
|
+
while True:
|
|
808
|
+
token = self.peek_token(offset)
|
|
809
|
+
if token.type == TokenType.EOF:
|
|
810
|
+
return False
|
|
811
|
+
if token.type in (TokenType.LPAREN, TokenType.LBRACKET):
|
|
812
|
+
depth += 1
|
|
813
|
+
elif token.type in (TokenType.RPAREN, TokenType.RBRACKET):
|
|
814
|
+
depth -= 1
|
|
815
|
+
if depth == 0:
|
|
816
|
+
return False
|
|
817
|
+
elif (depth == 1 and token.type == TokenType.IDENTIFIER and token.value == 'type'
|
|
818
|
+
and self.peek_token(offset + 1).type == TokenType.ASSIGN):
|
|
819
|
+
kind = self.peek_token(offset + 2)
|
|
820
|
+
# a bare `float`/`bool`/`string`/`color` lexes as TYPE_IDENTIFIER; parse_function_call rewrites it too
|
|
821
|
+
return (kind.type in (TokenType.IDENTIFIER, TokenType.TYPE_IDENTIFIER)
|
|
822
|
+
and kind.value.removeprefix('input.') in LEGACY_INPUT_TYPE_ARGUMENT_FUNCTIONS)
|
|
823
|
+
offset += 1
|
|
824
|
+
|
|
797
825
|
def parse_input_declaration(self) -> InputDecl:
|
|
798
826
|
"""Parse input declaration. Handles: name = input.*() and type name = input.*()"""
|
|
799
827
|
# Skip optional type hint
|
|
@@ -1526,6 +1554,14 @@ class Parser:
|
|
|
1526
1554
|
else:
|
|
1527
1555
|
func_name = func
|
|
1528
1556
|
|
|
1557
|
+
# Pine v2-v4 `input(defval, title, type=input.float)` is the v5 `input.float(defval, title)` (card #115)
|
|
1558
|
+
if func_name == 'input' and isinstance(kwargs.get('type'), Identifier):
|
|
1559
|
+
from .pine_builtins import LEGACY_INPUT_TYPE_ARGUMENT_FUNCTIONS
|
|
1560
|
+
typed_function = LEGACY_INPUT_TYPE_ARGUMENT_FUNCTIONS.get(kwargs['type'].name.removeprefix('input.'))
|
|
1561
|
+
if typed_function is not None:
|
|
1562
|
+
func_name = typed_function
|
|
1563
|
+
kwargs = {key: value for key, value in kwargs.items() if key != 'type'}
|
|
1564
|
+
|
|
1529
1565
|
# a call is placed at its callee's name, where a refusal points (card #28)
|
|
1530
1566
|
return FunctionCall(func=func_name, args=args, kwargs=kwargs,
|
|
1531
1567
|
line=getattr(func, 'line', 0), column=getattr(func, 'column', 0))
|
pine2pyne/pine_builtins.py
CHANGED
|
@@ -260,6 +260,14 @@ INPUT_POSITIONAL_PARAMS: Dict[str, list[str]] = {
|
|
|
260
260
|
'input.enum': ['options', 'tooltip', 'inline', 'group', 'confirm', 'display', 'active'], # card #97
|
|
261
261
|
}
|
|
262
262
|
|
|
263
|
+
# Pine v2-v4 `input(defval, title, type=input.X)`: the keyword names the kind the v5+ `input.X` functions
|
|
264
|
+
# carry in their names (card #115). The key is the spelling after an optional `input.` prefix.
|
|
265
|
+
LEGACY_INPUT_TYPE_ARGUMENT_FUNCTIONS: Dict[str, str] = {
|
|
266
|
+
'integer': 'input.int', 'float': 'input.float', 'bool': 'input.bool', 'string': 'input.string',
|
|
267
|
+
'symbol': 'input.symbol', 'resolution': 'input.timeframe', 'source': 'input.source',
|
|
268
|
+
'session': 'input.session', 'time': 'input.time', 'color': 'input.color', 'price': 'input.price',
|
|
269
|
+
}
|
|
270
|
+
|
|
263
271
|
# input.int/input.float have a second positional form after `title` (v6 reference): the options form,
|
|
264
272
|
# selected by a list third argument (pynecore's _bind_numeric_tail, _NUMERIC_OPTIONS_TAIL; card #95).
|
|
265
273
|
NUMERIC_INPUT_FUNCTIONS: FrozenSet[str] = frozenset({'input.int', 'input.float'})
|
pine2pyne/scope_renamer.py
CHANGED
|
@@ -322,12 +322,18 @@ class _ScopeRenamer:
|
|
|
322
322
|
shadows = any(name in scope for scope in self.scopes[:-1])
|
|
323
323
|
if shadows:
|
|
324
324
|
self.shadowed_names.add(name)
|
|
325
|
-
collides = name in self.library_names
|
|
326
325
|
# scopes[0] is the top level; every deeper scope is a function, a loop or a block
|
|
326
|
+
# a variable named like a function or method of the script: Python reads the name as the variable all through
|
|
327
|
+
# its function, so a call of the function in that scope reads the variable (card #118). At the top level too:
|
|
328
|
+
# the functions are defined inside main(), where the variable is a local that hides them from every function
|
|
329
|
+
names_a_function = name in self.user_function_names or name in self.method_names
|
|
330
|
+
if names_a_function:
|
|
331
|
+
self.moved_bindings = True
|
|
332
|
+
collides = name in self.library_names
|
|
327
333
|
hides_builtin_series = name in SERIES_VARIABLES and len(self.scopes) > 1
|
|
328
334
|
if hides_builtin_series:
|
|
329
335
|
self.moved_bindings = True
|
|
330
|
-
if shadows or collides or hides_builtin_series:
|
|
336
|
+
if shadows or collides or hides_builtin_series or names_a_function:
|
|
331
337
|
emitted = self.fresh_name(name)
|
|
332
338
|
elif self.holds_external_spelling(name):
|
|
333
339
|
self.moved_bindings = True
|
|
@@ -0,0 +1,101 @@
|
|
|
1
|
+
"""`when=` of a strategy call, lowered to an `if` (card #117).
|
|
2
|
+
|
|
3
|
+
strategy.entry("l", strategy.long, when=c) if c
|
|
4
|
+
-> strategy.entry("l", strategy.long)
|
|
5
|
+
|
|
6
|
+
Pine v1-v5 gave the order functions of `strategy.*` a `when` argument: the call acts only on the bars where it is
|
|
7
|
+
true. Pine v6 removed it, and the runtime's `strategy.entry` / `strategy.close` / ... take none, so a `when=` that
|
|
8
|
+
reached the generated code raised `TypeError: entry() got an unexpected keyword argument 'when'` at run. The
|
|
9
|
+
lowering runs at the end of parsing, so no later stage sees one.
|
|
10
|
+
|
|
11
|
+
Only the keyword form, only a call that stands as a statement (the functions return nothing), and only in a script
|
|
12
|
+
declaring //@version 1-5. A v6 script is left as written: TradingView refuses a `when` there, so no spelling is
|
|
13
|
+
invented for it. A positional `when` (the 9th slot of `strategy.entry`, the 2nd of v4 `strategy.close`) is not
|
|
14
|
+
lowered here (card #117 follow-up).
|
|
15
|
+
"""
|
|
16
|
+
import dataclasses
|
|
17
|
+
import itertools
|
|
18
|
+
|
|
19
|
+
from .ast_nodes import (Assignment, ExpressionStatement, FuncDecl, FunctionCall, Identifier, IfStatement, IndexAccess,
|
|
20
|
+
MethodCall, Script)
|
|
21
|
+
from .ast_walk import child_field_names, contains, is_node
|
|
22
|
+
|
|
23
|
+
STRATEGY_FUNCTIONS_WITH_WHEN = frozenset({
|
|
24
|
+
'strategy.entry', 'strategy.order', 'strategy.exit', 'strategy.close', 'strategy.close_all',
|
|
25
|
+
'strategy.cancel', 'strategy.cancel_all',
|
|
26
|
+
})
|
|
27
|
+
|
|
28
|
+
|
|
29
|
+
def lower_strategy_when_arguments(script: Script, pine_version: int) -> Script:
|
|
30
|
+
"""A copy of `script` with each `strategy.*(..., when=c)` statement turned into `if c` around the call
|
|
31
|
+
without `when`. A script without one, or of another version, is returned unchanged."""
|
|
32
|
+
if not 1 <= pine_version <= 5 or not contains(script, _is_call_with_when):
|
|
33
|
+
return script
|
|
34
|
+
return _WhenLowering().lowered(script)
|
|
35
|
+
|
|
36
|
+
|
|
37
|
+
def _is_call_with_when(node) -> bool:
|
|
38
|
+
# a callee that is not a name (`na(x)`: a NaLiteral node) is unhashable and is never a strategy function
|
|
39
|
+
return (isinstance(node, FunctionCall) and isinstance(node.func, str) and node.func in STRATEGY_FUNCTIONS_WITH_WHEN
|
|
40
|
+
and 'when' in node.kwargs)
|
|
41
|
+
|
|
42
|
+
|
|
43
|
+
def _is_stateful_or_history(node) -> bool:
|
|
44
|
+
return isinstance(node, (FunctionCall, MethodCall, IndexAccess))
|
|
45
|
+
|
|
46
|
+
|
|
47
|
+
class _Statements(list):
|
|
48
|
+
"""The statements one statement is replaced by (a list item that expands to several)."""
|
|
49
|
+
|
|
50
|
+
|
|
51
|
+
class _WhenLowering:
|
|
52
|
+
"""Pine evaluates every argument of a `strategy.*(..., when=c)` call on every bar, whatever `c` is. An argument
|
|
53
|
+
holding a call (`ta.lowest(low, 10)`, a user function) or a history index (`close[1]`) has state per bar, and
|
|
54
|
+
inside an `if` it would update only on the bars where `c` holds. So such an argument is assigned to a temporary
|
|
55
|
+
before the `if`, and the temporary is what the call inside the `if` receives."""
|
|
56
|
+
|
|
57
|
+
def __init__(self):
|
|
58
|
+
self.temporary_numbers = itertools.count(1)
|
|
59
|
+
|
|
60
|
+
def lowered(self, node):
|
|
61
|
+
children = {name: self._map(getattr(node, name)) for name in child_field_names(type(node))}
|
|
62
|
+
rebuilt = dataclasses.replace(node, **children)
|
|
63
|
+
if isinstance(rebuilt, ExpressionStatement) and _is_call_with_when(rebuilt.expr):
|
|
64
|
+
return _Statements(self._if_around(rebuilt))
|
|
65
|
+
if isinstance(rebuilt, FuncDecl) and _is_call_with_when(rebuilt.body):
|
|
66
|
+
# `f(c) => strategy.close("l", when=c)`: the one-line body is the call, made a statement
|
|
67
|
+
call = rebuilt.body
|
|
68
|
+
statement = ExpressionStatement(expr=call, line=call.line, column=call.column)
|
|
69
|
+
return dataclasses.replace(rebuilt, body=self._if_around(statement))
|
|
70
|
+
return rebuilt
|
|
71
|
+
|
|
72
|
+
def _map(self, value):
|
|
73
|
+
if isinstance(value, list):
|
|
74
|
+
items = []
|
|
75
|
+
for item in value:
|
|
76
|
+
mapped = self._map(item)
|
|
77
|
+
items.extend(mapped) if isinstance(mapped, _Statements) else items.append(mapped)
|
|
78
|
+
return items
|
|
79
|
+
if isinstance(value, tuple):
|
|
80
|
+
return tuple(self._map(item) for item in value)
|
|
81
|
+
if isinstance(value, dict):
|
|
82
|
+
return {key: self._map(item) for key, item in value.items()}
|
|
83
|
+
return self.lowered(value) if is_node(value) else value
|
|
84
|
+
|
|
85
|
+
def _if_around(self, statement: ExpressionStatement) -> list:
|
|
86
|
+
call = statement.expr
|
|
87
|
+
temporaries = []
|
|
88
|
+
args = [self._every_bar(argument, 'argument', statement, temporaries) for argument in call.args]
|
|
89
|
+
kwargs = {key: self._every_bar(value, key, statement, temporaries)
|
|
90
|
+
for key, value in call.kwargs.items() if key != 'when'}
|
|
91
|
+
without_when = dataclasses.replace(call, args=args, kwargs=kwargs)
|
|
92
|
+
guarded = IfStatement(condition=call.kwargs['when'], body=[dataclasses.replace(statement, expr=without_when)],
|
|
93
|
+
line=statement.line, column=statement.column)
|
|
94
|
+
return [*temporaries, guarded]
|
|
95
|
+
|
|
96
|
+
def _every_bar(self, argument, label, statement, temporaries):
|
|
97
|
+
if not contains(argument, _is_stateful_or_history):
|
|
98
|
+
return argument
|
|
99
|
+
name = f'_when_{label}_{next(self.temporary_numbers)}'
|
|
100
|
+
temporaries.append(Assignment(target=name, value=argument, line=statement.line, column=statement.column))
|
|
101
|
+
return Identifier(name=name, line=statement.line, column=statement.column)
|
|
@@ -1,301 +0,0 @@
|
|
|
1
|
-
Metadata-Version: 2.4
|
|
2
|
-
Name: opencode-pine2pyne
|
|
3
|
-
Version: 0.2.2
|
|
4
|
-
Summary: Pine Script v6 to PyneCore Python transpiler
|
|
5
|
-
Author: rubycell
|
|
6
|
-
License-Expression: Apache-2.0
|
|
7
|
-
Project-URL: Homepage, https://github.com/rubycell/pine2pyne
|
|
8
|
-
Project-URL: Repository, https://github.com/rubycell/pine2pyne
|
|
9
|
-
Keywords: pinescript,tradingview,transpiler,pynecore,backtesting
|
|
10
|
-
Classifier: Programming Language :: Python :: 3.11
|
|
11
|
-
Classifier: Programming Language :: Python :: 3.12
|
|
12
|
-
Classifier: Programming Language :: Python :: 3.13
|
|
13
|
-
Classifier: Topic :: Office/Business :: Financial :: Investment
|
|
14
|
-
Requires-Python: >=3.11
|
|
15
|
-
Description-Content-Type: text/markdown
|
|
16
|
-
License-File: LICENSE
|
|
17
|
-
Provides-Extra: runtime
|
|
18
|
-
Requires-Dist: opencode-pyneruntime[cli]>=6.9.1; extra == "runtime"
|
|
19
|
-
Dynamic: license-file
|
|
20
|
-
|
|
21
|
-
# PyneCore - Pine Script to Python Transpiler
|
|
22
|
-
|
|
23
|
-
> **Backlog & known bugs:** [Projects board #10](https://github.com/users/rubycell/projects/10)
|
|
24
|
-
> — open work is in **Backlog** (prioritised P0/P1/P2), shipped work in **Done**.
|
|
25
|
-
> Confirmed unfixed transpiler bugs live there too; several are pinned by strict
|
|
26
|
-
> `xfail` tests (see [Running Tests](#running-tests)).
|
|
27
|
-
|
|
28
|
-
## Installation (New Computer)
|
|
29
|
-
|
|
30
|
-
### Fastest: the sandbox-setup skill
|
|
31
|
-
|
|
32
|
-
From the repo root, run the bundled setup script (idempotent — safe to re-run):
|
|
33
|
-
|
|
34
|
-
```bash
|
|
35
|
-
bash ~/.claude/skills/opencode-pine2pyne-setup/setup.sh # from inside this clone; the skill lives in ~/.claude/skills
|
|
36
|
-
```
|
|
37
|
-
|
|
38
|
-
It builds `.venv`, installs everything below, detects `PINE_CPU_COUNT` /
|
|
39
|
-
`PINE_MAX_WORKERS`, seeds a demo dataset, and runs a smoke test that ends in
|
|
40
|
-
`smoke test : PASS` / `READY`. In Claude Code you can also invoke it as
|
|
41
|
-
`/sandbox-setup`.
|
|
42
|
-
|
|
43
|
-
### Manual (equivalent steps)
|
|
44
|
-
|
|
45
|
-
```bash
|
|
46
|
-
# From the repo root:
|
|
47
|
-
python3 -m venv .venv # 1. virtualenv
|
|
48
|
-
.venv/bin/pip install -e . # 2. this transpiler (editable)
|
|
49
|
-
.venv/bin/pip install 'opencode-pyneruntime[cli]>=6.9.1' # 3. PyneCore runtime + `pyne` CLI
|
|
50
|
-
.venv/bin/pip install optuna numpy # 4. optimizer deps (not pulled in by the runtime)
|
|
51
|
-
source .venv/bin/activate # 5. activate (Linux/macOS)
|
|
52
|
-
# .venv\Scripts\activate # (Windows)
|
|
53
|
-
```
|
|
54
|
-
|
|
55
|
-
Verify the install:
|
|
56
|
-
|
|
57
|
-
```bash
|
|
58
|
-
.venv/bin/python -c "import pynecore, pine2pyne, optuna, numpy; print('all ok')"
|
|
59
|
-
```
|
|
60
|
-
|
|
61
|
-
**Note**: All commands below assume you are at the repo root with the venv activated.
|
|
62
|
-
|
|
63
|
-
## How to Convert Pine Script to Python
|
|
64
|
-
|
|
65
|
-
### Single File Conversion
|
|
66
|
-
|
|
67
|
-
```bash
|
|
68
|
-
python -m pine2pyne path/to/script.pine -o path/to/output.py
|
|
69
|
-
```
|
|
70
|
-
|
|
71
|
-
### Batch Conversion
|
|
72
|
-
|
|
73
|
-
```bash
|
|
74
|
-
python -m pine2pyne "sample/pinescript/*.pine" -o workdir/scripts/
|
|
75
|
-
```
|
|
76
|
-
|
|
77
|
-
## Running Tests
|
|
78
|
-
|
|
79
|
-
There are **three** independent test systems. They answer different questions,
|
|
80
|
-
so a green run of one says nothing about the others.
|
|
81
|
-
|
|
82
|
-
| System | Question it answers | Runtime |
|
|
83
|
-
|---|---|---|
|
|
84
|
-
| **1. Unit tests** (`tests/`, pytest) | Are the transpiler's internals correct? | ~5 s |
|
|
85
|
-
| **2. Sample suites** (`tools/test_all_*.py`) | Does every corpus sample transpile and run? | ~35 s |
|
|
86
|
-
| **3. Ground truth** (`tools/*_ground_truth*`) | Does it produce the *same numbers* as before? | minutes |
|
|
87
|
-
|
|
88
|
-
### 1. Unit tests — pytest
|
|
89
|
-
|
|
90
|
-
6,870 tests over `pine2pyne/` (6,848 passed, 1 skipped, 21 xfailed at a16621b; 93% line / 86% branch coverage).
|
|
91
|
-
|
|
92
|
-
```bash
|
|
93
|
-
.venv/bin/python -m pytest tests/ -q
|
|
94
|
-
|
|
95
|
-
# with coverage
|
|
96
|
-
.venv/bin/python -m pytest tests/ \
|
|
97
|
-
--cov=pine2pyne --cov-report=term-missing
|
|
98
|
-
```
|
|
99
|
-
|
|
100
|
-
A bare `pytest tests/` needs no `--ignore`: `tests/test_screener_regression.py`, which
|
|
101
|
-
imported a `screener` module from a sibling repo that is absent here and aborted the
|
|
102
|
-
whole run at collection, was retired from the repo.
|
|
103
|
-
|
|
104
|
-
Some tests are `@pytest.mark.xfail(strict=True)` and pin **confirmed unfixed
|
|
105
|
-
bugs**, each with a card on [board #10](https://github.com/users/rubycell/projects/10).
|
|
106
|
-
Strict means fixing a bug turns its marker into a failure until the marker is
|
|
107
|
-
removed, so nothing regresses silently in either direction.
|
|
108
|
-
|
|
109
|
-
The former P0 — codegen dropping Pine grouping, so `10 - (5 - 3)` became
|
|
110
|
-
`10 - 5 - 3` — is fixed (card #6) and guarded by
|
|
111
|
-
`tests/test_codegen_operator_grouping.py`. See [docs/test_plan.md](docs/test_plan.md) for the full bug inventory and the
|
|
112
|
-
measured coverage/mutation results.
|
|
113
|
-
|
|
114
|
-
### 2. Sample suites — end-to-end transpile + run
|
|
115
|
-
|
|
116
|
-
Transpiles every `.pine` in the corpus and runs it through `pyne`.
|
|
117
|
-
|
|
118
|
-
```bash
|
|
119
|
-
python tools/test_all_samples.py # sample/pinescript (~411 files)
|
|
120
|
-
python tools/test_all_sample_pds.py # sample_pds (~328 files)
|
|
121
|
-
|
|
122
|
-
python tools/test_all_samples.py "ex_001*" # filter by pattern
|
|
123
|
-
python tools/test_all_samples.py --timeout 30 # default 15s
|
|
124
|
-
python tools/test_all_samples.py --verbose # stderr on failure
|
|
125
|
-
python tools/test_all_samples.py -j 0 # all CPU cores
|
|
126
|
-
python tools/test_all_samples.py --run-dir /tmp/run1 # private scripts/, output/ and report files (card #70)
|
|
127
|
-
```
|
|
128
|
-
|
|
129
|
-
- Source: `sample/pinescript/`, `sample_pds/` · Output: `workdir/scripts/`
|
|
130
|
-
- `--run-dir DIR` keeps a run's transpiled scripts (`DIR/scripts/`), pyne outputs (`DIR/output/`, passed to `pyne`
|
|
131
|
-
as explicit `--plot/--strat/--trade` paths) and `test_results*.{json,txt}` reports in `DIR`, so two runs, or a run
|
|
132
|
-
and an old report, cannot read each other's files. Without it nothing changes: the shared `workdir/` paths and
|
|
133
|
-
the repo-root reports. `--retry-failed` with `--run-dir` reads `DIR`'s own report.
|
|
134
|
-
- Data: `workdir/data/demo.ohlcv` · Results: `test_results{,_pds}.{json,txt}`
|
|
135
|
-
|
|
136
|
-
**Per-sample expectations** live in `tools/sample_expectations.toml`:
|
|
137
|
-
|
|
138
|
-
```toml
|
|
139
|
-
[expds_156_Out_of_bounds_index]
|
|
140
|
-
expect_error = "index_error" # MUST raise; running clean = FAIL
|
|
141
|
-
|
|
142
|
-
[ex_300_ticker_modify]
|
|
143
|
-
requires_feeds = { D = "demo", "2D" = "demo_2D" } # -> --security D=demo …
|
|
144
|
-
```
|
|
145
|
-
|
|
146
|
-
`expect_error` marks Pine documentation samples that exist to *demonstrate* an
|
|
147
|
-
error — they are reported as `XFAIL` and count as passes. `requires_feeds`
|
|
148
|
-
supplies extra OHLCV feeds for samples calling `request.security()` at a
|
|
149
|
-
timeframe other than the chart's; pynecore deliberately will not resample the
|
|
150
|
-
chart feed, so the feed must be explicit. Generate one with:
|
|
151
|
-
|
|
152
|
-
```bash
|
|
153
|
-
cd workdir && pyne data aggregate data/demo.ohlcv -tf 2D
|
|
154
|
-
```
|
|
155
|
-
|
|
156
|
-
Shared logic for both suites lives in `tools/_suite_common.py` — edit it there,
|
|
157
|
-
not in each harness.
|
|
158
|
-
|
|
159
|
-
### 3. Ground truth — numeric regression
|
|
160
|
-
|
|
161
|
-
Checks every freshly generated output file (~1,650 across ~550 samples) against
|
|
162
|
-
its recorded digest. This is the only system that catches *silently wrong
|
|
163
|
-
numbers*: output that runs fine but computes something different.
|
|
164
|
-
|
|
165
|
-
```bash
|
|
166
|
-
python tools/generate_ground_truth.py # (re)build baselines
|
|
167
|
-
python tools/compare_ground_truth.py # check current output vs baselines
|
|
168
|
-
python tools/compare_ground_truth.py --suite sample_pds -j 12
|
|
169
|
-
```
|
|
170
|
-
|
|
171
|
-
**The baseline is `ground_truth/digests.json`** (card #62). It holds a sha256 and a
|
|
172
|
-
line count for each output file, plus the sha256 of the data file. Git tracks
|
|
173
|
-
only it and `ground_truth/manifest.json` (under 1 MB together). The output
|
|
174
|
-
CSVs themselves (~920 MB) are gitignored: the generator still copies them to
|
|
175
|
-
`ground_truth/{sample,sample_pds}/` on your machine for diffing. Before writing a
|
|
176
|
-
new baseline, it moves the previous CSVs to `backup/deleteable/ground_truth_csv_<time>/`
|
|
177
|
-
(nothing is deleted; clear old ones out by hand). `manifest.json` lists its samples sorted by
|
|
178
|
-
suite, then filename (card #68), so a one-suite rebuild (`--suite sample`) and a full rebuild on the same data
|
|
179
|
-
give the same samples in the same order, and a rebuild never reorders the tracked file; only `generated_at`
|
|
180
|
-
changes by design.
|
|
181
|
-
|
|
182
|
-
- **Exit codes.** The compare exits 1 when `digests.json` is missing or empty, and
|
|
183
|
-
2 with `DATA CHANGED` when the data file is not the one the baseline was made
|
|
184
|
-
on (restore it, see *Test data* below, or rebaseline).
|
|
185
|
-
- **Every sample runs in its own directory** (card #70): the generator and the compare make one fresh run
|
|
186
|
-
root, `backup/deleteable/ground_truth_runs/<tool>-<time>/` (or an empty `--run-dir DIR`; a directory that already
|
|
187
|
-
holds files is refused), with `<suite>/<sample>/<data file>/` inside it holding the script, its 10B settings and the three
|
|
188
|
-
CSVs, and collect only from there. An earlier run's file of the same name cannot be read as this run's; a run
|
|
189
|
-
that exits 0 and writes nothing is `no_output` (a failure), never `ok` or `NEW PASS`. Run roots are never
|
|
190
|
-
deleted (see the paragraph below on where they go).
|
|
191
|
-
- **Where run roots go, and the disk they use.** The default run root of the generator, the compare and the
|
|
192
|
-
checker (`ground_truth_check.py capture` and `compare` each make one) is
|
|
193
|
-
`backup/deleteable/ground_truth_runs/<tool>-<time>/`, next to the other disposable output, and each run leaves
|
|
194
|
-
one tree of roughly 0.9 GB there (script, settings and three CSVs per sample). A generate run also copies the
|
|
195
|
-
collected files into `ground_truth/<suite>/` (gitignored). Nothing here is deleted by a tool: when you want the
|
|
196
|
-
space back, move old run roots (and the `gt_compare_<time>/` scratch copies) to
|
|
197
|
-
`/mnt/data970/Backup/mike-PC-deleteable/<date>-pine2pyne/`, never `rm` them. An explicit `--run-dir DIR` is
|
|
198
|
-
used as given (it must be empty).
|
|
199
|
-
- **A `REGRESSION` line** names the sample and file whose digest changed. To see
|
|
200
|
-
*what* changed, diff that sample's CSV in the run root's `<suite>/<sample>/<data file>/`
|
|
201
|
-
against a baseline copy. That copy is in `ground_truth/<suite>/<sample>/` if you
|
|
202
|
-
generated on the old commit, or, after a later generate, in the newest
|
|
203
|
-
`backup/deleteable/ground_truth_csv_<time>/`.
|
|
204
|
-
- **`--suite sample` or `--suite sample_pds`** rebuilds one suite and keeps the
|
|
205
|
-
other suite's digests and manifest entries. It refuses (exit 1) unless a
|
|
206
|
-
baseline made on the same data already exists.
|
|
207
|
-
- **Expected noise:** 11 `sample_pds` samples call unseeded `math.random()` and
|
|
208
|
-
differ on every run (expds_022, 037, 039, 043, 045, 081, 091, 092, 104, 114,
|
|
209
|
-
115).
|
|
210
|
-
|
|
211
|
-
The old end-to-end shell wrapper is retired (card #66): it read `ground_truth/samples/`,
|
|
212
|
-
`ground_truth/samples_pds/` and `ground_truth/strategies/`, which no tool writes, so its
|
|
213
|
-
`--quick` mode passed with zero checks. Its default and `--samples` modes are now
|
|
214
|
-
`python tools/compare_ground_truth.py` (`--suite sample`, `sample_pds` or `both`; it reads
|
|
215
|
-
`workdir/data/VN30F1M_15m.ohlcv` by default, see *Test data*); its `--quick` and
|
|
216
|
-
`--strategies` modes are `python tools/ground_truth_check.py capture`, then `compare`.
|
|
217
|
-
|
|
218
|
-
`tools/ground_truth_check.py` (`capture` / `compare`) is a separate, local check of the
|
|
219
|
-
bigtest strategies and the sample suites. It writes only to `ground_truth_check/` at the
|
|
220
|
-
repo root (gitignored), never to `ground_truth/`, so it cannot change the tracked baseline
|
|
221
|
-
above. `capture --force` moves the old `ground_truth_check/` to
|
|
222
|
-
`backup/deleteable/ground_truth_check_<time>/` (nothing is deleted) and touches nothing else. Both actions run the
|
|
223
|
-
bigtest strategies and the two suite harnesses (`--run-dir`) in one fresh run root under
|
|
224
|
-
`backup/deleteable/ground_truth_runs/` (`--run-dir DIR` names an empty one), read only that run's files, and fail the capture when a suite wrote no report
|
|
225
|
-
or a bigtest file did not transpile. `compare` writes its
|
|
226
|
-
scratch copy of the current outputs to `backup/deleteable/gt_compare_<time>/` and never deletes it (card #67); each
|
|
227
|
-
run leaves one such folder, which you move to the backup disk when you want the space back. A capture made
|
|
228
|
-
before card #63 sits in `ground_truth/` (`transpiled/`, `strategy_runs/`, `test_suite_*`);
|
|
229
|
-
`compare` no longer reads it and says so. Run `capture` again.
|
|
230
|
-
|
|
231
|
-
> Regenerate baselines only when you have *confirmed* the new output is correct
|
|
232
|
-
> — otherwise a bug gets frozen in as the expected result.
|
|
233
|
-
|
|
234
|
-
Both tools run every sample on `workdir/data/VN30F1M_15m.ohlcv`, with a
|
|
235
|
-
`<script>.toml` that sets `initial_capital` to 10,000,000,000, and with one
|
|
236
|
-
per-sample timeout (120 s). Both values live in `tools/_ground_truth_common.py`.
|
|
237
|
-
At pynecore's default capital of 1,000,000, one VN30F1M contract costs more than
|
|
238
|
-
equity, so nearly every strategy would record zero trades (card #57).
|
|
239
|
-
|
|
240
|
-
### Test data
|
|
241
|
-
|
|
242
|
-
`workdir/data/` is gitignored, so a fresh clone has none and the sample suites
|
|
243
|
-
abort with `Data file not found`. Restore the exact dataset the recorded results
|
|
244
|
-
were measured against:
|
|
245
|
-
|
|
246
|
-
```bash
|
|
247
|
-
mkdir -p workdir/data
|
|
248
|
-
# Sample suites: CCXT BTCUSDT 1D, plus the 2D/5D feeds `requires_feeds` names
|
|
249
|
-
git show 227f3d2^:workdir/data/demo.ohlcv > workdir/data/demo.ohlcv
|
|
250
|
-
cp tests/fixtures/data/demo.toml workdir/data/demo.toml
|
|
251
|
-
(cd workdir && pyne data aggregate data/demo.ohlcv -tf 2D && pyne data aggregate data/demo.ohlcv -tf 5D)
|
|
252
|
-
# Ground truth: VN30F1M 15m
|
|
253
|
-
git show 5d63d21:workdir/data/VN30F1M_15m.ohlcv > workdir/data/VN30F1M_15m.ohlcv
|
|
254
|
-
cp tests/fixtures/data/VN30F1M_15m.toml workdir/data/VN30F1M_15m.toml
|
|
255
|
-
```
|
|
256
|
-
|
|
257
|
-
The `.ohlcv` files come from those commits (sha256 `67e827cc1a5f71ed…` for
|
|
258
|
-
demo, `e9ef984b09eaa6d2…` for VN30F1M). Take the `.toml` files from
|
|
259
|
-
`tests/fixtures/data/`, not from the commits. The committed ones number weekdays
|
|
260
|
-
1..7, and pynecore rejects day 7 (it numbers 0 = Monday … 6 = Sunday). They also
|
|
261
|
-
declare one session per week, which leaves session-anchored values such as
|
|
262
|
-
`ta.vwap` NaN or weekly (card #57).
|
|
263
|
-
|
|
264
|
-
(`/sandbox-setup` instead seeds a small *synthetic* fixture — fine for a smoke
|
|
265
|
-
test, but results will not match the recorded rates.)
|
|
266
|
-
|
|
267
|
-
## For Claude Code / LLM Development
|
|
268
|
-
|
|
269
|
-
See `CLAUDE.md` in this directory for transpiler architecture, output format specs, optimizer documentation, and CSV rounding rules. See `pine2pyne/README.md` for transpiler internals and transformation rules.
|
|
270
|
-
|
|
271
|
-
## Optimizing Strategies
|
|
272
|
-
|
|
273
|
-
Quick start: `pyne optimize script.py data.ohlcv params.json -n 20`
|
|
274
|
-
|
|
275
|
-
See [Optimizer.md](./Optimizer.md) for full documentation (parameter JSON format, parallel execution, output files).
|
|
276
|
-
|
|
277
|
-
### Distributed Optimization (multi-machine)
|
|
278
|
-
|
|
279
|
-
Two tools distribute `pyne optimize` across SSH clusters:
|
|
280
|
-
|
|
281
|
-
| Tool | Model | Best for |
|
|
282
|
-
|------|-------|----------|
|
|
283
|
-
| `pyne-dynamic.sh` | Flat-queue (on-demand dispatch) | Multi-variant runs, heterogeneous clusters, long jobs |
|
|
284
|
-
| `pyne-parallel.sh` | Static pre-assignment | Quick jobs on similar-speed machines |
|
|
285
|
-
|
|
286
|
-
```bash
|
|
287
|
-
# Dynamic (recommended): flat-queue, pre-syncs once, auto-adapts to machine speed
|
|
288
|
-
# Run from workdir/ directory:
|
|
289
|
-
../tools/pyne-dynamic.sh scripts/strategy.py data/data.ohlcv optimize_variants/ \
|
|
290
|
-
-H ../tools/machines.txt -C 24 --name my_run --output-dir runs/output/
|
|
291
|
-
|
|
292
|
-
# Static: pre-assigns chunks by core count
|
|
293
|
-
./tools/pyne-parallel.sh scripts/strategy.py data/data.ohlcv optimize.json \
|
|
294
|
-
-H tools/machines.txt --sync
|
|
295
|
-
|
|
296
|
-
# Check progress / collect results
|
|
297
|
-
../tools/pyne-dynamic.sh --status
|
|
298
|
-
../tools/pyne-dynamic.sh --collect
|
|
299
|
-
```
|
|
300
|
-
|
|
301
|
-
Both use the same `machines.txt` format. See `CLAUDE.md` for cluster setup, machine file format, and troubleshooting.
|
|
File without changes
|
|
File without changes
|
|
File without changes
|