exness-data-preprocess 2.1.1__tar.gz → 3.0.1__tar.gz

{exness_data_preprocess-2.1.1 → exness_data_preprocess-3.0.1}/.github/workflows/publish.yml

@@ -10,9 +10,9 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v7
 
-      - uses: actions/setup-python@v5
+      - uses: actions/setup-python@v7
         with:
           python-version: '3.12'
 
@@ -23,7 +23,7 @@ jobs:
         run: python -m build
 
       - name: Upload distribution artifacts
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@v7
         with:
           name: dist
           path: dist/
@@ -41,7 +41,7 @@ jobs:
 
     steps:
       - name: Download distribution artifacts
-        uses: actions/download-artifact@v4
+        uses: actions/download-artifact@v8
         with:
           name: dist
           path: dist/

{exness_data_preprocess-2.1.1 → exness_data_preprocess-3.0.1}/.prototools

@@ -2,4 +2,4 @@
 # uv owns Python: the interpreter version is .python-version, and uv.toml makes uv use only its own
 # managed build. ClickHouse comes from the system PATH (e.g. Homebrew), not from proto.
 uv = "latest"
-moon = "2.6.1"
+moon = "2.6.2"

exness_data_preprocess-3.0.1/.python-version

@@ -0,0 +1 @@
+3.15

{exness_data_preprocess-2.1.1 → exness_data_preprocess-3.0.1}/CHANGELOG.md

@@ -1,3 +1,29 @@
+## [3.0.1](https://github.com/terrylica/exness-data-preprocess/compare/v3.0.0...v3.0.1) (2026-10-19)
+
+
+### Bug Fixes
+
+* **ohlc:** compute tick_count_standard and standard_spread_avg per minute from standard_ticks ([a26ec2c](https://github.com/terrylica/exness-data-preprocess/commit/a26ec2cca0e8e2b326c8b5c6f93f2c9f2a658986)), closes [#25](https://github.com/terrylica/exness-data-preprocess/issues/25)
+
+# [3.0.0](https://github.com/terrylica/exness-data-preprocess/compare/v2.1.1...v3.0.0) (2026-10-19)
+
+
+* fix!: keep distinct ticks that share a timestamp; read with FINAL; make force_redownload work ([e7ac22b](https://github.com/terrylica/exness-data-preprocess/commit/e7ac22ba659c7eb1c03f3ad1f223d23bff515304)), closes [#23](https://github.com/terrylica/exness-data-preprocess/issues/23)
+* refactor!: remove the unused config module, ExnessDataProcessor(config=...) and pyyaml ([b43be4a](https://github.com/terrylica/exness-data-preprocess/commit/b43be4a09702c6135a41b27256345952c83e6708)), closes [#21](https://github.com/terrylica/exness-data-preprocess/issues/21)
+
+
+### Bug Fixes
+
+* apply review findings (inclusive end_date day, cursor pruning, scoped ASOF build side, forced re-download) ([dd3a0b3](https://github.com/terrylica/exness-data-preprocess/commit/dd3a0b373689116d600f43ff489e3076a66eb13e)), closes [#23](https://github.com/terrylica/exness-data-preprocess/issues/23) [#24](https://github.com/terrylica/exness-data-preprocess/issues/24)
+* **query:** bind datetime parameters as DateTime64(6, 'UTC') so filters are not shifted on non-UTC servers ([1f88c41](https://github.com/terrylica/exness-data-preprocess/commit/1f88c41b372306bb5e929d131e1338d71e9ac539)), closes [#24](https://github.com/terrylica/exness-data-preprocess/issues/24)
+* report real storage_bytes and OHLC bar counts instead of placeholders ([a970be5](https://github.com/terrylica/exness-data-preprocess/commit/a970be5f75b8dbddff890e92a628388261f10026)), closes [#19](https://github.com/terrylica/exness-data-preprocess/issues/19)
+
+
+### BREAKING CHANGES
+
+* raw_spread_ticks and standard_ticks gain a `seq` column and sort by (instrument, timestamp, seq); existing tables are migrated in place by ensure_schema() (re-download months with update_data(..., force_redownload=True) to restore ticks the old key collapsed). CursorResult.next_cursor is now an opaque "<timestamp>#<seq>" string rather than a plain timestamp. Tick counts, tick queries and OHLC generation read with FINAL.
+* ExnessDataProcessor() no longer accepts a `config` argument, the exness_data_preprocess.config module (ConfigModel, load_config, get_default_config_path) is removed, and PyYAML is no longer installed as a dependency. Configure through environment variables (CLICKHOUSE_*, EXNESS_DOWNLOAD_DIR).
+
 ## [2.1.1](https://github.com/terrylica/exness-data-preprocess/compare/v2.1.0...v2.1.1) (2026-10-19)
 
 

{exness_data_preprocess-2.1.1 → exness_data_preprocess-3.0.1}/CLAUDE.md

@@ -16,7 +16,7 @@ moon run repo:build      # sdist + wheel into dist/
 
 ## Toolchain
 
-- proto pins uv (`.prototools`); uv owns Python. `.python-version` pins 3.11 and wins. If `UV_PYTHON` is exported in the shell, it overrides the pin, so run `env -u UV_PYTHON uv ...` or the venv is built on the wrong interpreter.
+- proto pins uv (`latest`) and moon (`.prototools`); uv owns Python. Development runs on the latest CPython (`.python-version`, currently 3.15); the published floor stays `requires-python >=3.11`, and ruff/mypy target 3.11 so code keeps working there. Never export `UV_PYTHON`: an env var outranks `.python-version`.
 - Releases: write `docs/release-notes/<next>.md` (narrative paragraph + Keep a Changelog bullets, one line each) in the PR before releasing; semantic-release (`npm run release`) refuses to release without it, runs `uv version <next> --frozen`, and `scripts/publish-release-notes.sh` rebuilds the GitHub Release body as narrative + generated section. pyproject.toml is the only place the version lives; `__version__` reads installed metadata. Publishing is `.github/workflows/publish.yml` (PyPI Trusted Publishing on a GitHub Release), the only publish path.
 - Dependency update PRs come from `.github/dependabot.yml`. No other CI runs in Actions; gates are local.
 
@@ -24,7 +24,8 @@ moon run repo:build      # sdist + wheel into dist/
 
 - `ensure_database_bootstrap()` must run before the first `get_client()` that binds the `exness` database (`tests/test_database_bootstrap_order.py`).
 - Insert DataFrames with `client.insert_df(...)`, never `client.insert(data=df)` (that takes row sequences).
-- Tables are ReplacingMergeTree: re-inserting a month is safe; duplicates collapse at merge time, so counts can be higher until a merge or `FINAL`.
+- Tables are ReplacingMergeTree. Tick tables dedupe on `(instrument, timestamp, seq)`: `seq` (`assign_tick_seq`) numbers same-timestamp ticks in source order, so distinct ticks sharing a timestamp are kept and re-inserting a month is idempotent. Duplicates only collapse at merge time, so every read whose answer depends on duplicates (counts, tick/OHLC rows, the OHLC build) uses `FINAL`; keep it that way.
+- Datetime query parameters are typed `DateTime64(6, 'UTC')`, never bare `DateTime64`: a bare one parses naive text in the server's time zone and shifts every filter on a non-UTC server. `repo:e2e` runs its server in America/New_York to catch this.
 - ClickHouse-backed tests are marked `clickhouse` or use `processor_with_clickhouse` (fixture users get the marker at collection, so `-m clickhouse` selects them); `conftest.py` skips them with the probe's reason unless `EXNESS_REQUIRE_CLICKHOUSE=1`. A server that holds :8123 but rejects the credentials (el02's shared system ClickHouse) counts as unavailable. Use `repo:e2e`, not that server.
 - `gapless-crypto-clickhouse` also uses localhost:8123. `clickhouse-stop` only stops the PID in this repo's pidfile.
 - ADRs and design specs under `docs/adr/` and `docs/design/` are historical records: add new ones, don't rewrite old ones.

{exness_data_preprocess-2.1.1 → exness_data_preprocess-3.0.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: exness-data-preprocess
-Version: 2.1.1
+Version: 3.0.1
 Summary: Professional Exness forex tick data preprocessing with ClickHouse backend. Provides efficient storage with lossless precision and direct queryability.
 Project-URL: Homepage, https://github.com/terrylica/exness-data-preprocess
 Project-URL: Documentation, https://github.com/terrylica/exness-data-preprocess#readme
@@ -22,15 +22,15 @@ Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.15
 Classifier: Topic :: Office/Business :: Financial :: Investment
 Classifier: Topic :: Scientific/Engineering :: Information Analysis
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Requires-Python: >=3.11
-Requires-Dist: clickhouse-connect>=0.7.0
+Requires-Dist: clickhouse-connect>=1.11.0
 Requires-Dist: exchange-calendars>=4.5.6
 Requires-Dist: pandas>=2.0.0
 Requires-Dist: pydantic>=2.0.0
-Requires-Dist: pyyaml>=6.0.0
 Provides-Extra: examples
 Requires-Dist: tqdm>=4.65.0; extra == 'examples'
 Description-Content-Type: text/markdown

exness_data_preprocess-3.0.1/docs/release-notes/3.0.0.md

@@ -0,0 +1,29 @@
+## 3.0.0 (2026-10-19)
+
+This release is about trusting the numbers this package stores and reports. The tick tables silently merged distinct Exness ticks that shared a timestamp: about 0.17% of Raw_Spread ticks on recent EURUSD months, gone without a trace. Date filters were shifted by the server's UTC offset whenever ClickHouse ran outside UTC. Storage size, the up-to-date bar count and `force_redownload` were placeholders that returned 0, ticks ÷ 1000 and nothing at all. Each is now measured and gated by `repo:e2e`, which loads real Exness months on a deliberately non-UTC server and checks that every inserted tick survives a merge, that re-loading changes nothing, and that pagination returns every tick sharing a timestamp. It is a major version for three reasons: the tick tables gain a `seq` column and a new sort key (existing tables are migrated in place automatically), the pagination cursor format changed, and the unused `config` parameter and PyYAML dependency are gone.
+
+### Upgrading
+
+- Nothing to do for the schema: the first `ExnessDataProcessor()` against a pre-3.0 database migrates both tick tables in place (`ADD COLUMN seq`, `MODIFY ORDER BY (instrument, timestamp, seq)`) and logs a warning.
+- To recover ticks the old key already merged away, re-load the affected months: `update_data(pair, start_date=..., force_redownload=True)`.
+- Remove any `ExnessDataProcessor(config=...)` argument; configure with `CLICKHOUSE_*` and `EXNESS_DOWNLOAD_DIR` environment variables.
+- Treat `CursorResult.next_cursor` as opaque (`"<timestamp>#<seq>"`); pass it back unchanged.
+- `clickhouse-connect` 1.11.0 or newer is now required.
+
+### Fixed
+
+- Distinct ticks that share a timestamp are kept: tick tables deduplicate on `(instrument, timestamp, seq)`, with `seq` numbering same-timestamp ticks in source order, so re-loading a month stays idempotent (#23).
+- Date filters are no longer shifted on non-UTC ClickHouse servers; every datetime parameter is bound as `DateTime64(6, 'UTC')` (#24).
+- A date-only `end_date` now includes that whole day, as documented; it used to stop at 00:00:00 of that day (#24).
+- Counts, tick and OHLC queries, and the OHLC build read with `FINAL`, so results are correct before ClickHouse's background merges; a re-load had reported 157,316 bars for 78,658 (#23).
+- `update_data(force_redownload=True)` now re-processes every month from `start_date`; it was previously accepted and ignored (#23).
+- Cursor pagination no longer skips ticks that share the last timestamp of a page (#23).
+- `storage_bytes` reports the real on-disk size of the package tables, and the up-to-date path of `update_data` reports the real OHLC bar count instead of ticks ÷ 1000 (#19, #20).
+
+### Removed
+
+- `exness_data_preprocess.config` (`ConfigModel`, `load_config`, `get_default_config_path`), the `ExnessDataProcessor(config=...)` parameter, and the PyYAML runtime dependency, none of which had any effect (#21, #22).
+
+### Known issues
+
+- `ohlc_1m.tick_count_standard` equals the Raw_Spread tick count of each minute rather than counting Standard ticks; this predates 3.0.0 and is tracked in #25.

exness_data_preprocess-3.0.1/docs/release-notes/3.0.1.md

@@ -0,0 +1,18 @@
+## 3.0.1 (2026-10-19)
+
+Two `ohlc_1m` columns did not measure what their documentation says. `tick_count_standard` is documented as the number of Standard ticks in the bar, but it always equalled the Raw_Spread count: it was computed as `countIf(... IS NOT NULL)` over an ASOF join that never produces NULLs. `standard_spread_avg` is documented as the average Standard spread in the bar, but it averaged whichever Standard quote preceded each Raw_Spread tick, and the choice among tied quotes could change between rebuilds. Each variant is now aggregated per minute on its own and the two are joined on the minute, so both columns match their definitions, rebuilds are deterministic, and the build no longer runs a per-tick join. On the E2E months the counts now differ on 64,889 of 78,658 bars, where before they were identical on every bar.
+
+### Upgrading
+
+- Bars already stored keep the old values until they are rebuilt. Rebuild an instrument's bars with `update_data(pair, start_date=..., force_redownload=True)`, which re-downloads and regenerates from `start_date`, or call `ExnessDataProcessor().ch_ohlc_generator.regenerate_ohlc(pair)` to rebuild every bar from the stored ticks.
+
+### Fixed
+
+- `tick_count_standard` counts the Standard ticks in the bar's minute (0 when there are none) (#25).
+- `standard_spread_avg` is the mean Standard `ask - bid` in the bar's minute (NULL when there are none), deterministic across rebuilds (#25).
+- `repo:e2e` checks that `sum(tick_count_raw_spread)` equals the stored Raw_Spread ticks and `sum(tick_count_standard)` equals the Standard ticks in bar minutes, that the two counts differ, and that a sampled `standard_spread_avg` equals a direct per-minute average (#25).
+- Stale docstrings describing the v1 30-column schema, normalized metrics and the ASOF join are corrected (#25).
+
+### Changed
+
+- Development runs on Python 3.15.0 and moon 2.6.2; the supported range is unchanged (`requires-python >=3.11`) (#27, #28).