exness-data-preprocess 2.1.1__tar.gz → 3.0.0__tar.gz

{exness_data_preprocess-2.1.1 → exness_data_preprocess-3.0.0}/.github/workflows/publish.yml

@@ -10,9 +10,9 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@v7
 
-      - uses: actions/setup-python@v5
+      - uses: actions/setup-python@v7
         with:
           python-version: '3.12'
 
@@ -23,7 +23,7 @@ jobs:
         run: python -m build
 
       - name: Upload distribution artifacts
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@v7
         with:
           name: dist
           path: dist/
@@ -41,7 +41,7 @@ jobs:
 
     steps:
       - name: Download distribution artifacts
-        uses: actions/download-artifact@v4
+        uses: actions/download-artifact@v8
         with:
           name: dist
           path: dist/

{exness_data_preprocess-2.1.1 → exness_data_preprocess-3.0.0}/CHANGELOG.md

@@ -1,3 +1,22 @@
+# [3.0.0](https://github.com/terrylica/exness-data-preprocess/compare/v2.1.1...v3.0.0) (2026-10-19)
+
+
+* fix!: keep distinct ticks that share a timestamp; read with FINAL; make force_redownload work ([e7ac22b](https://github.com/terrylica/exness-data-preprocess/commit/e7ac22ba659c7eb1c03f3ad1f223d23bff515304)), closes [#23](https://github.com/terrylica/exness-data-preprocess/issues/23)
+* refactor!: remove the unused config module, ExnessDataProcessor(config=...) and pyyaml ([b43be4a](https://github.com/terrylica/exness-data-preprocess/commit/b43be4a09702c6135a41b27256345952c83e6708)), closes [#21](https://github.com/terrylica/exness-data-preprocess/issues/21)
+
+
+### Bug Fixes
+
+* apply review findings (inclusive end_date day, cursor pruning, scoped ASOF build side, forced re-download) ([dd3a0b3](https://github.com/terrylica/exness-data-preprocess/commit/dd3a0b373689116d600f43ff489e3076a66eb13e)), closes [#23](https://github.com/terrylica/exness-data-preprocess/issues/23) [#24](https://github.com/terrylica/exness-data-preprocess/issues/24)
+* **query:** bind datetime parameters as DateTime64(6, 'UTC') so filters are not shifted on non-UTC servers ([1f88c41](https://github.com/terrylica/exness-data-preprocess/commit/1f88c41b372306bb5e929d131e1338d71e9ac539)), closes [#24](https://github.com/terrylica/exness-data-preprocess/issues/24)
+* report real storage_bytes and OHLC bar counts instead of placeholders ([a970be5](https://github.com/terrylica/exness-data-preprocess/commit/a970be5f75b8dbddff890e92a628388261f10026)), closes [#19](https://github.com/terrylica/exness-data-preprocess/issues/19)
+
+
+### BREAKING CHANGES
+
+* raw_spread_ticks and standard_ticks gain a `seq` column and sort by (instrument, timestamp, seq); existing tables are migrated in place by ensure_schema() (re-download months with update_data(..., force_redownload=True) to restore ticks the old key collapsed). CursorResult.next_cursor is now an opaque "<timestamp>#<seq>" string rather than a plain timestamp. Tick counts, tick queries and OHLC generation read with FINAL.
+* ExnessDataProcessor() no longer accepts a `config` argument, the exness_data_preprocess.config module (ConfigModel, load_config, get_default_config_path) is removed, and PyYAML is no longer installed as a dependency. Configure through environment variables (CLICKHOUSE_*, EXNESS_DOWNLOAD_DIR).
+
 ## [2.1.1](https://github.com/terrylica/exness-data-preprocess/compare/v2.1.0...v2.1.1) (2026-10-19)
 
 

{exness_data_preprocess-2.1.1 → exness_data_preprocess-3.0.0}/CLAUDE.md

@@ -24,7 +24,8 @@ moon run repo:build      # sdist + wheel into dist/
 
 - `ensure_database_bootstrap()` must run before the first `get_client()` that binds the `exness` database (`tests/test_database_bootstrap_order.py`).
 - Insert DataFrames with `client.insert_df(...)`, never `client.insert(data=df)` (that takes row sequences).
-- Tables are ReplacingMergeTree: re-inserting a month is safe; duplicates collapse at merge time, so counts can be higher until a merge or `FINAL`.
+- Tables are ReplacingMergeTree. Tick tables dedupe on `(instrument, timestamp, seq)`: `seq` (`assign_tick_seq`) numbers same-timestamp ticks in source order, so distinct ticks sharing a timestamp are kept and re-inserting a month is idempotent. Duplicates only collapse at merge time, so every read whose answer depends on duplicates (counts, tick/OHLC rows, the OHLC build) uses `FINAL`; keep it that way.
+- Datetime query parameters are typed `DateTime64(6, 'UTC')`, never bare `DateTime64`: a bare one parses naive text in the server's time zone and shifts every filter on a non-UTC server. `repo:e2e` runs its server in America/New_York to catch this.
 - ClickHouse-backed tests are marked `clickhouse` or use `processor_with_clickhouse` (fixture users get the marker at collection, so `-m clickhouse` selects them); `conftest.py` skips them with the probe's reason unless `EXNESS_REQUIRE_CLICKHOUSE=1`. A server that holds :8123 but rejects the credentials (el02's shared system ClickHouse) counts as unavailable. Use `repo:e2e`, not that server.
 - `gapless-crypto-clickhouse` also uses localhost:8123. `clickhouse-stop` only stops the PID in this repo's pidfile.
 - ADRs and design specs under `docs/adr/` and `docs/design/` are historical records: add new ones, don't rewrite old ones.

{exness_data_preprocess-2.1.1 → exness_data_preprocess-3.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: exness-data-preprocess
-Version: 2.1.1
+Version: 3.0.0
 Summary: Professional Exness forex tick data preprocessing with ClickHouse backend. Provides efficient storage with lossless precision and direct queryability.
 Project-URL: Homepage, https://github.com/terrylica/exness-data-preprocess
 Project-URL: Documentation, https://github.com/terrylica/exness-data-preprocess#readme
@@ -26,11 +26,10 @@ Classifier: Topic :: Office/Business :: Financial :: Investment
 Classifier: Topic :: Scientific/Engineering :: Information Analysis
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Requires-Python: >=3.11
-Requires-Dist: clickhouse-connect>=0.7.0
+Requires-Dist: clickhouse-connect>=1.11.0
 Requires-Dist: exchange-calendars>=4.5.6
 Requires-Dist: pandas>=2.0.0
 Requires-Dist: pydantic>=2.0.0
-Requires-Dist: pyyaml>=6.0.0
 Provides-Extra: examples
 Requires-Dist: tqdm>=4.65.0; extra == 'examples'
 Description-Content-Type: text/markdown

exness_data_preprocess-3.0.0/docs/release-notes/3.0.0.md

@@ -0,0 +1,29 @@
+## 3.0.0 (2026-10-19)
+
+This release is about trusting the numbers this package stores and reports. The tick tables silently merged distinct Exness ticks that shared a timestamp: about 0.17% of Raw_Spread ticks on recent EURUSD months, gone without a trace. Date filters were shifted by the server's UTC offset whenever ClickHouse ran outside UTC. Storage size, the up-to-date bar count and `force_redownload` were placeholders that returned 0, ticks ÷ 1000 and nothing at all. Each is now measured and gated by `repo:e2e`, which loads real Exness months on a deliberately non-UTC server and checks that every inserted tick survives a merge, that re-loading changes nothing, and that pagination returns every tick sharing a timestamp. It is a major version for three reasons: the tick tables gain a `seq` column and a new sort key (existing tables are migrated in place automatically), the pagination cursor format changed, and the unused `config` parameter and PyYAML dependency are gone.
+
+### Upgrading
+
+- Nothing to do for the schema: the first `ExnessDataProcessor()` against a pre-3.0 database migrates both tick tables in place (`ADD COLUMN seq`, `MODIFY ORDER BY (instrument, timestamp, seq)`) and logs a warning.
+- To recover ticks the old key already merged away, re-load the affected months: `update_data(pair, start_date=..., force_redownload=True)`.
+- Remove any `ExnessDataProcessor(config=...)` argument; configure with `CLICKHOUSE_*` and `EXNESS_DOWNLOAD_DIR` environment variables.
+- Treat `CursorResult.next_cursor` as opaque (`"<timestamp>#<seq>"`); pass it back unchanged.
+- `clickhouse-connect` 1.11.0 or newer is now required.
+
+### Fixed
+
+- Distinct ticks that share a timestamp are kept: tick tables deduplicate on `(instrument, timestamp, seq)`, with `seq` numbering same-timestamp ticks in source order, so re-loading a month stays idempotent (#23).
+- Date filters are no longer shifted on non-UTC ClickHouse servers; every datetime parameter is bound as `DateTime64(6, 'UTC')` (#24).
+- A date-only `end_date` now includes that whole day, as documented; it used to stop at 00:00:00 of that day (#24).
+- Counts, tick and OHLC queries, and the OHLC build read with `FINAL`, so results are correct before ClickHouse's background merges; a re-load had reported 157,316 bars for 78,658 (#23).
+- `update_data(force_redownload=True)` now re-processes every month from `start_date`; it was previously accepted and ignored (#23).
+- Cursor pagination no longer skips ticks that share the last timestamp of a page (#23).
+- `storage_bytes` reports the real on-disk size of the package tables, and the up-to-date path of `update_data` reports the real OHLC bar count instead of ticks ÷ 1000 (#19, #20).
+
+### Removed
+
+- `exness_data_preprocess.config` (`ConfigModel`, `load_config`, `get_default_config_path`), the `ExnessDataProcessor(config=...)` parameter, and the PyYAML runtime dependency, none of which had any effect (#21, #22).
+
+### Known issues
+
+- `ohlc_1m.tick_count_standard` equals the Raw_Spread tick count of each minute rather than counting Standard ticks; this predates 3.0.0 and is tracked in #25.