exness-data-preprocess 0.7.2__tar.gz → 1.0.0__tar.gz

{exness_data_preprocess-0.7.2 → exness_data_preprocess-1.0.0}/.gitignore

@@ -100,3 +100,10 @@ uv.lock
 # Pre-commit
 # .pre-commit-config.yaml should be tracked, not ignored
 .token-info.md
+
+# Node.js (semantic-release)
+node_modules/
+
+# Link checking (lychee)
+.lychee*
+.lycheecache

exness_data_preprocess-1.0.0/.lint-relative-paths-results.txt

@@ -0,0 +1,62 @@
+🔍 Scanning for non-repo-relative paths in: /Users/terryli/eon/exness-data-preprocess
+   Violations: paths not starting with / or http(s)
+
+❌ Found 141 violation(s):
+
+/Users/terryli/eon/exness-data-preprocess/README.md:112:- **Column Definitions**: See [`schema.py`](src/exness_data_preprocess/schema.py) - Single source of truth
+/Users/terryli/eon/exness-data-preprocess/README.md:113:- **Comprehensive Reference**: See [`DATABASE_SCHEMA.md`](docs/DATABASE_SCHEMA.md) - Query examples and usage patterns
+/Users/terryli/eon/exness-data-preprocess/README.md:401:- Validation: [`docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md`](docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md)
+/Users/terryli/eon/exness-data-preprocess/README.md:409:- SSoT: [`docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml`](docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml)
+/Users/terryli/eon/exness-data-preprocess/README.md:410:- Validation: [`docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md`](docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md)
+/Users/terryli/eon/exness-data-preprocess/README.md:417:- SSoT: [`docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml`](docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml)
+/Users/terryli/eon/exness-data-preprocess/README.md:419:**Release Notes**: See [`CHANGELOG.md`](CHANGELOG.md) for complete v0.5.0 details
+/Users/terryli/eon/exness-data-preprocess/README.md:485:**[📚 Complete Documentation Hub](docs/README.md)** - Organized guide from beginner to advanced (72+ documents)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:7:**Full Documentation**: [`README.md`](README.md) - Installation, usage, API reference
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:15:- **[README.md](README.md)** - Installation, API reference, usage examples
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:16:- **[examples/basic_usage.py](examples/basic_usage.py)** - Download, query, coverage operations
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:17:- **[examples/batch_processing.py](examples/batch_processing.py)** - Multi-instrument parallel processing
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:21:- **[docs/README.md](docs/README.md)** - Architecture, planning, research findings
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:22:- **[docs/MODULE_ARCHITECTURE.md](docs/MODULE_ARCHITECTURE.md)** - Complete module documentation with SLOs (v1.7.0)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:23:- **[docs/DATABASE_SCHEMA.md](docs/DATABASE_SCHEMA.md)** - Complete database schema with self-documentation
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:24:- **[docs/RESEARCH_PATTERNS.md](docs/RESEARCH_PATTERNS.md)** - Research lifecycle and tool selection (v1.0.0)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:25:- **[docs/UNIFIED_DUCKDB_PLAN_v2.md](docs/UNIFIED_DUCKDB_PLAN_v2.md)** - v2.0.0 architecture specification
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:26:- **[docs/EXNESS_DATA_SOURCES.md](docs/EXNESS_DATA_SOURCES.md)** - Data source variants and URLs
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:27:- **[Makefile](Makefile)** - Module introspection commands (module-stats, module-complexity, module-deps)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:31:- **[docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml](docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml)** - SSoT plan v2.0.0 (2.2x speedup)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:32:- **[docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml](docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml)** - SSoT plan v2.0.0 (complete coverage)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:33:- **[docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md](docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md)** - Phase 1 validation
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:34:- **[docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md](docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md)** - Phase 2 validation
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:40:- **[CONTRIBUTING.md](CONTRIBUTING.md)** - Contribution guidelines
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:55:**Complete Specification**: [`docs/UNIFIED_DUCKDB_PLAN_v2.md`](docs/UNIFIED_DUCKDB_PLAN_v2.md)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:76:**Complete Details**: [`docs/MODULE_ARCHITECTURE.md`](docs/MODULE_ARCHITECTURE.md)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:83:  - SSoT: [`docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md`](docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:89:  - SSoT: [`docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml`](docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:90:  - Validation: [`docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md`](docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:96:  - SSoT: [`docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml`](docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:109:**Complete Schema**: [`docs/DATABASE_SCHEMA.md`](docs/DATABASE_SCHEMA.md)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:118:**Complete Guide**: [`docs/EXNESS_DATA_SOURCES.md`](docs/EXNESS_DATA_SOURCES.md)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:128:**Complete Specification**: [`docs/RESEARCH_PATTERNS.md`](docs/RESEARCH_PATTERNS.md)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:196:**Complete Development Guide**: [`README.md`](README.md) (Installation section)
+/Users/terryli/eon/exness-data-preprocess/tests/README.md:30:- **New**: Phase7 13-column (v1.2.0) OHLC - See [`../src/exness_data_preprocess/schema.py`](../src/exness_data_preprocess/schema.py)
+/Users/terryli/eon/exness-data-preprocess/docs/EXNESS_DATA_SOURCES.md:310:**Full Specification**: [`docs/research/eurusd-zero-spread-deviations/data/plan/phase7_bid_ohlc_construction_v1.1.0.md`](research/eurusd-zero-spread-deviations/data/plan/phase7_bid_ohlc_construction_v1.1.0.md)
+/Users/terryli/eon/exness-data-preprocess/docs/EXNESS_DATA_SOURCES.md:386:- **Phase7 Methodology**: [`docs/research/eurusd-zero-spread-deviations/01-methodology.md`](research/eurusd-zero-spread-deviations/01-methodology.md)
+/Users/terryli/eon/exness-data-preprocess/docs/EXNESS_DATA_SOURCES.md:387:- **Zero-Spread Research**: [`docs/research/eurusd-zero-spread-deviations/README.md`](research/eurusd-zero-spread-deviations/README.md)
+/Users/terryli/eon/exness-data-preprocess/docs/EXNESS_DATA_SOURCES.md:391:- **Current Processor**: [`src/exness_data_preprocess/processor.py`](../src/exness_data_preprocess/processor.py) (needs refactoring)
+/Users/terryli/eon/exness-data-preprocess/docs/MODULE_ARCHITECTURE.md:5:**Related**: [`README.md`](../README.md) - Architecture overview
+/Users/terryli/eon/exness-data-preprocess/docs/MODULE_ARCHITECTURE.md:43:- See [`docs/validation/ARCHITECTURE_AUDIT_2025-10-17.md`](validation/ARCHITECTURE_AUDIT_2025-10-17.md) for complete audit findings
+/Users/terryli/eon/exness-data-preprocess/docs/MODULE_ARCHITECTURE.md:964:See [`Makefile`](../Makefile) for implementation.
+/Users/terryli/eon/exness-data-preprocess/docs/MODULE_ARCHITECTURE.md:1076:- **[README.md](../README.md)** - User-facing API reference
+/Users/terryli/eon/exness-data-preprocess/docs/MODULE_ARCHITECTURE.md:1077:- **[docs/README.md](README.md)** - Documentation hub
+/Users/terryli/eon/exness-data-preprocess/docs/MODULE_ARCHITECTURE.md:1078:- **[docs/DATABASE_SCHEMA.md](DATABASE_SCHEMA.md)** - Complete database schema
+/Users/terryli/eon/exness-data-preprocess/docs/MODULE_ARCHITECTURE.md:1079:- **[docs/UNIFIED_DUCKDB_PLAN_v2.md](UNIFIED_DUCKDB_PLAN_v2.md)** - v2.0.0 architecture specification
+/Users/terryli/eon/exness-data-preprocess/docs/MODULE_ARCHITECTURE.md:1080:- **[Makefile](../Makefile)** - Module introspection commands
+/Users/terryli/eon/exness-data-preprocess/docs/MODULE_ARCHITECTURE.md:1082:- **[docs/validation/ARCHITECTURE_AUDIT_2025-10-17.md](validation/ARCHITECTURE_AUDIT_2025-10-17.md)** - Complete audit findings and corrections
+/Users/terryli/eon/exness-data-preprocess/docs/DATABASE_SCHEMA.md:244:**Column Definitions**: See [`../src/exness_data_preprocess/schema.py`](../src/exness_data_preprocess/schema.py) for complete column definitions with types and descriptions.
+/Users/terryli/eon/exness-data-preprocess/docs/DATABASE_SCHEMA.md:258:**Architecture**: Exchange Registry Pattern (v1.6.0) - Session columns dynamically generated from centralized EXCHANGES dict in [`exchanges.py`](../src/exness_data_preprocess/exchanges.py)
+
+... and 91 more (showing first 50)
+
+📝 Fix: Convert relative paths to repo-relative paths starting with /
+   WRONG: [Link](../docs/foo.md)
+   WRONG: [Link](docs/foo.md)
+   RIGHT: [Link](/docs/foo.md)

exness_data_preprocess-1.0.0/.releaserc.yml

@@ -0,0 +1,22 @@
+# semantic-release configuration for Python project
+# Uses @semantic-release/exec instead of @semantic-release/npm
+branches:
+  - main
+
+plugins:
+  - "@semantic-release/commit-analyzer"
+  - "@semantic-release/release-notes-generator"
+  - "@semantic-release/changelog"
+  - - "@semantic-release/exec"
+    - prepareCmd: |
+        # Update version in pyproject.toml
+        sed -i '' 's/^version = ".*"/version = "${nextRelease.version}"/' pyproject.toml
+        # Update version in __init__.py
+        sed -i '' 's/^__version__ = ".*"/__version__ = "${nextRelease.version}"/' src/exness_data_preprocess/__init__.py
+  - - "@semantic-release/git"
+    - assets:
+        - pyproject.toml
+        - src/exness_data_preprocess/__init__.py
+        - CHANGELOG.md
+      message: "chore(release): bump version ${lastRelease.version} → ${nextRelease.version} [skip ci]"
+  - "@semantic-release/github"

{exness_data_preprocess-0.7.2 → exness_data_preprocess-1.0.0}/CHANGELOG.md

@@ -1,3 +1,34 @@
+# [1.0.0](https://github.com/terrylica/exness-data-preprocess/compare/v0.8.0...v1.0.0) (2025-12-10)
+
+
+* feat!: prune codebase - remove unused deps, archive docs, extract mixin ([e0d48b8](https://github.com/terrylica/exness-data-preprocess/commit/e0d48b8fe7ba44d2c3bc326661102a0b3d3b33b7))
+
+
+### BREAKING CHANGES
+
+* Remove deprecated api.py and cli.py modules
+
+Dependency cleanup:
+- Remove httpx, pyarrow, polars (unused)
+- Move tqdm to optional [examples] extra
+- Consolidate dev dependencies to [dependency-groups]
+
+Documentation archival:
+- Archive 20 planning/research docs to docs/archive/
+- Update .gitignore for node_modules, lychee patterns
+
+Code refactoring:
+- Extract ClickHouseClientMixin from 4 modules (~45 lines saved)
+- Delete api.py (304 lines) and cli.py (252 lines)
+
+ADR: docs/adr/2025-12-09-codebase-pruning.md
+
+# [0.8.0](https://github.com/terrylica/exness-data-preprocess/compare/v0.7.2...v0.8.0) (2025-12-10)
+
+### Features
+
+- add ClickHouse backend for cloud deployment ([c9b51a3](https://github.com/terrylica/exness-data-preprocess/commit/c9b51a3b374eea74375cc86afdd8cb9391fe73e8))
+
 # Changelog
 
 All notable changes to exness-data-preprocess will be documented in this file.
@@ -5,12 +36,10 @@ All notable changes to exness-data-preprocess will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-
 ### 💅 Code Style
 
 - Format markdown files with prettier Standardized formatting for CHANGELOG.md.
 
-
 ### 📚 Documentation
 
 - Move test plan to docs/plans/ - Move tests/archive/E2E_TESTING_PLAN_v1.0.0.md -> docs/plans/ - Remove empty tests/archive/ directory - Consistent with docs/ organization strategy Impact: Planning docs consolidated in docs/plans/ Risk: Low (internal doc, no external references)
@@ -19,7 +48,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - Remove redundant DOCUMENTATION.md - Remove DOCUMENTATION.md (content duplicates docs/README.md) - Update README.md link to point to docs/README.md - Both files serve as navigation hubs with identical purpose Impact: -1 root file, single documentation entry point Risk: Low (historical CHANGELOG references preserved, README updated)
 
-- Organize phase plans into docs/phases/ - Move PHASE2*.yaml, PHASE3*.yaml, PHASE4*.yaml -> docs/phases/ - Update 4 references in CLAUDE.md - Consolidate SSoT planning files in dedicated subdirectory Impact: -3 docs/ root files, improved organization Risk: Low (all references updated, git mv preserves history)
+- Organize phase plans into docs/phases/ - Move PHASE2*.yaml, PHASE3*.yaml, PHASE4\*.yaml -> docs/phases/ - Update 4 references in CLAUDE.md - Consolidate SSoT planning files in dedicated subdirectory Impact: -3 docs/ root files, improved organization Risk: Low (all references updated, git mv preserves history)
 
 - Move GitHub/PyPI setup to docs/setup/ - Move GITHUB_PYPI_SETUP.md -> docs/setup/ - No active references in README.md or CONTRIBUTING.md - Historical CHANGELOG references preserved Impact: -1 root file, infrastructure docs consolidated Risk: Low (no external links, CHANGELOG preserved)
 
@@ -31,7 +60,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - Update refactoring SSoT plans to completion status - REPOSITORY_REFACTORING_PLAN.yaml: v2.0.0 -> v3.0.0 - PRIORITY3_CLEANUP_PLAN.yaml: v1.0.0 -> v2.0.0 (Completed) Combined Priority 1-3 Impact: - Root MD files: 10 -> 4 (-60%) - docs/ root files: 12 -> 8 (-33%) - Total clutter: 30 -> 12 items (-60%) Total: 11 commits, ~40 minutes, zero regressions
 
-
 ### 🧰 Maintenance
 
 - Untrack lychee cache files - Remove .lychee-results.json, .lychee-results.txt, .lycheecache from tracking - Files remain gitignored via .gitignore rules - Add REPOSITORY_REFACTORING_PLAN.yaml SSoT (v1.0.0) Impact: Cleaner git status, prevent cache pollution Risk: None (cache files, no external references)

{exness_data_preprocess-0.7.2 → exness_data_preprocess-1.0.0}/CLAUDE.md

@@ -56,7 +56,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 
 ### Module Pattern (v1.7.0)
 
-- **Facade orchestrator** (processor.py) coordinating 7 specialized modules
+- **Facade orchestrator** (processor.py) coordinating 6 module instances + 1 static utility
 - **Separation of concerns** with single-responsibility modules
 - **SLO-based design** (Availability, Correctness, Observability, Maintainability)
 - **Off-the-shelf libraries** (httpx, pandas, DuckDB, exchange_calendars)

{exness_data_preprocess-0.7.2 → exness_data_preprocess-1.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: exness-data-preprocess
-Version: 0.7.2
+Version: 1.0.0
 Summary: Professional Exness forex tick data preprocessing with optimal compression (Parquet Zstd-22) and DuckDB OHLC generation. Provides efficient storage (9% smaller than ZIP) with lossless precision and direct queryability.
 Project-URL: Homepage, https://github.com/terrylica/exness-data-preprocess
 Project-URL: Documentation, https://github.com/terrylica/exness-data-preprocess#readme
@@ -12,39 +12,28 @@ Author-email: Eon Labs <terry@eonlabs.com>
 Maintainer-email: Terry Li <terry@eonlabs.com>
 License: MIT
 License-File: LICENSE
-Keywords: backtesting,compression,data-engineering,duckdb,efficient-storage,eurusd,exness,financial-data,forex,forex-data,lossless,microstructure,ohlc,pandas,parquet,preprocessing,pyarrow,quantitative-finance,queryable,tick-data,time-series,zstd
+Keywords: backtesting,compression,data-engineering,duckdb,efficient-storage,eurusd,exness,financial-data,forex,forex-data,lossless,microstructure,ohlc,pandas,parquet,preprocessing,quantitative-finance,queryable,tick-data,time-series,zstd
 Classifier: Development Status :: 3 - Alpha
 Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Financial and Insurance Industry
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Classifier: Topic :: Office/Business :: Financial :: Investment
 Classifier: Topic :: Scientific/Engineering :: Information Analysis
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
-Requires-Python: >=3.9
+Requires-Python: >=3.11
+Requires-Dist: clickhouse-connect>=0.7.0
 Requires-Dist: duckdb>=0.9.0
 Requires-Dist: exchange-calendars>=4.5.6
-Requires-Dist: httpx>=0.25.0
 Requires-Dist: pandas>=2.0.0
-Requires-Dist: pyarrow>=16.0.0
 Requires-Dist: pydantic>=2.0.0
 Requires-Dist: pyyaml>=6.0.0
-Requires-Dist: tqdm>=4.65.0
-Provides-Extra: dev
-Requires-Dist: black>=23.0.0; extra == 'dev'
-Requires-Dist: mypy>=1.5.0; extra == 'dev'
-Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
-Requires-Dist: pytest>=7.0.0; extra == 'dev'
-Requires-Dist: ruff>=0.1.0; extra == 'dev'
-Provides-Extra: test
-Requires-Dist: pytest-cov>=4.0.0; extra == 'test'
-Requires-Dist: pytest>=7.0.0; extra == 'test'
+Provides-Extra: examples
+Requires-Dist: tqdm>=4.65.0; extra == 'examples'
 Description-Content-Type: text/markdown
 
 # Exness Data Preprocess v2.0.0
@@ -455,7 +444,7 @@ Data is sourced from Exness's public tick data repository:
 - Vectorized approach: 2.69s (302K bars, 10 exchanges)
 - Combined Phase 1+2: ~16x total speedup (8.05s → 0.50s)
 - Implementation: Pre-compute trading minutes, vectorized `.isin()` lookup
-- SSoT: [`docs/PHASE2_SESSION_VECTORIZATION_PLAN.yaml`](docs/PHASE2_SESSION_VECTORIZATION_PLAN.yaml)
+- SSoT: [`docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml`](docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml)
 - Validation: [`docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md`](docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md)
 
 **SQL Gap Detection** - Complete coverage with 46% code reduction:
@@ -463,7 +452,7 @@ Data is sourced from Exness's public tick data repository:
 - Bug fix: Python approach missed internal gaps (41 detected vs 42 actual)
 - SQL EXCEPT operator detects ALL gaps (before + within + after existing data)
 - Code reduced from 62 lines to 34 lines (46% reduction)
-- SSoT: [`docs/PHASE3_SQL_GAP_DETECTION_PLAN.yaml`](docs/PHASE3_SQL_GAP_DETECTION_PLAN.yaml)
+- SSoT: [`docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml`](docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml)
 
 **Release Notes**: See [`CHANGELOG.md`](CHANGELOG.md) for complete v0.5.0 details
 

{exness_data_preprocess-0.7.2 → exness_data_preprocess-1.0.0}/README.md

@@ -406,7 +406,7 @@ Data is sourced from Exness's public tick data repository:
 - Vectorized approach: 2.69s (302K bars, 10 exchanges)
 - Combined Phase 1+2: ~16x total speedup (8.05s → 0.50s)
 - Implementation: Pre-compute trading minutes, vectorized `.isin()` lookup
-- SSoT: [`docs/PHASE2_SESSION_VECTORIZATION_PLAN.yaml`](docs/PHASE2_SESSION_VECTORIZATION_PLAN.yaml)
+- SSoT: [`docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml`](docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml)
 - Validation: [`docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md`](docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md)
 
 **SQL Gap Detection** - Complete coverage with 46% code reduction:
@@ -414,7 +414,7 @@ Data is sourced from Exness's public tick data repository:
 - Bug fix: Python approach missed internal gaps (41 detected vs 42 actual)
 - SQL EXCEPT operator detects ALL gaps (before + within + after existing data)
 - Code reduced from 62 lines to 34 lines (46% reduction)
-- SSoT: [`docs/PHASE3_SQL_GAP_DETECTION_PLAN.yaml`](docs/PHASE3_SQL_GAP_DETECTION_PLAN.yaml)
+- SSoT: [`docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml`](docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml)
 
 **Release Notes**: See [`CHANGELOG.md`](CHANGELOG.md) for complete v0.5.0 details
 

{exness_data_preprocess-0.7.2 → exness_data_preprocess-1.0.0}/docs/README.md

@@ -55,11 +55,6 @@
 
 ### Implementation Architecture v1.7.0
 
-**Refactoring Plans**:
-
-- [`plans/PHASE7_v1.6.0_REFACTORING_PROGRESS.md`](plans/PHASE7_v1.6.0_REFACTORING_PROGRESS.md) - Complete refactoring progress
-- [`plans/REFACTORING_CHECKLIST.md`](plans/REFACTORING_CHECKLIST.md) - Quick reference checklist
-
 **Pattern**: Facade with 7 specialized modules
 
 **Key Modules**:
@@ -261,5 +256,11 @@ curl -s "https://ticks.ex2archive.com/ticks/" | jq -r '.[] | .name' | grep -i "E
 
 ---
 
-**Last Updated**: 2025-10-18
+## Archived Documentation
+
+Historical planning, research, and completed phase documents are archived in [`archive/`](archive/).
+
+---
+
+**Last Updated**: 2025-12-09
 **Maintainer**: Terry Li <terry@eonlabs.com>

exness_data_preprocess-1.0.0/docs/adr/2025-12-09-codebase-pruning.md

@@ -0,0 +1,150 @@
+---
+status: accepted
+date: 2025-12-09
+decision-maker: Terry Li
+consulted:
+  [Code-Pruning-Agent, Dependency-Pruning-Agent, Documentation-Pruning-Agent]
+research-method: multi-agent
+clarification-iterations: 1
+perspectives: [Maintainability, Technical-Debt, Developer-Experience]
+---
+
+# Codebase Pruning: Dependencies, Documentation, and Code Cleanup
+
+**Design Spec**: [Implementation Spec](/docs/design/2025-12-09-codebase-pruning/spec.md)
+
+## Context and Problem Statement
+
+After the ClickHouse migration (v0.8.0), the codebase has accumulated:
+
+1. **Unused dependencies**: httpx (never imported), polars (abandoned experiment), pyarrow (research-only)
+2. **Orphaned documentation**: 18 planning documents from completed Oct 2025 phases
+3. **Code duplication**: 60+ lines of repeated client lifecycle code across ClickHouse modules
+4. **Deprecated API layer**: 304-line api.py marked for removal
+
+This technical debt increases install size by 80+ MB, confuses navigation, and violates DRY principles.
+
+## Decision Drivers
+
+- **Install size**: 200 MB → 120 MB (40% reduction)
+- **Codebase clarity**: Remove orphaned files that confuse AI assistants
+- **DRY compliance**: Eliminate duplicate ClickHouse client lifecycle code
+- **Breaking change acceptance**: User approved api.py removal
+
+## Considered Options
+
+1. **Quick wins only**: Dependencies + documentation archival
+2. **Quick wins + code refactoring**: Above + ClickHouseClientMixin
+3. **Full pruning** (SELECTED): All above + remove deprecated api.py
+
+## Decision Outcome
+
+**Chosen option**: Full pruning including api.py removal
+
+### Consequences
+
+**Good**:
+
+- 80+ MB smaller install size
+- 400+ lines of code removed
+- Cleaner documentation structure
+- DRY-compliant ClickHouse modules
+
+**Bad**:
+
+- api.py removal is breaking change for any external consumers
+- Research scripts in docs/research/ will need manual pyarrow install
+
+## Architecture
+
+### Before/After Diagram
+
+<!-- graph-easy source:
+[ pyproject.toml ] -- httpx (unused) --> [ 1 MB waste ]
+
+[ pyproject.toml ] -- polars (unused) --> [ 50 MB waste ]
+[ pyproject.toml ] -- pyarrow (research) --> [ 30 MB waste ]
+[ docs/plans/ ] -- 15 orphaned files --> [ 7,405 lines ]
+[ clickhouse_*.py (4) ] -- duplicate lifecycle --> [ 60 lines ]
+[ api.py ] -- deprecated --> [ 304 lines ]
+-->
+
+```
++------------------+     httpx (unused)      +-------------+
+| pyproject.toml   | ----------------------> | 1 MB waste  |
++------------------+                         +-------------+
+        |
+        |            polars (unused)         +-------------+
+        +------------------------------->    | 50 MB waste |
+        |                                    +-------------+
+        |
+        |           pyarrow (research)       +-------------+
+        +------------------------------->    | 30 MB waste |
+                                             +-------------+
+
++------------------+   15 orphaned files     +-------------+
+|   docs/plans/    | ----------------------> | 7,405 lines |
++------------------+                         +-------------+
+
++------------------+  duplicate lifecycle    +-------------+
+|clickhouse_*.py(4)| ----------------------> |  60 lines   |
++------------------+                         +-------------+
+
++------------------+     deprecated          +-------------+
+|     api.py       | ----------------------> | 304 lines   |
++------------------+                         +-------------+
+```
+
+### After Pruning
+
+<!-- graph-easy source:
+[ pyproject.toml ] -- clean deps --> [ 120 MB install ]
+
+[ docs/archive/ ] -- preserved --> [ Historical context ]
+[ clickhouse_base.py ] -- mixin --> [ DRY code ]
+[ api.py ] -- removed --> [ Breaking change logged ]
+-->
+
+```
++------------------+     clean deps          +----------------+
+| pyproject.toml   | ----------------------> | 120 MB install |
++------------------+                         +----------------+
+
++------------------+     preserved           +------------------+
+|  docs/archive/   | ----------------------> | Historical ctxt  |
++------------------+                         +------------------+
+
++--------------------+      mixin            +----------+
+| clickhouse_base.py | --------------------> | DRY code |
++--------------------+                       +----------+
+
++------------------+     removed             +-------------------+
+|     api.py       | ----------------------> | Breaking chg log  |
++------------------+                         +-------------------+
+```
+
+## Implementation Details
+
+### Phase A: Dependency Cleanup
+
+- Remove: httpx, pyarrow, polars
+- Move: tqdm to `[project.optional-dependencies] examples`
+- Consolidate: dev dependencies to `[dependency-groups]`
+
+### Phase B: Documentation Cleanup
+
+- Update .gitignore: node_modules/, .lychee\*, .token-info.md
+- Archive: 18 planning documents to docs/archive/
+- Update: docs/README.md
+
+### Phase C: Code Cleanup
+
+- Create: `clickhouse_base.py` with `ClickHouseClientMixin`
+- Refactor: 4 ClickHouse modules to use mixin
+- Delete: `api.py` (304 lines)
+- Update: `__init__.py` exports
+
+## More Information
+
+- Related: [ClickHouse Migration ADR](/docs/adr/2025-12-09-exness-clickhouse-migration.md)
+- Source Plan: `~/.claude/plans/lively-splashing-steele.md`

exness_data_preprocess-1.0.0/docs/adr/2025-12-09-exness-clickhouse-migration.md

@@ -0,0 +1,196 @@
+---
+status: accepted
+date: 2025-12-09
+decision-maker: Terry Li
+consulted:
+  [
+    ZSTD-vs-LZ4-Codec-Agent,
+    Delta-Gorilla-Corruption-Agent,
+    Dictionary-vs-JOIN-Agent,
+    AI-Schema-Documentation-Agent,
+    ClickHouse-Architect-Skill-Agent,
+  ]
+research-method: 9-agent-parallel-dctl
+clarification-iterations: 4
+perspectives:
+  [Performance, Simplicity, AICodeGeneration, DataModeling, FutureScalability]
+---
+
+# ADR: Exness Data Preprocess DuckDB to ClickHouse Migration
+
+**Design Spec**: [Implementation Spec](/docs/design/2025-12-09-exness-clickhouse-migration/spec.md)
+
+## Context and Problem Statement
+
+The exness-data-preprocess project currently uses embedded DuckDB for storing forex tick data and OHLC bars. While DuckDB works well for single-file analytics, the project needs to migrate to ClickHouse to:
+
+1. **Unify infrastructure** with other ClickHouse-based projects (deribit data, MLflow tracking)
+2. **Support future scale** (100+ instruments, multi-terabyte datasets)
+3. **Enable real-time dashboards** with sub-second query latency
+4. **Leverage ClickHouse-native features** (ReplacingMergeTree, parameterized views, compression codecs)
+
+The migration must preserve all existing functionality while adopting ClickHouse-native patterns validated through independent research.
+
+### Before/After
+
+```
+   ┌────────────┐            ┌────────────┐
+   │   DuckDB   │  migrate   │ ClickHouse │
+   │ (embedded) │ ─────────> │  (local)   │
+   └────────────┘            └────────────┘
+```
+
+<details>
+<summary>graph-easy source</summary>
+
+```
+graph { flow: east; }
+[duckdb] { label: "DuckDB\n(embedded)"; }
+[clickhouse] { label: "ClickHouse\n(local)"; }
+[duckdb] -- migrate --> [clickhouse]
+```
+
+</details>
+
+## Research Summary
+
+Five independent research agents validated the ClickHouse Architect skill's recommendations:
+
+| Agent Perspective          | Key Finding                                                                       | Confidence |
+| -------------------------- | --------------------------------------------------------------------------------- | ---------- |
+| ZSTD-vs-LZ4-Codec          | DoubleDelta + LZ4 is 1.76x faster for timestamps; ZSTD provides no benefit        | High       |
+| Delta-Gorilla-Corruption   | Bug fixed in v23.2 (PR #45615); combination is redundant, not dangerous           | High       |
+| Dictionary-vs-JOIN         | Dictionaries only benefit >364 rows; standard tables + JOINs better for <100 rows | High       |
+| AI-Schema-Documentation    | COMMENT statements provide 20-27% AI accuracy improvement (AtScale 2025 study)    | High       |
+| ClickHouse-Architect-Skill | Skill already updated with PR #45615 fix documentation                            | Medium     |
+
+## Decision Log
+
+| Decision Area        | Options Evaluated             | Chosen              | Rationale                                                                       |
+| -------------------- | ----------------------------- | ------------------- | ------------------------------------------------------------------------------- |
+| Timestamp codec      | ZSTD, LZ4                     | LZ4                 | 1.76x faster decompression; ZSTD provides no additional benefit for DoubleDelta |
+| Lookup tables        | Dictionaries, Standard tables | Standard tables     | <364 rows threshold; v24.4 optimizer handles small JOINs efficiently            |
+| Float codec          | LZ4, ZSTD                     | ZSTD                | ZSTD(1) optimal for Gorilla-encoded floats                                      |
+| Schema documentation | None, Comments                | COMMENT statements  | 20-27% AI accuracy improvement                                                  |
+| Data access pattern  | Repository, Views             | Parameterized Views | ClickHouse-native OLAP pattern                                                  |
+
+### Trade-offs Accepted
+
+| Trade-off                   | Choice                            | Accepted Cost                                               |
+| --------------------------- | --------------------------------- | ----------------------------------------------------------- |
+| Simplicity vs Complexity    | Standard tables over Dictionaries | Marginally slower JOINs (unmeasurable for 10-20 rows)       |
+| LZ4 vs ZSTD for timestamps  | LZ4                               | Slightly worse compression ratio (unnoticeable in practice) |
+| Full rewrite vs incremental | Full rewrite in 3 batches         | Higher initial effort, cleaner final architecture           |
+
+## Decision Drivers
+
+- **Research validation**: All recommendations independently verified before adoption
+- **ClickHouse v25.11.2 compatibility**: Ensure patterns work with installed version
+- **AI code generation**: Schema must be understandable by AI coding agents
+- **Operational simplicity**: Standard SQL patterns over specialized ClickHouse features where equivalent
+- **Future scalability**: Design for 100+ instruments even if starting with 2
+
+## Considered Options
+
+- **Option A**: Direct port (keep DuckDB patterns, just change connection)
+- **Option B**: ClickHouse Architect skill recommendations (Dictionaries, ZSTD everywhere)
+- **Option C**: Research-validated hybrid (standard tables, LZ4 for timestamps, ZSTD for floats) <- Selected
+
+## Decision Outcome
+
+Chosen option: **Option C (Research-validated hybrid)**, because independent agent research proved that:
+
+1. **Dictionaries are overkill** for <364 row lookup tables
+2. **DoubleDelta + LZ4** outperforms DoubleDelta + ZSTD by 1.76x
+3. **COMMENT statements** provide measurable AI accuracy improvements
+4. The Delta+Gorilla corruption warning is **outdated** (fixed 3 years ago)
+
+## Synthesis
+
+**Convergent findings**: All agents agreed on:
+
+- ReplacingMergeTree for deduplication
+- LowCardinality(String) for instrument column
+- PARTITION BY toYYYYMM() for monthly partitions
+- Never use FINAL (100x slower)
+
+**Divergent findings**: ClickHouse Architect skill vs independent research:
+
+- Skill recommended Dictionaries; research showed they're overkill for small tables
+- Skill recommended ZSTD everywhere; research showed LZ4 better for DoubleDelta
+- Skill had outdated corruption warning; research confirmed it's fixed in v23.2+
+
+**Resolution**: User selected research-validated recommendations over skill defaults where they diverged.
+
+## Consequences
+
+### Positive
+
+- Unified infrastructure with other ClickHouse projects
+- 70% storage reduction with LowCardinality instrument column
+- 15-50x compression with optimized codec selection
+- AI-friendly schema with comprehensive COMMENT statements
+- Simpler architecture without unnecessary Dictionaries
+
+### Negative
+
+- Full rewrite required (7 modules)
+- Testing overhead (validate each batch)
+- Documentation updates needed
+- Temporary loss of backward compatibility during migration
+
+## Architecture
+
+```
+                                                 ┌──────────────────────────────┐
+                                                 │       Application Code       │
+                                                 └──────────────────────────────┘
+                                                   │
+                                                   │
+                                                   ∨
+┌──────────────────────────────────────┐         ┌──────────────────────────────┐         ┌────────────────────────────────────┐
+│ raw_spread_ticks(ReplacingMergeTree) │ <────── │     Parameterized Views      │ ──────> │ standard_ticks(ReplacingMergeTree) │
+└──────────────────────────────────────┘         └──────────────────────────────┘         └────────────────────────────────────┘
+                                                   │
+                                                   │
+                                                   ∨
+┌──────────────────────────────────────┐  JOIN   ┌──────────────────────────────┐
+│         holidays(MergeTree)          │ <────── │ ohlc_1m(ReplacingMergeTree)  │
+└──────────────────────────────────────┘         └──────────────────────────────┘
+                                                   │
+                                                   │ JOIN
+                                                   ∨
+                                                 ┌──────────────────────────────┐
+                                                 │ exchange_sessions(MergeTree) │
+                                                 └──────────────────────────────┘
+```
+
+<details>
+<summary>graph-easy source</summary>
+
+```
+graph { flow: south; }
+[ Application Code ]
+[ Parameterized Views ]
+[ raw_spread_ticks ] { label: "raw_spread_ticks(ReplacingMergeTree)"; }
+[ standard_ticks ] { label: "standard_ticks(ReplacingMergeTree)"; }
+[ ohlc_1m ] { label: "ohlc_1m(ReplacingMergeTree)"; }
+[ exchange_sessions ] { label: "exchange_sessions(MergeTree)"; }
+[ holidays ] { label: "holidays(MergeTree)"; }
+
+[ Application Code ] -> [ Parameterized Views ]
+[ Parameterized Views ] -> [ raw_spread_ticks ]
+[ Parameterized Views ] -> [ standard_ticks ]
+[ Parameterized Views ] -> [ ohlc_1m ]
+[ ohlc_1m ] -- JOIN --> [ exchange_sessions ]
+[ ohlc_1m ] -- JOIN --> [ holidays ]
+```
+
+</details>
+
+## References
+
+- [Migration Plan](/docs/design/2025-12-09-exness-clickhouse-migration/spec.md)
+- [ClickHouse Architect Skill](/Users/terryli/.claude/plugins/marketplaces/cc-skills/plugins/quality-tools/skills/clickhouse-architect/SKILL.md)
+- [PR #45615: Delta+Gorilla fix](https://github.com/ClickHouse/ClickHouse/pull/45615)
+- [PR #45652: Safety guardrail](https://github.com/ClickHouse/ClickHouse/pull/45652)