exness-data-preprocess 0.7.2__tar.gz → 0.8.0__tar.gz

exness_data_preprocess-0.8.0/.lint-relative-paths-results.txt

@@ -0,0 +1,62 @@
+🔍 Scanning for non-repo-relative paths in: /Users/terryli/eon/exness-data-preprocess
+   Violations: paths not starting with / or http(s)
+
+❌ Found 141 violation(s):
+
+/Users/terryli/eon/exness-data-preprocess/README.md:112:- **Column Definitions**: See [`schema.py`](src/exness_data_preprocess/schema.py) - Single source of truth
+/Users/terryli/eon/exness-data-preprocess/README.md:113:- **Comprehensive Reference**: See [`DATABASE_SCHEMA.md`](docs/DATABASE_SCHEMA.md) - Query examples and usage patterns
+/Users/terryli/eon/exness-data-preprocess/README.md:401:- Validation: [`docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md`](docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md)
+/Users/terryli/eon/exness-data-preprocess/README.md:409:- SSoT: [`docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml`](docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml)
+/Users/terryli/eon/exness-data-preprocess/README.md:410:- Validation: [`docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md`](docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md)
+/Users/terryli/eon/exness-data-preprocess/README.md:417:- SSoT: [`docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml`](docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml)
+/Users/terryli/eon/exness-data-preprocess/README.md:419:**Release Notes**: See [`CHANGELOG.md`](CHANGELOG.md) for complete v0.5.0 details
+/Users/terryli/eon/exness-data-preprocess/README.md:485:**[📚 Complete Documentation Hub](docs/README.md)** - Organized guide from beginner to advanced (72+ documents)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:7:**Full Documentation**: [`README.md`](README.md) - Installation, usage, API reference
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:15:- **[README.md](README.md)** - Installation, API reference, usage examples
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:16:- **[examples/basic_usage.py](examples/basic_usage.py)** - Download, query, coverage operations
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:17:- **[examples/batch_processing.py](examples/batch_processing.py)** - Multi-instrument parallel processing
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:21:- **[docs/README.md](docs/README.md)** - Architecture, planning, research findings
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:22:- **[docs/MODULE_ARCHITECTURE.md](docs/MODULE_ARCHITECTURE.md)** - Complete module documentation with SLOs (v1.7.0)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:23:- **[docs/DATABASE_SCHEMA.md](docs/DATABASE_SCHEMA.md)** - Complete database schema with self-documentation
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:24:- **[docs/RESEARCH_PATTERNS.md](docs/RESEARCH_PATTERNS.md)** - Research lifecycle and tool selection (v1.0.0)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:25:- **[docs/UNIFIED_DUCKDB_PLAN_v2.md](docs/UNIFIED_DUCKDB_PLAN_v2.md)** - v2.0.0 architecture specification
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:26:- **[docs/EXNESS_DATA_SOURCES.md](docs/EXNESS_DATA_SOURCES.md)** - Data source variants and URLs
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:27:- **[Makefile](Makefile)** - Module introspection commands (module-stats, module-complexity, module-deps)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:31:- **[docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml](docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml)** - SSoT plan v2.0.0 (2.2x speedup)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:32:- **[docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml](docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml)** - SSoT plan v2.0.0 (complete coverage)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:33:- **[docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md](docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md)** - Phase 1 validation
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:34:- **[docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md](docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md)** - Phase 2 validation
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:40:- **[CONTRIBUTING.md](CONTRIBUTING.md)** - Contribution guidelines
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:55:**Complete Specification**: [`docs/UNIFIED_DUCKDB_PLAN_v2.md`](docs/UNIFIED_DUCKDB_PLAN_v2.md)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:76:**Complete Details**: [`docs/MODULE_ARCHITECTURE.md`](docs/MODULE_ARCHITECTURE.md)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:83:  - SSoT: [`docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md`](docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:89:  - SSoT: [`docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml`](docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:90:  - Validation: [`docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md`](docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:96:  - SSoT: [`docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml`](docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:109:**Complete Schema**: [`docs/DATABASE_SCHEMA.md`](docs/DATABASE_SCHEMA.md)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:118:**Complete Guide**: [`docs/EXNESS_DATA_SOURCES.md`](docs/EXNESS_DATA_SOURCES.md)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:128:**Complete Specification**: [`docs/RESEARCH_PATTERNS.md`](docs/RESEARCH_PATTERNS.md)
+/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:196:**Complete Development Guide**: [`README.md`](README.md) (Installation section)
+/Users/terryli/eon/exness-data-preprocess/tests/README.md:30:- **New**: Phase7 13-column (v1.2.0) OHLC - See [`../src/exness_data_preprocess/schema.py`](../src/exness_data_preprocess/schema.py)
+/Users/terryli/eon/exness-data-preprocess/docs/EXNESS_DATA_SOURCES.md:310:**Full Specification**: [`docs/research/eurusd-zero-spread-deviations/data/plan/phase7_bid_ohlc_construction_v1.1.0.md`](research/eurusd-zero-spread-deviations/data/plan/phase7_bid_ohlc_construction_v1.1.0.md)
+/Users/terryli/eon/exness-data-preprocess/docs/EXNESS_DATA_SOURCES.md:386:- **Phase7 Methodology**: [`docs/research/eurusd-zero-spread-deviations/01-methodology.md`](research/eurusd-zero-spread-deviations/01-methodology.md)
+/Users/terryli/eon/exness-data-preprocess/docs/EXNESS_DATA_SOURCES.md:387:- **Zero-Spread Research**: [`docs/research/eurusd-zero-spread-deviations/README.md`](research/eurusd-zero-spread-deviations/README.md)
+/Users/terryli/eon/exness-data-preprocess/docs/EXNESS_DATA_SOURCES.md:391:- **Current Processor**: [`src/exness_data_preprocess/processor.py`](../src/exness_data_preprocess/processor.py) (needs refactoring)
+/Users/terryli/eon/exness-data-preprocess/docs/MODULE_ARCHITECTURE.md:5:**Related**: [`README.md`](../README.md) - Architecture overview
+/Users/terryli/eon/exness-data-preprocess/docs/MODULE_ARCHITECTURE.md:43:- See [`docs/validation/ARCHITECTURE_AUDIT_2025-10-17.md`](validation/ARCHITECTURE_AUDIT_2025-10-17.md) for complete audit findings
+/Users/terryli/eon/exness-data-preprocess/docs/MODULE_ARCHITECTURE.md:964:See [`Makefile`](../Makefile) for implementation.
+/Users/terryli/eon/exness-data-preprocess/docs/MODULE_ARCHITECTURE.md:1076:- **[README.md](../README.md)** - User-facing API reference
+/Users/terryli/eon/exness-data-preprocess/docs/MODULE_ARCHITECTURE.md:1077:- **[docs/README.md](README.md)** - Documentation hub
+/Users/terryli/eon/exness-data-preprocess/docs/MODULE_ARCHITECTURE.md:1078:- **[docs/DATABASE_SCHEMA.md](DATABASE_SCHEMA.md)** - Complete database schema
+/Users/terryli/eon/exness-data-preprocess/docs/MODULE_ARCHITECTURE.md:1079:- **[docs/UNIFIED_DUCKDB_PLAN_v2.md](UNIFIED_DUCKDB_PLAN_v2.md)** - v2.0.0 architecture specification
+/Users/terryli/eon/exness-data-preprocess/docs/MODULE_ARCHITECTURE.md:1080:- **[Makefile](../Makefile)** - Module introspection commands
+/Users/terryli/eon/exness-data-preprocess/docs/MODULE_ARCHITECTURE.md:1082:- **[docs/validation/ARCHITECTURE_AUDIT_2025-10-17.md](validation/ARCHITECTURE_AUDIT_2025-10-17.md)** - Complete audit findings and corrections
+/Users/terryli/eon/exness-data-preprocess/docs/DATABASE_SCHEMA.md:244:**Column Definitions**: See [`../src/exness_data_preprocess/schema.py`](../src/exness_data_preprocess/schema.py) for complete column definitions with types and descriptions.
+/Users/terryli/eon/exness-data-preprocess/docs/DATABASE_SCHEMA.md:258:**Architecture**: Exchange Registry Pattern (v1.6.0) - Session columns dynamically generated from centralized EXCHANGES dict in [`exchanges.py`](../src/exness_data_preprocess/exchanges.py)
+
+... and 91 more (showing first 50)
+
+📝 Fix: Convert relative paths to repo-relative paths starting with /
+   WRONG: [Link](../docs/foo.md)
+   WRONG: [Link](docs/foo.md)
+   RIGHT: [Link](/docs/foo.md)

{exness_data_preprocess-0.7.2 → exness_data_preprocess-0.8.0}/CHANGELOG.md

@@ -1,3 +1,10 @@
+# [0.8.0](https://github.com/terrylica/exness-data-preprocess/compare/v0.7.2...v0.8.0) (2025-12-10)
+
+
+### Features
+
+* add ClickHouse backend for cloud deployment ([c9b51a3](https://github.com/terrylica/exness-data-preprocess/commit/c9b51a3b374eea74375cc86afdd8cb9391fe73e8))
+
 # Changelog
 
 All notable changes to exness-data-preprocess will be documented in this file.
@@ -5,12 +12,10 @@ All notable changes to exness-data-preprocess will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-
 ### 💅 Code Style
 
 - Format markdown files with prettier Standardized formatting for CHANGELOG.md.
 
-
 ### 📚 Documentation
 
 - Move test plan to docs/plans/ - Move tests/archive/E2E_TESTING_PLAN_v1.0.0.md -> docs/plans/ - Remove empty tests/archive/ directory - Consistent with docs/ organization strategy Impact: Planning docs consolidated in docs/plans/ Risk: Low (internal doc, no external references)
@@ -19,7 +24,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - Remove redundant DOCUMENTATION.md - Remove DOCUMENTATION.md (content duplicates docs/README.md) - Update README.md link to point to docs/README.md - Both files serve as navigation hubs with identical purpose Impact: -1 root file, single documentation entry point Risk: Low (historical CHANGELOG references preserved, README updated)
 
-- Organize phase plans into docs/phases/ - Move PHASE2*.yaml, PHASE3*.yaml, PHASE4*.yaml -> docs/phases/ - Update 4 references in CLAUDE.md - Consolidate SSoT planning files in dedicated subdirectory Impact: -3 docs/ root files, improved organization Risk: Low (all references updated, git mv preserves history)
+- Organize phase plans into docs/phases/ - Move PHASE2*.yaml, PHASE3*.yaml, PHASE4\*.yaml -> docs/phases/ - Update 4 references in CLAUDE.md - Consolidate SSoT planning files in dedicated subdirectory Impact: -3 docs/ root files, improved organization Risk: Low (all references updated, git mv preserves history)
 
 - Move GitHub/PyPI setup to docs/setup/ - Move GITHUB_PYPI_SETUP.md -> docs/setup/ - No active references in README.md or CONTRIBUTING.md - Historical CHANGELOG references preserved Impact: -1 root file, infrastructure docs consolidated Risk: Low (no external links, CHANGELOG preserved)
 
@@ -31,7 +36,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - Update refactoring SSoT plans to completion status - REPOSITORY_REFACTORING_PLAN.yaml: v2.0.0 -> v3.0.0 - PRIORITY3_CLEANUP_PLAN.yaml: v1.0.0 -> v2.0.0 (Completed) Combined Priority 1-3 Impact: - Root MD files: 10 -> 4 (-60%) - docs/ root files: 12 -> 8 (-33%) - Total clutter: 30 -> 12 items (-60%) Total: 11 commits, ~40 minutes, zero regressions
 
-
 ### 🧰 Maintenance
 
 - Untrack lychee cache files - Remove .lychee-results.json, .lychee-results.txt, .lycheecache from tracking - Files remain gitignored via .gitignore rules - Add REPOSITORY_REFACTORING_PLAN.yaml SSoT (v1.0.0) Impact: Cleaner git status, prevent cache pollution Risk: None (cache files, no external references)

{exness_data_preprocess-0.7.2 → exness_data_preprocess-0.8.0}/CLAUDE.md

@@ -56,7 +56,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 
 ### Module Pattern (v1.7.0)
 
-- **Facade orchestrator** (processor.py) coordinating 7 specialized modules
+- **Facade orchestrator** (processor.py) coordinating 6 module instances + 1 static utility
 - **Separation of concerns** with single-responsibility modules
 - **SLO-based design** (Availability, Correctness, Observability, Maintainability)
 - **Off-the-shelf libraries** (httpx, pandas, DuckDB, exchange_calendars)

{exness_data_preprocess-0.7.2 → exness_data_preprocess-0.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: exness-data-preprocess
-Version: 0.7.2
+Version: 0.8.0
 Summary: Professional Exness forex tick data preprocessing with optimal compression (Parquet Zstd-22) and DuckDB OHLC generation. Provides efficient storage (9% smaller than ZIP) with lossless precision and direct queryability.
 Project-URL: Homepage, https://github.com/terrylica/exness-data-preprocess
 Project-URL: Documentation, https://github.com/terrylica/exness-data-preprocess#readme
@@ -19,15 +19,14 @@ Classifier: Intended Audience :: Financial and Insurance Industry
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Classifier: Topic :: Office/Business :: Financial :: Investment
 Classifier: Topic :: Scientific/Engineering :: Information Analysis
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
-Requires-Python: >=3.9
+Requires-Python: >=3.11
+Requires-Dist: clickhouse-connect>=0.7.0
 Requires-Dist: duckdb>=0.9.0
 Requires-Dist: exchange-calendars>=4.5.6
 Requires-Dist: httpx>=0.25.0
@@ -455,7 +454,7 @@ Data is sourced from Exness's public tick data repository:
 - Vectorized approach: 2.69s (302K bars, 10 exchanges)
 - Combined Phase 1+2: ~16x total speedup (8.05s → 0.50s)
 - Implementation: Pre-compute trading minutes, vectorized `.isin()` lookup
-- SSoT: [`docs/PHASE2_SESSION_VECTORIZATION_PLAN.yaml`](docs/PHASE2_SESSION_VECTORIZATION_PLAN.yaml)
+- SSoT: [`docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml`](docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml)
 - Validation: [`docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md`](docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md)
 
 **SQL Gap Detection** - Complete coverage with 46% code reduction:
@@ -463,7 +462,7 @@ Data is sourced from Exness's public tick data repository:
 - Bug fix: Python approach missed internal gaps (41 detected vs 42 actual)
 - SQL EXCEPT operator detects ALL gaps (before + within + after existing data)
 - Code reduced from 62 lines to 34 lines (46% reduction)
-- SSoT: [`docs/PHASE3_SQL_GAP_DETECTION_PLAN.yaml`](docs/PHASE3_SQL_GAP_DETECTION_PLAN.yaml)
+- SSoT: [`docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml`](docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml)
 
 **Release Notes**: See [`CHANGELOG.md`](CHANGELOG.md) for complete v0.5.0 details
 

{exness_data_preprocess-0.7.2 → exness_data_preprocess-0.8.0}/README.md

@@ -406,7 +406,7 @@ Data is sourced from Exness's public tick data repository:
 - Vectorized approach: 2.69s (302K bars, 10 exchanges)
 - Combined Phase 1+2: ~16x total speedup (8.05s → 0.50s)
 - Implementation: Pre-compute trading minutes, vectorized `.isin()` lookup
-- SSoT: [`docs/PHASE2_SESSION_VECTORIZATION_PLAN.yaml`](docs/PHASE2_SESSION_VECTORIZATION_PLAN.yaml)
+- SSoT: [`docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml`](docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml)
 - Validation: [`docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md`](docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md)
 
 **SQL Gap Detection** - Complete coverage with 46% code reduction:
@@ -414,7 +414,7 @@ Data is sourced from Exness's public tick data repository:
 - Bug fix: Python approach missed internal gaps (41 detected vs 42 actual)
 - SQL EXCEPT operator detects ALL gaps (before + within + after existing data)
 - Code reduced from 62 lines to 34 lines (46% reduction)
-- SSoT: [`docs/PHASE3_SQL_GAP_DETECTION_PLAN.yaml`](docs/PHASE3_SQL_GAP_DETECTION_PLAN.yaml)
+- SSoT: [`docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml`](docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml)
 
 **Release Notes**: See [`CHANGELOG.md`](CHANGELOG.md) for complete v0.5.0 details
 

exness_data_preprocess-0.8.0/docs/adr/2025-12-09-exness-clickhouse-migration.md

@@ -0,0 +1,196 @@
+---
+status: accepted
+date: 2025-12-09
+decision-maker: Terry Li
+consulted:
+  [
+    ZSTD-vs-LZ4-Codec-Agent,
+    Delta-Gorilla-Corruption-Agent,
+    Dictionary-vs-JOIN-Agent,
+    AI-Schema-Documentation-Agent,
+    ClickHouse-Architect-Skill-Agent,
+  ]
+research-method: 9-agent-parallel-dctl
+clarification-iterations: 4
+perspectives:
+  [Performance, Simplicity, AICodeGeneration, DataModeling, FutureScalability]
+---
+
+# ADR: Exness Data Preprocess DuckDB to ClickHouse Migration
+
+**Design Spec**: [Implementation Spec](/docs/design/2025-12-09-exness-clickhouse-migration/spec.md)
+
+## Context and Problem Statement
+
+The exness-data-preprocess project currently uses embedded DuckDB for storing forex tick data and OHLC bars. While DuckDB works well for single-file analytics, the project needs to migrate to ClickHouse to:
+
+1. **Unify infrastructure** with other ClickHouse-based projects (deribit data, MLflow tracking)
+2. **Support future scale** (100+ instruments, multi-terabyte datasets)
+3. **Enable real-time dashboards** with sub-second query latency
+4. **Leverage ClickHouse-native features** (ReplacingMergeTree, parameterized views, compression codecs)
+
+The migration must preserve all existing functionality while adopting ClickHouse-native patterns validated through independent research.
+
+### Before/After
+
+```
+   ┌────────────┐            ┌────────────┐
+   │   DuckDB   │  migrate   │ ClickHouse │
+   │ (embedded) │ ─────────> │  (local)   │
+   └────────────┘            └────────────┘
+```
+
+<details>
+<summary>graph-easy source</summary>
+
+```
+graph { flow: east; }
+[duckdb] { label: "DuckDB\n(embedded)"; }
+[clickhouse] { label: "ClickHouse\n(local)"; }
+[duckdb] -- migrate --> [clickhouse]
+```
+
+</details>
+
+## Research Summary
+
+Five independent research agents validated the ClickHouse Architect skill's recommendations:
+
+| Agent Perspective          | Key Finding                                                                       | Confidence |
+| -------------------------- | --------------------------------------------------------------------------------- | ---------- |
+| ZSTD-vs-LZ4-Codec          | DoubleDelta + LZ4 is 1.76x faster for timestamps; ZSTD provides no benefit        | High       |
+| Delta-Gorilla-Corruption   | Bug fixed in v23.2 (PR #45615); combination is redundant, not dangerous           | High       |
+| Dictionary-vs-JOIN         | Dictionaries only benefit >364 rows; standard tables + JOINs better for <100 rows | High       |
+| AI-Schema-Documentation    | COMMENT statements provide 20-27% AI accuracy improvement (AtScale 2025 study)    | High       |
+| ClickHouse-Architect-Skill | Skill already updated with PR #45615 fix documentation                            | Medium     |
+
+## Decision Log
+
+| Decision Area        | Options Evaluated             | Chosen              | Rationale                                                                       |
+| -------------------- | ----------------------------- | ------------------- | ------------------------------------------------------------------------------- |
+| Timestamp codec      | ZSTD, LZ4                     | LZ4                 | 1.76x faster decompression; ZSTD provides no additional benefit for DoubleDelta |
+| Lookup tables        | Dictionaries, Standard tables | Standard tables     | <364 rows threshold; v24.4 optimizer handles small JOINs efficiently            |
+| Float codec          | LZ4, ZSTD                     | ZSTD                | ZSTD(1) optimal for Gorilla-encoded floats                                      |
+| Schema documentation | None, Comments                | COMMENT statements  | 20-27% AI accuracy improvement                                                  |
+| Data access pattern  | Repository, Views             | Parameterized Views | ClickHouse-native OLAP pattern                                                  |
+
+### Trade-offs Accepted
+
+| Trade-off                   | Choice                            | Accepted Cost                                               |
+| --------------------------- | --------------------------------- | ----------------------------------------------------------- |
+| Simplicity vs Complexity    | Standard tables over Dictionaries | Marginally slower JOINs (unmeasurable for 10-20 rows)       |
+| LZ4 vs ZSTD for timestamps  | LZ4                               | Slightly worse compression ratio (unnoticeable in practice) |
+| Full rewrite vs incremental | Full rewrite in 3 batches         | Higher initial effort, cleaner final architecture           |
+
+## Decision Drivers
+
+- **Research validation**: All recommendations independently verified before adoption
+- **ClickHouse v25.11.2 compatibility**: Ensure patterns work with installed version
+- **AI code generation**: Schema must be understandable by AI coding agents
+- **Operational simplicity**: Standard SQL patterns over specialized ClickHouse features where equivalent
+- **Future scalability**: Design for 100+ instruments even if starting with 2
+
+## Considered Options
+
+- **Option A**: Direct port (keep DuckDB patterns, just change connection)
+- **Option B**: ClickHouse Architect skill recommendations (Dictionaries, ZSTD everywhere)
+- **Option C**: Research-validated hybrid (standard tables, LZ4 for timestamps, ZSTD for floats) <- Selected
+
+## Decision Outcome
+
+Chosen option: **Option C (Research-validated hybrid)**, because independent agent research proved that:
+
+1. **Dictionaries are overkill** for <364 row lookup tables
+2. **DoubleDelta + LZ4** outperforms DoubleDelta + ZSTD by 1.76x
+3. **COMMENT statements** provide measurable AI accuracy improvements
+4. The Delta+Gorilla corruption warning is **outdated** (fixed 3 years ago)
+
+## Synthesis
+
+**Convergent findings**: All agents agreed on:
+
+- ReplacingMergeTree for deduplication
+- LowCardinality(String) for instrument column
+- PARTITION BY toYYYYMM() for monthly partitions
+- Never use FINAL (100x slower)
+
+**Divergent findings**: ClickHouse Architect skill vs independent research:
+
+- Skill recommended Dictionaries; research showed they're overkill for small tables
+- Skill recommended ZSTD everywhere; research showed LZ4 better for DoubleDelta
+- Skill had outdated corruption warning; research confirmed it's fixed in v23.2+
+
+**Resolution**: User selected research-validated recommendations over skill defaults where they diverged.
+
+## Consequences
+
+### Positive
+
+- Unified infrastructure with other ClickHouse projects
+- 70% storage reduction with LowCardinality instrument column
+- 15-50x compression with optimized codec selection
+- AI-friendly schema with comprehensive COMMENT statements
+- Simpler architecture without unnecessary Dictionaries
+
+### Negative
+
+- Full rewrite required (7 modules)
+- Testing overhead (validate each batch)
+- Documentation updates needed
+- Temporary loss of backward compatibility during migration
+
+## Architecture
+
+```
+                                                 ┌──────────────────────────────┐
+                                                 │       Application Code       │
+                                                 └──────────────────────────────┘
+                                                   │
+                                                   │
+                                                   ∨
+┌──────────────────────────────────────┐         ┌──────────────────────────────┐         ┌────────────────────────────────────┐
+│ raw_spread_ticks(ReplacingMergeTree) │ <────── │     Parameterized Views      │ ──────> │ standard_ticks(ReplacingMergeTree) │
+└──────────────────────────────────────┘         └──────────────────────────────┘         └────────────────────────────────────┘
+                                                   │
+                                                   │
+                                                   ∨
+┌──────────────────────────────────────┐  JOIN   ┌──────────────────────────────┐
+│         holidays(MergeTree)          │ <────── │ ohlc_1m(ReplacingMergeTree)  │
+└──────────────────────────────────────┘         └──────────────────────────────┘
+                                                   │
+                                                   │ JOIN
+                                                   ∨
+                                                 ┌──────────────────────────────┐
+                                                 │ exchange_sessions(MergeTree) │
+                                                 └──────────────────────────────┘
+```
+
+<details>
+<summary>graph-easy source</summary>
+
+```
+graph { flow: south; }
+[ Application Code ]
+[ Parameterized Views ]
+[ raw_spread_ticks ] { label: "raw_spread_ticks(ReplacingMergeTree)"; }
+[ standard_ticks ] { label: "standard_ticks(ReplacingMergeTree)"; }
+[ ohlc_1m ] { label: "ohlc_1m(ReplacingMergeTree)"; }
+[ exchange_sessions ] { label: "exchange_sessions(MergeTree)"; }
+[ holidays ] { label: "holidays(MergeTree)"; }
+
+[ Application Code ] -> [ Parameterized Views ]
+[ Parameterized Views ] -> [ raw_spread_ticks ]
+[ Parameterized Views ] -> [ standard_ticks ]
+[ Parameterized Views ] -> [ ohlc_1m ]
+[ ohlc_1m ] -- JOIN --> [ exchange_sessions ]
+[ ohlc_1m ] -- JOIN --> [ holidays ]
+```
+
+</details>
+
+## References
+
+- [Migration Plan](/docs/design/2025-12-09-exness-clickhouse-migration/spec.md)
+- [ClickHouse Architect Skill](/Users/terryli/.claude/plugins/marketplaces/cc-skills/plugins/quality-tools/skills/clickhouse-architect/SKILL.md)
+- [PR #45615: Delta+Gorilla fix](https://github.com/ClickHouse/ClickHouse/pull/45615)
+- [PR #45652: Safety guardrail](https://github.com/ClickHouse/ClickHouse/pull/45652)

{exness_data_preprocess-0.7.2 → exness_data_preprocess-0.8.0}/docs/archive/UNIFIED_DUCKDB_PLAN_v1.0.0_LEGACY.md

@@ -700,6 +700,6 @@ Based on real data validation (Sep 2024):
 
 - **Validation Report**: [`/tmp/exness-duckdb-test/FINDINGS.md`](/tmp/exness-duckdb-test/FINDINGS.md)
 - **Test Script**: [`/tmp/exness-duckdb-test/test_real_unified.py`](/tmp/exness-duckdb-test/test_real_unified.py)
-- **Phase7 Spec**: [`docs/research/eurusd-zero-spread-deviations/data/plan/phase7_bid_ohlc_construction_v1.1.0.md`](research/eurusd-zero-spread-deviations/data/plan/phase7_bid_ohlc_construction_v1.1.0.md)
-- **Compression Benchmarks**: [`docs/research/compression-benchmarks/README.md`](research/compression-benchmarks/README.md)
-- **Data Sources Guide**: [`docs/EXNESS_DATA_SOURCES.md`](EXNESS_DATA_SOURCES.md)
+- **Phase7 Spec**: Archived - see DATABASE_SCHEMA.md for current schema
+- **Compression Benchmarks**: Archived - DuckDB compression validated
+- **Data Sources Guide**: [`docs/EXNESS_DATA_SOURCES.md`](../EXNESS_DATA_SOURCES.md)