exness-data-preprocess 1.2.0__tar.gz → 1.2.1__tar.gz

{exness_data_preprocess-1.2.0 → exness_data_preprocess-1.2.1}/.mise.toml

@@ -2,8 +2,8 @@
 # E2E validation workflow for codebase pruning verification
 
 [env]
-# Python version for validation
-PYTHON_VERSION = "3.14"
+# Python version for validation (pinned for alpha-forge compatibility)
+PYTHON_VERSION = "3.11"
 # Package name
 PACKAGE_NAME = "exness_data_preprocess"
 # Validation temp directory
@@ -26,7 +26,8 @@ DBEAVER_CONFIG_PATH = ".dbeaver/data-sources.json"
 # CLICKHOUSE_PASSWORD_READONLY - set in .env
 
 [tools]
-python = "3.14"
+# Pin Python 3.11 for alpha-forge compatibility (numpy requirements)
+python = "3.11"
 
 # =============================================================================
 # Validation Tasks - E2E Testing Pipeline

{exness_data_preprocess-1.2.0 → exness_data_preprocess-1.2.1}/CHANGELOG.md

@@ -1,3 +1,10 @@
+## [1.2.1](https://github.com/terrylica/exness-data-preprocess/compare/v1.2.0...v1.2.1) (2025-12-11)
+
+
+### Bug Fixes
+
+* **clickhouse:** bootstrap database creation without existing database context ([0fdfef1](https://github.com/terrylica/exness-data-preprocess/commit/0fdfef17cf797d64a66b3b163037a01c1e8245a7))
+
 # [1.2.0](https://github.com/terrylica/exness-data-preprocess/compare/v1.1.0...v1.2.0) (2025-12-10)
 
 

exness_data_preprocess-1.2.1/CLAUDE.md

@@ -0,0 +1,150 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+**Package**: `exness-data-preprocess` — Forex tick data preprocessing with DuckDB (local) and ClickHouse (cloud) backends
+
+**Version**: See [`pyproject.toml`](pyproject.toml) line 3 (SSoT via semantic-release)
+
+**Quick Start**: [`README.md`](README.md) — Installation, API reference, usage examples
+
+---
+
+## Documentation Hub (Progressive Disclosure)
+
+### Essential Links
+
+| Topic            | SSoT Document                                                      | Purpose                               |
+| ---------------- | ------------------------------------------------------------------ | ------------------------------------- |
+| **User Guide**   | [`README.md`](README.md)                                           | Installation, API, examples           |
+| **Architecture** | [`docs/UNIFIED_DUCKDB_PLAN_v2.md`](docs/UNIFIED_DUCKDB_PLAN_v2.md) | Unified single-file DuckDB design     |
+| **Schema**       | [`docs/DATABASE_SCHEMA.md`](docs/DATABASE_SCHEMA.md)               | Phase7 30-column OHLC specification   |
+| **Modules**      | [`docs/MODULE_ARCHITECTURE.md`](docs/MODULE_ARCHITECTURE.md)       | 19 modules with SLOs                  |
+| **Data Sources** | [`docs/EXNESS_DATA_SOURCES.md`](docs/EXNESS_DATA_SOURCES.md)       | Exness tick data variants             |
+| **Research**     | [`docs/RESEARCH_PATTERNS.md`](docs/RESEARCH_PATTERNS.md)           | Hybrid materialization lifecycle      |
+| **Tasks**        | [`.mise.toml`](.mise.toml)                                         | Validation, ClickHouse, DBeaver tasks |
+| **Contributing** | [`CONTRIBUTING.md`](CONTRIBUTING.md)                               | Development guidelines                |
+
+### Deep Dives (On-Demand)
+
+| Topic              | Location                               | When to Read                     |
+| ------------------ | -------------------------------------- | -------------------------------- |
+| ADRs               | [`docs/adr/`](docs/adr/)               | Architecture decisions           |
+| Design Specs       | [`docs/design/`](docs/design/)         | Implementation blueprints        |
+| Optimization Plans | [`docs/phases/`](docs/phases/)         | Phase 2/3 SSoT YAML plans        |
+| Validation Results | [`docs/validation/`](docs/validation/) | Spike test proofs                |
+| Research Analysis  | [`docs/research/`](docs/research/)     | Zero-spread, compression studies |
+
+---
+
+## Architecture Overview
+
+### Dual Backend Design
+
+| Backend        | Storage                                   | Use Case          | Entry Point           |
+| -------------- | ----------------------------------------- | ----------------- | --------------------- |
+| **DuckDB**     | One file per instrument (`eurusd.duckdb`) | Local development | `ExnessDataProcessor` |
+| **ClickHouse** | Single table with instrument column       | Cloud deployment  | `ClickHouseManager`   |
+
+### Module Structure (19 Modules)
+
+**DuckDB Backend (8 modules)**:
+
+- `processor.py` — Thin orchestrator facade
+- `downloader.py` — HTTP download operations
+- `tick_loader.py` — CSV parsing
+- `database_manager.py` — Database operations
+- `session_detector.py` — Holiday/session detection (vectorized)
+- `gap_detector.py` — Incremental update logic (SQL-based)
+- `ohlc_generator.py` — Phase7 OHLC generation
+- `query_engine.py` — Query operations
+
+**ClickHouse Backend (7 modules)**:
+
+- `clickhouse_manager.py` — Schema management, data loading
+- `clickhouse_gap_detector.py` — Gap detection for ClickHouse
+- `clickhouse_query_engine.py` — Query with pagination
+- `clickhouse_ohlc_generator.py` — OHLC generation
+- `clickhouse_client.py` — Connection factory with error handling
+- `clickhouse_base.py` — Shared mixin for client lifecycle
+- `clickhouse_cloud.py` — Cloud-specific configuration
+
+**Shared (4 modules)**:
+
+- `config.py` — Configuration management
+- `models.py` — Pydantic models (UpdateResult, CoverageInfo, etc.)
+- `schema.py` — SQL schema definitions
+- `exchanges.py` — Exchange calendar integration
+
+**Complete Details**: [`docs/MODULE_ARCHITECTURE.md`](docs/MODULE_ARCHITECTURE.md)
+
+---
+
+## Development Commands
+
+### Quick Reference
+
+```bash
+# Setup
+uv sync --dev
+
+# Testing (48 tests)
+uv run pytest                          # All tests
+mise run validate                      # Full E2E validation
+
+# Linting
+uv run ruff check --fix src/ tests/
+
+# ClickHouse
+mise run clickhouse:start              # Start local server
+mise run clickhouse:status             # Check connection
+mise run validate:clickhouse           # E2E ClickHouse tests
+
+# Build & Publish
+uv build
+mise run db-client-generate            # Generate DBeaver config
+```
+
+### mise Tasks
+
+All validation and tooling tasks are defined in [`.mise.toml`](.mise.toml):
+
+| Task                           | Description                                                               |
+| ------------------------------ | ------------------------------------------------------------------------- |
+| `mise run validate`            | Full E2E pipeline (imports, lint, typecheck, test, build, install, mixin) |
+| `mise run validate:clickhouse` | ClickHouse E2E tests (requires running server)                            |
+| `mise run clickhouse:start`    | Start mise-installed ClickHouse                                           |
+| `mise run db-client-generate`  | Generate DBeaver config from Pydantic model                               |
+| `mise run ci`                  | Simulate CI pipeline locally                                              |
+
+---
+
+## Key Patterns
+
+### Performance Optimizations (~16x total speedup)
+
+| Phase | Optimization          | Speedup           | SSoT                                                                                                                 |
+| ----- | --------------------- | ----------------- | -------------------------------------------------------------------------------------------------------------------- |
+| 1     | Incremental OHLC      | 7.3x              | [`docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md`](docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md) |
+| 2     | Session Vectorization | 2.2x              | [`docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml`](docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml)           |
+| 3     | SQL Gap Detection     | 46% LOC reduction | [`docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml`](docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml)                   |
+
+### Research Lifecycle
+
+```
+Explore (pandas/Polars) → Validate → Graduate (DuckDB) → Query (SQL)
+```
+
+**SSoT**: [`docs/RESEARCH_PATTERNS.md`](docs/RESEARCH_PATTERNS.md)
+
+---
+
+## References
+
+- **Exness Data**: <https://ticks.ex2archive.com/>
+- **DuckDB**: <https://duckdb.org/>
+- **ClickHouse**: <https://clickhouse.com/>
+
+---
+
+**Last Updated**: 2025-12-10

{exness_data_preprocess-1.2.0 → exness_data_preprocess-1.2.1}/CONTRIBUTING.md

@@ -6,7 +6,7 @@ Thank you for your interest in contributing to Exness Data Preprocess! This docu
 
 ### Prerequisites
 
-- Python 3.9 or higher
+- Python 3.11 or higher (see `pyproject.toml` requires-python)
 - uv package manager (recommended) or pip
 - Git
 
@@ -114,6 +114,7 @@ When reporting issues, please include:
    ```
 
 7. **Push to your fork** and submit a pull request:
+
    ```bash
    git push origin feature/your-feature-name
    ```
@@ -143,31 +144,24 @@ test: add integration tests for DuckDB operations
 
 ### Project Structure
 
-```
-exness-data-preprocess/
-├── src/exness_data_preprocess/
-│   ├── __init__.py          # Package entry point
-│   ├── processor.py         # Core processor class
-│   ├── api.py              # Simple API functions
-│   └── cli.py              # Command-line interface
-├── tests/
-│   ├── conftest.py         # Pytest fixtures
-│   ├── test_processor.py   # Processor tests
-│   ├── test_api.py         # API tests
-│   └── test_cli.py         # CLI tests
-├── docs/                   # Documentation
-├── pyproject.toml         # Package configuration
-└── README.md             # Main documentation
-```
+**See [`CLAUDE.md`](CLAUDE.md) for complete module structure (19 modules).**
+
+Key directories:
+
+- `src/exness_data_preprocess/` — Source modules (DuckDB + ClickHouse backends)
+- `tests/` — Test suite (48 tests)
+- `docs/` — Documentation hub
+- `.mise.toml` — Validation and tooling tasks
+- `pyproject.toml` — Package configuration (SSoT for version via semantic-release)
 
 ### Adding New Features
 
 When adding new features:
 
-1. **Add implementation** in appropriate module (processor.py, api.py, or cli.py)
-2. **Add tests** in corresponding test file
+1. **Add implementation** in appropriate module (see CLAUDE.md Module Structure)
+2. **Add tests** in corresponding test file under `tests/`
 3. **Update API exports** in `__init__.py` if needed
-4. **Update README.md** with usage examples
+4. **Update CLAUDE.md** with module links if adding new modules
 5. **Add docstrings** following Google style
 
 Example docstring:

{exness_data_preprocess-1.2.0 → exness_data_preprocess-1.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: exness-data-preprocess
-Version: 1.2.0
+Version: 1.2.1
 Summary: Professional Exness forex tick data preprocessing with optimal compression (Parquet Zstd-22) and DuckDB OHLC generation. Provides efficient storage (9% smaller than ZIP) with lossless precision and direct queryability.
 Project-URL: Homepage, https://github.com/terrylica/exness-data-preprocess
 Project-URL: Documentation, https://github.com/terrylica/exness-data-preprocess#readme

{exness_data_preprocess-1.2.0 → exness_data_preprocess-1.2.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "exness-data-preprocess"
-version = "1.2.0"
+version = "1.2.1"
 description = "Professional Exness forex tick data preprocessing with optimal compression (Parquet Zstd-22) and DuckDB OHLC generation. Provides efficient storage (9% smaller than ZIP) with lossless precision and direct queryability."
 readme = { file = "README.md", content-type = "text/markdown" }
 authors = [

{exness_data_preprocess-1.2.0 → exness_data_preprocess-1.2.1}/src/exness_data_preprocess/__init__.py

@@ -57,7 +57,7 @@ AI Agent Discovery:
     >>> print(get_args(TimeframeType))  # ('1m', '5m', '15m', ...)
 """
 
-__version__ = "1.2.0"
+__version__ = "1.2.1"
 __author__ = "Terry Li <terry@eonlabs.com>"
 __license__ = "MIT"
 

{exness_data_preprocess-1.2.0 → exness_data_preprocess-1.2.1}/src/exness_data_preprocess/clickhouse_client.py

@@ -79,7 +79,8 @@ def get_client(
     resolved_host = host or os.environ.get("CLICKHOUSE_HOST", DEFAULT_HOST)
     resolved_user = user or os.environ.get("CLICKHOUSE_USER", DEFAULT_USER)
     resolved_password = password or os.environ.get("CLICKHOUSE_PASSWORD", DEFAULT_PASSWORD)
-    resolved_database = database or os.environ.get("CLICKHOUSE_DATABASE", DEFAULT_DATABASE)
+    # Database can be None for initial schema creation (CREATE DATABASE needs no database context)
+    resolved_database = database if database is not None else os.environ.get("CLICKHOUSE_DATABASE", DEFAULT_DATABASE)
 
     # Mode-based SSL detection (ADR: 2025-12-10-clickhouse-pydantic-config)
     # CLICKHOUSE_MODE takes precedence over CLICKHOUSE_SECURE for clarity

{exness_data_preprocess-1.2.0 → exness_data_preprocess-1.2.1}/src/exness_data_preprocess/clickhouse_manager.py

@@ -78,8 +78,15 @@ class ClickHouseManager(ClickHouseClientMixin):
         Raises:
             ClickHouseQueryError: If schema creation fails
         """
-        # Create database
-        execute_command(self.client, f"CREATE DATABASE IF NOT EXISTS {self.DATABASE}")
+        # Create database using a client without database context
+        # (database may not exist yet, so we can't connect to it)
+        from exness_data_preprocess.clickhouse_client import get_client as get_client_fn
+
+        bootstrap_client = get_client_fn(database="")  # Empty string = no database
+        try:
+            execute_command(bootstrap_client, f"CREATE DATABASE IF NOT EXISTS {self.DATABASE}")
+        finally:
+            bootstrap_client.close()
 
         # Create tick tables
         self._create_raw_spread_ticks_table()

exness_data_preprocess-1.2.0/.lint-relative-paths-results.txt

@@ -1,62 +0,0 @@
-🔍 Scanning for non-repo-relative paths in: /Users/terryli/eon/exness-data-preprocess
-   Violations: paths not starting with / or http(s)
-
-❌ Found 140 violation(s):
-
-/Users/terryli/eon/exness-data-preprocess/README.md:112:- **Column Definitions**: See [`schema.py`](src/exness_data_preprocess/schema.py) - Single source of truth
-/Users/terryli/eon/exness-data-preprocess/README.md:113:- **Comprehensive Reference**: See [`DATABASE_SCHEMA.md`](docs/DATABASE_SCHEMA.md) - Query examples and usage patterns
-/Users/terryli/eon/exness-data-preprocess/README.md:401:- Validation: [`docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md`](docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md)
-/Users/terryli/eon/exness-data-preprocess/README.md:409:- SSoT: [`docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml`](docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml)
-/Users/terryli/eon/exness-data-preprocess/README.md:410:- Validation: [`docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md`](docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md)
-/Users/terryli/eon/exness-data-preprocess/README.md:417:- SSoT: [`docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml`](docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml)
-/Users/terryli/eon/exness-data-preprocess/README.md:419:**Release Notes**: See [`CHANGELOG.md`](CHANGELOG.md) for complete v0.5.0 details
-/Users/terryli/eon/exness-data-preprocess/README.md:485:**[📚 Complete Documentation Hub](docs/README.md)** - Organized guide from beginner to advanced (72+ documents)
-/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:7:**Full Documentation**: [`README.md`](README.md) - Installation, usage, API reference
-/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:15:- **[README.md](README.md)** - Installation, API reference, usage examples
-/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:16:- **[examples/basic_usage.py](examples/basic_usage.py)** - Download, query, coverage operations
-/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:17:- **[examples/batch_processing.py](examples/batch_processing.py)** - Multi-instrument parallel processing
-/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:21:- **[docs/README.md](docs/README.md)** - Architecture, planning, research findings
-/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:22:- **[docs/MODULE_ARCHITECTURE.md](docs/MODULE_ARCHITECTURE.md)** - Complete module documentation with SLOs (v1.7.0)
-/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:23:- **[docs/DATABASE_SCHEMA.md](docs/DATABASE_SCHEMA.md)** - Complete database schema with self-documentation
-/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:24:- **[docs/RESEARCH_PATTERNS.md](docs/RESEARCH_PATTERNS.md)** - Research lifecycle and tool selection (v1.0.0)
-/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:25:- **[docs/UNIFIED_DUCKDB_PLAN_v2.md](docs/UNIFIED_DUCKDB_PLAN_v2.md)** - v2.0.0 architecture specification
-/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:26:- **[docs/EXNESS_DATA_SOURCES.md](docs/EXNESS_DATA_SOURCES.md)** - Data source variants and URLs
-/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:27:- **[Makefile](Makefile)** - Module introspection commands (module-stats, module-complexity, module-deps)
-/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:31:- **[docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml](docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml)** - SSoT plan v2.0.0 (2.2x speedup)
-/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:32:- **[docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml](docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml)** - SSoT plan v2.0.0 (complete coverage)
-/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:33:- **[docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md](docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md)** - Phase 1 validation
-/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:34:- **[docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md](docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md)** - Phase 2 validation
-/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:40:- **[CONTRIBUTING.md](CONTRIBUTING.md)** - Contribution guidelines
-/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:55:**Complete Specification**: [`docs/UNIFIED_DUCKDB_PLAN_v2.md`](docs/UNIFIED_DUCKDB_PLAN_v2.md)
-/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:76:**Complete Details**: [`docs/MODULE_ARCHITECTURE.md`](docs/MODULE_ARCHITECTURE.md)
-/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:83:  - SSoT: [`docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md`](docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md)
-/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:89:  - SSoT: [`docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml`](docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml)
-/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:90:  - Validation: [`docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md`](docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md)
-/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:96:  - SSoT: [`docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml`](docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml)
-/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:109:**Complete Schema**: [`docs/DATABASE_SCHEMA.md`](docs/DATABASE_SCHEMA.md)
-/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:118:**Complete Guide**: [`docs/EXNESS_DATA_SOURCES.md`](docs/EXNESS_DATA_SOURCES.md)
-/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:128:**Complete Specification**: [`docs/RESEARCH_PATTERNS.md`](docs/RESEARCH_PATTERNS.md)
-/Users/terryli/eon/exness-data-preprocess/CLAUDE.md:196:**Complete Development Guide**: [`README.md`](README.md) (Installation section)
-/Users/terryli/eon/exness-data-preprocess/tests/README.md:30:- **New**: Phase7 13-column (v1.2.0) OHLC - See [`../src/exness_data_preprocess/schema.py`](../src/exness_data_preprocess/schema.py)
-/Users/terryli/eon/exness-data-preprocess/docs/EXNESS_DATA_SOURCES.md:310:**Full Specification**: [`docs/research/eurusd-zero-spread-deviations/data/plan/phase7_bid_ohlc_construction_v1.1.0.md`](research/eurusd-zero-spread-deviations/data/plan/phase7_bid_ohlc_construction_v1.1.0.md)
-/Users/terryli/eon/exness-data-preprocess/docs/EXNESS_DATA_SOURCES.md:386:- **Phase7 Methodology**: [`docs/research/eurusd-zero-spread-deviations/01-methodology.md`](research/eurusd-zero-spread-deviations/01-methodology.md)
-/Users/terryli/eon/exness-data-preprocess/docs/EXNESS_DATA_SOURCES.md:387:- **Zero-Spread Research**: [`docs/research/eurusd-zero-spread-deviations/README.md`](research/eurusd-zero-spread-deviations/README.md)
-/Users/terryli/eon/exness-data-preprocess/docs/EXNESS_DATA_SOURCES.md:391:- **Current Processor**: [`src/exness_data_preprocess/processor.py`](../src/exness_data_preprocess/processor.py) (needs refactoring)
-/Users/terryli/eon/exness-data-preprocess/docs/MODULE_ARCHITECTURE.md:5:**Related**: [`README.md`](../README.md) - Architecture overview
-/Users/terryli/eon/exness-data-preprocess/docs/MODULE_ARCHITECTURE.md:43:- See [`docs/validation/ARCHITECTURE_AUDIT_2025-10-17.md`](validation/ARCHITECTURE_AUDIT_2025-10-17.md) for complete audit findings
-/Users/terryli/eon/exness-data-preprocess/docs/MODULE_ARCHITECTURE.md:964:See [`Makefile`](../Makefile) for implementation.
-/Users/terryli/eon/exness-data-preprocess/docs/MODULE_ARCHITECTURE.md:1076:- **[README.md](../README.md)** - User-facing API reference
-/Users/terryli/eon/exness-data-preprocess/docs/MODULE_ARCHITECTURE.md:1077:- **[docs/README.md](README.md)** - Documentation hub
-/Users/terryli/eon/exness-data-preprocess/docs/MODULE_ARCHITECTURE.md:1078:- **[docs/DATABASE_SCHEMA.md](DATABASE_SCHEMA.md)** - Complete database schema
-/Users/terryli/eon/exness-data-preprocess/docs/MODULE_ARCHITECTURE.md:1079:- **[docs/UNIFIED_DUCKDB_PLAN_v2.md](UNIFIED_DUCKDB_PLAN_v2.md)** - v2.0.0 architecture specification
-/Users/terryli/eon/exness-data-preprocess/docs/MODULE_ARCHITECTURE.md:1080:- **[Makefile](../Makefile)** - Module introspection commands
-/Users/terryli/eon/exness-data-preprocess/docs/MODULE_ARCHITECTURE.md:1082:- **[docs/validation/ARCHITECTURE_AUDIT_2025-10-17.md](validation/ARCHITECTURE_AUDIT_2025-10-17.md)** - Complete audit findings and corrections
-/Users/terryli/eon/exness-data-preprocess/docs/DATABASE_SCHEMA.md:244:**Column Definitions**: See [`../src/exness_data_preprocess/schema.py`](../src/exness_data_preprocess/schema.py) for complete column definitions with types and descriptions.
-/Users/terryli/eon/exness-data-preprocess/docs/DATABASE_SCHEMA.md:258:**Architecture**: Exchange Registry Pattern (v1.6.0) - Session columns dynamically generated from centralized EXCHANGES dict in [`exchanges.py`](../src/exness_data_preprocess/exchanges.py)
-
-... and 90 more (showing first 50)
-
-📝 Fix: Convert relative paths to repo-relative paths starting with /
-   WRONG: [Link](../docs/foo.md)
-   WRONG: [Link](docs/foo.md)
-   RIGHT: [Link](/docs/foo.md)

exness_data_preprocess-1.2.0/CLAUDE.md

@@ -1,213 +0,0 @@
-# CLAUDE.md
-
-This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
-
-**Architecture**: Forex tick data preprocessing with unified single-file DuckDB storage
-
-**Full Documentation**: [`README.md`](README.md) - Installation, usage, API reference
-
----
-
-## Documentation Hub
-
-### User Documentation
-
-- **[README.md](README.md)** - Installation, API reference, usage examples
-- **[examples/basic_usage.py](examples/basic_usage.py)** - Download, query, coverage operations
-- **[examples/batch_processing.py](examples/batch_processing.py)** - Multi-instrument parallel processing
-
-### AI Assistant Documentation
-
-- **[docs/README.md](docs/README.md)** - Architecture, planning, research findings
-- **[docs/MODULE_ARCHITECTURE.md](docs/MODULE_ARCHITECTURE.md)** - Complete module documentation with SLOs (v1.7.0)
-- **[docs/DATABASE_SCHEMA.md](docs/DATABASE_SCHEMA.md)** - Complete database schema with self-documentation
-- **[docs/RESEARCH_PATTERNS.md](docs/RESEARCH_PATTERNS.md)** - Research lifecycle and tool selection (v1.0.0)
-- **[docs/UNIFIED_DUCKDB_PLAN_v2.md](docs/UNIFIED_DUCKDB_PLAN_v2.md)** - v2.0.0 architecture specification
-- **[docs/EXNESS_DATA_SOURCES.md](docs/EXNESS_DATA_SOURCES.md)** - Data source variants and URLs
-- **[Makefile](Makefile)** - Module introspection commands (module-stats, module-complexity, module-deps)
-
-### Optimization Plans (v1.7.0)
-
-- **[docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml](docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml)** - SSoT plan v2.0.0 (2.2x speedup)
-- **[docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml](docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml)** - SSoT plan v2.0.0 (complete coverage)
-- **[docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md](docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md)** - Phase 1 validation
-- **[docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md](docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md)** - Phase 2 validation
-
-### Implementation
-
-- **src/exness_data_preprocess/** - Source code (7 specialized modules + facade)
-- **tests/** - Test suite (48 tests, 100% passing)
-- **[CONTRIBUTING.md](CONTRIBUTING.md)** - Contribution guidelines
-
----
-
-## Architecture Summary
-
-### Storage Pattern (v2.0.0)
-
-- **Single DuckDB file per instrument** (e.g., eurusd.duckdb containing all historical data)
-- **Dual-variant storage** (Raw_Spread + Standard in same database)
-- **Incremental updates** with automatic gap detection
-- **PRIMARY KEY constraints** prevent duplicates during updates
-- **Sub-15ms query performance** with date range filtering
-- **On-demand resampling** to any timeframe (5m, 1h, 1d)
-
-**Complete Specification**: [`docs/UNIFIED_DUCKDB_PLAN_v2.md`](docs/UNIFIED_DUCKDB_PLAN_v2.md)
-
-### Module Pattern (v1.7.0)
-
-- **Facade orchestrator** (processor.py) coordinating 6 module instances + 1 static utility
-- **Separation of concerns** with single-responsibility modules
-- **SLO-based design** (Availability, Correctness, Observability, Maintainability)
-- **Off-the-shelf libraries** (httpx, pandas, DuckDB, exchange_calendars)
-- **Zero regressions** (48 tests pass)
-
-**Modules**:
-
-1. **processor.py** - Thin orchestrator facade
-2. **downloader.py** - HTTP download operations
-3. **tick_loader.py** - CSV parsing
-4. **database_manager.py** - Database operations
-5. **session_detector.py** - Holiday and session detection (v1.7.0: vectorized, 2.2x speedup)
-6. **gap_detector.py** - Incremental update logic (v1.7.0: SQL gap detection, complete coverage)
-7. **ohlc_generator.py** - Phase7 OHLC generation (v1.7.0: incremental, 7.3x speedup)
-8. **query_engine.py** - Query operations
-
-**Complete Details**: [`docs/MODULE_ARCHITECTURE.md`](docs/MODULE_ARCHITECTURE.md)
-
-### Performance Optimizations (v1.7.0)
-
-- **Incremental OHLC Generation** (Phase 1): 7.3x speedup
-  - Optional date-range filtering for regeneration
-  - Measured: 8.05s → 1.10s for incremental updates (7 months)
-  - SSoT: [`docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md`](docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md)
-
-- **Session Vectorization** (Phase 2): 2.2x speedup
-  - Pre-compute trading minutes, vectorized .isin() lookup
-  - Measured: 5.99s → 2.69s for 302K bars across 10 exchanges
-  - Combined Phase 1+2: **~16x total speedup** (8.05s → 0.50s)
-  - SSoT: [`docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml`](docs/phases/PHASE2_SESSION_VECTORIZATION_PLAN.yaml)
-  - Validation: [`docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md`](docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md)
-
-- **SQL Gap Detection** (Phase 3): Complete coverage + 46% LOC reduction
-  - SQL EXCEPT query detects ALL gaps (internal + edges)
-  - Bug fix: Now detects internal gaps missed by Python MIN/MAX
-  - Measured: 41 → 42 gaps detected (spike test validation)
-  - SSoT: [`docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml`](docs/phases/PHASE3_SQL_GAP_DETECTION_PLAN.yaml)
-
-**Pattern**: Spike test first (validate theory) → Implement → Iterative SSoT plan updates
-
-### Schema (v1.6.0)
-
-- **Phase7 30-column OHLC** with dual-variant spreads
-- **BID-only OHLC** from Raw_Spread execution prices
-- **Normalized metrics** (range_per_spread, range_per_tick, body_per_spread, body_per_tick)
-- **10 global exchange sessions with trading hour detection** (NYSE, LSE, XSWX, XFRA, XTSE, XNZE, XTKS, XASX, XHKG, XSES)
-- **Holiday tracking** (US, UK, major holidays)
-- **Self-documenting** via COMMENT ON statements
-
-**Complete Schema**: [`docs/DATABASE_SCHEMA.md`](docs/DATABASE_SCHEMA.md)
-
-### Data Sources
-
-- **Source**: https://ticks.ex2archive.com/ (public Exness tick data)
-- **Primary**: Raw_Spread variant (97.81% zero-spreads, execution prices)
-- **Reference**: Standard variant (0% zero-spreads, market quotes)
-- **Format**: Monthly ZIP files with microsecond-precision CSV
-
-**Complete Guide**: [`docs/EXNESS_DATA_SOURCES.md`](docs/EXNESS_DATA_SOURCES.md)
-
-### Research Patterns (v1.0.0)
-
-- **Lifecycle**: Explore (pandas/Polars) → Validate → Graduate (DuckDB) → Query (SQL)
-- **ASOF Performance**: pandas/Polars 0.04s, DuckDB 0.89s at 880K ticks (24x difference)
-- **Tool Selection**: pandas/Polars for tick-level temporal matching, DuckDB for materialized results
-- **Hybrid Pattern**: ASOF operations stay in pandas, validated findings materialize to DuckDB tables
-- **Single Source of Truth**: DuckDB stores validated research outcomes with COMMENT ON documentation
-
-**Complete Specification**: [`docs/RESEARCH_PATTERNS.md`](docs/RESEARCH_PATTERNS.md)
-
----
-
-## Development Commands
-
-### Setup
-
-```bash
-# Install with development dependencies
-uv sync --dev
-
-# Or with pip
-pip install -e ".[dev]"
-```
-
-### Testing
-
-```bash
-# Run all tests
-uv run pytest
-
-# Run with coverage
-uv run pytest --cov=exness_data_preprocess --cov-report=html
-
-# Run specific test file
-uv run pytest tests/test_basic.py -v
-```
-
-### Code Quality
-
-```bash
-# Format code
-uv run ruff format .
-
-# Lint and auto-fix
-uv run ruff check --fix .
-
-# Type checking
-uv run mypy src/
-```
-
-### Module Introspection
-
-```bash
-# Show current line counts (always accurate)
-make module-stats
-
-# Show cyclomatic complexity (requires radon)
-make module-complexity
-
-# Show dependency tree (requires pipdeptree)
-make module-deps
-```
-
-### Building and Publishing
-
-```bash
-# Build package
-uv build
-
-# Test installation locally
-uv tool install --editable .
-
-# Publish to PyPI (requires PYPI_TOKEN)
-doppler run --project claude-config --config dev -- uv publish --token "$PYPI_TOKEN"
-```
-
-**Complete Development Guide**: [`README.md`](README.md) (Installation section)
-
----
-
-## References
-
-- **Exness Data**: https://ticks.ex2archive.com/
-- **DuckDB**: https://duckdb.org/
-- **Apache Parquet**: https://parquet.apache.org/
-- **Zstd Compression**: https://facebook.github.io/zstd/
-
----
-
-**Version**: 2.0.0 (Architecture) + 1.7.0 (Implementation) + 1.0.0 (Research Patterns)
-**Last Updated**: 2025-10-18
-**Architecture**: Unified Single-File DuckDB Storage with Incremental Updates
-**Implementation**: Facade Pattern with 7 Specialized Modules + Performance Optimizations (~16x speedup)
-**Research**: Hybrid Materialization (pandas/Polars exploration, DuckDB validated results)