exness-data-preprocess 0.3.1__tar.gz → 0.5.0__tar.gz

{exness_data_preprocess-0.3.1 → exness_data_preprocess-0.5.0}/.cz.toml

@@ -1,6 +1,6 @@
 [tool.commitizen]
 name = "cz_conventional_commits"
-version = "0.3.0"
+version = "0.4.0"
 tag_format = "v$major.$minor.$patch"
 update_changelog_on_bump = true
 version_files = [

exness_data_preprocess-0.5.0/CHANGELOG.md

@@ -0,0 +1,169 @@
+# Changelog
+
+All notable changes to exness-data-preprocess will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.5.0] - 2025-10-18
+
+### ⚡ Performance
+
+- **ohlc**: Implement incremental OHLC generation with 7.3x speedup
+
+  Replace full regeneration with incremental updates for new data only.
+
+  Implementation:
+  - Add optional `start_date`/`end_date` parameters to `regenerate_ohlc()`
+  - Support three modes: full regeneration, incremental append, range update
+  - Use `INSERT OR IGNORE` pattern with PRIMARY KEY for duplicate handling
+  - Track `earliest_added_month` in processor for incremental triggers
+
+  Performance (measured via spike test):
+  - Full regeneration: 8.05s (303K bars, 7 months)
+  - Incremental update: 1.10s (43K new bars, 1 month)
+  - Speedup: 7.3x (86.3% time reduction)
+
+  Validation:
+  - Spike test: 100% row count match, zero duplicates
+  - All 48 existing tests pass (backward compatible)
+
+  SSoT: docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md
+
+  BREAKING CHANGE: None (optional parameters maintain existing API)
+
+- **session**: Implement vectorized session detection with 2.2x speedup
+
+  Pre-compute trading minutes for vectorized `.isin()` lookup instead of per-timestamp `.apply()` calls.
+
+  Implementation:
+  - Add `_precompute_trading_minutes()` helper in SessionDetector
+  - Replace `.apply()` loop with vectorized `.isin()` lookup
+  - Preserve accuracy via exchange_calendars.is_open_on_minute()
+  - Handle lunch breaks (Tokyo 11:30-12:30, Hong Kong/Singapore 12:00-13:00)
+
+  Performance (measured via spike test):
+  - Current approach: 5.99s (302K bars, 10 exchanges)
+  - Vectorized approach: 2.69s (302K bars, 10 exchanges)
+  - Speedup: 2.2x (55.2% time reduction)
+  - Combined Phase 1+2: ~16x total speedup (8.05s → 0.50s)
+
+  Validation:
+  - Spike test: 100% accuracy match across all 10 exchanges
+  - All 48 existing tests pass (backward compatible)
+
+  SSoT: docs/PHASE2_SESSION_VECTORIZATION_PLAN.yaml (v2.0.0)
+  Validation: docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md
+
+  BREAKING CHANGE: None (transparent drop-in replacement)
+
+- **gap**: Implement SQL gap detection with complete coverage
+
+  Replace Python iteration with SQL EXCEPT query to detect ALL gaps (before + within + after), not just edge gaps.
+
+  Implementation:
+  - Replace lines 94-155 (62 lines) with SQL EXCEPT query (34 lines)
+  - Use DuckDB `generate_series()` + EXCEPT operator
+  - Detect internal gaps missed by Python MIN/MAX approach
+
+  Correctness (measured via spike test):
+  - Python approach: 41 gaps detected (missed internal gap 2022-03)
+  - SQL approach: 42 gaps detected (correctly found 2022-03)
+  - Bug fixed: Internal gaps now detected
+
+  Maintainability:
+  - LOC reduction: 46% (62 → 34 lines)
+  - Complexity: O(n) Python loops → O(1) SQL query
+
+  Validation:
+  - Spike test: SQL detects gaps Python iteration misses
+  - All 48 existing tests pass (backward compatible)
+
+  SSoT: docs/PHASE3_SQL_GAP_DETECTION_PLAN.yaml (v2.0.0)
+
+  BREAKING CHANGE: None (same method signature, better correctness)
+
+### 📚 Documentation
+
+- Add comprehensive SSoT plans with spike test validation
+
+  Create machine-readable OpenAPI 3.1.0 specifications documenting implementation plans with SLOs and discovered nuances.
+
+  Files Added:
+  - docs/PHASE2_SESSION_VECTORIZATION_PLAN.yaml (v1.0.0 → v2.0.0)
+  - docs/PHASE3_SQL_GAP_DETECTION_PLAN.yaml (v1.0.0 → v2.0.0)
+  - docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md
+  - docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md
+  - docs/validation/DOCUMENTATION_SURVEY_v1.7.0.md
+
+  SLOs Defined:
+  - Availability: API surface backward compatible
+  - Correctness: Exact match with existing implementation
+  - Observability: Print statements show progress
+  - Maintainability: Off-the-shelf libraries, no custom code
+
+  Pattern Established:
+  - Spike test first (validate theory before implementation)
+  - Iterative SSoT plan updates (v1.0.0 → v2.0.0 with discoveries)
+  - Measured results (no theoretical claims without validation)
+
+### ♻️ Refactoring
+
+- **schema**: Centralize OHLC schema in schema.py module - Create schema.py with OHLCSchema class as single source of truth - Reduce coupling from 42 to 5 update locations (88% reduction) - Replace hardcoded column lists in docs with pointers to schema.py - Delete obsolete PYDANTIC_TEST_STRATEGY.md (tests implemented) - Delete add_schema_comments.py (functionality merged into processor) - Update tests to use OHLCSchema.get_required_columns() - Achieve maximum DRY: future column additions require 1-2 file updates BREAKING CHANGE: None (additive refactoring, no API changes)
+
+- Extract 7 specialized modules from processor.py (Phase 1-5) Refactor processor.py using facade pattern with 7 focused modules implementing separation of concerns. Modules Created: - downloader.py: HTTP download operations - tick_loader.py: CSV parsing - database_manager.py: Database operations - session_detector.py: Holiday/session detection - gap_detector.py: Incremental update logic - ohlc_generator.py: Phase7 OHLC generation - query_engine.py: Query operations Design Principles: - Facade pattern: processor.py delegates to specialized modules - SLO-based: All modules define Availability, Correctness, Observability, Maintainability - Off-the-shelf: httpx, pandas, DuckDB, exchange_calendars (no custom implementations) - Zero regressions: All 48 tests pass, all ruff checks pass Documentation: - Updated CLAUDE.md with module structure and facade pattern - Updated docs/README.md with implementation architecture v1.3.0 - Added planning documents (REFACTORING_CHECKLIST.md, PHASE7_v1.6.0_REFACTORING_PROGRESS.md) Version: 1.3.0 (Implementation) Validation: 48 tests pass, ruff checks pass, backward compatible
+
+
+### ✨ Features
+
+- Implement pydantic v2 models with dual-variant e2e testing - Add Pydantic v2 models (UpdateResult, CoverageInfo) for type-safe API - Update processor to use Pydantic types (PairType, TimeframeType, VariantType) - Remove deprecated api.py module (broken functionality) - Implement comprehensive test suite (48 tests, 100% passing): * test_models.py - Pydantic model validation (13 tests) * test_types.py - Type safety and helpers (15 tests) * test_processor_pydantic.py - Integration tests (6 tests) * test_functional_regression.py - v2.0.0 regression tests (10 tests) - Fix Standard variant downloads (variant="" not "Standard") - Add true end-to-end testing with real Exness downloads - Test data: EURUSD August 2024 (815K Raw_Spread + 877K Standard ticks) - Update documentation and add refactoring status tracking - Coverage: models.py 100%, __init__.py 100%, processor.py 45% BREAKING CHANGE: api.py removed (use ExnessDataProcessor directly)
+
+
+### 📚 Documentation
+
+- Update all schema references from 9-column to 13-column (v1.2.0) Update outdated "9-column" or "Phase7 9 columns" references to "13-column (v1.2.0)" across all documentation files to reflect the current schema version with normalized spread metrics. Files updated: - GITHUB_PYPI_SETUP.md - Release notes reference - PYDANTIC_REFACTORING_PLAN.md - API documentation and examples (4 locations) - docs/DATABASE_SCHEMA.md - Version history (2 locations) - docs/README.md - Validation results - docs/UNIFIED_DUCKDB_PLAN_v2.md - Table descriptions and return types (2 locations) - tests/README.md - Test descriptions (2 locations) All outdated schema references now consistently reference the Phase7 13-column (v1.2.0) schema with normalized metrics (range_per_spread, range_per_tick, body_per_spread, body_per_tick). CHANGELOG.md and research/archive files intentionally preserved for historical accuracy.
+
+
+### 🔧 Continuous Integration
+
+- Modernize publish workflow with uv and validation pipeline Update GitHub Actions publish workflow to match gapless-crypto-data standards: Build System: - Replace pip+build with uv for consistency with local development - Add UV_SYSTEM_PYTHON=1 environment variable - Use uv sync --dev for dependency installation Validation Pipeline: - Add file encoding validation (UTF-8/ASCII enforcement) - Add ruff linting and format checking - Add pytest test suite execution before build Security: - Add Sigstore artifact signing for published packages - Maintain PyPI Trusted Publishing (OIDC) authentication Configuration: - Change environment name: release → pypi (semantic clarity) - Change artifact name: dist → python-package-distributions (consistency) - Add verbose output to PyPI publish step Security Note: - Added .token-info.md to .gitignore to prevent accidental token documentation commits - Workflow uses OIDC (no API token needed)
+
+
+### 🧰 Maintenance
+
+- **release**: Bump version 0.1.0 → 0.2.0 [skip ci]
+
+- **release**: Bump version 0.2.0 → 0.3.0 [skip ci]
+
+- **release**: Bump version 0.3.0 → 0.3.1 [skip ci]
+
+
+### ✨ Features
+
+- Initial commit with v2.0.0 unified DuckDB architecture - Unified single-file DuckDB storage per instrument (eurusd.duckdb) - Phase7 9-column OHLC schema with dual-variant spreads - Sub-15ms query performance with date range filtering - PRIMARY KEY constraints prevent duplicates - Automatic gap detection for incremental updates - DuckDB self-documentation with COMMENT ON statements - GitHub Actions CI/CD with PyPI Trusted Publishing - Pre-commit hooks (ruff, commitizen, basic checks) - Professional README with shields.io badges
+
+
+### 🐛 Bug Fixes
+
+- Force add .pre-commit-config.yaml (was incorrectly gitignored)
+
+- Remove .pre-commit-config.yaml from .gitignore Pre-commit config should be version controlled, not ignored.
+
+- **pre-commit**: Enable unsafe fixes for ruff to resolve remaining errors
+
+- **research**: Resolve remaining linting errors in research scripts - Fix bare except clause in 02_volatility_model_simple.py (E722) - Move math.erf imports to top in 03_liquidity_crisis_detection.py (E402) - Move math.erf imports to top in 04_regime_detection_analysis.py (E402)
+
+
+### 💅 Code Style
+
+- Apply pre-commit formatting fixes - Fix end-of-file newlines in 2 MD files - Apply ruff quote style fixes (200 errors fixed) - Reformat 26 files with ruff-format
+
+
+### 🔧 CI/CD Improvements
+
+- **ci**: Add pre-commit config and basic tests - Add .pre-commit-config.yaml to repository - Create basic test suite (test_basic.py) for CI - Add GITHUB_PYPI_SETUP.md with complete setup instructions Fixes CI failures: - Pre-commit hook now finds config file - Pytest now finds and runs tests successfully
+
+
+### 🧰 Maintenance
+
+- Prepare v0.1.0 release

exness_data_preprocess-0.5.0/CLAUDE.md

@@ -0,0 +1,197 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+**Architecture**: Forex tick data preprocessing with unified single-file DuckDB storage
+
+**Full Documentation**: [`README.md`](README.md) - Installation, usage, API reference
+
+---
+
+## Documentation Hub
+
+### User Documentation
+- **[README.md](README.md)** - Installation, API reference, usage examples
+- **[examples/basic_usage.py](examples/basic_usage.py)** - Download, query, coverage operations
+- **[examples/batch_processing.py](examples/batch_processing.py)** - Multi-instrument parallel processing
+
+### AI Assistant Documentation
+- **[docs/README.md](docs/README.md)** - Architecture, planning, research findings
+- **[docs/MODULE_ARCHITECTURE.md](docs/MODULE_ARCHITECTURE.md)** - Complete module documentation with SLOs (v1.7.0)
+- **[docs/DATABASE_SCHEMA.md](docs/DATABASE_SCHEMA.md)** - Complete database schema with self-documentation
+- **[docs/RESEARCH_PATTERNS.md](docs/RESEARCH_PATTERNS.md)** - Research lifecycle and tool selection (v1.0.0)
+- **[docs/UNIFIED_DUCKDB_PLAN_v2.md](docs/UNIFIED_DUCKDB_PLAN_v2.md)** - v2.0.0 architecture specification
+- **[docs/EXNESS_DATA_SOURCES.md](docs/EXNESS_DATA_SOURCES.md)** - Data source variants and URLs
+- **[Makefile](Makefile)** - Module introspection commands (module-stats, module-complexity, module-deps)
+
+### Optimization Plans (v1.7.0)
+- **[docs/PHASE2_SESSION_VECTORIZATION_PLAN.yaml](docs/PHASE2_SESSION_VECTORIZATION_PLAN.yaml)** - SSoT plan v2.0.0 (2.2x speedup)
+- **[docs/PHASE3_SQL_GAP_DETECTION_PLAN.yaml](docs/PHASE3_SQL_GAP_DETECTION_PLAN.yaml)** - SSoT plan v2.0.0 (complete coverage)
+- **[docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md](docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md)** - Phase 1 validation
+- **[docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md](docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md)** - Phase 2 validation
+
+### Implementation
+- **src/exness_data_preprocess/** - Source code (7 specialized modules + facade)
+- **tests/** - Test suite (48 tests, 100% passing)
+- **[CONTRIBUTING.md](CONTRIBUTING.md)** - Contribution guidelines
+
+---
+
+## Architecture Summary
+
+### Storage Pattern (v2.0.0)
+- **Single DuckDB file per instrument** (e.g., eurusd.duckdb containing all historical data)
+- **Dual-variant storage** (Raw_Spread + Standard in same database)
+- **Incremental updates** with automatic gap detection
+- **PRIMARY KEY constraints** prevent duplicates during updates
+- **Sub-15ms query performance** with date range filtering
+- **On-demand resampling** to any timeframe (5m, 1h, 1d)
+
+**Complete Specification**: [`docs/UNIFIED_DUCKDB_PLAN_v2.md`](docs/UNIFIED_DUCKDB_PLAN_v2.md)
+
+### Module Pattern (v1.7.0)
+- **Facade orchestrator** (processor.py) coordinating 7 specialized modules
+- **Separation of concerns** with single-responsibility modules
+- **SLO-based design** (Availability, Correctness, Observability, Maintainability)
+- **Off-the-shelf libraries** (httpx, pandas, DuckDB, exchange_calendars)
+- **Zero regressions** (48 tests pass)
+
+**Modules**:
+1. **processor.py** - Thin orchestrator facade
+2. **downloader.py** - HTTP download operations
+3. **tick_loader.py** - CSV parsing
+4. **database_manager.py** - Database operations
+5. **session_detector.py** - Holiday and session detection (v1.7.0: vectorized, 2.2x speedup)
+6. **gap_detector.py** - Incremental update logic (v1.7.0: SQL gap detection, complete coverage)
+7. **ohlc_generator.py** - Phase7 OHLC generation (v1.7.0: incremental, 7.3x speedup)
+8. **query_engine.py** - Query operations
+
+**Complete Details**: [`docs/MODULE_ARCHITECTURE.md`](docs/MODULE_ARCHITECTURE.md)
+
+### Performance Optimizations (v1.7.0)
+- **Incremental OHLC Generation** (Phase 1): 7.3x speedup
+  - Optional date-range filtering for regeneration
+  - Measured: 8.05s → 1.10s for incremental updates (7 months)
+  - SSoT: [`docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md`](docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md)
+
+- **Session Vectorization** (Phase 2): 2.2x speedup
+  - Pre-compute trading minutes, vectorized .isin() lookup
+  - Measured: 5.99s → 2.69s for 302K bars across 10 exchanges
+  - Combined Phase 1+2: **~16x total speedup** (8.05s → 0.50s)
+  - SSoT: [`docs/PHASE2_SESSION_VECTORIZATION_PLAN.yaml`](docs/PHASE2_SESSION_VECTORIZATION_PLAN.yaml)
+  - Validation: [`docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md`](docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md)
+
+- **SQL Gap Detection** (Phase 3): Complete coverage + 46% LOC reduction
+  - SQL EXCEPT query detects ALL gaps (internal + edges)
+  - Bug fix: Now detects internal gaps missed by Python MIN/MAX
+  - Measured: 41 → 42 gaps detected (spike test validation)
+  - SSoT: [`docs/PHASE3_SQL_GAP_DETECTION_PLAN.yaml`](docs/PHASE3_SQL_GAP_DETECTION_PLAN.yaml)
+
+**Pattern**: Spike test first (validate theory) → Implement → Iterative SSoT plan updates
+
+### Schema (v1.6.0)
+- **Phase7 30-column OHLC** with dual-variant spreads
+- **BID-only OHLC** from Raw_Spread execution prices
+- **Normalized metrics** (range_per_spread, range_per_tick, body_per_spread, body_per_tick)
+- **10 global exchange sessions with trading hour detection** (NYSE, LSE, XSWX, XFRA, XTSE, XNZE, XTKS, XASX, XHKG, XSES)
+- **Holiday tracking** (US, UK, major holidays)
+- **Self-documenting** via COMMENT ON statements
+
+**Complete Schema**: [`docs/DATABASE_SCHEMA.md`](docs/DATABASE_SCHEMA.md)
+
+### Data Sources
+- **Source**: https://ticks.ex2archive.com/ (public Exness tick data)
+- **Primary**: Raw_Spread variant (97.81% zero-spreads, execution prices)
+- **Reference**: Standard variant (0% zero-spreads, market quotes)
+- **Format**: Monthly ZIP files with microsecond-precision CSV
+
+**Complete Guide**: [`docs/EXNESS_DATA_SOURCES.md`](docs/EXNESS_DATA_SOURCES.md)
+
+### Research Patterns (v1.0.0)
+- **Lifecycle**: Explore (pandas/Polars) → Validate → Graduate (DuckDB) → Query (SQL)
+- **ASOF Performance**: pandas/Polars 0.04s, DuckDB 0.89s at 880K ticks (24x difference)
+- **Tool Selection**: pandas/Polars for tick-level temporal matching, DuckDB for materialized results
+- **Hybrid Pattern**: ASOF operations stay in pandas, validated findings materialize to DuckDB tables
+- **Single Source of Truth**: DuckDB stores validated research outcomes with COMMENT ON documentation
+
+**Complete Specification**: [`docs/RESEARCH_PATTERNS.md`](docs/RESEARCH_PATTERNS.md)
+
+---
+
+## Development Commands
+
+### Setup
+```bash
+# Install with development dependencies
+uv sync --dev
+
+# Or with pip
+pip install -e ".[dev]"
+```
+
+### Testing
+```bash
+# Run all tests
+uv run pytest
+
+# Run with coverage
+uv run pytest --cov=exness_data_preprocess --cov-report=html
+
+# Run specific test file
+uv run pytest tests/test_basic.py -v
+```
+
+### Code Quality
+```bash
+# Format code
+uv run ruff format .
+
+# Lint and auto-fix
+uv run ruff check --fix .
+
+# Type checking
+uv run mypy src/
+```
+
+### Module Introspection
+```bash
+# Show current line counts (always accurate)
+make module-stats
+
+# Show cyclomatic complexity (requires radon)
+make module-complexity
+
+# Show dependency tree (requires pipdeptree)
+make module-deps
+```
+
+### Building and Publishing
+```bash
+# Build package
+uv build
+
+# Test installation locally
+uv tool install --editable .
+
+# Publish to PyPI (requires PYPI_TOKEN)
+doppler run --project claude-config --config dev -- uv publish --token "$PYPI_TOKEN"
+```
+
+**Complete Development Guide**: [`README.md`](README.md) (Installation section)
+
+---
+
+## References
+
+- **Exness Data**: https://ticks.ex2archive.com/
+- **DuckDB**: https://duckdb.org/
+- **Apache Parquet**: https://parquet.apache.org/
+- **Zstd Compression**: https://facebook.github.io/zstd/
+
+---
+
+**Version**: 2.0.0 (Architecture) + 1.7.0 (Implementation) + 1.0.0 (Research Patterns)
+**Last Updated**: 2025-10-18
+**Architecture**: Unified Single-File DuckDB Storage with Incremental Updates
+**Implementation**: Facade Pattern with 7 Specialized Modules + Performance Optimizations (~16x speedup)
+**Research**: Hybrid Materialization (pandas/Polars exploration, DuckDB validated results)

exness_data_preprocess-0.5.0/DOCUMENTATION.md

@@ -0,0 +1,270 @@
+# 📚 Documentation Hub
+
+**Welcome!** This page organizes all documentation from beginner to advanced, helping you find exactly what you need.
+
+---
+
+## 🎯 Quick Start (5 Minutes)
+
+New to this project? Start here:
+
+1. **[README.md](README.md)** - Installation, features, and quick start guide
+2. **[Basic Usage Examples](examples/basic_usage.py)** - Download data, query OHLC/ticks, coverage checks
+3. **[Contributing Guide](CONTRIBUTING.md)** - Want to contribute? Start here
+
+**Next Steps**: Read [Data Sources](#-understanding-the-data) below to understand what data you're working with.
+
+---
+
+## 📖 Beginner Path (Your First Hour)
+
+Follow this path if you're new to the project:
+
+### 1. Getting Started
+- **[README.md](README.md)** - Main documentation with installation and API reference
+- **[Basic Usage Examples](examples/basic_usage.py)** - 6 practical code examples you can run immediately
+- **[Contributing Guide](CONTRIBUTING.md)** - Development setup, code quality, testing
+
+### 2. Understanding the Data
+- **[Exness Data Sources](docs/EXNESS_DATA_SOURCES.md)** - Raw_Spread vs Standard variants, data quality characteristics
+  - Learn about 97.81% zero-spreads in Raw_Spread (execution prices)
+  - Understand tick density and storage estimates
+  - See CSV format and download patterns
+
+### 3. First Queries
+- **[Database Schema](docs/DATABASE_SCHEMA.md)** - Phase7 30-column OHLC structure
+  - 4 tables: raw_spread_ticks, standard_ticks, ohlc_1m, metadata
+  - SQL query examples and performance benchmarks
+  - 10 global exchange sessions with trading hours
+
+**Checkpoint**: After this section, you should be able to download data and run basic queries.
+
+---
+
+## 🚀 Intermediate Path (Production Usage)
+
+Ready for multi-instrument processing and advanced queries:
+
+### 4. Advanced Workflows
+- **[Batch Processing Examples](examples/batch_processing.py)** - Multi-instrument parallel processing
+  - Sequential and parallel processing patterns
+  - Error handling and retry logic
+  - Progress monitoring with tqdm
+
+### 5. Research Methodology
+- **[Research Patterns](docs/RESEARCH_PATTERNS.md)** - pandas/Polars vs DuckDB tool selection
+  - Research lifecycle: Explore → Validate → Graduate → Query
+  - Hybrid materialization pattern
+  - Performance benchmarks (ASOF joins: pandas 0.04s vs DuckDB 0.89s)
+
+### 6. Schema Deep Dive
+- **[Schema v1.6.0 Migration Guide](docs/plans/SCHEMA_v1.6.0_MIGRATION_GUIDE.md)** - Breaking changes from v1.5.0
+  - 19 → 30 columns (added 10 exchange sessions + 1 major holiday flag)
+  - Midnight detection bug fix (critical for session flags)
+  - Migration steps and validation
+
+**Checkpoint**: You can now process multiple instruments efficiently and choose the right tools for research.
+
+---
+
+## 🏗️ Advanced Path (Architecture & Development)
+
+For contributors and researchers who need deep understanding:
+
+### 7. System Architecture
+- **[Module Architecture](docs/MODULE_ARCHITECTURE.md)** ⭐ *v1.3.1 - Verified Accurate*
+  - Facade pattern with 6 module instances + 1 static utility
+  - SLO-based design (Availability, Correctness, Observability, Maintainability)
+  - Complete data flow with 3 Mermaid diagrams
+  - See [Architecture Audit](docs/validation/ARCHITECTURE_AUDIT_2025-10-17.md) for verification
+
+- **[Unified DuckDB Plan v2.0.0](docs/UNIFIED_DUCKDB_PLAN_v2.md)** - Storage architecture
+  - Single-file per instrument (eurusd.duckdb)
+  - Incremental updates with gap detection
+  - PRIMARY KEY prevents duplicates
+
+### 8. Trading Hours Implementation
+- **[Trading Hours Research](docs/TRADING_HOURS_RESEARCH.md)** - Industry analysis (49KB comprehensive)
+  - 5 quant finance libraries studied
+  - 3 professional platforms analyzed
+  - 4 time-series databases compared
+  - Unanimous consensus for calendar abstraction
+
+- **[Hybrid Session Detection Analysis](docs/research/HYBRID_SESSION_DETECTION_ANALYSIS.md)** - Implementation rationale
+  - Why holidays are date-level but sessions are minute-level
+  - DuckDB performance optimization
+  - Bulk UPDATE strategies
+
+### 9. Validation & Quality Assurance
+- **[E2E Validation Results v1.6.0](docs/validation/E2E_VALIDATION_RESULTS_v1.6.0.md)** ⭐ *100% Accuracy*
+  - 355,970 bars of real EURUSD data (12 months, 1.92 GB)
+  - Tokyo lunch breaks: 0/59 lunch, 150/150 morning, 151/151 afternoon
+  - Holidays, DST, weekend gaps, exchange overlaps all verified
+
+- **[Weekend Gap Validation](docs/validation/WEEKEND_GAP_VALIDATION_SUMMARY.md)** - Timezone-aware weekend handling
+  - 48-hour forex gap (Friday 22:00 UTC → Sunday 22:00 UTC)
+  - New Zealand opens first (Monday 10:00 NZST = Sunday 21:00 UTC)
+
+- **[Architecture Audit Report](docs/validation/ARCHITECTURE_AUDIT_2025-10-17.md)** - Documentation accuracy verification
+  - 78% → 100% accuracy after comprehensive audit
+  - All constructor signatures verified
+  - HTTP library, type signatures, data flow corrected
+
+### 10. Performance Optimizations (v0.5.0)
+- **[Phase 1 Spike Test Results](docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md)** ⭐ *7.3x Speedup*
+  - Incremental OHLC generation: 8.05s → 1.10s for updates
+  - 303K bars regeneration vs 43K incremental update
+  - Measured: 86.3% time reduction
+
+- **[Phase 2 SSoT Plan](docs/PHASE2_SESSION_VECTORIZATION_PLAN.yaml)** - Session vectorization specification
+  - OpenAPI 3.1.0 machine-readable plan with SLOs
+  - Version tracking: v1.0.0 (initial) → v2.0.0 (completed)
+  - Implementation discoveries and nuances documented
+
+- **[Phase 2 Spike Test Results](docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md)** ⭐ *2.2x Speedup*
+  - Vectorized session detection: 5.99s → 2.69s for 302K bars
+  - 10 global exchange sessions processed in parallel
+  - Combined Phase 1+2: ~16x total speedup (8.05s → 0.50s)
+
+- **[Phase 3 SSoT Plan](docs/PHASE3_SQL_GAP_DETECTION_PLAN.yaml)** - SQL gap detection specification
+  - OpenAPI 3.1.0 specification with implementation strategy
+  - Bug fix documentation: Internal gaps now detected
+  - 46% LOC reduction (62 → 34 lines)
+
+- **[CHANGELOG v0.5.0](CHANGELOG.md)** - Complete release notes with measured results
+
+**Checkpoint**: You understand the optimization strategies and spike-test-first validation methodology.
+
+---
+
+## 📊 Research Projects (Deep Dives)
+
+Published research with reproduction guides:
+
+### Zero-Spread Deviation Analysis
+- **[Project Overview](docs/research/eurusd-zero-spread-deviations/README.md)** - 16-month validation study
+- **[Methodology](docs/research/eurusd-zero-spread-deviations/01-methodology.md)** - Research approach
+- **[Baseline Analysis](docs/research/eurusd-zero-spread-deviations/02-baseline-sep2024.md)** - September 2024 baseline
+- **[Multi-Period Validation](docs/research/eurusd-zero-spread-deviations/03-multiperiod-validation.md)** - Extended validation
+- **[Discoveries](docs/research/eurusd-zero-spread-deviations/04-discoveries-and-plan-evolution.md)** - Plan evolution
+- **[Reproduction Guide](docs/research/eurusd-zero-spread-deviations/scripts/reproduction_guide.md)** - How to reproduce results
+
+### Spread Variant Analysis
+- **[Project Overview](docs/research/eurusd-spread-analysis/README.md)** - Modal-band-excluded variance estimation
+- **[Methodology](docs/research/eurusd-spread-analysis/01-methodology.md)** - Research design
+- **[Mode-Truncated Analysis](docs/research/eurusd-spread-analysis/02-mode-truncated-analysis.md)** - Truncation approach
+- **[Hurdle Model](docs/research/eurusd-spread-analysis/03-hurdle-model-analysis.md)** - Two-part model
+- **[Temporal Patterns](docs/research/eurusd-spread-analysis/04-temporal-patterns.md)** - Time-based analysis
+- **[Recommendations](docs/research/eurusd-spread-analysis/05-final-recommendations.md)** - Final guidance
+- **[Reproduction Guide](docs/research/eurusd-spread-analysis/scripts/reproduction_guide.md)** - How to reproduce
+
+### Compression Benchmarks
+- **[Compression Study](docs/research/compression-benchmarks/README.md)** - Why Parquet + Zstd-22
+
+---
+
+## 📚 Reference Documentation
+
+Quick lookups and specific information:
+
+### API & Schema
+- **[README.md](README.md)** - Complete API reference with examples
+- **[Database Schema](docs/DATABASE_SCHEMA.md)** - 30-column Phase7 OHLC specification
+- **[CLAUDE.md](CLAUDE.md)** - AI assistant quick reference (architecture summary)
+
+### Version History
+- **[CHANGELOG.md](CHANGELOG.md)** - Version history (Keep a Changelog format)
+- **[RELEASE_NOTES.md](RELEASE_NOTES.md)** - Human-readable release notes
+- **[AUTHORS.md](AUTHORS.md)** - Project contributors
+
+### Testing
+- **[Test Suite Documentation](tests/README.md)** - 48 tests, coverage reports
+- **[Makefile](Makefile)** - Build commands (`make test`, `make module-stats`, etc.)
+
+---
+
+## 🔧 Development & Planning
+
+Internal documentation for contributors:
+
+### Current Plans
+- **[Documentation Hub](docs/README.md)** - Hub-and-spoke architecture with progressive disclosure
+- **[Implementation Plan](docs/implementation-plan.yaml)** - Machine-readable implementation plan with SLOs
+- **[Planning Index](docs/planning-index.yaml)** - Index of all planning documents
+
+### Migration & Refactoring
+- **[Schema v1.6.0 Migration Guide](docs/plans/SCHEMA_v1.6.0_MIGRATION_GUIDE.md)** - v1.5.0 → v1.6.0 breaking changes
+- **[Phase7 v1.6.0 Progress](docs/plans/PHASE7_v1.6.0_REFACTORING_PROGRESS.md)** - Refactoring progress tracking
+- **[Refactoring Checklist](docs/plans/REFACTORING_CHECKLIST.md)** - Comprehensive checklist
+
+### Quality Audits
+- **[Workspace Survey](docs/validation/WORKSPACE_SURVEY_2025-10-17.md)** - Organization assessment (2025-10-17)
+- **[Architecture Audit](docs/validation/ARCHITECTURE_AUDIT_2025-10-17.md)** - Documentation accuracy (2025-10-17)
+- **[Functionality Validation](docs/plans/FUNCTIONALITY_VALIDATION_REPORT_2025-10-16.md)** - Feature testing (2025-10-16)
+
+---
+
+## 📁 Archive
+
+Historical documentation (for reference only):
+
+- **[Unified DuckDB Plan v1.0.0](docs/archive/UNIFIED_DUCKDB_PLAN_v1.0.0_LEGACY.md)** - Legacy architecture (superseded by v2.0.0)
+- **[Pydantic Refactoring Plan](docs/archive/PYDANTIC_REFACTORING_PLAN_v0.2.0_COMPLETE.md)** - Completed v0.2.0
+- **[E2E Testing Plan](tests/archive/E2E_TESTING_PLAN_v1.0.0.md)** - Archived testing approach
+
+---
+
+## 🗺️ Documentation Organization
+
+This project uses a **hub-and-spoke model** with progressive disclosure:
+
+```
+DOCUMENTATION.md (You are here - Hub)
+├── Beginner Path → README → Examples → Data Sources
+├── Intermediate Path → Batch Processing → Research Patterns → Migration Guide
+├── Advanced Path → Module Architecture → DuckDB Plan → Trading Hours Research
+├── Research Projects → Zero-Spread Analysis → Spread Variant Analysis
+└── Reference → API Docs → Schema → Testing
+```
+
+**Navigation Tips**:
+- 🎯 = Start here if you're new
+- ⭐ = Critical documentation (verified accurate)
+- All links use relative URLs from project root
+- Use browser search (Ctrl+F / Cmd+F) to find specific topics
+
+---
+
+## 🤔 Which Documentation Should I Read?
+
+### "I just want to download forex data and analyze it"
+→ [README.md](README.md) → [Basic Usage Examples](examples/basic_usage.py)
+
+### "I need to process multiple currency pairs efficiently"
+→ [Batch Processing Examples](examples/batch_processing.py)
+
+### "I want to write SQL queries against tick data"
+→ [Database Schema](docs/DATABASE_SCHEMA.md)
+
+### "I'm doing quantitative research with temporal analysis"
+→ [Research Patterns](docs/RESEARCH_PATTERNS.md) → [Database Schema](docs/DATABASE_SCHEMA.md)
+
+### "I want to contribute code to this project"
+→ [Module Architecture](docs/MODULE_ARCHITECTURE.md) → [Contributing Guide](CONTRIBUTING.md)
+
+### "I need to understand trading hour detection"
+→ [Trading Hours Research](docs/TRADING_HOURS_RESEARCH.md) → [E2E Validation](docs/validation/E2E_VALIDATION_RESULTS_v1.6.0.md)
+
+---
+
+## 📞 Getting Help
+
+- **Issues**: [GitHub Issues](https://github.com/terrylica/exness-data-preprocess/issues)
+- **Contributing**: See [CONTRIBUTING.md](CONTRIBUTING.md)
+- **License**: [MIT License](LICENSE)
+
+---
+
+**Last Updated**: 2025-10-18
+**Documentation Version**: v1.7.0
+**Total Documentation Files**: 70 markdown files + 2 Python examples + 4 YAML specs