exness-data-preprocess 0.6.0__tar.gz → 0.7.0__tar.gz

{exness_data_preprocess-0.6.0 → exness_data_preprocess-0.7.0}/.cz.toml

@@ -1,6 +1,6 @@
 [tool.commitizen]
 name = "cz_conventional_commits"
-version = "0.6.0"
+version = "0.7.0"
 tag_format = "v$major.$minor.$patch"
 update_changelog_on_bump = true
 version_files = [

{exness_data_preprocess-0.6.0 → exness_data_preprocess-0.7.0}/AUTHORS.md

@@ -9,6 +9,7 @@
 ## Organization
 
 **Eon Labs**
+
 - Website: https://eonlabs.com
 - GitHub: https://github.com/Eon-Labs
 

{exness_data_preprocess-0.6.0 → exness_data_preprocess-0.7.0}/CHANGELOG.md

@@ -6,11 +6,20 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 
+### ✨ Features
+
+- **phase4**: Add dry-run mode and config file support Implements Phase 4 features from adversarial probe recommendations. Both features follow SSoT methodology with iterative plan updates. **Dry-Run Mode**: - Preview download operations without file system changes - Estimates based on historical averages (~9.5M ticks/month, ~11 MB/month) - Returns DryRunResult with gap analysis and size estimates - CLI: exness-preprocess process --dry-run **Config File Support**: - Persistent user preferences in ~/.exness-preprocess.yaml - Pydantic validation with off-the-shelf PyYAML - Priority resolution: CLI flags > config > defaults - Optional (no error if missing) - CLI: exness-preprocess --config /path/to/config.yaml **Implementation**: - config.py (new): ConfigModel with Pydantic + PyYAML loader - models.py: DryRunResult model with 2 computed fields - processor.py: dry_run + config parameters, union return type - api.py: dry_run forwarding for v1.0.0 API compatibility - cli.py: --dry-run + --config flags with precedence logic - pyproject.toml: pyyaml>=6.0.0 dependency **SLOs**: - Availability: Features optional, no breaking changes - Correctness: Gap detector reused, Pydantic validation on config - Observability: Dry-run prints estimate summary - Maintainability: Off-the-shelf PyYAML, no custom config parser **Testing**: - All 48 tests passing - Backward compatible (all new parameters optional) - Config file errors propagate (no silent fallbacks) **Time**: ~1.5 hours (as estimated in probe report) Related: v0.6.0 probe recommendations (items 4-5) SSoT Plan: docs/PHASE4_DRY_RUN_CONFIG_PLAN.yaml v2.0.0
+
+
+### 💅 Code Style
+
+- Format markdown files with prettier Auto-formatted 66 markdown files across docs, root, and tests directories to maintain consistent style and readability.
+
 ### 🔌 API Enhancements
 
-- **api**: Add AI-agent-friendly input validation (Phase 1 critical fixes) Implement adversarial probe recommendations for improved AI agent usability. This addresses the three critical issues identified in the v0.5.0 probe report. ## Changes ### 1. Early Input Validation (Critical Issue #1) - Add validation helper methods: _validate_pair(), _validate_variant(), _validate_timeframe(), _validate_date_format() - Call validators BEFORE file operations in all public methods - Clear ValueError messages instead of confusing FileNotFoundError - Example: "Invalid pair 'INVALID'. Must be one of: EURUSD, GBPUSD, ..." instead of "FileNotFoundError: Database not found" ### 2. Pydantic Field Constraints (Critical Issue #2) - Add pattern validation to CoverageInfo.earliest_date and latest_date fields - Validates date format: r'^\d{4}-\d{2}-\d{2}' (YYYY-MM-DD) - Note: ge=0 constraints already existed on all numeric fields ✓ ### 3. Raises Documentation (Critical Issue #3) - Add comprehensive Raises sections to 4 public methods: - update_data(): ValueError, FileNotFoundError, ConnectionError, pd.errors.EmptyDataError - query_ticks(): ValueError, FileNotFoundError, duckdb.Error, pd.errors.EmptyDataError - query_ohlc(): ValueError, FileNotFoundError, duckdb.Error, pd.errors.EmptyDataError - get_data_coverage(): ValueError, duckdb.Error - AI agents can now understand error handling before calling methods ## Impact Before (confusing): ```python proc.query_ticks(pair='INVALID') # FileNotFoundError: Database not found: /path/to/invalid.duckdb ``` After (clear): ```python proc.query_ticks(pair='INVALID') # ValueError: Invalid pair 'INVALID'. Must be one of: EURUSD, GBPUSD, XAUUSD, ... ``` ## Testing ✓ Import successful ✓ Validation works correctly ✓ All error messages clear and actionable ✓ No regressions (backward compatible) ## Performance - Validation overhead: <1ms per method call - Zero impact on existing functionality ## References - Adversarial Probe Report: exness-data-preprocess v0.5.0 - Priority: 🔴 CRITICAL (Issues #1, #2, #3) - Estimated time: 45 minutes (actual: 45 minutes) - ROI: ⭐⭐⭐⭐⭐ (10x better AI-agent discoverability) Closes: Critical Issues #1, #2, #3 from adversarial probe report
+- **api**: Add AI-agent-friendly input validation (Phase 1 critical fixes) Implement adversarial probe recommendations for improved AI agent usability. This addresses the three critical issues identified in the v0.5.0 probe report. ## Changes ### 1. Early Input Validation (Critical Issue #1) - Add validation helper methods: \_validate_pair(), \_validate_variant(), \_validate_timeframe(), \_validate_date_format() - Call validators BEFORE file operations in all public methods - Clear ValueError messages instead of confusing FileNotFoundError - Example: "Invalid pair 'INVALID'. Must be one of: EURUSD, GBPUSD, ..." instead of "FileNotFoundError: Database not found" ### 2. Pydantic Field Constraints (Critical Issue #2) - Add pattern validation to CoverageInfo.earliest_date and latest_date fields - Validates date format: r'^\d{4}-\d{2}-\d{2}' (YYYY-MM-DD) - Note: ge=0 constraints already existed on all numeric fields ✓ ### 3. Raises Documentation (Critical Issue #3) - Add comprehensive Raises sections to 4 public methods: - update_data(): ValueError, FileNotFoundError, ConnectionError, pd.errors.EmptyDataError - query_ticks(): ValueError, FileNotFoundError, duckdb.Error, pd.errors.EmptyDataError - query_ohlc(): ValueError, FileNotFoundError, duckdb.Error, pd.errors.EmptyDataError - get_data_coverage(): ValueError, duckdb.Error - AI agents can now understand error handling before calling methods ## Impact Before (confusing): `python proc.query_ticks(pair='INVALID') # FileNotFoundError: Database not found: /path/to/invalid.duckdb ` After (clear): `python proc.query_ticks(pair='INVALID') # ValueError: Invalid pair 'INVALID'. Must be one of: EURUSD, GBPUSD, XAUUSD, ... ` ## Testing ✓ Import successful ✓ Validation works correctly ✓ All error messages clear and actionable ✓ No regressions (backward compatible) ## Performance - Validation overhead: <1ms per method call - Zero impact on existing functionality ## References - Adversarial Probe Report: exness-data-preprocess v0.5.0 - Priority: 🔴 CRITICAL (Issues #1, #2, #3) - Estimated time: 45 minutes (actual: 45 minutes) - ROI: ⭐⭐⭐⭐⭐ (10x better AI-agent discoverability) Closes: Critical Issues #1, #2, #3 from adversarial probe report
 
-- **api**: Add computed fields, convenience methods, and context manager (Phase 2) Implements high-impact improvements for AI-agent usability following adversarial probe report recommendations. **Pydantic Computed Fields** (models.py): - UpdateResult: - avg_ticks_per_month: Average ticks per month downloaded - storage_efficiency_mb_per_million_ticks: Storage efficiency metric - CoverageInfo: - total_ticks: Sum of both data variants - coverage_percentage: Estimated trading day coverage - storage_efficiency_mb_per_million_ticks: Storage efficiency metric **Convenience Methods** (processor.py): - get_available_dates(): Returns (earliest, latest) tuple for quick boundary checks - validate_date_range(): Pre-flight validation returning (bool, error_msg) - estimate_download_size(): Estimates MB based on date range (~11 MB/month) **Context Manager Support** (processor.py): - __enter__(): Returns self for 'with' statement usage - __exit__(): Cleans up temporary ZIP files, doesn't suppress exceptions - Enables: with ExnessDataProcessor() as proc: ... **CLI Enhancement** (cli.py): - Added --version flag using importlib.metadata - Standard version display for automation scripts All methods include comprehensive docstrings with examples and type hints. Computed fields auto-appear in JSON Schema for AI agent discovery. Related: Phase 1 (f8c3ad2) - Input validation, field constraints, Raises docs Time: ~1 hour implementation
+- **api**: Add computed fields, convenience methods, and context manager (Phase 2) Implements high-impact improvements for AI-agent usability following adversarial probe report recommendations. **Pydantic Computed Fields** (models.py): - UpdateResult: - avg_ticks_per_month: Average ticks per month downloaded - storage_efficiency_mb_per_million_ticks: Storage efficiency metric - CoverageInfo: - total_ticks: Sum of both data variants - coverage_percentage: Estimated trading day coverage - storage_efficiency_mb_per_million_ticks: Storage efficiency metric **Convenience Methods** (processor.py): - get_available_dates(): Returns (earliest, latest) tuple for quick boundary checks - validate_date_range(): Pre-flight validation returning (bool, error_msg) - estimate_download_size(): Estimates MB based on date range (~11 MB/month) **Context Manager Support** (processor.py): - **enter**(): Returns self for 'with' statement usage - **exit**(): Cleans up temporary ZIP files, doesn't suppress exceptions - Enables: with ExnessDataProcessor() as proc: ... **CLI Enhancement** (cli.py): - Added --version flag using importlib.metadata - Standard version display for automation scripts All methods include comprehensive docstrings with examples and type hints. Computed fields auto-appear in JSON Schema for AI agent discovery. Related: Phase 1 (f8c3ad2) - Input validation, field constraints, Raises docs Time: ~1 hour implementation
 
 ### ✨ Features
 

{exness_data_preprocess-0.6.0 → exness_data_preprocess-0.7.0}/CLAUDE.md

@@ -11,11 +11,13 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 ## Documentation Hub
 
 ### User Documentation
+
 - **[README.md](README.md)** - Installation, API reference, usage examples
 - **[examples/basic_usage.py](examples/basic_usage.py)** - Download, query, coverage operations
 - **[examples/batch_processing.py](examples/batch_processing.py)** - Multi-instrument parallel processing
 
 ### AI Assistant Documentation
+
 - **[docs/README.md](docs/README.md)** - Architecture, planning, research findings
 - **[docs/MODULE_ARCHITECTURE.md](docs/MODULE_ARCHITECTURE.md)** - Complete module documentation with SLOs (v1.7.0)
 - **[docs/DATABASE_SCHEMA.md](docs/DATABASE_SCHEMA.md)** - Complete database schema with self-documentation
@@ -25,12 +27,14 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 - **[Makefile](Makefile)** - Module introspection commands (module-stats, module-complexity, module-deps)
 
 ### Optimization Plans (v1.7.0)
+
 - **[docs/PHASE2_SESSION_VECTORIZATION_PLAN.yaml](docs/PHASE2_SESSION_VECTORIZATION_PLAN.yaml)** - SSoT plan v2.0.0 (2.2x speedup)
 - **[docs/PHASE3_SQL_GAP_DETECTION_PLAN.yaml](docs/PHASE3_SQL_GAP_DETECTION_PLAN.yaml)** - SSoT plan v2.0.0 (complete coverage)
 - **[docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md](docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md)** - Phase 1 validation
 - **[docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md](docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md)** - Phase 2 validation
 
 ### Implementation
+
 - **src/exness_data_preprocess/** - Source code (7 specialized modules + facade)
 - **tests/** - Test suite (48 tests, 100% passing)
 - **[CONTRIBUTING.md](CONTRIBUTING.md)** - Contribution guidelines
@@ -40,6 +44,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 ## Architecture Summary
 
 ### Storage Pattern (v2.0.0)
+
 - **Single DuckDB file per instrument** (e.g., eurusd.duckdb containing all historical data)
 - **Dual-variant storage** (Raw_Spread + Standard in same database)
 - **Incremental updates** with automatic gap detection
@@ -50,6 +55,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 **Complete Specification**: [`docs/UNIFIED_DUCKDB_PLAN_v2.md`](docs/UNIFIED_DUCKDB_PLAN_v2.md)
 
 ### Module Pattern (v1.7.0)
+
 - **Facade orchestrator** (processor.py) coordinating 7 specialized modules
 - **Separation of concerns** with single-responsibility modules
 - **SLO-based design** (Availability, Correctness, Observability, Maintainability)
@@ -57,6 +63,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 - **Zero regressions** (48 tests pass)
 
 **Modules**:
+
 1. **processor.py** - Thin orchestrator facade
 2. **downloader.py** - HTTP download operations
 3. **tick_loader.py** - CSV parsing
@@ -69,6 +76,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 **Complete Details**: [`docs/MODULE_ARCHITECTURE.md`](docs/MODULE_ARCHITECTURE.md)
 
 ### Performance Optimizations (v1.7.0)
+
 - **Incremental OHLC Generation** (Phase 1): 7.3x speedup
   - Optional date-range filtering for regeneration
   - Measured: 8.05s → 1.10s for incremental updates (7 months)
@@ -90,6 +98,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 **Pattern**: Spike test first (validate theory) → Implement → Iterative SSoT plan updates
 
 ### Schema (v1.6.0)
+
 - **Phase7 30-column OHLC** with dual-variant spreads
 - **BID-only OHLC** from Raw_Spread execution prices
 - **Normalized metrics** (range_per_spread, range_per_tick, body_per_spread, body_per_tick)
@@ -100,6 +109,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 **Complete Schema**: [`docs/DATABASE_SCHEMA.md`](docs/DATABASE_SCHEMA.md)
 
 ### Data Sources
+
 - **Source**: https://ticks.ex2archive.com/ (public Exness tick data)
 - **Primary**: Raw_Spread variant (97.81% zero-spreads, execution prices)
 - **Reference**: Standard variant (0% zero-spreads, market quotes)
@@ -108,6 +118,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 **Complete Guide**: [`docs/EXNESS_DATA_SOURCES.md`](docs/EXNESS_DATA_SOURCES.md)
 
 ### Research Patterns (v1.0.0)
+
 - **Lifecycle**: Explore (pandas/Polars) → Validate → Graduate (DuckDB) → Query (SQL)
 - **ASOF Performance**: pandas/Polars 0.04s, DuckDB 0.89s at 880K ticks (24x difference)
 - **Tool Selection**: pandas/Polars for tick-level temporal matching, DuckDB for materialized results
@@ -121,6 +132,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 ## Development Commands
 
 ### Setup
+
 ```bash
 # Install with development dependencies
 uv sync --dev
@@ -130,6 +142,7 @@ pip install -e ".[dev]"
 ```
 
 ### Testing
+
 ```bash
 # Run all tests
 uv run pytest
@@ -142,6 +155,7 @@ uv run pytest tests/test_basic.py -v
 ```
 
 ### Code Quality
+
 ```bash
 # Format code
 uv run ruff format .
@@ -154,6 +168,7 @@ uv run mypy src/
 ```
 
 ### Module Introspection
+
 ```bash
 # Show current line counts (always accurate)
 make module-stats
@@ -166,6 +181,7 @@ make module-deps
 ```
 
 ### Building and Publishing
+
 ```bash
 # Build package
 uv build

{exness_data_preprocess-0.6.0 → exness_data_preprocess-0.7.0}/CONTRIBUTING.md

@@ -87,6 +87,7 @@ When reporting issues, please include:
 ### Submitting Pull Requests
 
 1. **Fork the repository** and create a new branch:
+
    ```bash
    git checkout -b feature/your-feature-name
    ```
@@ -98,6 +99,7 @@ When reporting issues, please include:
 4. **Update documentation** if needed (README.md, docstrings)
 
 5. **Run all quality checks**:
+
    ```bash
    uv run ruff format .
    uv run ruff check --fix .
@@ -106,6 +108,7 @@ When reporting issues, please include:
    ```
 
 6. **Commit your changes** with clear messages:
+
    ```bash
    git commit -m "feat: add support for custom compression levels"
    ```
@@ -128,6 +131,7 @@ We follow conventional commit format:
 - `chore:` Build process or tooling changes
 
 Examples:
+
 ```
 feat: add support for multiple currency pairs
 fix: handle timezone conversion edge case
@@ -167,6 +171,7 @@ When adding new features:
 5. **Add docstrings** following Google style
 
 Example docstring:
+
 ```python
 def my_function(param1: int, param2: str) -> bool:
     """
@@ -215,6 +220,7 @@ def test_feature_name(fixture1, fixture2):
 ### Integration Tests
 
 Mark integration tests (requiring network access) with:
+
 ```python
 @pytest.mark.integration
 def test_download_real_data():
@@ -263,6 +269,7 @@ def process_data(data: pd.DataFrame, compression: str = "zstd") -> Path:
 ### README Updates
 
 When adding features:
+
 - Add usage examples
 - Update feature list
 - Document any breaking changes

{exness_data_preprocess-0.6.0 → exness_data_preprocess-0.7.0}/DOCUMENTATION.md

@@ -21,17 +21,20 @@ New to this project? Start here:
 Follow this path if you're new to the project:
 
 ### 1. Getting Started
+
 - **[README.md](README.md)** - Main documentation with installation and API reference
 - **[Basic Usage Examples](examples/basic_usage.py)** - 6 practical code examples you can run immediately
 - **[Contributing Guide](CONTRIBUTING.md)** - Development setup, code quality, testing
 
 ### 2. Understanding the Data
+
 - **[Exness Data Sources](docs/EXNESS_DATA_SOURCES.md)** - Raw_Spread vs Standard variants, data quality characteristics
   - Learn about 97.81% zero-spreads in Raw_Spread (execution prices)
   - Understand tick density and storage estimates
   - See CSV format and download patterns
 
 ### 3. First Queries
+
 - **[Database Schema](docs/DATABASE_SCHEMA.md)** - Phase7 30-column OHLC structure
   - 4 tables: raw_spread_ticks, standard_ticks, ohlc_1m, metadata
   - SQL query examples and performance benchmarks
@@ -46,18 +49,21 @@ Follow this path if you're new to the project:
 Ready for multi-instrument processing and advanced queries:
 
 ### 4. Advanced Workflows
+
 - **[Batch Processing Examples](examples/batch_processing.py)** - Multi-instrument parallel processing
   - Sequential and parallel processing patterns
   - Error handling and retry logic
   - Progress monitoring with tqdm
 
 ### 5. Research Methodology
+
 - **[Research Patterns](docs/RESEARCH_PATTERNS.md)** - pandas/Polars vs DuckDB tool selection
   - Research lifecycle: Explore → Validate → Graduate → Query
   - Hybrid materialization pattern
   - Performance benchmarks (ASOF joins: pandas 0.04s vs DuckDB 0.89s)
 
 ### 6. Schema Deep Dive
+
 - **[Schema v1.6.0 Migration Guide](docs/plans/SCHEMA_v1.6.0_MIGRATION_GUIDE.md)** - Breaking changes from v1.5.0
   - 19 → 30 columns (added 10 exchange sessions + 1 major holiday flag)
   - Midnight detection bug fix (critical for session flags)
@@ -72,7 +78,8 @@ Ready for multi-instrument processing and advanced queries:
 For contributors and researchers who need deep understanding:
 
 ### 7. System Architecture
-- **[Module Architecture](docs/MODULE_ARCHITECTURE.md)** ⭐ *v1.3.1 - Verified Accurate*
+
+- **[Module Architecture](docs/MODULE_ARCHITECTURE.md)** ⭐ _v1.3.1 - Verified Accurate_
   - Facade pattern with 6 module instances + 1 static utility
   - SLO-based design (Availability, Correctness, Observability, Maintainability)
   - Complete data flow with 3 Mermaid diagrams
@@ -84,6 +91,7 @@ For contributors and researchers who need deep understanding:
   - PRIMARY KEY prevents duplicates
 
 ### 8. Trading Hours Implementation
+
 - **[Trading Hours Research](docs/TRADING_HOURS_RESEARCH.md)** - Industry analysis (49KB comprehensive)
   - 5 quant finance libraries studied
   - 3 professional platforms analyzed
@@ -96,7 +104,8 @@ For contributors and researchers who need deep understanding:
   - Bulk UPDATE strategies
 
 ### 9. Validation & Quality Assurance
-- **[E2E Validation Results v1.6.0](docs/validation/E2E_VALIDATION_RESULTS_v1.6.0.md)** ⭐ *100% Accuracy*
+
+- **[E2E Validation Results v1.6.0](docs/validation/E2E_VALIDATION_RESULTS_v1.6.0.md)** ⭐ _100% Accuracy_
   - 355,970 bars of real EURUSD data (12 months, 1.92 GB)
   - Tokyo lunch breaks: 0/59 lunch, 150/150 morning, 151/151 afternoon
   - Holidays, DST, weekend gaps, exchange overlaps all verified
@@ -111,7 +120,8 @@ For contributors and researchers who need deep understanding:
   - HTTP library, type signatures, data flow corrected
 
 ### 10. Performance Optimizations (v0.5.0)
-- **[Phase 1 Spike Test Results](docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md)** ⭐ *7.3x Speedup*
+
+- **[Phase 1 Spike Test Results](docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md)** ⭐ _7.3x Speedup_
   - Incremental OHLC generation: 8.05s → 1.10s for updates
   - 303K bars regeneration vs 43K incremental update
   - Measured: 86.3% time reduction
@@ -121,7 +131,7 @@ For contributors and researchers who need deep understanding:
   - Version tracking: v1.0.0 (initial) → v2.0.0 (completed)
   - Implementation discoveries and nuances documented
 
-- **[Phase 2 Spike Test Results](docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md)** ⭐ *2.2x Speedup*
+- **[Phase 2 Spike Test Results](docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md)** ⭐ _2.2x Speedup_
   - Vectorized session detection: 5.99s → 2.69s for 302K bars
   - 10 global exchange sessions processed in parallel
   - Combined Phase 1+2: ~16x total speedup (8.05s → 0.50s)
@@ -142,6 +152,7 @@ For contributors and researchers who need deep understanding:
 Published research with reproduction guides:
 
 ### Zero-Spread Deviation Analysis
+
 - **[Project Overview](docs/research/eurusd-zero-spread-deviations/README.md)** - 16-month validation study
 - **[Methodology](docs/research/eurusd-zero-spread-deviations/01-methodology.md)** - Research approach
 - **[Baseline Analysis](docs/research/eurusd-zero-spread-deviations/02-baseline-sep2024.md)** - September 2024 baseline
@@ -150,6 +161,7 @@ Published research with reproduction guides:
 - **[Reproduction Guide](docs/research/eurusd-zero-spread-deviations/scripts/reproduction_guide.md)** - How to reproduce results
 
 ### Spread Variant Analysis
+
 - **[Project Overview](docs/research/eurusd-spread-analysis/README.md)** - Modal-band-excluded variance estimation
 - **[Methodology](docs/research/eurusd-spread-analysis/01-methodology.md)** - Research design
 - **[Mode-Truncated Analysis](docs/research/eurusd-spread-analysis/02-mode-truncated-analysis.md)** - Truncation approach
@@ -159,6 +171,7 @@ Published research with reproduction guides:
 - **[Reproduction Guide](docs/research/eurusd-spread-analysis/scripts/reproduction_guide.md)** - How to reproduce
 
 ### Compression Benchmarks
+
 - **[Compression Study](docs/research/compression-benchmarks/README.md)** - Why Parquet + Zstd-22
 
 ---
@@ -168,16 +181,19 @@ Published research with reproduction guides:
 Quick lookups and specific information:
 
 ### API & Schema
+
 - **[README.md](README.md)** - Complete API reference with examples
 - **[Database Schema](docs/DATABASE_SCHEMA.md)** - 30-column Phase7 OHLC specification
 - **[CLAUDE.md](CLAUDE.md)** - AI assistant quick reference (architecture summary)
 
 ### Version History
+
 - **[CHANGELOG.md](CHANGELOG.md)** - Version history (Keep a Changelog format)
 - **[RELEASE_NOTES.md](RELEASE_NOTES.md)** - Human-readable release notes
 - **[AUTHORS.md](AUTHORS.md)** - Project contributors
 
 ### Testing
+
 - **[Test Suite Documentation](tests/README.md)** - 48 tests, coverage reports
 - **[Makefile](Makefile)** - Build commands (`make test`, `make module-stats`, etc.)
 
@@ -188,16 +204,19 @@ Quick lookups and specific information:
 Internal documentation for contributors:
 
 ### Current Plans
+
 - **[Documentation Hub](docs/README.md)** - Hub-and-spoke architecture with progressive disclosure
 - **[Implementation Plan](docs/implementation-plan.yaml)** - Machine-readable implementation plan with SLOs
 - **[Planning Index](docs/planning-index.yaml)** - Index of all planning documents
 
 ### Migration & Refactoring
+
 - **[Schema v1.6.0 Migration Guide](docs/plans/SCHEMA_v1.6.0_MIGRATION_GUIDE.md)** - v1.5.0 → v1.6.0 breaking changes
 - **[Phase7 v1.6.0 Progress](docs/plans/PHASE7_v1.6.0_REFACTORING_PROGRESS.md)** - Refactoring progress tracking
 - **[Refactoring Checklist](docs/plans/REFACTORING_CHECKLIST.md)** - Comprehensive checklist
 
 ### Quality Audits
+
 - **[Workspace Survey](docs/validation/WORKSPACE_SURVEY_2025-10-17.md)** - Organization assessment (2025-10-17)
 - **[Architecture Audit](docs/validation/ARCHITECTURE_AUDIT_2025-10-17.md)** - Documentation accuracy (2025-10-17)
 - **[Functionality Validation](docs/plans/FUNCTIONALITY_VALIDATION_REPORT_2025-10-16.md)** - Feature testing (2025-10-16)
@@ -228,6 +247,7 @@ DOCUMENTATION.md (You are here - Hub)
 ```
 
 **Navigation Tips**:
+
 - 🎯 = Start here if you're new
 - ⭐ = Critical documentation (verified accurate)
 - All links use relative URLs from project root
@@ -238,21 +258,27 @@ DOCUMENTATION.md (You are here - Hub)
 ## 🤔 Which Documentation Should I Read?
 
 ### "I just want to download forex data and analyze it"
+
 → [README.md](README.md) → [Basic Usage Examples](examples/basic_usage.py)
 
 ### "I need to process multiple currency pairs efficiently"
+
 → [Batch Processing Examples](examples/batch_processing.py)
 
 ### "I want to write SQL queries against tick data"
+
 → [Database Schema](docs/DATABASE_SCHEMA.md)
 
 ### "I'm doing quantitative research with temporal analysis"
+
 → [Research Patterns](docs/RESEARCH_PATTERNS.md) → [Database Schema](docs/DATABASE_SCHEMA.md)
 
 ### "I want to contribute code to this project"
+
 → [Module Architecture](docs/MODULE_ARCHITECTURE.md) → [Contributing Guide](CONTRIBUTING.md)
 
 ### "I need to understand trading hour detection"
+
 → [Trading Hours Research](docs/TRADING_HOURS_RESEARCH.md) → [E2E Validation](docs/validation/E2E_VALIDATION_RESULTS_v1.6.0.md)
 
 ---

{exness_data_preprocess-0.6.0 → exness_data_preprocess-0.7.0}/GITHUB_PYPI_SETUP.md

@@ -7,21 +7,25 @@
 ## ✅ Completed Automatically
 
 ### 1. Code Changes
+
 - ✅ Updated `pyproject.toml` URLs → `terrylica/exness-data-preprocess`
 - ✅ Updated `CHANGELOG.md` comparison links
 - ✅ Added 6 shields.io badges to `README.md`
 
 ### 2. GitHub Actions Workflows Created
+
 - ✅ `.github/workflows/ci.yml` - Pre-commit CI + Python 3.9-3.13 testing
 - ✅ `.github/workflows/publish.yml` - PyPI Trusted Publishing
 - ✅ `.github/workflows/pre-commit-autoupdate.yml` - Weekly hook updates
 
 ### 3. Git Repository
+
 - ✅ Initialized git repository
 - ✅ Committed all files (75 files, 19,478 lines)
 - ✅ Initial commit: `a982494`
 
 ### 4. GitHub Repository Created
+
 - ✅ Repository: https://github.com/terrylica/exness-data-preprocess
 - ✅ Public repository with description
 - ✅ Code pushed to `main` branch
@@ -30,6 +34,7 @@
 - ✅ Wiki disabled
 
 ### 5. Repository Configuration
+
 ```json
 {
   "name": "exness-data-preprocess",
@@ -37,7 +42,24 @@
   "url": "https://github.com/terrylica/exness-data-preprocess",
   "isPrivate": false,
   "hasIssuesEnabled": true,
-  "topics": ["backtesting", "data-engineering", "duckdb", "eurusd", "financial-data", "forex", "microstructure", "ohlc", "pandas", "parquet", "pyarrow", "python", "quantitative-finance", "tick-data", "time-series", "zstd"]
+  "topics": [
+    "backtesting",
+    "data-engineering",
+    "duckdb",
+    "eurusd",
+    "financial-data",
+    "forex",
+    "microstructure",
+    "ohlc",
+    "pandas",
+    "parquet",
+    "pyarrow",
+    "python",
+    "quantitative-finance",
+    "tick-data",
+    "time-series",
+    "zstd"
+  ]
 }
 ```
 
@@ -72,10 +94,12 @@ You should see a confirmation: "Pending publisher registered!"
 ### Step 5: Verify
 
 You should see the pending publisher listed on the page:
+
 - Project: `exness-data-preprocess`
 - Status: Pending (will change to Active after first successful publish)
 
 **What This Does:**
+
 - Enables password-less publishing to PyPI via GitHub Actions
 - Uses GitHub's OIDC tokens for authentication (no API tokens needed!)
 - Automatically verifies your GitHub URLs with green checkmarks on PyPI
@@ -100,11 +124,13 @@ Open: https://github.com/terrylica/exness-data-preprocess/settings/environments
 ### Step 3: Add Protection Rules (Optional)
 
 **Required Reviewers** (recommended):
+
 - Check "Required reviewers"
 - Add yourself (`terrylica`)
 - This will require you to manually approve each release
 
 **Wait Timer** (optional):
+
 - Set to 0 minutes (or add delay if wanted)
 
 ### Step 4: Save
@@ -112,6 +138,7 @@ Open: https://github.com/terrylica/exness-data-preprocess/settings/environments
 Click **"Save protection rules"**
 
 **What This Does:**
+
 - Requires manual approval before publishing to PyPI
 - Prevents accidental releases
 - Adds extra safety layer
@@ -134,6 +161,7 @@ Open: https://github.com/terrylica/exness-data-preprocess/settings/branches
 ### Step 3: Enable Protection
 
 **Recommended settings:**
+
 - ✅ Require a pull request before merging
 - ✅ Require status checks to pass before merging
   - Search and add: `Pre-commit hooks`
@@ -227,6 +255,7 @@ gh release create v0.1.0 \
 ## 📊 Monitoring & Verification
 
 ### Check CI Status
+
 ```bash
 gh run list --limit 5
 ```
@@ -234,6 +263,7 @@ gh run list --limit 5
 Or visit: https://github.com/terrylica/exness-data-preprocess/actions
 
 ### Check Latest Release
+
 ```bash
 gh release list
 ```
@@ -241,6 +271,7 @@ gh release list
 Or visit: https://github.com/terrylica/exness-data-preprocess/releases
 
 ### Test Package Installation (After First Release)
+
 ```bash
 # Create clean environment
 python -m venv /tmp/test-env
@@ -258,27 +289,32 @@ python -c "import exness_data_preprocess as edp; print(edp.__version__)"
 ## 🔧 Common Commands
 
 ### Run Pre-commit Locally
+
 ```bash
 pre-commit run --all-files
 ```
 
 ### Run Tests Locally
+
 ```bash
 uv run pytest --cov
 ```
 
 ### Build Package Locally
+
 ```bash
 uv build
 ```
 
 ### Check Package Metadata
+
 ```bash
 uv run python -m build --outdir dist/
 twine check dist/*
 ```
 
 ### Manual PyPI Upload (Not Recommended - Use Trusted Publishing Instead!)
+
 ```bash
 # Only if Trusted Publishing fails
 doppler run --project claude-config --config dev -- \
@@ -313,6 +349,7 @@ After your first successful release:
 ✅ **Environment Protection Ready** - Manual approval gate available
 
 **Total Complexity:**
+
 - Workflows: 3 files, ~120 lines YAML
 - Secrets: 0 (Trusted Publishing!)
 - Manual Token Management: None

{exness_data_preprocess-0.6.0 → exness_data_preprocess-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: exness-data-preprocess
-Version: 0.6.0
+Version: 0.7.0
 Summary: Professional Exness forex tick data preprocessing with optimal compression (Parquet Zstd-22) and DuckDB OHLC generation. Provides efficient storage (9% smaller than ZIP) with lossless precision and direct queryability.
 Project-URL: Homepage, https://github.com/terrylica/exness-data-preprocess
 Project-URL: Documentation, https://github.com/terrylica/exness-data-preprocess#readme
@@ -35,6 +35,7 @@ Requires-Dist: httpx>=0.25.0
 Requires-Dist: pandas>=2.0.0
 Requires-Dist: pyarrow>=16.0.0
 Requires-Dist: pydantic>=2.0.0
+Requires-Dist: pyyaml>=6.0.0
 Requires-Dist: tqdm>=4.65.0
 Provides-Extra: dev
 Requires-Dist: black>=23.0.0; extra == 'dev'
@@ -443,12 +444,14 @@ Data is sourced from Exness's public tick data repository:
 ### Performance Optimizations (v0.5.0)
 
 **Incremental OHLC Generation** - 7.3x speedup for updates:
+
 - Full regeneration: 8.05s (303K bars, 7 months)
 - Incremental update: 1.10s (43K new bars, 1 month)
 - Implementation: Optional date-range parameters for partial regeneration
 - Validation: [`docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md`](docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md)
 
 **Vectorized Session Detection** - 2.2x speedup for trading hour detection:
+
 - Current approach: 5.99s (302K bars, 10 exchanges)
 - Vectorized approach: 2.69s (302K bars, 10 exchanges)
 - Combined Phase 1+2: ~16x total speedup (8.05s → 0.50s)
@@ -457,6 +460,7 @@ Data is sourced from Exness's public tick data repository:
 - Validation: [`docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md`](docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md)
 
 **SQL Gap Detection** - Complete coverage with 46% code reduction:
+
 - Bug fix: Python approach missed internal gaps (41 detected vs 42 actual)
 - SQL EXCEPT operator detects ALL gaps (before + within + after existing data)
 - Code reduced from 62 lines to 34 lines (46% reduction)

{exness_data_preprocess-0.6.0 → exness_data_preprocess-0.7.0}/README.md

@@ -394,12 +394,14 @@ Data is sourced from Exness's public tick data repository:
 ### Performance Optimizations (v0.5.0)
 
 **Incremental OHLC Generation** - 7.3x speedup for updates:
+
 - Full regeneration: 8.05s (303K bars, 7 months)
 - Incremental update: 1.10s (43K new bars, 1 month)
 - Implementation: Optional date-range parameters for partial regeneration
 - Validation: [`docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md`](docs/validation/SPIKE_TEST_RESULTS_PHASE1_2025-10-18.md)
 
 **Vectorized Session Detection** - 2.2x speedup for trading hour detection:
+
 - Current approach: 5.99s (302K bars, 10 exchanges)
 - Vectorized approach: 2.69s (302K bars, 10 exchanges)
 - Combined Phase 1+2: ~16x total speedup (8.05s → 0.50s)
@@ -408,6 +410,7 @@ Data is sourced from Exness's public tick data repository:
 - Validation: [`docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md`](docs/validation/SPIKE_TEST_RESULTS_PHASE2_2025-10-18.md)
 
 **SQL Gap Detection** - Complete coverage with 46% code reduction:
+
 - Bug fix: Python approach missed internal gaps (41 detected vs 42 actual)
 - SQL EXCEPT operator detects ALL gaps (before + within + after existing data)
 - Code reduced from 62 lines to 34 lines (46% reduction)