exness-data-preprocess 0.3.0__tar.gz → 0.4.0__tar.gz

{exness_data_preprocess-0.3.0 → exness_data_preprocess-0.4.0}/.cz.toml

@@ -1,6 +1,6 @@
 [tool.commitizen]
 name = "cz_conventional_commits"
-version = "0.2.0"
+version = "0.4.0"
 tag_format = "v$major.$minor.$patch"
 update_changelog_on_bump = true
 version_files = [

{exness_data_preprocess-0.3.0 → exness_data_preprocess-0.4.0}/.gitignore

@@ -72,6 +72,7 @@ data/
 *.zip
 *.csv
 temp/
+eurusd
 
 # macOS
 .DS_Store

{exness_data_preprocess-0.3.0 → exness_data_preprocess-0.4.0}/CHANGELOG.md

@@ -6,11 +6,23 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 
+### ♻️ Refactoring
+
+- **schema**: Centralize OHLC schema in schema.py module - Create schema.py with OHLCSchema class as single source of truth - Reduce coupling from 42 to 5 update locations (88% reduction) - Replace hardcoded column lists in docs with pointers to schema.py - Delete obsolete PYDANTIC_TEST_STRATEGY.md (tests implemented) - Delete add_schema_comments.py (functionality merged into processor) - Update tests to use OHLCSchema.get_required_columns() - Achieve maximum DRY: future column additions require 1-2 file updates BREAKING CHANGE: None (additive refactoring, no API changes)
+
+- Extract 7 specialized modules from processor.py (Phase 1-5) Refactor processor.py using facade pattern with 7 focused modules implementing separation of concerns. Modules Created: - downloader.py: HTTP download operations - tick_loader.py: CSV parsing - database_manager.py: Database operations - session_detector.py: Holiday/session detection - gap_detector.py: Incremental update logic - ohlc_generator.py: Phase7 OHLC generation - query_engine.py: Query operations Design Principles: - Facade pattern: processor.py delegates to specialized modules - SLO-based: All modules define Availability, Correctness, Observability, Maintainability - Off-the-shelf: httpx, pandas, DuckDB, exchange_calendars (no custom implementations) - Zero regressions: All 48 tests pass, all ruff checks pass Documentation: - Updated CLAUDE.md with module structure and facade pattern - Updated docs/README.md with implementation architecture v1.3.0 - Added planning documents (REFACTORING_CHECKLIST.md, PHASE7_v1.6.0_REFACTORING_PROGRESS.md) Version: 1.3.0 (Implementation) Validation: 48 tests pass, ruff checks pass, backward compatible
+
+
 ### ✨ Features
 
 - Implement pydantic v2 models with dual-variant e2e testing - Add Pydantic v2 models (UpdateResult, CoverageInfo) for type-safe API - Update processor to use Pydantic types (PairType, TimeframeType, VariantType) - Remove deprecated api.py module (broken functionality) - Implement comprehensive test suite (48 tests, 100% passing): * test_models.py - Pydantic model validation (13 tests) * test_types.py - Type safety and helpers (15 tests) * test_processor_pydantic.py - Integration tests (6 tests) * test_functional_regression.py - v2.0.0 regression tests (10 tests) - Fix Standard variant downloads (variant="" not "Standard") - Add true end-to-end testing with real Exness downloads - Test data: EURUSD August 2024 (815K Raw_Spread + 877K Standard ticks) - Update documentation and add refactoring status tracking - Coverage: models.py 100%, __init__.py 100%, processor.py 45% BREAKING CHANGE: api.py removed (use ExnessDataProcessor directly)
 
 
+### 📚 Documentation
+
+- Update all schema references from 9-column to 13-column (v1.2.0) Update outdated "9-column" or "Phase7 9 columns" references to "13-column (v1.2.0)" across all documentation files to reflect the current schema version with normalized spread metrics. Files updated: - GITHUB_PYPI_SETUP.md - Release notes reference - PYDANTIC_REFACTORING_PLAN.md - API documentation and examples (4 locations) - docs/DATABASE_SCHEMA.md - Version history (2 locations) - docs/README.md - Validation results - docs/UNIFIED_DUCKDB_PLAN_v2.md - Table descriptions and return types (2 locations) - tests/README.md - Test descriptions (2 locations) All outdated schema references now consistently reference the Phase7 13-column (v1.2.0) schema with normalized metrics (range_per_spread, range_per_tick, body_per_spread, body_per_tick). CHANGELOG.md and research/archive files intentionally preserved for historical accuracy.
+
+
 ### 🔧 Continuous Integration
 
 - Modernize publish workflow with uv and validation pipeline Update GitHub Actions publish workflow to match gapless-crypto-data standards: Build System: - Replace pip+build with uv for consistency with local development - Add UV_SYSTEM_PYTHON=1 environment variable - Use uv sync --dev for dependency installation Validation Pipeline: - Add file encoding validation (UTF-8/ASCII enforcement) - Add ruff linting and format checking - Add pytest test suite execution before build Security: - Add Sigstore artifact signing for published packages - Maintain PyPI Trusted Publishing (OIDC) authentication Configuration: - Change environment name: release → pypi (semantic clarity) - Change artifact name: dist → python-package-distributions (consistency) - Add verbose output to PyPI publish step Security Note: - Added .token-info.md to .gitignore to prevent accidental token documentation commits - Workflow uses OIDC (no API token needed)
@@ -20,6 +32,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - **release**: Bump version 0.1.0 → 0.2.0 [skip ci]
 
+- **release**: Bump version 0.2.0 → 0.3.0 [skip ci]
+
+- **release**: Bump version 0.3.0 → 0.3.1 [skip ci]
+
 
 ### ✨ Features
 
@@ -50,4 +66,3 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### 🧰 Maintenance
 
 - Prepare v0.1.0 release
-

exness_data_preprocess-0.4.0/CLAUDE.md

@@ -0,0 +1,170 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+**Architecture**: Forex tick data preprocessing with unified single-file DuckDB storage
+
+**Full Documentation**: [`README.md`](README.md) - Installation, usage, API reference
+
+---
+
+## Documentation Hub
+
+### User Documentation
+- **[README.md](README.md)** - Installation, API reference, usage examples
+- **[examples/basic_usage.py](examples/basic_usage.py)** - Download, query, coverage operations
+- **[examples/batch_processing.py](examples/batch_processing.py)** - Multi-instrument parallel processing
+
+### AI Assistant Documentation
+- **[docs/README.md](docs/README.md)** - Architecture, planning, research findings
+- **[docs/MODULE_ARCHITECTURE.md](docs/MODULE_ARCHITECTURE.md)** - Complete module documentation with SLOs
+- **[docs/DATABASE_SCHEMA.md](docs/DATABASE_SCHEMA.md)** - Complete database schema with self-documentation
+- **[docs/RESEARCH_PATTERNS.md](docs/RESEARCH_PATTERNS.md)** - Research lifecycle and tool selection (v1.0.0)
+- **[docs/UNIFIED_DUCKDB_PLAN_v2.md](docs/UNIFIED_DUCKDB_PLAN_v2.md)** - v2.0.0 architecture specification
+- **[docs/EXNESS_DATA_SOURCES.md](docs/EXNESS_DATA_SOURCES.md)** - Data source variants and URLs
+- **[Makefile](Makefile)** - Module introspection commands (module-stats, module-complexity, module-deps)
+
+### Implementation
+- **src/exness_data_preprocess/** - Source code (7 specialized modules + facade)
+- **tests/** - Test suite (48 tests, 100% passing)
+- **[CONTRIBUTING.md](CONTRIBUTING.md)** - Contribution guidelines
+
+---
+
+## Architecture Summary
+
+### Storage Pattern (v2.0.0)
+- **Single DuckDB file per instrument** (e.g., eurusd.duckdb containing all historical data)
+- **Dual-variant storage** (Raw_Spread + Standard in same database)
+- **Incremental updates** with automatic gap detection
+- **PRIMARY KEY constraints** prevent duplicates during updates
+- **Sub-15ms query performance** with date range filtering
+- **On-demand resampling** to any timeframe (5m, 1h, 1d)
+
+**Complete Specification**: [`docs/UNIFIED_DUCKDB_PLAN_v2.md`](docs/UNIFIED_DUCKDB_PLAN_v2.md)
+
+### Module Pattern (v1.3.0)
+- **Facade orchestrator** (processor.py) coordinating 7 specialized modules
+- **Separation of concerns** with single-responsibility modules
+- **SLO-based design** (Availability, Correctness, Observability, Maintainability)
+- **Off-the-shelf libraries** (httpx, pandas, DuckDB, exchange_calendars)
+- **Zero regressions** (48 tests pass)
+
+**Modules**:
+1. **processor.py** - Thin orchestrator facade
+2. **downloader.py** - HTTP download operations
+3. **tick_loader.py** - CSV parsing
+4. **database_manager.py** - Database operations
+5. **session_detector.py** - Holiday and session detection
+6. **gap_detector.py** - Incremental update logic
+7. **ohlc_generator.py** - Phase7 OHLC generation
+8. **query_engine.py** - Query operations
+
+**Complete Details**: [`docs/MODULE_ARCHITECTURE.md`](docs/MODULE_ARCHITECTURE.md)
+
+### Schema (v1.6.0)
+- **Phase7 30-column OHLC** with dual-variant spreads
+- **BID-only OHLC** from Raw_Spread execution prices
+- **Normalized metrics** (range_per_spread, range_per_tick, body_per_spread, body_per_tick)
+- **10 global exchange sessions with trading hour detection** (NYSE, LSE, XSWX, XFRA, XTSE, XNZE, XTKS, XASX, XHKG, XSES)
+- **Holiday tracking** (US, UK, major holidays)
+- **Self-documenting** via COMMENT ON statements
+
+**Complete Schema**: [`docs/DATABASE_SCHEMA.md`](docs/DATABASE_SCHEMA.md)
+
+### Data Sources
+- **Source**: https://ticks.ex2archive.com/ (public Exness tick data)
+- **Primary**: Raw_Spread variant (97.81% zero-spreads, execution prices)
+- **Reference**: Standard variant (0% zero-spreads, market quotes)
+- **Format**: Monthly ZIP files with microsecond-precision CSV
+
+**Complete Guide**: [`docs/EXNESS_DATA_SOURCES.md`](docs/EXNESS_DATA_SOURCES.md)
+
+### Research Patterns (v1.0.0)
+- **Lifecycle**: Explore (pandas/Polars) → Validate → Graduate (DuckDB) → Query (SQL)
+- **ASOF Performance**: pandas/Polars 0.04s, DuckDB 0.89s at 880K ticks (24x difference)
+- **Tool Selection**: pandas/Polars for tick-level temporal matching, DuckDB for materialized results
+- **Hybrid Pattern**: ASOF operations stay in pandas, validated findings materialize to DuckDB tables
+- **Single Source of Truth**: DuckDB stores validated research outcomes with COMMENT ON documentation
+
+**Complete Specification**: [`docs/RESEARCH_PATTERNS.md`](docs/RESEARCH_PATTERNS.md)
+
+---
+
+## Development Commands
+
+### Setup
+```bash
+# Install with development dependencies
+uv sync --dev
+
+# Or with pip
+pip install -e ".[dev]"
+```
+
+### Testing
+```bash
+# Run all tests
+uv run pytest
+
+# Run with coverage
+uv run pytest --cov=exness_data_preprocess --cov-report=html
+
+# Run specific test file
+uv run pytest tests/test_basic.py -v
+```
+
+### Code Quality
+```bash
+# Format code
+uv run ruff format .
+
+# Lint and auto-fix
+uv run ruff check --fix .
+
+# Type checking
+uv run mypy src/
+```
+
+### Module Introspection
+```bash
+# Show current line counts (always accurate)
+make module-stats
+
+# Show cyclomatic complexity (requires radon)
+make module-complexity
+
+# Show dependency tree (requires pipdeptree)
+make module-deps
+```
+
+### Building and Publishing
+```bash
+# Build package
+uv build
+
+# Test installation locally
+uv tool install --editable .
+
+# Publish to PyPI (requires PYPI_TOKEN)
+doppler run --project claude-config --config dev -- uv publish --token "$PYPI_TOKEN"
+```
+
+**Complete Development Guide**: [`README.md`](README.md) (Installation section)
+
+---
+
+## References
+
+- **Exness Data**: https://ticks.ex2archive.com/
+- **DuckDB**: https://duckdb.org/
+- **Apache Parquet**: https://parquet.apache.org/
+- **Zstd Compression**: https://facebook.github.io/zstd/
+
+---
+
+**Version**: 2.0.0 (Architecture) + 1.3.0 (Implementation) + 1.0.0 (Research Patterns)
+**Last Updated**: 2025-10-17
+**Architecture**: Unified Single-File DuckDB Storage with Incremental Updates
+**Implementation**: Facade Pattern with 7 Specialized Modules
+**Research**: Hybrid Materialization (pandas/Polars exploration, DuckDB validated results)

{exness_data_preprocess-0.3.0 → exness_data_preprocess-0.4.0}/GITHUB_PYPI_SETUP.md

@@ -173,7 +173,7 @@ gh release create v0.1.0 \
 
 **Architecture v2.0.0:**
 - Unified single-file DuckDB storage per instrument
-- Phase7 9-column OHLC schema with dual-variant spreads
+- Phase7 13-column (v1.2.0) OHLC schema with dual-variant spreads and normalized metrics
 - Sub-15ms query performance with date range filtering
 - PRIMARY KEY constraints prevent duplicates
 - Automatic gap detection for incremental updates

exness_data_preprocess-0.4.0/Makefile

@@ -0,0 +1,62 @@
+# Makefile for exness-data-preprocess
+#
+# Introspection commands for module statistics and project metrics
+
+.PHONY: module-stats module-complexity module-deps clean test help
+
+# Show current line counts for all modules
+module-stats:
+	@echo "Module Statistics (current line counts)"
+	@echo "========================================"
+	@cd src/exness_data_preprocess && \
+	for f in *.py; do \
+		echo "$$f: $$(wc -l < "$$f") lines"; \
+	done | sort
+	@echo ""
+	@echo "Note: Line counts include comments, docstrings, and blank lines"
+
+# Show module complexity using radon (requires: pip install radon)
+module-complexity:
+	@echo "Module Complexity Analysis"
+	@echo "=========================="
+	@which radon > /dev/null 2>&1 || (echo "Error: radon not installed. Run: uv pip install radon" && exit 1)
+	@radon cc src/exness_data_preprocess/*.py -s -a
+	@echo ""
+	@echo "Complexity Ratings: A=simple, B=low, C=moderate, D=high, E=very high, F=extremely high"
+
+# Show module dependencies using pipdeptree (requires: pip install pipdeptree)
+module-deps:
+	@echo "Module Dependencies"
+	@echo "==================="
+	@which pipdeptree > /dev/null 2>&1 || (echo "Error: pipdeptree not installed. Run: uv pip install pipdeptree" && exit 1)
+	@pipdeptree --packages exness-data-preprocess
+
+# Run test suite
+test:
+	@uv run pytest -v --tb=short
+
+# Run test suite with coverage
+test-cov:
+	@uv run pytest --cov=exness_data_preprocess --cov-report=html --cov-report=term
+
+# Clean build artifacts
+clean:
+	@rm -rf dist/ build/ *.egg-info
+	@rm -rf .pytest_cache .ruff_cache .mypy_cache
+	@rm -rf htmlcov/
+	@find . -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null || true
+	@echo "✓ Cleaned build artifacts"
+
+# Show help
+help:
+	@echo "Available commands:"
+	@echo ""
+	@echo "  make module-stats       Show current line counts for all modules"
+	@echo "  make module-complexity  Show cyclomatic complexity analysis (requires radon)"
+	@echo "  make module-deps        Show package dependency tree (requires pipdeptree)"
+	@echo "  make test               Run test suite"
+	@echo "  make test-cov           Run test suite with coverage report"
+	@echo "  make clean              Remove build artifacts and caches"
+	@echo "  make help               Show this help message"
+	@echo ""
+	@echo "Documentation: See CLAUDE.md for architecture and design patterns"

{exness_data_preprocess-0.3.0 → exness_data_preprocess-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: exness-data-preprocess
-Version: 0.3.0
+Version: 0.4.0
 Summary: Professional Exness forex tick data preprocessing with optimal compression (Parquet Zstd-22) and DuckDB OHLC generation. Provides efficient storage (9% smaller than ZIP) with lossless precision and direct queryability.
 Project-URL: Homepage, https://github.com/terrylica/exness-data-preprocess
 Project-URL: Documentation, https://github.com/terrylica/exness-data-preprocess#readme
@@ -30,6 +30,7 @@ Classifier: Topic :: Scientific/Engineering :: Information Analysis
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Requires-Python: >=3.9
 Requires-Dist: duckdb>=0.9.0
+Requires-Dist: exchange-calendars>=4.5.6
 Requires-Dist: httpx>=0.25.0
 Requires-Dist: pandas>=2.0.0
 Requires-Dist: pyarrow>=16.0.0
@@ -55,14 +56,14 @@ Description-Content-Type: text/markdown
 [![Downloads](https://img.shields.io/pypi/dm/exness-data-preprocess.svg)](https://pypi.org/project/exness-data-preprocess/)
 [![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-000000.svg)](https://github.com/astral-sh/ruff)
 
-Professional forex tick data preprocessing with unified single-file DuckDB storage. Provides incremental updates, dual-variant storage (Raw_Spread + Standard), and Phase7 13-column OHLC schema (v1.2.0) with sub-15ms query performance.
+Professional forex tick data preprocessing with unified single-file DuckDB storage. Provides incremental updates, dual-variant storage (Raw_Spread + Standard), and Phase7 30-column OHLC schema (v1.6.0) with 10 global exchange sessions (trading hour detection) and sub-15ms query performance.
 
 ## Features
 
 - **Unified Single-File Architecture**: One DuckDB file per instrument (eurusd.duckdb)
 - **Incremental Updates**: Automatic gap detection and download only missing months
 - **Dual-Variant Storage**: Raw_Spread (primary) + Standard (reference) in same database
-- **Phase7 OHLC Schema**: 13-column bars (v1.2.0) with dual spreads, tick counts, and normalized metrics
+- **Phase7 OHLC Schema**: 30-column bars (v1.6.0) with dual spreads, tick counts, normalized metrics, and 10 global exchange sessions with trading hour detection
 - **Fast Queries**: Date range queries with sub-15ms performance
 - **On-Demand Resampling**: Any timeframe (5m, 1h, 1d) resampled in <15ms
 - **PRIMARY KEY Constraints**: Prevents duplicate data during incremental updates
@@ -138,7 +139,7 @@ Download Only Missing Months (Raw_Spread + Standard)
            ↓
 DuckDB Single-File Storage (PRIMARY KEY prevents duplicates)
            ↓
-Phase7 13-Column OHLC Generation (v1.2.0 - dual spreads, tick counts, normalized metrics)
+Phase7 30-Column OHLC Generation (v1.6.0 - dual spreads, tick counts, normalized metrics, 10 global exchange sessions with trading hour detection)
            ↓
 Query Interface (date ranges, SQL filters, on-demand resampling)
 ```
@@ -148,15 +149,17 @@ Query Interface (date ranges, SQL filters, on-demand resampling)
 **Single File Per Instrument**: `~/eon/exness-data/eurusd.duckdb`
 
 **Schema**:
+
 - `raw_spread_ticks` table: Timestamp (PK), Bid, Ask
 - `standard_ticks` table: Timestamp (PK), Bid, Ask
-- `ohlc_1m` table: Phase7 13-column schema (v1.2.0)
+- `ohlc_1m` table: Phase7 30-column schema (v1.6.0)
 - `metadata` table: Coverage tracking
 
-**Phase7 13-Column OHLC (v1.2.0)**:
+**Phase7 30-Column OHLC (v1.6.0)**:
+
 - **Column Definitions**: See [`schema.py`](src/exness_data_preprocess/schema.py) - Single source of truth
 - **Comprehensive Reference**: See [`DATABASE_SCHEMA.md`](docs/DATABASE_SCHEMA.md) - Query examples and usage patterns
-- **Key Features**: BID-only OHLC with dual spreads (Raw_Spread + Standard) and normalized spread metrics
+- **Key Features**: BID-only OHLC with dual spreads (Raw_Spread + Standard), normalized spread metrics, and 10 global exchange sessions with trading hour detection (XNYS, XLON, XSWX, XFRA, XTSE, XNZE, XTKS, XASX, XHKG, XSES)
 
 ### Directory Structure
 
@@ -172,6 +175,7 @@ Query Interface (date ranges, SQL filters, on-demand resampling)
 ```
 
 **Why Single-File Per Instrument?**
+
 - **Unified Storage**: All years in one database
 - **Incremental Updates**: Automatic gap detection and download only missing months
 - **No Duplicates**: PRIMARY KEY constraints prevent duplicate data
@@ -394,6 +398,7 @@ uv tool install --editable .
 ## Data Source
 
 Data is sourced from Exness's public tick data repository:
+
 - **URL**: https://ticks.ex2archive.com/
 - **Format**: Monthly ZIP files with CSV tick data
 - **Variants**: Raw_Spread (zero-spreads) + Standard (market spreads)
@@ -404,35 +409,35 @@ Data is sourced from Exness's public tick data repository:
 
 ### Database Size (3-Year History, EURUSD)
 
-| Metric | Value |
-|--------|-------|
-| Raw_Spread ticks | ~18.6M |
-| Standard ticks | ~19.6M |
-| OHLC bars (1m) | ~413K |
-| Database size | ~2.08 GB |
-| Date range | 2022-01-01 to 2025-01-10 |
+| Metric           | Value                    |
+| ---------------- | ------------------------ |
+| Raw_Spread ticks | ~18.6M                   |
+| Standard ticks   | ~19.6M                   |
+| OHLC bars (1m)   | ~413K                    |
+| Database size    | ~2.08 GB                 |
+| Date range       | 2022-01-01 to 2025-01-10 |
 
 ### Query Performance
 
-| Operation | Time |
-|-----------|------|
+| Operation                  | Time  |
+| -------------------------- | ----- |
 | Query 880K ticks (1 month) | <15ms |
-| Query 1m OHLC (1 month) | <10ms |
-| Resample to 1h (1 month) | <15ms |
-| Resample to 1d (1 year) | <20ms |
+| Query 1m OHLC (1 month)    | <10ms |
+| Resample to 1h (1 month)   | <15ms |
+| Resample to 1d (1 year)    | <20ms |
 
 ### Architecture Benefits
 
-| Feature | Benefit |
-|---------|---------|
-| Single file per instrument | Unified storage, no file fragmentation |
-| PRIMARY KEY constraints | Prevents duplicates during incremental updates |
-| Automatic gap detection | Download only missing months |
-| Dual-variant storage | Raw_Spread + Standard in same database |
-| Phase7 OHLC schema | Dual spreads + dual tick counts |
-| Date range queries | Efficient filtering without loading entire dataset |
-| On-demand resampling | Any timeframe in <15ms |
-| SQL filter support | Direct SQL WHERE clauses on ticks |
+| Feature                    | Benefit                                            |
+| -------------------------- | -------------------------------------------------- |
+| Single file per instrument | Unified storage, no file fragmentation             |
+| PRIMARY KEY constraints    | Prevents duplicates during incremental updates     |
+| Automatic gap detection    | Download only missing months                       |
+| Dual-variant storage       | Raw_Spread + Standard in same database             |
+| Phase7 OHLC schema         | Dual spreads + dual tick counts                    |
+| Date range queries         | Efficient filtering without loading entire dataset |
+| On-demand resampling       | Any timeframe in <15ms                             |
+| SQL filter support         | Direct SQL WHERE clauses on ticks                  |
 
 ## API Reference
 
@@ -461,16 +466,19 @@ processor = edp.ExnessDataProcessor(base_dir="~/eon/exness-data")
 ## Migration from v1.0.0
 
 **v1.0.0 (Legacy)**:
+
 - Monthly DuckDB files: `eurusd_ohlc_2024_08.duckdb`
 - Parquet tick storage: `eurusd_ticks_2024_08.parquet`
 - Functions: `process_month()`, `process_date_range()`, `analyze_ticks()`
 
 **v2.0.0 (Current)**:
+
 - Single DuckDB file: `eurusd.duckdb`
 - No Parquet files (everything in DuckDB)
 - Unified API: `processor.update_data()`, `processor.query_ohlc()`, `processor.query_ticks()`
 
 **Migration Steps**:
+
 1. Run `processor.update_data(pair, start_date)` to create new unified database
 2. Delete old monthly files: `rm eurusd_ohlc_2024_*.duckdb eurusd_ticks_2024_*.parquet`
 3. Update code to use new API methods

{exness_data_preprocess-0.3.0 → exness_data_preprocess-0.4.0}/README.md

@@ -7,14 +7,14 @@
 [![Downloads](https://img.shields.io/pypi/dm/exness-data-preprocess.svg)](https://pypi.org/project/exness-data-preprocess/)
 [![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-000000.svg)](https://github.com/astral-sh/ruff)
 
-Professional forex tick data preprocessing with unified single-file DuckDB storage. Provides incremental updates, dual-variant storage (Raw_Spread + Standard), and Phase7 13-column OHLC schema (v1.2.0) with sub-15ms query performance.
+Professional forex tick data preprocessing with unified single-file DuckDB storage. Provides incremental updates, dual-variant storage (Raw_Spread + Standard), and Phase7 30-column OHLC schema (v1.6.0) with 10 global exchange sessions (trading hour detection) and sub-15ms query performance.
 
 ## Features
 
 - **Unified Single-File Architecture**: One DuckDB file per instrument (eurusd.duckdb)
 - **Incremental Updates**: Automatic gap detection and download only missing months
 - **Dual-Variant Storage**: Raw_Spread (primary) + Standard (reference) in same database
-- **Phase7 OHLC Schema**: 13-column bars (v1.2.0) with dual spreads, tick counts, and normalized metrics
+- **Phase7 OHLC Schema**: 30-column bars (v1.6.0) with dual spreads, tick counts, normalized metrics, and 10 global exchange sessions with trading hour detection
 - **Fast Queries**: Date range queries with sub-15ms performance
 - **On-Demand Resampling**: Any timeframe (5m, 1h, 1d) resampled in <15ms
 - **PRIMARY KEY Constraints**: Prevents duplicate data during incremental updates
@@ -90,7 +90,7 @@ Download Only Missing Months (Raw_Spread + Standard)
            ↓
 DuckDB Single-File Storage (PRIMARY KEY prevents duplicates)
            ↓
-Phase7 13-Column OHLC Generation (v1.2.0 - dual spreads, tick counts, normalized metrics)
+Phase7 30-Column OHLC Generation (v1.6.0 - dual spreads, tick counts, normalized metrics, 10 global exchange sessions with trading hour detection)
            ↓
 Query Interface (date ranges, SQL filters, on-demand resampling)
 ```
@@ -100,15 +100,17 @@ Query Interface (date ranges, SQL filters, on-demand resampling)
 **Single File Per Instrument**: `~/eon/exness-data/eurusd.duckdb`
 
 **Schema**:
+
 - `raw_spread_ticks` table: Timestamp (PK), Bid, Ask
 - `standard_ticks` table: Timestamp (PK), Bid, Ask
-- `ohlc_1m` table: Phase7 13-column schema (v1.2.0)
+- `ohlc_1m` table: Phase7 30-column schema (v1.6.0)
 - `metadata` table: Coverage tracking
 
-**Phase7 13-Column OHLC (v1.2.0)**:
+**Phase7 30-Column OHLC (v1.6.0)**:
+
 - **Column Definitions**: See [`schema.py`](src/exness_data_preprocess/schema.py) - Single source of truth
 - **Comprehensive Reference**: See [`DATABASE_SCHEMA.md`](docs/DATABASE_SCHEMA.md) - Query examples and usage patterns
-- **Key Features**: BID-only OHLC with dual spreads (Raw_Spread + Standard) and normalized spread metrics
+- **Key Features**: BID-only OHLC with dual spreads (Raw_Spread + Standard), normalized spread metrics, and 10 global exchange sessions with trading hour detection (XNYS, XLON, XSWX, XFRA, XTSE, XNZE, XTKS, XASX, XHKG, XSES)
 
 ### Directory Structure
 
@@ -124,6 +126,7 @@ Query Interface (date ranges, SQL filters, on-demand resampling)
 ```
 
 **Why Single-File Per Instrument?**
+
 - **Unified Storage**: All years in one database
 - **Incremental Updates**: Automatic gap detection and download only missing months
 - **No Duplicates**: PRIMARY KEY constraints prevent duplicate data
@@ -346,6 +349,7 @@ uv tool install --editable .
 ## Data Source
 
 Data is sourced from Exness's public tick data repository:
+
 - **URL**: https://ticks.ex2archive.com/
 - **Format**: Monthly ZIP files with CSV tick data
 - **Variants**: Raw_Spread (zero-spreads) + Standard (market spreads)
@@ -356,35 +360,35 @@ Data is sourced from Exness's public tick data repository:
 
 ### Database Size (3-Year History, EURUSD)
 
-| Metric | Value |
-|--------|-------|
-| Raw_Spread ticks | ~18.6M |
-| Standard ticks | ~19.6M |
-| OHLC bars (1m) | ~413K |
-| Database size | ~2.08 GB |
-| Date range | 2022-01-01 to 2025-01-10 |
+| Metric           | Value                    |
+| ---------------- | ------------------------ |
+| Raw_Spread ticks | ~18.6M                   |
+| Standard ticks   | ~19.6M                   |
+| OHLC bars (1m)   | ~413K                    |
+| Database size    | ~2.08 GB                 |
+| Date range       | 2022-01-01 to 2025-01-10 |
 
 ### Query Performance
 
-| Operation | Time |
-|-----------|------|
+| Operation                  | Time  |
+| -------------------------- | ----- |
 | Query 880K ticks (1 month) | <15ms |
-| Query 1m OHLC (1 month) | <10ms |
-| Resample to 1h (1 month) | <15ms |
-| Resample to 1d (1 year) | <20ms |
+| Query 1m OHLC (1 month)    | <10ms |
+| Resample to 1h (1 month)   | <15ms |
+| Resample to 1d (1 year)    | <20ms |
 
 ### Architecture Benefits
 
-| Feature | Benefit |
-|---------|---------|
-| Single file per instrument | Unified storage, no file fragmentation |
-| PRIMARY KEY constraints | Prevents duplicates during incremental updates |
-| Automatic gap detection | Download only missing months |
-| Dual-variant storage | Raw_Spread + Standard in same database |
-| Phase7 OHLC schema | Dual spreads + dual tick counts |
-| Date range queries | Efficient filtering without loading entire dataset |
-| On-demand resampling | Any timeframe in <15ms |
-| SQL filter support | Direct SQL WHERE clauses on ticks |
+| Feature                    | Benefit                                            |
+| -------------------------- | -------------------------------------------------- |
+| Single file per instrument | Unified storage, no file fragmentation             |
+| PRIMARY KEY constraints    | Prevents duplicates during incremental updates     |
+| Automatic gap detection    | Download only missing months                       |
+| Dual-variant storage       | Raw_Spread + Standard in same database             |
+| Phase7 OHLC schema         | Dual spreads + dual tick counts                    |
+| Date range queries         | Efficient filtering without loading entire dataset |
+| On-demand resampling       | Any timeframe in <15ms                             |
+| SQL filter support         | Direct SQL WHERE clauses on ticks                  |
 
 ## API Reference
 
@@ -413,16 +417,19 @@ processor = edp.ExnessDataProcessor(base_dir="~/eon/exness-data")
 ## Migration from v1.0.0
 
 **v1.0.0 (Legacy)**:
+
 - Monthly DuckDB files: `eurusd_ohlc_2024_08.duckdb`
 - Parquet tick storage: `eurusd_ticks_2024_08.parquet`
 - Functions: `process_month()`, `process_date_range()`, `analyze_ticks()`
 
 **v2.0.0 (Current)**:
+
 - Single DuckDB file: `eurusd.duckdb`
 - No Parquet files (everything in DuckDB)
 - Unified API: `processor.update_data()`, `processor.query_ohlc()`, `processor.query_ticks()`
 
 **Migration Steps**:
+
 1. Run `processor.update_data(pair, start_date)` to create new unified database
 2. Delete old monthly files: `rm eurusd_ohlc_2024_*.duckdb eurusd_ticks_2024_*.parquet`
 3. Update code to use new API methods

exness_data_preprocess-0.4.0/RELEASE_NOTES.md

@@ -0,0 +1,26 @@
+## 0.3.1 - 2025-10-16
+
+### Architecture Improvements
+
+Refactored processor.py from monolithic file into 7 focused modules using facade pattern with separation of concerns.
+
+**Modules Created**:
+- `downloader.py`: HTTP download operations
+- `tick_loader.py`: CSV parsing
+- `database_manager.py`: Database operations
+- `session_detector.py`: Holiday and session detection
+- `gap_detector.py`: Incremental update logic
+- `ohlc_generator.py`: Phase7 OHLC generation
+- `query_engine.py`: Query operations
+
+**Design Principles**:
+- Facade pattern: processor.py delegates to specialized modules
+- SLO-based: All modules define Availability, Correctness, Observability, Maintainability
+- Off-the-shelf libraries: httpx, pandas, DuckDB, exchange_calendars (no custom implementations)
+- Zero regressions: All 48 tests pass, backward compatible
+
+**For Users**:
+No API changes - all existing code continues to work. Internal refactoring improves maintainability and sets foundation for future enhancements.
+
+---
+**Full Changelog**: https://github.com/terrylica/exness-data-preprocess/compare/v0.3.0...v0.3.1