exness-data-preprocess 0.1.0__tar.gz → 0.3.0__tar.gz

{exness_data_preprocess-0.1.0 → exness_data_preprocess-0.3.0}/.cz.toml

@@ -1,6 +1,6 @@
 [tool.commitizen]
 name = "cz_conventional_commits"
-version = "0.1.0"
+version = "0.2.0"
 tag_format = "v$major.$minor.$patch"
 update_changelog_on_bump = true
 version_files = [

{exness_data_preprocess-0.1.0 → exness_data_preprocess-0.3.0}/.gitignore

@@ -98,3 +98,4 @@ uv.lock
 
 # Pre-commit
 # .pre-commit-config.yaml should be tracked, not ignored
+.token-info.md

exness_data_preprocess-0.3.0/CHANGELOG.md

@@ -0,0 +1,53 @@
+# Changelog
+
+All notable changes to exness-data-preprocess will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+
+### ✨ Features
+
+- Implement pydantic v2 models with dual-variant e2e testing - Add Pydantic v2 models (UpdateResult, CoverageInfo) for type-safe API - Update processor to use Pydantic types (PairType, TimeframeType, VariantType) - Remove deprecated api.py module (broken functionality) - Implement comprehensive test suite (48 tests, 100% passing): * test_models.py - Pydantic model validation (13 tests) * test_types.py - Type safety and helpers (15 tests) * test_processor_pydantic.py - Integration tests (6 tests) * test_functional_regression.py - v2.0.0 regression tests (10 tests) - Fix Standard variant downloads (variant="" not "Standard") - Add true end-to-end testing with real Exness downloads - Test data: EURUSD August 2024 (815K Raw_Spread + 877K Standard ticks) - Update documentation and add refactoring status tracking - Coverage: models.py 100%, __init__.py 100%, processor.py 45% BREAKING CHANGE: api.py removed (use ExnessDataProcessor directly)
+
+
+### 🔧 Continuous Integration
+
+- Modernize publish workflow with uv and validation pipeline Update GitHub Actions publish workflow to match gapless-crypto-data standards: Build System: - Replace pip+build with uv for consistency with local development - Add UV_SYSTEM_PYTHON=1 environment variable - Use uv sync --dev for dependency installation Validation Pipeline: - Add file encoding validation (UTF-8/ASCII enforcement) - Add ruff linting and format checking - Add pytest test suite execution before build Security: - Add Sigstore artifact signing for published packages - Maintain PyPI Trusted Publishing (OIDC) authentication Configuration: - Change environment name: release → pypi (semantic clarity) - Change artifact name: dist → python-package-distributions (consistency) - Add verbose output to PyPI publish step Security Note: - Added .token-info.md to .gitignore to prevent accidental token documentation commits - Workflow uses OIDC (no API token needed)
+
+
+### 🧰 Maintenance
+
+- **release**: Bump version 0.1.0 → 0.2.0 [skip ci]
+
+
+### ✨ Features
+
+- Initial commit with v2.0.0 unified DuckDB architecture - Unified single-file DuckDB storage per instrument (eurusd.duckdb) - Phase7 9-column OHLC schema with dual-variant spreads - Sub-15ms query performance with date range filtering - PRIMARY KEY constraints prevent duplicates - Automatic gap detection for incremental updates - DuckDB self-documentation with COMMENT ON statements - GitHub Actions CI/CD with PyPI Trusted Publishing - Pre-commit hooks (ruff, commitizen, basic checks) - Professional README with shields.io badges
+
+
+### 🐛 Bug Fixes
+
+- Force add .pre-commit-config.yaml (was incorrectly gitignored)
+
+- Remove .pre-commit-config.yaml from .gitignore Pre-commit config should be version controlled, not ignored.
+
+- **pre-commit**: Enable unsafe fixes for ruff to resolve remaining errors
+
+- **research**: Resolve remaining linting errors in research scripts - Fix bare except clause in 02_volatility_model_simple.py (E722) - Move math.erf imports to top in 03_liquidity_crisis_detection.py (E402) - Move math.erf imports to top in 04_regime_detection_analysis.py (E402)
+
+
+### 💅 Code Style
+
+- Apply pre-commit formatting fixes - Fix end-of-file newlines in 2 MD files - Apply ruff quote style fixes (200 errors fixed) - Reformat 26 files with ruff-format
+
+
+### 🔧 CI/CD Improvements
+
+- **ci**: Add pre-commit config and basic tests - Add .pre-commit-config.yaml to repository - Create basic test suite (test_basic.py) for CI - Add GITHUB_PYPI_SETUP.md with complete setup instructions Fixes CI failures: - Pre-commit hook now finds config file - Pytest now finds and runs tests successfully
+
+
+### 🧰 Maintenance
+
+- Prepare v0.1.0 release
+

{exness_data_preprocess-0.1.0 → exness_data_preprocess-0.3.0}/CLAUDE.md

@@ -1,4 +1,6 @@
-# exness-data-preprocess - Project Memory
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
 
 **Architecture**: Professional forex tick data preprocessing with unified single-file DuckDB storage
 
@@ -6,6 +8,147 @@
 
 ---
 
+## Development Commands
+
+### Setup
+```bash
+# Install with development dependencies
+uv sync --dev
+
+# Or with pip
+pip install -e ".[dev]"
+```
+
+### Testing
+```bash
+# Run all tests
+uv run pytest
+
+# Run with coverage
+uv run pytest --cov=exness_data_preprocess --cov-report=html
+
+# Run specific test file
+uv run pytest tests/test_processor.py -v
+
+# Run specific test
+uv run pytest tests/test_processor.py::test_compression_ratio -v
+```
+
+### Code Quality
+```bash
+# Format code
+uv run ruff format .
+
+# Check formatting without changes
+uv run ruff format --check .
+
+# Lint and auto-fix
+uv run ruff check --fix .
+
+# Lint without changes
+uv run ruff check .
+
+# Type checking
+uv run mypy src/
+```
+
+### Building and Publishing
+```bash
+# Build package
+uv build
+
+# Test installation locally
+uv tool install --editable .
+
+# Publish to PyPI (requires PYPI_TOKEN in environment)
+doppler run --project claude-config --config dev -- uv publish --token "$PYPI_TOKEN"
+```
+
+---
+
+## Codebase Architecture
+
+### Module Structure
+
+**Core Components**:
+
+1. **`processor.py`** (`/Users/terryli/eon/exness-data-preprocess/src/exness_data_preprocess/processor.py`) - Main `ExnessDataProcessor` class
+   - **Lines 29-899**: Complete processor implementation
+   - **Lines 75-113**: `download_exness_zip()` - Downloads monthly ZIP files from ticks.ex2archive.com
+   - **Lines 115-232**: `_get_or_create_db()` - Database initialization with schema and COMMENT ON statements
+   - **Lines 234-245**: `_load_ticks_from_zip()` - CSV parsing from ZIP files
+   - **Lines 247-275**: `_append_ticks_to_db()` - Tick insertion with PRIMARY KEY duplicate prevention
+   - **Lines 277-377**: `_discover_missing_months()` - Gap detection for incremental updates
+   - **Lines 379-516**: `update_data()` - Main entry point for downloading and updating data
+   - **Lines 518-549**: `_regenerate_ohlc()` - Phase7 13-column OHLC (v1.2.0) generation with LEFT JOIN
+   - **Lines 551-603**: `query_ticks()` - Tick queries with date range and SQL filters
+   - **Lines 605-706**: `query_ohlc()` - OHLC queries with on-demand resampling (1m/5m/15m/1h/4h/1d)
+   - **Lines 708-821**: `add_schema_comments()` / `add_schema_comments_all()` - Retrofit self-documentation
+   - **Lines 823-899**: `get_data_coverage()` - Coverage statistics and metadata
+
+2. **`api.py`** (`/Users/terryli/eon/exness-data-preprocess/src/exness_data_preprocess/api.py`) - Simple wrapper functions
+   - Convenience functions wrapping `ExnessDataProcessor` methods
+   - Legacy v1.0.0 API compatibility layer (to be deprecated)
+
+3. **`cli.py`** (`/Users/terryli/eon/exness-data-preprocess/src/exness_data_preprocess/cli.py`) - Command-line interface
+   - Entry point: `exness-preprocess` command
+   - Commands for download, query, coverage operations
+
+### Key Design Patterns
+
+**Unified Single-File Architecture** (v2.0.0):
+- ONE DuckDB file per instrument (e.g., `eurusd.duckdb`) containing all historical data
+- NO monthly file separation (major change from v1.0.0)
+- Incremental updates with automatic gap detection
+- PRIMARY KEY constraints prevent duplicates during updates
+
+**Phase7 13-Column OHLC Schema** (v1.2.0):
+- BID-only OHLC from Raw_Spread variant
+- Dual spreads: `raw_spread_avg` and `standard_spread_avg`
+- Dual tick counts: `tick_count_raw_spread` and `tick_count_standard`
+- Normalized metrics: `range_per_spread`, `range_per_tick`, `body_per_spread`, `body_per_tick`
+- Generated via LEFT JOIN between Raw_Spread and Standard variants
+
+**Self-Documentation**:
+- All tables and columns have embedded `COMMENT ON` statements
+- Queryable via `duckdb_tables()` and `duckdb_columns()` system functions
+- Enables BI tools and IDEs to display inline help
+
+**Data Flow**:
+```
+Exness Repository (monthly ZIPs)
+  → Automatic Gap Detection
+  → Download Missing Months (Raw_Spread + Standard)
+  → DuckDB Storage (PRIMARY KEY prevents duplicates)
+  → Phase7 OHLC Generation
+  → Query Interface (date ranges, SQL filters, resampling)
+```
+
+### Database Schema (per instrument)
+
+Each `.duckdb` file contains:
+
+1. **`raw_spread_ticks`** table:
+   - Columns: `Timestamp` (PK), `Bid`, `Ask`
+   - Exness Raw_Spread variant (~98% zero-spreads, execution prices)
+   - Primary data source for OHLC construction
+
+2. **`standard_ticks`** table:
+   - Columns: `Timestamp` (PK), `Bid`, `Ask`
+   - Exness Standard variant (0% zero-spreads, always Bid < Ask)
+   - Reference data for spread comparison
+
+3. **`ohlc_1m`** table:
+   - **Schema**: Phase7 13-column (v1.2.0) - See [`schema.py`](src/exness_data_preprocess/schema.py)
+   - **Details**: BID-only OHLC with dual-variant spreads, tick counts, and normalized metrics
+   - **Reference**: [`DATABASE_SCHEMA.md`](docs/DATABASE_SCHEMA.md)
+
+4. **`metadata`** table:
+   - Columns: `key` (PK), `value`, `updated_at`
+   - Tracks coverage (earliest_date, latest_date, etc.)
+
+---
+
 ## Quick Links
 
 ### Documentation
@@ -22,7 +165,7 @@
 
 ### Development
 - **[CONTRIBUTING.md](CONTRIBUTING.md)** - Contribution guidelines
-- **[tests/README.md](tests/README.md)** - Test suite documentation
+- **[tests/README.md](tests/README.md)** - Test suite documentation (v2.0.0 tests needed)
 
 ---
 
@@ -37,7 +180,7 @@
 - ✅ **Dual-variant storage**: Raw_Spread + Standard in same database
 - ✅ **PRIMARY KEY constraints**: Prevents duplicates during incremental updates
 - ✅ **Automatic gap detection**: Downloads only missing months
-- ✅ **Phase7 9-column OHLC**: Dual spreads + dual tick counts
+- ✅ **Phase7 13-column OHLC (v1.2.0)**: Dual spreads + dual tick counts
 - ✅ **Date range queries**: Sub-15ms query performance
 - ✅ **On-demand resampling**: Any timeframe (5m, 1h, 1d) in <15ms
 
@@ -64,11 +207,13 @@
 - `/tmp/exness-duckdb-test/test_refactored_processor.py` - Validation test
 - `/tmp/exness-duckdb-test/test_queries_only.py` - Query validation test
 
-### Phase7 9-Column OHLC Schema (✅ Implemented)
+### Phase7 13-Column OHLC Schema v1.2.0 (✅ Implemented)
 
-**Decision**: Dual-variant BID-only OHLC with 9 columns capturing both Raw_Spread and Standard characteristics
+**Decision**: Dual-variant BID-only OHLC with 13 columns capturing both Raw_Spread and Standard characteristics plus normalized metrics
 
-**Schema**: `Timestamp | Open | High | Low | Close | raw_spread_avg | standard_spread_avg | tick_count_raw_spread | tick_count_standard`
+**Schema**: Phase7 13-column (v1.2.0)
+- **Definition**: See [`schema.py`](src/exness_data_preprocess/schema.py)
+- **Documentation**: See [`DATABASE_SCHEMA.md`](docs/DATABASE_SCHEMA.md)
 
 **Key Features**:
 - ✅ **BID-only OHLC**: Uses Raw_Spread Bid prices (execution prices)
@@ -76,7 +221,7 @@
 - ✅ **Dual tick counts**: Records tick counts from both variants
 - ✅ **LEFT JOIN methodology**: Raw_Spread primary, Standard reference
 
-**Implementation**: [`src/exness_data_preprocess/processor.py`](src/exness_data_preprocess/processor.py) (lines 453-485)
+**Implementation**: [`src/exness_data_preprocess/processor.py`](src/exness_data_preprocess/processor.py) (lines 518-549)
 
 **Specification**: [`docs/research/eurusd-zero-spread-deviations/data/plan/phase7_bid_ohlc_construction_v1.1.0.md`](docs/research/eurusd-zero-spread-deviations/data/plan/phase7_bid_ohlc_construction_v1.1.0.md)
 
@@ -111,7 +256,7 @@ SELECT table_name, column_name, data_type, comment FROM duckdb_columns();
 - `add_schema_comments(pair)` - Add comments to single database
 - `add_schema_comments_all()` - Batch-update all databases
 
-**Implementation**: [`src/exness_data_preprocess/processor.py`](src/exness_data_preprocess/processor.py) (lines 138-215, 698-797)
+**Implementation**: [`src/exness_data_preprocess/processor.py`](src/exness_data_preprocess/processor.py) (lines 138-229, 708-821)
 
 **Usage Example**: [`examples/add_schema_comments.py`](examples/add_schema_comments.py)
 
@@ -161,21 +306,6 @@ SELECT table_name, column_name, data_type, comment FROM duckdb_columns();
 
 ---
 
-## Development Setup
-
-**Quick Start**:
-```bash
-git clone https://github.com/Eon-Labs/exness-data-preprocess.git
-cd exness-data-preprocess
-uv sync --dev
-```
-
-**Complete Setup Guide**: [`README.md`](README.md) - Installation, testing, code quality, building
-
-**Contributing**: [`CONTRIBUTING.md`](CONTRIBUTING.md) - Contribution guidelines and workflow
-
----
-
 ## Current Implementation Status
 
 ### v2.0.0 Architecture (✅ Completed 2025-10-12)
@@ -184,7 +314,7 @@ uv sync --dev
 - ✅ **Dual-variant storage** - Raw_Spread + Standard in same database
 - ✅ **PRIMARY KEY constraints** - Prevents duplicates during incremental updates
 - ✅ **Automatic gap detection** - Downloads only missing months
-- ✅ **Phase7 9-column OHLC** - Dual spreads + dual tick counts
+- ✅ **Phase7 13-column OHLC (v1.2.0)** - Dual spreads + dual tick counts
 - ✅ **Date range queries** - Sub-15ms query performance
 - ✅ **On-demand resampling** - Any timeframe in <15ms
 - ✅ **SQL filter support** - Direct SQL WHERE clauses on ticks
@@ -212,7 +342,7 @@ uv sync --dev
 
 ## File Locations
 
-**Project Root**: `/Users/terryli/eon/exness-data-preprocess/ `
+**Project Root**: `/Users/terryli/eon/exness-data-preprocess/`
 
 **Data Storage** (default): `~/eon/exness-data/`
 ```
@@ -229,7 +359,7 @@ uv sync --dev
 eurusd.duckdb:
 ├── raw_spread_ticks   # Timestamp (PK), Bid, Ask
 ├── standard_ticks     # Timestamp (PK), Bid, Ask
-├── ohlc_1m            # Phase7 9-column schema
+├── ohlc_1m            # Phase7 13-column schema (v1.2.0)
 └── metadata           # Coverage tracking
 ```
 

{exness_data_preprocess-0.1.0 → exness_data_preprocess-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: exness-data-preprocess
-Version: 0.1.0
+Version: 0.3.0
 Summary: Professional Exness forex tick data preprocessing with optimal compression (Parquet Zstd-22) and DuckDB OHLC generation. Provides efficient storage (9% smaller than ZIP) with lossless precision and direct queryability.
 Project-URL: Homepage, https://github.com/terrylica/exness-data-preprocess
 Project-URL: Documentation, https://github.com/terrylica/exness-data-preprocess#readme
@@ -33,6 +33,7 @@ Requires-Dist: duckdb>=0.9.0
 Requires-Dist: httpx>=0.25.0
 Requires-Dist: pandas>=2.0.0
 Requires-Dist: pyarrow>=16.0.0
+Requires-Dist: pydantic>=2.0.0
 Requires-Dist: tqdm>=4.65.0
 Provides-Extra: dev
 Requires-Dist: black>=23.0.0; extra == 'dev'
@@ -54,14 +55,14 @@ Description-Content-Type: text/markdown
 [![Downloads](https://img.shields.io/pypi/dm/exness-data-preprocess.svg)](https://pypi.org/project/exness-data-preprocess/)
 [![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-000000.svg)](https://github.com/astral-sh/ruff)
 
-Professional forex tick data preprocessing with unified single-file DuckDB storage. Provides incremental updates, dual-variant storage (Raw_Spread + Standard), and Phase7 9-column OHLC schema with sub-15ms query performance.
+Professional forex tick data preprocessing with unified single-file DuckDB storage. Provides incremental updates, dual-variant storage (Raw_Spread + Standard), and Phase7 13-column OHLC schema (v1.2.0) with sub-15ms query performance.
 
 ## Features
 
 - **Unified Single-File Architecture**: One DuckDB file per instrument (eurusd.duckdb)
 - **Incremental Updates**: Automatic gap detection and download only missing months
 - **Dual-Variant Storage**: Raw_Spread (primary) + Standard (reference) in same database
-- **Phase7 OHLC Schema**: 9-column bars with dual spreads and dual tick counts
+- **Phase7 OHLC Schema**: 13-column bars (v1.2.0) with dual spreads, tick counts, and normalized metrics
 - **Fast Queries**: Date range queries with sub-15ms performance
 - **On-Demand Resampling**: Any timeframe (5m, 1h, 1d) resampled in <15ms
 - **PRIMARY KEY Constraints**: Prevents duplicate data during incremental updates
@@ -137,7 +138,7 @@ Download Only Missing Months (Raw_Spread + Standard)
            ↓
 DuckDB Single-File Storage (PRIMARY KEY prevents duplicates)
            ↓
-Phase7 9-Column OHLC Generation (dual spreads + dual tick counts)
+Phase7 13-Column OHLC Generation (v1.2.0 - dual spreads, tick counts, normalized metrics)
            ↓
 Query Interface (date ranges, SQL filters, on-demand resampling)
 ```
@@ -149,19 +150,13 @@ Query Interface (date ranges, SQL filters, on-demand resampling)
 **Schema**:
 - `raw_spread_ticks` table: Timestamp (PK), Bid, Ask
 - `standard_ticks` table: Timestamp (PK), Bid, Ask
-- `ohlc_1m` table: Phase7 9-column schema
+- `ohlc_1m` table: Phase7 13-column schema (v1.2.0)
 - `metadata` table: Coverage tracking
 
-**Phase7 9-Column OHLC**:
-1. `Timestamp` - Bar timestamp
-2. `Open` - Opening price
-3. `High` - High price
-4. `Low` - Low price
-5. `Close` - Closing price
-6. `raw_spread_avg` - Average spread from Raw_Spread variant
-7. `standard_spread_avg` - Average spread from Standard variant
-8. `tick_count_raw_spread` - Tick count from Raw_Spread variant
-9. `tick_count_standard` - Tick count from Standard variant
+**Phase7 13-Column OHLC (v1.2.0)**:
+- **Column Definitions**: See [`schema.py`](src/exness_data_preprocess/schema.py) - Single source of truth
+- **Comprehensive Reference**: See [`DATABASE_SCHEMA.md`](docs/DATABASE_SCHEMA.md) - Query examples and usage patterns
+- **Key Features**: BID-only OHLC with dual spreads (Raw_Spread + Standard) and normalized spread metrics
 
 ### Directory Structure