exness-data-preprocess 0.2.0__tar.gz → 0.3.0__tar.gz

{exness_data_preprocess-0.2.0 → exness_data_preprocess-0.3.0}/.cz.toml

@@ -1,6 +1,6 @@
 [tool.commitizen]
 name = "cz_conventional_commits"
-version = "0.1.0"
+version = "0.2.0"
 tag_format = "v$major.$minor.$patch"
 update_changelog_on_bump = true
 version_files = [

exness_data_preprocess-0.3.0/CHANGELOG.md

@@ -0,0 +1,53 @@
+# Changelog
+
+All notable changes to exness-data-preprocess will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+
+### ✨ Features
+
+- Implement pydantic v2 models with dual-variant e2e testing - Add Pydantic v2 models (UpdateResult, CoverageInfo) for type-safe API - Update processor to use Pydantic types (PairType, TimeframeType, VariantType) - Remove deprecated api.py module (broken functionality) - Implement comprehensive test suite (48 tests, 100% passing): * test_models.py - Pydantic model validation (13 tests) * test_types.py - Type safety and helpers (15 tests) * test_processor_pydantic.py - Integration tests (6 tests) * test_functional_regression.py - v2.0.0 regression tests (10 tests) - Fix Standard variant downloads (variant="" not "Standard") - Add true end-to-end testing with real Exness downloads - Test data: EURUSD August 2024 (815K Raw_Spread + 877K Standard ticks) - Update documentation and add refactoring status tracking - Coverage: models.py 100%, __init__.py 100%, processor.py 45% BREAKING CHANGE: api.py removed (use ExnessDataProcessor directly)
+
+
+### 🔧 Continuous Integration
+
+- Modernize publish workflow with uv and validation pipeline Update GitHub Actions publish workflow to match gapless-crypto-data standards: Build System: - Replace pip+build with uv for consistency with local development - Add UV_SYSTEM_PYTHON=1 environment variable - Use uv sync --dev for dependency installation Validation Pipeline: - Add file encoding validation (UTF-8/ASCII enforcement) - Add ruff linting and format checking - Add pytest test suite execution before build Security: - Add Sigstore artifact signing for published packages - Maintain PyPI Trusted Publishing (OIDC) authentication Configuration: - Change environment name: release → pypi (semantic clarity) - Change artifact name: dist → python-package-distributions (consistency) - Add verbose output to PyPI publish step Security Note: - Added .token-info.md to .gitignore to prevent accidental token documentation commits - Workflow uses OIDC (no API token needed)
+
+
+### 🧰 Maintenance
+
+- **release**: Bump version 0.1.0 → 0.2.0 [skip ci]
+
+
+### ✨ Features
+
+- Initial commit with v2.0.0 unified DuckDB architecture - Unified single-file DuckDB storage per instrument (eurusd.duckdb) - Phase7 9-column OHLC schema with dual-variant spreads - Sub-15ms query performance with date range filtering - PRIMARY KEY constraints prevent duplicates - Automatic gap detection for incremental updates - DuckDB self-documentation with COMMENT ON statements - GitHub Actions CI/CD with PyPI Trusted Publishing - Pre-commit hooks (ruff, commitizen, basic checks) - Professional README with shields.io badges
+
+
+### 🐛 Bug Fixes
+
+- Force add .pre-commit-config.yaml (was incorrectly gitignored)
+
+- Remove .pre-commit-config.yaml from .gitignore Pre-commit config should be version controlled, not ignored.
+
+- **pre-commit**: Enable unsafe fixes for ruff to resolve remaining errors
+
+- **research**: Resolve remaining linting errors in research scripts - Fix bare except clause in 02_volatility_model_simple.py (E722) - Move math.erf imports to top in 03_liquidity_crisis_detection.py (E402) - Move math.erf imports to top in 04_regime_detection_analysis.py (E402)
+
+
+### 💅 Code Style
+
+- Apply pre-commit formatting fixes - Fix end-of-file newlines in 2 MD files - Apply ruff quote style fixes (200 errors fixed) - Reformat 26 files with ruff-format
+
+
+### 🔧 CI/CD Improvements
+
+- **ci**: Add pre-commit config and basic tests - Add .pre-commit-config.yaml to repository - Create basic test suite (test_basic.py) for CI - Add GITHUB_PYPI_SETUP.md with complete setup instructions Fixes CI failures: - Pre-commit hook now finds config file - Pytest now finds and runs tests successfully
+
+
+### 🧰 Maintenance
+
+- Prepare v0.1.0 release
+

{exness_data_preprocess-0.2.0 → exness_data_preprocess-0.3.0}/CLAUDE.md

@@ -80,7 +80,7 @@ doppler run --project claude-config --config dev -- uv publish --token "$PYPI_TO
    - **Lines 247-275**: `_append_ticks_to_db()` - Tick insertion with PRIMARY KEY duplicate prevention
    - **Lines 277-377**: `_discover_missing_months()` - Gap detection for incremental updates
    - **Lines 379-516**: `update_data()` - Main entry point for downloading and updating data
-   - **Lines 518-549**: `_regenerate_ohlc()` - Phase7 9-column OHLC generation with LEFT JOIN
+   - **Lines 518-549**: `_regenerate_ohlc()` - Phase7 13-column OHLC (v1.2.0) generation with LEFT JOIN
    - **Lines 551-603**: `query_ticks()` - Tick queries with date range and SQL filters
    - **Lines 605-706**: `query_ohlc()` - OHLC queries with on-demand resampling (1m/5m/15m/1h/4h/1d)
    - **Lines 708-821**: `add_schema_comments()` / `add_schema_comments_all()` - Retrofit self-documentation
@@ -102,10 +102,11 @@ doppler run --project claude-config --config dev -- uv publish --token "$PYPI_TO
 - Incremental updates with automatic gap detection
 - PRIMARY KEY constraints prevent duplicates during updates
 
-**Phase7 9-Column OHLC Schema**:
+**Phase7 13-Column OHLC Schema** (v1.2.0):
 - BID-only OHLC from Raw_Spread variant
 - Dual spreads: `raw_spread_avg` and `standard_spread_avg`
 - Dual tick counts: `tick_count_raw_spread` and `tick_count_standard`
+- Normalized metrics: `range_per_spread`, `range_per_tick`, `body_per_spread`, `body_per_tick`
 - Generated via LEFT JOIN between Raw_Spread and Standard variants
 
 **Self-Documentation**:
@@ -138,9 +139,9 @@ Each `.duckdb` file contains:
    - Reference data for spread comparison
 
 3. **`ohlc_1m`** table:
-   - Columns: `Timestamp` (PK), `Open`, `High`, `Low`, `Close`, `raw_spread_avg`, `standard_spread_avg`, `tick_count_raw_spread`, `tick_count_standard`
-   - Phase7 9-column schema
-   - Generated from Raw_Spread BID prices with dual-variant spreads
+   - **Schema**: Phase7 13-column (v1.2.0) - See [`schema.py`](src/exness_data_preprocess/schema.py)
+   - **Details**: BID-only OHLC with dual-variant spreads, tick counts, and normalized metrics
+   - **Reference**: [`DATABASE_SCHEMA.md`](docs/DATABASE_SCHEMA.md)
 
 4. **`metadata`** table:
    - Columns: `key` (PK), `value`, `updated_at`
@@ -179,7 +180,7 @@ Each `.duckdb` file contains:
 - ✅ **Dual-variant storage**: Raw_Spread + Standard in same database
 - ✅ **PRIMARY KEY constraints**: Prevents duplicates during incremental updates
 - ✅ **Automatic gap detection**: Downloads only missing months
-- ✅ **Phase7 9-column OHLC**: Dual spreads + dual tick counts
+- ✅ **Phase7 13-column OHLC (v1.2.0)**: Dual spreads + dual tick counts
 - ✅ **Date range queries**: Sub-15ms query performance
 - ✅ **On-demand resampling**: Any timeframe (5m, 1h, 1d) in <15ms
 
@@ -206,11 +207,13 @@ Each `.duckdb` file contains:
 - `/tmp/exness-duckdb-test/test_refactored_processor.py` - Validation test
 - `/tmp/exness-duckdb-test/test_queries_only.py` - Query validation test
 
-### Phase7 9-Column OHLC Schema (✅ Implemented)
+### Phase7 13-Column OHLC Schema v1.2.0 (✅ Implemented)
 
-**Decision**: Dual-variant BID-only OHLC with 9 columns capturing both Raw_Spread and Standard characteristics
+**Decision**: Dual-variant BID-only OHLC with 13 columns capturing both Raw_Spread and Standard characteristics plus normalized metrics
 
-**Schema**: `Timestamp | Open | High | Low | Close | raw_spread_avg | standard_spread_avg | tick_count_raw_spread | tick_count_standard`
+**Schema**: Phase7 13-column (v1.2.0)
+- **Definition**: See [`schema.py`](src/exness_data_preprocess/schema.py)
+- **Documentation**: See [`DATABASE_SCHEMA.md`](docs/DATABASE_SCHEMA.md)
 
 **Key Features**:
 - ✅ **BID-only OHLC**: Uses Raw_Spread Bid prices (execution prices)
@@ -311,7 +314,7 @@ SELECT table_name, column_name, data_type, comment FROM duckdb_columns();
 - ✅ **Dual-variant storage** - Raw_Spread + Standard in same database
 - ✅ **PRIMARY KEY constraints** - Prevents duplicates during incremental updates
 - ✅ **Automatic gap detection** - Downloads only missing months
-- ✅ **Phase7 9-column OHLC** - Dual spreads + dual tick counts
+- ✅ **Phase7 13-column OHLC (v1.2.0)** - Dual spreads + dual tick counts
 - ✅ **Date range queries** - Sub-15ms query performance
 - ✅ **On-demand resampling** - Any timeframe in <15ms
 - ✅ **SQL filter support** - Direct SQL WHERE clauses on ticks
@@ -356,7 +359,7 @@ SELECT table_name, column_name, data_type, comment FROM duckdb_columns();
 eurusd.duckdb:
 ├── raw_spread_ticks   # Timestamp (PK), Bid, Ask
 ├── standard_ticks     # Timestamp (PK), Bid, Ask
-├── ohlc_1m            # Phase7 9-column schema
+├── ohlc_1m            # Phase7 13-column schema (v1.2.0)
 └── metadata           # Coverage tracking
 ```
 

{exness_data_preprocess-0.2.0 → exness_data_preprocess-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: exness-data-preprocess
-Version: 0.2.0
+Version: 0.3.0
 Summary: Professional Exness forex tick data preprocessing with optimal compression (Parquet Zstd-22) and DuckDB OHLC generation. Provides efficient storage (9% smaller than ZIP) with lossless precision and direct queryability.
 Project-URL: Homepage, https://github.com/terrylica/exness-data-preprocess
 Project-URL: Documentation, https://github.com/terrylica/exness-data-preprocess#readme
@@ -55,14 +55,14 @@ Description-Content-Type: text/markdown
 [![Downloads](https://img.shields.io/pypi/dm/exness-data-preprocess.svg)](https://pypi.org/project/exness-data-preprocess/)
 [![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-000000.svg)](https://github.com/astral-sh/ruff)
 
-Professional forex tick data preprocessing with unified single-file DuckDB storage. Provides incremental updates, dual-variant storage (Raw_Spread + Standard), and Phase7 9-column OHLC schema with sub-15ms query performance.
+Professional forex tick data preprocessing with unified single-file DuckDB storage. Provides incremental updates, dual-variant storage (Raw_Spread + Standard), and Phase7 13-column OHLC schema (v1.2.0) with sub-15ms query performance.
 
 ## Features
 
 - **Unified Single-File Architecture**: One DuckDB file per instrument (eurusd.duckdb)
 - **Incremental Updates**: Automatic gap detection and download only missing months
 - **Dual-Variant Storage**: Raw_Spread (primary) + Standard (reference) in same database
-- **Phase7 OHLC Schema**: 9-column bars with dual spreads and dual tick counts
+- **Phase7 OHLC Schema**: 13-column bars (v1.2.0) with dual spreads, tick counts, and normalized metrics
 - **Fast Queries**: Date range queries with sub-15ms performance
 - **On-Demand Resampling**: Any timeframe (5m, 1h, 1d) resampled in <15ms
 - **PRIMARY KEY Constraints**: Prevents duplicate data during incremental updates
@@ -138,7 +138,7 @@ Download Only Missing Months (Raw_Spread + Standard)
            ↓
 DuckDB Single-File Storage (PRIMARY KEY prevents duplicates)
            ↓
-Phase7 9-Column OHLC Generation (dual spreads + dual tick counts)
+Phase7 13-Column OHLC Generation (v1.2.0 - dual spreads, tick counts, normalized metrics)
            ↓
 Query Interface (date ranges, SQL filters, on-demand resampling)
 ```
@@ -150,19 +150,13 @@ Query Interface (date ranges, SQL filters, on-demand resampling)
 **Schema**:
 - `raw_spread_ticks` table: Timestamp (PK), Bid, Ask
 - `standard_ticks` table: Timestamp (PK), Bid, Ask
-- `ohlc_1m` table: Phase7 9-column schema
+- `ohlc_1m` table: Phase7 13-column schema (v1.2.0)
 - `metadata` table: Coverage tracking
 
-**Phase7 9-Column OHLC**:
-1. `Timestamp` - Bar timestamp
-2. `Open` - Opening price
-3. `High` - High price
-4. `Low` - Low price
-5. `Close` - Closing price
-6. `raw_spread_avg` - Average spread from Raw_Spread variant
-7. `standard_spread_avg` - Average spread from Standard variant
-8. `tick_count_raw_spread` - Tick count from Raw_Spread variant
-9. `tick_count_standard` - Tick count from Standard variant
+**Phase7 13-Column OHLC (v1.2.0)**:
+- **Column Definitions**: See [`schema.py`](src/exness_data_preprocess/schema.py) - Single source of truth
+- **Comprehensive Reference**: See [`DATABASE_SCHEMA.md`](docs/DATABASE_SCHEMA.md) - Query examples and usage patterns
+- **Key Features**: BID-only OHLC with dual spreads (Raw_Spread + Standard) and normalized spread metrics
 
 ### Directory Structure
 

{exness_data_preprocess-0.2.0 → exness_data_preprocess-0.3.0}/README.md

@@ -7,14 +7,14 @@
 [![Downloads](https://img.shields.io/pypi/dm/exness-data-preprocess.svg)](https://pypi.org/project/exness-data-preprocess/)
 [![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-000000.svg)](https://github.com/astral-sh/ruff)
 
-Professional forex tick data preprocessing with unified single-file DuckDB storage. Provides incremental updates, dual-variant storage (Raw_Spread + Standard), and Phase7 9-column OHLC schema with sub-15ms query performance.
+Professional forex tick data preprocessing with unified single-file DuckDB storage. Provides incremental updates, dual-variant storage (Raw_Spread + Standard), and Phase7 13-column OHLC schema (v1.2.0) with sub-15ms query performance.
 
 ## Features
 
 - **Unified Single-File Architecture**: One DuckDB file per instrument (eurusd.duckdb)
 - **Incremental Updates**: Automatic gap detection and download only missing months
 - **Dual-Variant Storage**: Raw_Spread (primary) + Standard (reference) in same database
-- **Phase7 OHLC Schema**: 9-column bars with dual spreads and dual tick counts
+- **Phase7 OHLC Schema**: 13-column bars (v1.2.0) with dual spreads, tick counts, and normalized metrics
 - **Fast Queries**: Date range queries with sub-15ms performance
 - **On-Demand Resampling**: Any timeframe (5m, 1h, 1d) resampled in <15ms
 - **PRIMARY KEY Constraints**: Prevents duplicate data during incremental updates
@@ -90,7 +90,7 @@ Download Only Missing Months (Raw_Spread + Standard)
            ↓
 DuckDB Single-File Storage (PRIMARY KEY prevents duplicates)
            ↓
-Phase7 9-Column OHLC Generation (dual spreads + dual tick counts)
+Phase7 13-Column OHLC Generation (v1.2.0 - dual spreads, tick counts, normalized metrics)
            ↓
 Query Interface (date ranges, SQL filters, on-demand resampling)
 ```
@@ -102,19 +102,13 @@ Query Interface (date ranges, SQL filters, on-demand resampling)
 **Schema**:
 - `raw_spread_ticks` table: Timestamp (PK), Bid, Ask
 - `standard_ticks` table: Timestamp (PK), Bid, Ask
-- `ohlc_1m` table: Phase7 9-column schema
+- `ohlc_1m` table: Phase7 13-column schema (v1.2.0)
 - `metadata` table: Coverage tracking
 
-**Phase7 9-Column OHLC**:
-1. `Timestamp` - Bar timestamp
-2. `Open` - Opening price
-3. `High` - High price
-4. `Low` - Low price
-5. `Close` - Closing price
-6. `raw_spread_avg` - Average spread from Raw_Spread variant
-7. `standard_spread_avg` - Average spread from Standard variant
-8. `tick_count_raw_spread` - Tick count from Raw_Spread variant
-9. `tick_count_standard` - Tick count from Standard variant
+**Phase7 13-Column OHLC (v1.2.0)**:
+- **Column Definitions**: See [`schema.py`](src/exness_data_preprocess/schema.py) - Single source of truth
+- **Comprehensive Reference**: See [`DATABASE_SCHEMA.md`](docs/DATABASE_SCHEMA.md) - Query examples and usage patterns
+- **Key Features**: BID-only OHLC with dual spreads (Raw_Spread + Standard) and normalized spread metrics
 
 ### Directory Structure
 

exness_data_preprocess-0.3.0/RELEASE_NOTES.md

@@ -0,0 +1,22 @@
+# Changelog
+
+All notable changes to exness-data-preprocess will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+
+### ✨ Features
+
+- Implement pydantic v2 models with dual-variant e2e testing - Add Pydantic v2 models (UpdateResult, CoverageInfo) for type-safe API - Update processor to use Pydantic types (PairType, TimeframeType, VariantType) - Remove deprecated api.py module (broken functionality) - Implement comprehensive test suite (48 tests, 100% passing): * test_models.py - Pydantic model validation (13 tests) * test_types.py - Type safety and helpers (15 tests) * test_processor_pydantic.py - Integration tests (6 tests) * test_functional_regression.py - v2.0.0 regression tests (10 tests) - Fix Standard variant downloads (variant="" not "Standard") - Add true end-to-end testing with real Exness downloads - Test data: EURUSD August 2024 (815K Raw_Spread + 877K Standard ticks) - Update documentation and add refactoring status tracking - Coverage: models.py 100%, __init__.py 100%, processor.py 45% BREAKING CHANGE: api.py removed (use ExnessDataProcessor directly)
+
+
+### 🔧 Continuous Integration
+
+- Modernize publish workflow with uv and validation pipeline Update GitHub Actions publish workflow to match gapless-crypto-data standards: Build System: - Replace pip+build with uv for consistency with local development - Add UV_SYSTEM_PYTHON=1 environment variable - Use uv sync --dev for dependency installation Validation Pipeline: - Add file encoding validation (UTF-8/ASCII enforcement) - Add ruff linting and format checking - Add pytest test suite execution before build Security: - Add Sigstore artifact signing for published packages - Maintain PyPI Trusted Publishing (OIDC) authentication Configuration: - Change environment name: release → pypi (semantic clarity) - Change artifact name: dist → python-package-distributions (consistency) - Add verbose output to PyPI publish step Security Note: - Added .token-info.md to .gitignore to prevent accidental token documentation commits - Workflow uses OIDC (no API token needed)
+
+
+### 🧰 Maintenance
+
+- **release**: Bump version 0.1.0 → 0.2.0 [skip ci]
+

{exness_data_preprocess-0.2.0 → exness_data_preprocess-0.3.0}/docs/DATABASE_SCHEMA.md

@@ -1,9 +1,9 @@
-# Database Schema Documentation - exness-data-preprocess v2.0.0
+# Database Schema Documentation - exness-data-preprocess v1.2.0
 
 **Database Type**: DuckDB (embedded OLAP database)
 **Architecture**: Single-file per instrument (unified multi-year storage)
-**Schema Version**: 2.0.0 (Phase7 9-column OHLC)
-**Last Updated**: 2025-10-12
+**Schema Version**: v1.2.0 (Phase7 13-column OHLC with normalized metrics)
+**Last Updated**: 2025-10-14
 
 ---
 
@@ -11,7 +11,7 @@
 
 Each currency pair is stored in a **single DuckDB file** containing:
 - **All historical tick data** from both variants (Raw_Spread + Standard)
-- **Pre-computed 1-minute OHLC bars** with Phase7 9-column schema
+- **Pre-computed 1-minute OHLC bars** with Phase7 13-column schema (includes normalized spread metrics)
 - **Metadata** tracking coverage and update history
 
 **File Naming Convention**: `{instrument_lowercase}.duckdb`
@@ -229,25 +229,23 @@ ORDER BY hour;
 
 ## Table 3: `ohlc_1m`
 
-**Purpose**: Pre-computed 1-minute OHLC bars with Phase7 9-column schema
+**Purpose**: Pre-computed 1-minute OHLC bars with Phase7 13-column schema
 
 **Use Case**: Primary data source for backtesting and technical analysis
 
 **Generation Method**: Aggregated from `raw_spread_ticks` and `standard_ticks` tables
 
-### Schema (Phase7 9-Column)
+### Schema (Phase7 13-Column - v1.2.0)
 
-| Column                   | Type                       | Constraints   | Description                                           |
-|--------------------------|----------------------------|---------------|-------------------------------------------------------|
-| `Timestamp`              | TIMESTAMP WITH TIME ZONE   | PRIMARY KEY   | Minute-aligned bar timestamp                          |
-| `Open`                   | DOUBLE                     | NOT NULL      | Opening price (first Raw_Spread Bid)                  |
-| `High`                   | DOUBLE                     | NOT NULL      | High price (max Raw_Spread Bid)                       |
-| `Low`                    | DOUBLE                     | NOT NULL      | Low price (min Raw_Spread Bid)                        |
-| `Close`                  | DOUBLE                     | NOT NULL      | Closing price (last Raw_Spread Bid)                   |
-| `raw_spread_avg`         | DOUBLE                     | NULL          | Average spread from Raw_Spread variant (NULL if no ticks) |
-| `standard_spread_avg`    | DOUBLE                     | NULL          | Average spread from Standard variant (NULL if no Standard ticks for that minute) |
-| `tick_count_raw_spread`  | BIGINT                     | NULL          | Number of ticks from Raw_Spread variant (NULL if no ticks) |
-| `tick_count_standard`    | BIGINT                     | NULL          | Number of ticks from Standard variant (NULL if no Standard ticks for that minute) |
+**Column Definitions**: See [`../src/exness_data_preprocess/schema.py`](../src/exness_data_preprocess/schema.py) for complete column definitions with types and descriptions.
+
+**Quick Summary**: 13 columns comprising:
+- **Core OHLC** (5): Timestamp, Open, High, Low, Close
+- **Dual Spreads** (2): raw_spread_avg, standard_spread_avg
+- **Dual Tick Counts** (2): tick_count_raw_spread, tick_count_standard
+- **Normalized Metrics** (4): range_per_spread, range_per_tick, body_per_spread, body_per_tick
+
+**Schema Version**: v1.2.0
 
 ### Indexes
 
@@ -258,27 +256,25 @@ ORDER BY hour;
 
 - **PRIMARY KEY on Timestamp**: Ensures unique 1-minute bars
 - **NOT NULL constraints**: Only on OHLC price columns (Open, High, Low, Close) to ensure price data integrity
-- **NULLABLE columns**: Spread and tick count columns (raw_spread_avg, standard_spread_avg, tick_count_raw_spread, tick_count_standard) can be NULL when LEFT JOIN with Standard ticks yields no matches for that minute
+- **NULLABLE columns**:
+  - Spread and tick count columns (raw_spread_avg, standard_spread_avg, tick_count_raw_spread, tick_count_standard) can be NULL when LEFT JOIN with Standard ticks yields no matches for that minute
+  - Normalized metrics (range_per_spread, range_per_tick, body_per_spread, body_per_tick) are NULL when Standard data is missing or when division by zero would occur
 
 ### Table Comments
 
-**Stored in Database**: The following comments are automatically added when the database is created (see `processor.py` lines 201-215).
+**Stored in Database**: All 13 column comments are automatically added when the database is created, dynamically generated from `schema.py` (see `OHLCSchema.get_column_comment_sqls()`).
 
 ```sql
 COMMENT ON TABLE ohlc_1m IS
-'Phase7 v1.1.0 1-minute OHLC bars (BID-only from Raw_Spread, dual-variant spreads and tick counts).
+'Phase7 v1.2.0 1-minute OHLC bars (BID-only from Raw_Spread, dual-variant spreads and tick counts, normalized metrics).
  OHLC Source: Raw_Spread BID prices. Spreads: Dual-variant (Raw_Spread + Standard).';
 
--- Column comments
+-- All 13 column comments (see schema.py for complete definitions)
+-- Examples:
 COMMENT ON COLUMN ohlc_1m.Timestamp IS 'Minute-aligned bar timestamp';
 COMMENT ON COLUMN ohlc_1m.Open IS 'Opening price (first Raw_Spread Bid)';
-COMMENT ON COLUMN ohlc_1m.High IS 'High price (max Raw_Spread Bid)';
-COMMENT ON COLUMN ohlc_1m.Low IS 'Low price (min Raw_Spread Bid)';
-COMMENT ON COLUMN ohlc_1m.Close IS 'Closing price (last Raw_Spread Bid)';
-COMMENT ON COLUMN ohlc_1m.raw_spread_avg IS 'Average spread from Raw_Spread variant (NULL if no ticks)';
-COMMENT ON COLUMN ohlc_1m.standard_spread_avg IS 'Average spread from Standard variant (NULL if no Standard ticks for that minute)';
-COMMENT ON COLUMN ohlc_1m.tick_count_raw_spread IS 'Number of ticks from Raw_Spread variant (NULL if no ticks)';
-COMMENT ON COLUMN ohlc_1m.tick_count_standard IS 'Number of ticks from Standard variant (NULL if no Standard ticks for that minute)';
+COMMENT ON COLUMN ohlc_1m.range_per_spread IS '(High-Low)/standard_spread_avg - Range normalized by spread (NULL if no Standard ticks)';
+-- ... (see schema.py OHLCSchema.COLUMNS for all 13 column comments)
 ```
 
 **Retrieve Comments**:
@@ -301,7 +297,28 @@ SELECT
     AVG(r.Ask - r.Bid) as raw_spread_avg,
     AVG(s.Ask - s.Bid) as standard_spread_avg,
     COUNT(r.Timestamp) as tick_count_raw_spread,
-    COUNT(s.Timestamp) as tick_count_standard
+    COUNT(s.Timestamp) as tick_count_standard,
+    -- Normalized spread metrics (v1.2.0) - NULL-safe calculations
+    CASE
+        WHEN AVG(s.Ask - s.Bid) > 0
+        THEN (MAX(r.Bid) - MIN(r.Bid)) / AVG(s.Ask - s.Bid)
+        ELSE NULL
+    END as range_per_spread,
+    CASE
+        WHEN COUNT(s.Timestamp) > 0
+        THEN (MAX(r.Bid) - MIN(r.Bid)) / COUNT(s.Timestamp)
+        ELSE NULL
+    END as range_per_tick,
+    CASE
+        WHEN AVG(s.Ask - s.Bid) > 0
+        THEN ABS(LAST(r.Bid ORDER BY r.Timestamp) - FIRST(r.Bid ORDER BY r.Timestamp)) / AVG(s.Ask - s.Bid)
+        ELSE NULL
+    END as body_per_spread,
+    CASE
+        WHEN COUNT(s.Timestamp) > 0
+        THEN ABS(LAST(r.Bid ORDER BY r.Timestamp) - FIRST(r.Bid ORDER BY r.Timestamp)) / COUNT(s.Timestamp)
+        ELSE NULL
+    END as body_per_tick
 FROM raw_spread_ticks r
 LEFT JOIN standard_ticks s
     ON DATE_TRUNC('minute', r.Timestamp) = DATE_TRUNC('minute', s.Timestamp)
@@ -309,6 +326,40 @@ GROUP BY DATE_TRUNC('minute', r.Timestamp)
 ORDER BY Timestamp;
 ```
 
+###  Normalized Spread Metrics (v1.2.0+)
+
+**Purpose**: Normalize OHLC movements by market activity (spread and tick count) to enable comparison across different market conditions.
+
+**Calculations** (NULL-safe to handle missing Standard data):
+
+1. **`range_per_spread`** = (High - Low) / standard_spread_avg
+   - **Interpretation**: How many spreads wide is the bar's range?
+   - **High values**: Large price movement relative to typical spread
+   - **Example**: If range = 0.0002 and avg_spread = 0.00004, then range_per_spread = 5.0 spreads
+
+2. **`range_per_tick`** = (High - Low) / tick_count_standard
+   - **Interpretation**: Average price movement per tick
+   - **High values**: High volatility with few ticks (fast moves)
+   - **Low values**: Choppy price action with many ticks
+
+3. **`body_per_spread`** = abs(Close - Open) / standard_spread_avg
+   - **Interpretation**: How many spreads is the directional movement?
+   - **High values**: Strong directional movement relative to spread
+   - **Low values**: Indecision/ranging behavior
+
+4. **`body_per_tick`** = abs(Close - Open) / tick_count_standard
+   - **Interpretation**: Directional efficiency (movement per tick)
+   - **High values**: Strong directional movement per tick
+   - **Low values**: Many ticks but little net movement
+
+**NULL Handling**: All normalized metrics are NULL when `standard_spread_avg` or `tick_count_standard` are 0 or NULL (occurs when LEFT JOIN yields no Standard ticks for that minute).
+
+**Use Cases**:
+- Identify high-volatility bars independent of absolute price levels
+- Compare market activity across different sessions
+- Filter for significant directional moves
+- Detect ranging vs trending market conditions
+
 ### Characteristics
 
 - **Timeframe**: 1-minute bars (minute-aligned)
@@ -320,22 +371,29 @@ ORDER BY Timestamp;
 ### Example Data
 
 ```
-Timestamp            | Open    | High    | Low     | Close   | raw_spread_avg | standard_spread_avg | tick_count_raw_spread | tick_count_standard
----------------------|---------|---------|---------|---------|----------------|---------------------|-----------------------|--------------------
-2024-09-01 14:05:00  | 1.10500 | 1.10520 | 1.10495 | 1.10515 | 0.00001        | 0.00004             | 25                    | 28
-2024-09-01 14:06:00  | 1.10515 | 1.10525 | 1.10510 | 1.10520 | 0.00001        | 0.00004             | 23                    | 26
-2024-09-01 14:07:00  | 1.10520 | 1.10530 | 1.10515 | 1.10525 | 0.00001        | 0.00004             | 27                    | 30
+Timestamp            | Open    | High    | Low     | Close   | raw_spread_avg | standard_spread_avg | tick_count_raw_spread | tick_count_standard | range_per_spread | range_per_tick | body_per_spread | body_per_tick
+---------------------|---------|---------|---------|---------|----------------|---------------------|-----------------------|---------------------|------------------|----------------|-----------------|---------------
+2024-09-01 14:05:00  | 1.10500 | 1.10520 | 1.10495 | 1.10515 | 0.00001        | 0.00004             | 25                    | 28                  | 6.25             | 0.00089        | 3.75            | 0.00054
+2024-09-01 14:06:00  | 1.10515 | 1.10525 | 1.10510 | 1.10520 | 0.00001        | 0.00004             | 23                    | 26                  | 3.75             | 0.00058        | 1.25            | 0.00019
+2024-09-01 14:07:00  | 1.10520 | 1.10530 | 1.10515 | 1.10525 | 0.00001        | 0.00004             | 27                    | 30                  | 3.75             | 0.00050        | 1.25            | 0.00017
 ```
 
+**Notes**:
+- All 13 columns shown including normalized metrics (v1.2.0+)
+- `range_per_spread = (High-Low) / standard_spread_avg` (e.g., 0.00025 / 0.00004 = 6.25 spreads)
+- `range_per_tick = (High-Low) / tick_count_standard` (e.g., 0.00025 / 28 = 0.00089)
+- `body_per_spread = abs(Close-Open) / standard_spread_avg` (e.g., 0.00015 / 0.00004 = 3.75 spreads)
+- `body_per_tick = abs(Close-Open) / tick_count_standard` (e.g., 0.00015 / 28 = 0.00054)
+
 ### Query Examples
 
 ```sql
--- Get 1-minute bars for September 2024
+-- Get 1-minute bars for September 2024 (all 13 columns)
 SELECT * FROM ohlc_1m
 WHERE Timestamp >= '2024-09-01' AND Timestamp < '2024-10-01'
 ORDER BY Timestamp;
 
--- Resample to 1-hour bars
+-- Resample to 1-hour bars (all 13 columns)
 SELECT
     DATE_TRUNC('hour', Timestamp) as hour,
     FIRST(Open ORDER BY Timestamp) as Open,
@@ -345,21 +403,41 @@ SELECT
     AVG(raw_spread_avg) as raw_spread_avg,
     AVG(standard_spread_avg) as standard_spread_avg,
     SUM(tick_count_raw_spread) as tick_count_raw_spread,
-    SUM(tick_count_standard) as tick_count_standard
+    SUM(tick_count_standard) as tick_count_standard,
+    AVG(range_per_spread) as range_per_spread,
+    AVG(range_per_tick) as range_per_tick,
+    AVG(body_per_spread) as body_per_spread,
+    AVG(body_per_tick) as body_per_tick
 FROM ohlc_1m
 WHERE Timestamp >= '2024-09-01'
 GROUP BY hour
 ORDER BY hour;
 
--- Get daily statistics
+-- Filter high-volatility bars (range > 5 spreads)
+SELECT * FROM ohlc_1m
+WHERE Timestamp >= '2024-09-01'
+  AND range_per_spread > 5.0
+ORDER BY range_per_spread DESC
+LIMIT 10;
+
+-- Find strong directional bars (body > 3 spreads, high body efficiency)
+SELECT * FROM ohlc_1m
+WHERE Timestamp >= '2024-09-01'
+  AND body_per_spread > 3.0
+  AND body_per_tick > 0.0005
+ORDER BY body_per_spread DESC
+LIMIT 10;
+
+-- Compare normalized metrics across time periods
 SELECT
     DATE_TRUNC('day', Timestamp) as day,
     COUNT(*) as bar_count,
-    MIN(Low) as day_low,
-    MAX(High) as day_high,
+    AVG(range_per_spread) as avg_range_per_spread,
+    AVG(body_per_spread) as avg_body_per_spread,
     AVG(raw_spread_avg) * 10000 as avg_raw_spread_pips
 FROM ohlc_1m
 WHERE Timestamp >= '2024-09-01'
+  AND range_per_spread IS NOT NULL  -- Ensure Standard data exists
 GROUP BY day
 ORDER BY day;
 ```
@@ -703,10 +781,22 @@ print(f"Months added: {result['months_added']}")
 
 ## Version History
 
-### v2.0.0 (2025-10-12)
+### v1.2.0 (2025-10-14) - Current
+
+- **Added**: 4 normalized spread metrics to OHLC schema (range_per_spread, range_per_tick, body_per_spread, body_per_tick)
+- **Refactored**: Centralized schema definition in `schema.py` module
+- **OHLC Schema**: 13 columns (Phase7 with normalized metrics)
+- **Architecture**: Single source of truth for all schema definitions
+- **Breaking Change**: None - additive only (existing 9 columns unchanged)
+
+### v1.1.0 (2025-10-12) - Phase7 Initial
+
+- **Added**: Phase7 9-column OHLC schema (dual spreads + dual tick counts)
+- **OHLC Schema**: BID-only OHLC from Raw_Spread with dual-variant metrics
+
+### v2.0.0 (2025-10-12) - Unified Architecture
 
 - **Change**: Single-file per instrument (not monthly files)
-- **Added**: Phase7 9-column OHLC schema
 - **Added**: Dual-variant storage (Raw_Spread + Standard)
 - **Added**: PRIMARY KEY constraints for duplicate prevention
 - **Added**: Metadata table for coverage tracking
@@ -723,6 +813,7 @@ print(f"Months added: {result['months_added']}")
 
 ## Related Documentation
 
+- **Schema Definition**: [`../src/exness_data_preprocess/schema.py`](../src/exness_data_preprocess/schema.py) - Single source of truth for all schema definitions
 - **Architecture Plan**: [`UNIFIED_DUCKDB_PLAN_v2.md`](UNIFIED_DUCKDB_PLAN_v2.md)
 - **Phase7 OHLC Spec**: [`research/eurusd-zero-spread-deviations/data/plan/phase7_bid_ohlc_construction_v1.1.0.md`](research/eurusd-zero-spread-deviations/data/plan/phase7_bid_ohlc_construction_v1.1.0.md)
 - **Data Sources**: [`EXNESS_DATA_SOURCES.md`](EXNESS_DATA_SOURCES.md)
@@ -731,6 +822,6 @@ print(f"Months added: {result['months_added']}")
 
 ---
 
-**Last Updated**: 2025-10-12
+**Last Updated**: 2025-10-14
 **Maintainer**: Terry Li <terry@eonlabs.com>
-**Schema Version**: 2.0.0
+**Schema Version**: v1.2.0 (Phase7 13-column OHLC with normalized metrics)

{exness_data_preprocess-0.2.0 → exness_data_preprocess-0.3.0}/docs/README.md

@@ -24,7 +24,7 @@
 - ✅ **Incremental updates**: Automatic gap detection, download only missing months
 - ✅ **PRIMARY KEY constraints**: Prevents duplicates during incremental updates
 - ✅ **Performance**: <15ms queries for all timeframes
-- ✅ **Phase7 Compliant**: Dual-variant 9-column schema
+- ✅ **Phase7 Compliant**: Dual-variant 13-column (v1.2.0) schema
 - ✅ **SQL on Ticks**: Direct tick analysis with date range filtering
 - ✅ **Scalability**: 2.08 GB for 13 months, ~4.8 GB for 3 years
 
@@ -98,13 +98,9 @@ curl -s "https://ticks.ex2archive.com/ticks/" | jq -r '.[] | .name' | grep -i "E
 - [`04-discoveries-and-plan-evolution.md`](research/eurusd-zero-spread-deviations/04-discoveries-and-plan-evolution.md) - Version-tracked findings
 - [`data/plan/phase7_bid_ohlc_construction_v1.1.0.md`](research/eurusd-zero-spread-deviations/data/plan/phase7_bid_ohlc_construction_v1.1.0.md) - OHLC specification
 
-**Phase7 Schema** (9 columns):
-
-```sql
-Timestamp, Open, High, Low, Close,
-raw_spread_avg, standard_spread_avg,
-tick_count_raw_spread, tick_count_standard
-```
+**Phase7 Schema** (13 columns, v1.2.0):
+- **Definition**: [`../src/exness_data_preprocess/schema.py`](../src/exness_data_preprocess/schema.py)
+- **Comprehensive Guide**: [`DATABASE_SCHEMA.md`](DATABASE_SCHEMA.md)
 
 ---
 

{exness_data_preprocess-0.2.0 → exness_data_preprocess-0.3.0}/docs/UNIFIED_DUCKDB_PLAN_v2.md

@@ -22,7 +22,7 @@
 - ✅ **Maximum Simplicity**: ONE file = ONE instrument (eurusd.duckdb, xauusd.duckdb)
 - ✅ **Continuous Time Series**: No gaps between months, seamless queries across years
 - ✅ **Incremental Updates**: Check for new data on Exness, append automatically
-- ✅ **Phase7 Compliant**: Dual-variant 9-column schema
+- ✅ **Phase7 Compliant**: Dual-variant 13-column (v1.2.0) schema
 - ✅ **Scalable**: ~400-500 MB per instrument for 3 years (manageable)
 - ✅ **Simple Backups**: Copy one file to backup entire instrument history
 
@@ -587,7 +587,7 @@ Based on real data validation:
 - ✅ **Multi-year storage** (minimum 3 years)
 - ✅ **Incremental updates** (automatic gap detection)
 - ✅ **Duplicate prevention** (PRIMARY KEY constraints)
-- ✅ **Phase7 compliance** (9-column OHLC schema)
+- ✅ **Phase7 compliance** (13-column (v1.2.0) OHLC schema)
 - ✅ **Time-range queries** (start_date/end_date support)
 
 ### Performance

{exness_data_preprocess-0.2.0 → exness_data_preprocess-0.3.0}/examples/basic_usage.py

@@ -11,7 +11,7 @@ Architecture v2.0.0:
 - One DuckDB file per instrument (eurusd.duckdb, not monthly files)
 - Dual-variant storage (Raw_Spread + Standard) for Phase7
 - Automatic gap detection and incremental updates
-- 9-column OHLC schema with dual spreads
+- 13-column (v1.2.0) OHLC schema with dual spreads
 """
 
 from pathlib import Path
@@ -33,7 +33,7 @@ print("=" * 80)
 #   1. Create eurusd.duckdb (single file)
 #   2. Download missing months from start_date to present
 #   3. Store dual variants (Raw_Spread + Standard)
-#   4. Generate Phase7 9-column OHLC
+#   4. Generate Phase7 13-column (v1.2.0) OHLC
 processor = edp.ExnessDataProcessor(base_dir=BASE_DIR)
 
 result = processor.update_data(
@@ -206,7 +206,7 @@ print("\nKey Features of v2.0.0:")
 print("   ✅ Single file per instrument (eurusd.duckdb)")
 print("   ✅ Automatic incremental updates")
 print("   ✅ Dual-variant storage (Phase7 compliant)")
-print("   ✅ 9-column OHLC schema")
+print("   ✅ 13-column (v1.2.0) OHLC schema")
 print("   ✅ Date range queries")
 print("   ✅ On-demand resampling (<15ms)")
 print("   ✅ SQL filters on ticks")