gapless-crypto-data 2.8.0__tar.gz → 2.9.0__tar.gz

{gapless_crypto_data-2.8.0 → gapless_crypto_data-2.9.0}/.gitignore

@@ -12,3 +12,5 @@ venv/
 env/
 test-env/
 *-env/
+# Gapless Crypto Data checkpoint files (auto-generated during collection)
+.gapless_checkpoints/

gapless_crypto_data-2.8.0/README.md → gapless_crypto_data-2.9.0/PKG-INFO

@@ -1,3 +1,51 @@
+Metadata-Version: 2.4
+Name: gapless-crypto-data
+Version: 2.9.0
+Summary: Ultra-fast cryptocurrency data collection with zero gaps guarantee. 22x faster via Binance public repository with complete 13-timeframe support (1s-1d) and intelligent monthly-to-daily fallback. Provides 11-column microstructure format with order flow metrics.
+Project-URL: Homepage, https://github.com/Eon-Labs/gapless-crypto-data
+Project-URL: Documentation, https://github.com/Eon-Labs/gapless-crypto-data#readme
+Project-URL: Repository, https://github.com/Eon-Labs/gapless-crypto-data.git
+Project-URL: Issues, https://github.com/Eon-Labs/gapless-crypto-data/issues
+Project-URL: Changelog, https://github.com/Eon-Labs/gapless-crypto-data/blob/main/CHANGELOG.md
+Author-email: Eon Labs <terry@eonlabs.com>
+Maintainer-email: Terry Li <terry@eonlabs.com>
+License: MIT
+License-File: AUTHORS.md
+License-File: LICENSE
+Keywords: 13-timeframes,1s-1d,22x-faster,OHLCV,api,authentic-data,backward-compatibility,binance,ccxt,collection,crypto,cryptocurrency,data,download,dual-parameter,fetch-data,financial-data,function-based,gap-filling,gapless,interval,liquidity,microstructure,monthly-daily-fallback,order-flow,pandas,performance,taker-volume,time-series,timeframe,trading,ultra-high-frequency,zero-gaps
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Financial and Insurance Industry
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Office/Business :: Financial :: Investment
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.9
+Requires-Dist: httpx>=0.25.0
+Requires-Dist: joblib>=1.5.2
+Requires-Dist: pandas>=2.0.0
+Requires-Dist: polars>=1.33.1
+Requires-Dist: pyarrow>=21.0.0
+Requires-Dist: pyod>=2.0.5
+Provides-Extra: dev
+Requires-Dist: black>=23.0.0; extra == 'dev'
+Requires-Dist: mypy>=1.5.0; extra == 'dev'
+Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
+Requires-Dist: pytest>=7.0.0; extra == 'dev'
+Requires-Dist: ruff>=0.1.0; extra == 'dev'
+Provides-Extra: test
+Requires-Dist: pytest-asyncio>=0.21.0; extra == 'test'
+Requires-Dist: pytest-cov>=4.0.0; extra == 'test'
+Requires-Dist: pytest>=7.0.0; extra == 'test'
+Description-Content-Type: text/markdown
+
 # Gapless Crypto Data
 
 [![PyPI version](https://badge.fury.io/py/gapless-crypto-data.svg)](https://badge.fury.io/py/gapless-crypto-data)
@@ -5,18 +53,22 @@
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![UV Managed](https://img.shields.io/badge/uv-managed-blue.svg)](https://github.com/astral-sh/uv)
 
-Cryptocurrency data collection with gap detection and filling capabilities. Provides 11-column microstructure format through Binance public data repository.
+Ultra-fast cryptocurrency data collection with zero gaps guarantee. Provides 11-column microstructure format through Binance public data repository with intelligent monthly-to-daily fallback for seamless coverage.
 
 ## Features
 
-- Data collection via Binance public data repository
-- 11-column microstructure format with order flow and liquidity metrics
-- Gap detection and filling with API validation
-- UV-based Python tooling
-- Atomic file operations
-- Multi-symbol & multi-timeframe support (1m, 3m, 5m, 15m, 30m, 1h, 2h, 4h)
-- CCXT-compatible dual parameter support (timeframe/interval)
-- Backward compatibility with 5-year deprecation period
+- **22x faster** data collection via Binance public data repository
+- **Zero gaps guarantee** through intelligent monthly-to-daily fallback
+- **Complete 13-timeframe support**: 1s, 1m, 3m, 5m, 15m, 30m, 1h, 2h, 4h, 6h, 8h, 12h, 1d
+- **Ultra-high frequency** to daily data collection (1-second to 1-day intervals)
+- **11-column microstructure format** with order flow and liquidity metrics
+- **Intelligent fallback system** automatically switches to daily files when monthly files unavailable
+- **Gap detection and filling** with authentic Binance API data only
+- **UV-based Python tooling** for modern dependency management
+- **Atomic file operations** ensuring data integrity
+- **Multi-symbol & multi-timeframe** concurrent collection
+- **CCXT-compatible** dual parameter support (timeframe/interval)
+- **Production-grade** with comprehensive test coverage
 
 ## Quick Start
 
@@ -39,11 +91,17 @@ pip install gapless-crypto-data
 ### CLI Usage
 
 ```bash
-# Collect data for multiple timeframes (default output location)
-gapless-crypto-data --symbol SOLUSDT --timeframes 1m,3m,5m,15m,30m,1h,2h,4h
+# Collect data for multiple timeframes (all 13 timeframes supported)
+gapless-crypto-data --symbol SOLUSDT --timeframes 1s,1m,5m,1h,4h,1d
+
+# Ultra-high frequency data collection (1-second intervals)
+gapless-crypto-data --symbol BTCUSDT --timeframes 1s,1m,3m
+
+# Extended timeframes with intelligent fallback
+gapless-crypto-data --symbol ETHUSDT --timeframes 6h,8h,12h,1d
 
 # Collect multiple symbols at once (native multi-symbol support)
-gapless-crypto-data --symbol BTCUSDT,ETHUSDT,SOLUSDT --timeframes 1h,4h
+gapless-crypto-data --symbol BTCUSDT,ETHUSDT,SOLUSDT --timeframes 1h,4h,1d
 
 # Collect specific date range with custom output directory
 gapless-crypto-data --symbol BTCUSDT --timeframes 1h --start 2023-01-01 --end 2023-12-31 --output-dir ./crypto_data
@@ -405,16 +463,23 @@ gapless-crypto-data/
 
 ## 🔍 Supported Timeframes
 
-| Timeframe | Code | Description |
-|-----------|------|-------------|
-| 1 minute  | `1m` | Highest resolution |
-| 3 minutes | `3m` | Short-term analysis |
-| 5 minutes | `5m` | Common trading timeframe |
-| 15 minutes| `15m`| Medium-term signals |
-| 30 minutes| `30m`| Longer-term patterns |
-| 1 hour    | `1h` | Popular for backtesting |
-| 2 hours   | `2h` | Extended analysis |
-| 4 hours   | `4h` | Daily cycle patterns |
+All 13 Binance timeframes supported for complete market coverage:
+
+| Timeframe | Code | Description | Use Case |
+|-----------|------|-------------|----------|
+| 1 second  | `1s` | Ultra-high frequency | HFT, microstructure analysis |
+| 1 minute  | `1m` | High resolution | Scalping, order flow |
+| 3 minutes | `3m` | Short-term analysis | Quick trend detection |
+| 5 minutes | `5m` | Common trading timeframe | Day trading signals |
+| 15 minutes| `15m`| Medium-term signals | Swing trading entry |
+| 30 minutes| `30m`| Longer-term patterns | Position management |
+| 1 hour    | `1h` | Popular for backtesting | Strategy development |
+| 2 hours   | `2h` | Extended analysis | Multi-timeframe confluence |
+| 4 hours   | `4h` | Daily cycle patterns | Trend following |
+| 6 hours   | `6h` | Quarter-day analysis | Position sizing |
+| 8 hours   | `8h` | Third-day cycles | Risk management |
+| 12 hours  | `12h`| Half-day patterns | Overnight positions |
+| 1 day     | `1d` | Daily analysis | Long-term trends |
 
 ## ⚠️ Requirements
 
@@ -564,23 +629,104 @@ success = merger.merge_gap_data_safe(
 )
 ```
 
-### Data Format
+## Output Formats
+
+### DataFrame Structure (Python API)
+
+Returns pandas DataFrame with 11-column microstructure format:
+
+| Column | Type | Description | Example |
+|--------|------|-------------|---------|
+| `date` | datetime64[ns] | Open timestamp | `2024-01-01 12:00:00` |
+| `open` | float64 | Opening price | `42150.50` |
+| `high` | float64 | Highest price | `42200.00` |
+| `low` | float64 | Lowest price | `42100.25` |
+| `close` | float64 | Closing price | `42175.75` |
+| `volume` | float64 | Base asset volume | `15.250000` |
+| `close_time` | datetime64[ns] | Close timestamp | `2024-01-01 12:59:59` |
+| `quote_asset_volume` | float64 | Quote asset volume | `643238.125` |
+| `number_of_trades` | int64 | Trade count | `1547` |
+| `taker_buy_base_asset_volume` | float64 | Taker buy base volume | `7.825000` |
+| `taker_buy_quote_asset_volume` | float64 | Taker buy quote volume | `329891.750` |
+
+### CSV File Structure
+
+CSV files include header comments with metadata followed by data:
+
+```csv
+# Binance Spot Market Data v2.5.0
+# Generated: 2025-09-18T23:09:25.391126+00:00Z
+# Source: Binance Public Data Repository
+# Market: SPOT | Symbol: BTCUSDT | Timeframe: 1h
+# Coverage: 48 bars
+# Period: 2024-01-01 00:00:00 to 2024-01-02 23:00:00
+# Collection: direct_download in 0.0s
+# Data Hash: 5fba9d2e5d3db849...
+# Compliance: Zero-Magic-Numbers, Temporal-Integrity, Official-Binance-Source
+#
+date,open,high,low,close,volume,close_time,quote_asset_volume,number_of_trades,taker_buy_base_asset_volume,taker_buy_quote_asset_volume
+2024-01-01 00:00:00,42283.58,42554.57,42261.02,42475.23,1271.68108,2024-01-01 00:59:59,53957248.973789,47134,682.57581,28957416.819645
+```
+
+### Metadata JSON Structure
+
+Each CSV file includes comprehensive metadata in `.metadata.json`:
+
+```json
+{
+  "version": "v2.5.0",
+  "generator": "BinancePublicDataCollector",
+  "data_source": "Binance Public Data Repository",
+  "symbol": "BTCUSDT",
+  "timeframe": "1h",
+  "enhanced_microstructure_format": {
+    "total_columns": 11,
+    "analysis_capabilities": [
+      "order_flow_analysis",
+      "liquidity_metrics",
+      "market_microstructure",
+      "trade_weighted_prices",
+      "institutional_data_patterns"
+    ]
+  },
+  "gap_analysis": {
+    "total_gaps_detected": 0,
+    "data_completeness_score": 1.0,
+    "gap_filling_method": "authentic_binance_api"
+  },
+  "data_integrity": {
+    "chronological_order": true,
+    "corruption_detected": false
+  }
+}
+```
+
+### Streaming Output (Memory-Efficient)
+
+For large datasets, Polars streaming provides constant memory usage:
 
-All classes work with the standardized 11-column microstructure format:
+```python
+from gapless_crypto_data.streaming import StreamingDataProcessor
+
+processor = StreamingDataProcessor(chunk_size=10_000, memory_limit_mb=100)
+for chunk in processor.stream_csv_chunks("large_dataset.csv"):
+    # Process chunk with constant memory usage
+    print(f"Chunk shape: {chunk.shape}")
+```
+
+### File Naming Convention
+
+Output files follow consistent naming pattern:
+
+```
+binance_spot_{SYMBOL}-{TIMEFRAME}_{START_DATE}-{END_DATE}_v{VERSION}.csv
+binance_spot_{SYMBOL}-{TIMEFRAME}_{START_DATE}-{END_DATE}_v{VERSION}.metadata.json
+```
 
-| Column | Description | Example |
-|--------|-------------|---------|
-| `date` | Open timestamp | `2024-01-01 12:00:00` |
-| `open` | Opening price | `42150.50` |
-| `high` | Highest price | `42200.00` |
-| `low` | Lowest price | `42100.25` |
-| `close` | Closing price | `42175.75` |
-| `volume` | Base asset volume | `15.250000` |
-| `close_time` | Close timestamp | `2024-01-01 12:59:59` |
-| `quote_asset_volume` | Quote asset volume | `643238.125` |
-| `number_of_trades` | Trade count | `1547` |
-| `taker_buy_base_asset_volume` | Taker buy base volume | `7.825000` |
-| `taker_buy_quote_asset_volume` | Taker buy quote volume | `329891.750` |
+Examples:
+- `binance_spot_BTCUSDT-1h_20240101-20240102_v2.5.0.csv`
+- `binance_spot_ETHUSDT-4h_20240101-20240201_v2.5.0.csv`
+- `binance_spot_SOLUSDT-1d_20240101-20241231_v2.5.0.csv`
 
 ### Error Handling
 

gapless_crypto_data-2.8.0/PKG-INFO → gapless_crypto_data-2.9.0/README.md

@@ -1,51 +1,3 @@
-Metadata-Version: 2.4
-Name: gapless-crypto-data
-Version: 2.8.0
-Summary: Cryptocurrency data collection with gap detection and filling capabilities. Provides 11-column microstructure format with dual parameter support (timeframe/interval) for CCXT compatibility.
-Project-URL: Homepage, https://github.com/Eon-Labs/gapless-crypto-data
-Project-URL: Documentation, https://github.com/Eon-Labs/gapless-crypto-data#readme
-Project-URL: Repository, https://github.com/Eon-Labs/gapless-crypto-data.git
-Project-URL: Issues, https://github.com/Eon-Labs/gapless-crypto-data/issues
-Project-URL: Changelog, https://github.com/Eon-Labs/gapless-crypto-data/blob/main/CHANGELOG.md
-Author-email: Eon Labs <terry@eonlabs.com>
-Maintainer-email: Terry Li <terry@eonlabs.com>
-License: MIT
-License-File: AUTHORS.md
-License-File: LICENSE
-Keywords: OHLCV,api,authentic-data,backward-compatibility,binance,ccxt,collection,crypto,cryptocurrency,data,download,dual-parameter,fetch-data,financial-data,function-based,gap-filling,interval,liquidity,microstructure,order-flow,pandas,taker-volume,time-series,timeframe,trading
-Classifier: Development Status :: 5 - Production/Stable
-Classifier: Intended Audience :: Developers
-Classifier: Intended Audience :: Financial and Insurance Industry
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Operating System :: OS Independent
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Programming Language :: Python :: 3.13
-Classifier: Topic :: Office/Business :: Financial :: Investment
-Classifier: Topic :: Scientific/Engineering :: Information Analysis
-Classifier: Topic :: Software Development :: Libraries :: Python Modules
-Requires-Python: >=3.9
-Requires-Dist: httpx>=0.25.0
-Requires-Dist: joblib>=1.5.2
-Requires-Dist: pandas>=2.0.0
-Requires-Dist: polars>=1.33.1
-Requires-Dist: pyarrow>=21.0.0
-Requires-Dist: pyod>=2.0.5
-Provides-Extra: dev
-Requires-Dist: black>=23.0.0; extra == 'dev'
-Requires-Dist: mypy>=1.5.0; extra == 'dev'
-Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
-Requires-Dist: pytest>=7.0.0; extra == 'dev'
-Requires-Dist: ruff>=0.1.0; extra == 'dev'
-Provides-Extra: test
-Requires-Dist: pytest-asyncio>=0.21.0; extra == 'test'
-Requires-Dist: pytest-cov>=4.0.0; extra == 'test'
-Requires-Dist: pytest>=7.0.0; extra == 'test'
-Description-Content-Type: text/markdown
-
 # Gapless Crypto Data
 
 [![PyPI version](https://badge.fury.io/py/gapless-crypto-data.svg)](https://badge.fury.io/py/gapless-crypto-data)
@@ -53,18 +5,22 @@ Description-Content-Type: text/markdown
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![UV Managed](https://img.shields.io/badge/uv-managed-blue.svg)](https://github.com/astral-sh/uv)
 
-Cryptocurrency data collection with gap detection and filling capabilities. Provides 11-column microstructure format through Binance public data repository.
+Ultra-fast cryptocurrency data collection with zero gaps guarantee. Provides 11-column microstructure format through Binance public data repository with intelligent monthly-to-daily fallback for seamless coverage.
 
 ## Features
 
-- Data collection via Binance public data repository
-- 11-column microstructure format with order flow and liquidity metrics
-- Gap detection and filling with API validation
-- UV-based Python tooling
-- Atomic file operations
-- Multi-symbol & multi-timeframe support (1m, 3m, 5m, 15m, 30m, 1h, 2h, 4h)
-- CCXT-compatible dual parameter support (timeframe/interval)
-- Backward compatibility with 5-year deprecation period
+- **22x faster** data collection via Binance public data repository
+- **Zero gaps guarantee** through intelligent monthly-to-daily fallback
+- **Complete 13-timeframe support**: 1s, 1m, 3m, 5m, 15m, 30m, 1h, 2h, 4h, 6h, 8h, 12h, 1d
+- **Ultra-high frequency** to daily data collection (1-second to 1-day intervals)
+- **11-column microstructure format** with order flow and liquidity metrics
+- **Intelligent fallback system** automatically switches to daily files when monthly files unavailable
+- **Gap detection and filling** with authentic Binance API data only
+- **UV-based Python tooling** for modern dependency management
+- **Atomic file operations** ensuring data integrity
+- **Multi-symbol & multi-timeframe** concurrent collection
+- **CCXT-compatible** dual parameter support (timeframe/interval)
+- **Production-grade** with comprehensive test coverage
 
 ## Quick Start
 
@@ -87,11 +43,17 @@ pip install gapless-crypto-data
 ### CLI Usage
 
 ```bash
-# Collect data for multiple timeframes (default output location)
-gapless-crypto-data --symbol SOLUSDT --timeframes 1m,3m,5m,15m,30m,1h,2h,4h
+# Collect data for multiple timeframes (all 13 timeframes supported)
+gapless-crypto-data --symbol SOLUSDT --timeframes 1s,1m,5m,1h,4h,1d
+
+# Ultra-high frequency data collection (1-second intervals)
+gapless-crypto-data --symbol BTCUSDT --timeframes 1s,1m,3m
+
+# Extended timeframes with intelligent fallback
+gapless-crypto-data --symbol ETHUSDT --timeframes 6h,8h,12h,1d
 
 # Collect multiple symbols at once (native multi-symbol support)
-gapless-crypto-data --symbol BTCUSDT,ETHUSDT,SOLUSDT --timeframes 1h,4h
+gapless-crypto-data --symbol BTCUSDT,ETHUSDT,SOLUSDT --timeframes 1h,4h,1d
 
 # Collect specific date range with custom output directory
 gapless-crypto-data --symbol BTCUSDT --timeframes 1h --start 2023-01-01 --end 2023-12-31 --output-dir ./crypto_data
@@ -453,16 +415,23 @@ gapless-crypto-data/
 
 ## 🔍 Supported Timeframes
 
-| Timeframe | Code | Description |
-|-----------|------|-------------|
-| 1 minute  | `1m` | Highest resolution |
-| 3 minutes | `3m` | Short-term analysis |
-| 5 minutes | `5m` | Common trading timeframe |
-| 15 minutes| `15m`| Medium-term signals |
-| 30 minutes| `30m`| Longer-term patterns |
-| 1 hour    | `1h` | Popular for backtesting |
-| 2 hours   | `2h` | Extended analysis |
-| 4 hours   | `4h` | Daily cycle patterns |
+All 13 Binance timeframes supported for complete market coverage:
+
+| Timeframe | Code | Description | Use Case |
+|-----------|------|-------------|----------|
+| 1 second  | `1s` | Ultra-high frequency | HFT, microstructure analysis |
+| 1 minute  | `1m` | High resolution | Scalping, order flow |
+| 3 minutes | `3m` | Short-term analysis | Quick trend detection |
+| 5 minutes | `5m` | Common trading timeframe | Day trading signals |
+| 15 minutes| `15m`| Medium-term signals | Swing trading entry |
+| 30 minutes| `30m`| Longer-term patterns | Position management |
+| 1 hour    | `1h` | Popular for backtesting | Strategy development |
+| 2 hours   | `2h` | Extended analysis | Multi-timeframe confluence |
+| 4 hours   | `4h` | Daily cycle patterns | Trend following |
+| 6 hours   | `6h` | Quarter-day analysis | Position sizing |
+| 8 hours   | `8h` | Third-day cycles | Risk management |
+| 12 hours  | `12h`| Half-day patterns | Overnight positions |
+| 1 day     | `1d` | Daily analysis | Long-term trends |
 
 ## ⚠️ Requirements
 
@@ -612,23 +581,104 @@ success = merger.merge_gap_data_safe(
 )
 ```
 
-### Data Format
+## Output Formats
+
+### DataFrame Structure (Python API)
+
+Returns pandas DataFrame with 11-column microstructure format:
+
+| Column | Type | Description | Example |
+|--------|------|-------------|---------|
+| `date` | datetime64[ns] | Open timestamp | `2024-01-01 12:00:00` |
+| `open` | float64 | Opening price | `42150.50` |
+| `high` | float64 | Highest price | `42200.00` |
+| `low` | float64 | Lowest price | `42100.25` |
+| `close` | float64 | Closing price | `42175.75` |
+| `volume` | float64 | Base asset volume | `15.250000` |
+| `close_time` | datetime64[ns] | Close timestamp | `2024-01-01 12:59:59` |
+| `quote_asset_volume` | float64 | Quote asset volume | `643238.125` |
+| `number_of_trades` | int64 | Trade count | `1547` |
+| `taker_buy_base_asset_volume` | float64 | Taker buy base volume | `7.825000` |
+| `taker_buy_quote_asset_volume` | float64 | Taker buy quote volume | `329891.750` |
+
+### CSV File Structure
+
+CSV files include header comments with metadata followed by data:
+
+```csv
+# Binance Spot Market Data v2.5.0
+# Generated: 2025-09-18T23:09:25.391126+00:00Z
+# Source: Binance Public Data Repository
+# Market: SPOT | Symbol: BTCUSDT | Timeframe: 1h
+# Coverage: 48 bars
+# Period: 2024-01-01 00:00:00 to 2024-01-02 23:00:00
+# Collection: direct_download in 0.0s
+# Data Hash: 5fba9d2e5d3db849...
+# Compliance: Zero-Magic-Numbers, Temporal-Integrity, Official-Binance-Source
+#
+date,open,high,low,close,volume,close_time,quote_asset_volume,number_of_trades,taker_buy_base_asset_volume,taker_buy_quote_asset_volume
+2024-01-01 00:00:00,42283.58,42554.57,42261.02,42475.23,1271.68108,2024-01-01 00:59:59,53957248.973789,47134,682.57581,28957416.819645
+```
+
+### Metadata JSON Structure
+
+Each CSV file includes comprehensive metadata in `.metadata.json`:
+
+```json
+{
+  "version": "v2.5.0",
+  "generator": "BinancePublicDataCollector",
+  "data_source": "Binance Public Data Repository",
+  "symbol": "BTCUSDT",
+  "timeframe": "1h",
+  "enhanced_microstructure_format": {
+    "total_columns": 11,
+    "analysis_capabilities": [
+      "order_flow_analysis",
+      "liquidity_metrics",
+      "market_microstructure",
+      "trade_weighted_prices",
+      "institutional_data_patterns"
+    ]
+  },
+  "gap_analysis": {
+    "total_gaps_detected": 0,
+    "data_completeness_score": 1.0,
+    "gap_filling_method": "authentic_binance_api"
+  },
+  "data_integrity": {
+    "chronological_order": true,
+    "corruption_detected": false
+  }
+}
+```
+
+### Streaming Output (Memory-Efficient)
+
+For large datasets, Polars streaming provides constant memory usage:
 
-All classes work with the standardized 11-column microstructure format:
+```python
+from gapless_crypto_data.streaming import StreamingDataProcessor
+
+processor = StreamingDataProcessor(chunk_size=10_000, memory_limit_mb=100)
+for chunk in processor.stream_csv_chunks("large_dataset.csv"):
+    # Process chunk with constant memory usage
+    print(f"Chunk shape: {chunk.shape}")
+```
+
+### File Naming Convention
+
+Output files follow consistent naming pattern:
+
+```
+binance_spot_{SYMBOL}-{TIMEFRAME}_{START_DATE}-{END_DATE}_v{VERSION}.csv
+binance_spot_{SYMBOL}-{TIMEFRAME}_{START_DATE}-{END_DATE}_v{VERSION}.metadata.json
+```
 
-| Column | Description | Example |
-|--------|-------------|---------|
-| `date` | Open timestamp | `2024-01-01 12:00:00` |
-| `open` | Opening price | `42150.50` |
-| `high` | Highest price | `42200.00` |
-| `low` | Lowest price | `42100.25` |
-| `close` | Closing price | `42175.75` |
-| `volume` | Base asset volume | `15.250000` |
-| `close_time` | Close timestamp | `2024-01-01 12:59:59` |
-| `quote_asset_volume` | Quote asset volume | `643238.125` |
-| `number_of_trades` | Trade count | `1547` |
-| `taker_buy_base_asset_volume` | Taker buy base volume | `7.825000` |
-| `taker_buy_quote_asset_volume` | Taker buy quote volume | `329891.750` |
+Examples:
+- `binance_spot_BTCUSDT-1h_20240101-20240102_v2.5.0.csv`
+- `binance_spot_ETHUSDT-4h_20240101-20240201_v2.5.0.csv`
+- `binance_spot_SOLUSDT-1d_20240101-20241231_v2.5.0.csv`
 
 ### Error Handling