ml4t-data 0.1.0__tar.gz

ml4t_data-0.1.0/.gitignore

@@ -0,0 +1,45 @@
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+.env
+.venv/
+venv/
+ENV/
+.mypy_cache/
+.ruff_cache/
+.pytest_cache/
+.coverage
+htmlcov/
+*.log
+.DS_Store
+*.tmp
+/data/
+logs/
+site/
+src/ml4t/data/_version.py
+coverage/
+
+# Claude Code (local development only)
+CLAUDE.md
+.claude/
+.agentic_documentation/
+.skills-outputs/
+planning/
+.idea/

ml4t_data-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025-2026 Stefan Jansen
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

ml4t_data-0.1.0/PKG-INFO

@@ -0,0 +1,457 @@
+Metadata-Version: 2.4
+Name: ml4t-data
+Version: 0.1.0
+Summary: High-performance market data management library with unified multi-provider interface
+Project-URL: Homepage, https://www.ml4trading.io/docs/data/
+Project-URL: Documentation, https://www.ml4trading.io/docs/data/
+Project-URL: Repository, https://github.com/ml4t/data
+Project-URL: Issues, https://github.com/ml4t/data/issues
+Project-URL: Changelog, https://github.com/ml4t/data/blob/main/CHANGELOG.md
+Author-email: ML4T Team <info@ml4trading.io>
+Maintainer-email: ML4T Contributors <dev@ml4trading.io>
+License: MIT
+License-File: LICENSE
+Keywords: backtesting,binance,cryptocompare,databento,finance,market-data,oanda,parquet,polars,quantitative-finance,trading,yahoo-finance
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Financial and Insurance Industry
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: MacOS
+Classifier: Operating System :: Microsoft :: Windows
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Office/Business :: Financial :: Investment
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Typing :: Typed
+Requires-Python: >=3.12
+Requires-Dist: aiofiles>=23.0.0
+Requires-Dist: click>=8.0.0
+Requires-Dist: filelock>=3.19.1
+Requires-Dist: httpx>=0.25.0
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: openpyxl>=3.1.5
+Requires-Dist: pandas-market-calendars>=4.3.0
+Requires-Dist: pandas>=2.0.0
+Requires-Dist: platformdirs>=4.0.0
+Requires-Dist: polars>=0.20.0
+Requires-Dist: pybreaker>=1.0.0
+Requires-Dist: pydantic-settings>=2.0.0
+Requires-Dist: pydantic<3,>=2.12
+Requires-Dist: pydantic>=2.14.0b1; python_version >= '3.15'
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.0.0
+Requires-Dist: structlog>=23.0.0
+Requires-Dist: tenacity>=8.0.0
+Provides-Extra: all
+Requires-Dist: cot-reports>=0.1.0; extra == 'all'
+Requires-Dist: databento>=0.38.0; extra == 'all'
+Requires-Dist: hypothesis>=6.80.0; extra == 'all'
+Requires-Dist: ipdb>=0.13.0; extra == 'all'
+Requires-Dist: ipython>=8.14.0; extra == 'all'
+Requires-Dist: mkdocs-git-revision-date-localized-plugin>=1.2.0; extra == 'all'
+Requires-Dist: mkdocs-material>=9.5.0; extra == 'all'
+Requires-Dist: mkdocs<2,>=1.6; extra == 'all'
+Requires-Dist: mkdocstrings[python]>=0.24.0; extra == 'all'
+Requires-Dist: oandapyv20>=0.7.0; extra == 'all'
+Requires-Dist: pre-commit>=3.3.0; extra == 'all'
+Requires-Dist: pyarrow>=14.0.0; extra == 'all'
+Requires-Dist: pytest-asyncio>=0.21.0; extra == 'all'
+Requires-Dist: pytest-cov>=4.1.0; extra == 'all'
+Requires-Dist: pytest-timeout>=2.1.0; extra == 'all'
+Requires-Dist: pytest-xdist>=3.3.0; extra == 'all'
+Requires-Dist: pytest>=7.4.0; extra == 'all'
+Requires-Dist: requests>=2.32.0; extra == 'all'
+Requires-Dist: ruff>=0.1.0; extra == 'all'
+Requires-Dist: ty; extra == 'all'
+Requires-Dist: xlsxwriter>=3.1.0; extra == 'all'
+Requires-Dist: yfinance>=0.2.0; extra == 'all'
+Provides-Extra: all-providers
+Requires-Dist: cot-reports>=0.1.0; extra == 'all-providers'
+Requires-Dist: databento>=0.38.0; extra == 'all-providers'
+Requires-Dist: oandapyv20>=0.7.0; extra == 'all-providers'
+Requires-Dist: pyarrow>=14.0.0; extra == 'all-providers'
+Requires-Dist: requests>=2.32.0; extra == 'all-providers'
+Requires-Dist: yfinance>=0.2.0; extra == 'all-providers'
+Provides-Extra: cot
+Requires-Dist: cot-reports>=0.1.0; extra == 'cot'
+Provides-Extra: databento
+Requires-Dist: databento>=0.38.0; extra == 'databento'
+Requires-Dist: pyarrow>=14.0.0; extra == 'databento'
+Provides-Extra: dev
+Requires-Dist: hypothesis>=6.80.0; extra == 'dev'
+Requires-Dist: ipdb>=0.13.0; extra == 'dev'
+Requires-Dist: ipython>=8.14.0; extra == 'dev'
+Requires-Dist: oandapyv20>=0.7.0; extra == 'dev'
+Requires-Dist: pre-commit>=3.3.0; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.21.0; extra == 'dev'
+Requires-Dist: pytest-cov>=4.1.0; extra == 'dev'
+Requires-Dist: pytest-timeout>=2.1.0; extra == 'dev'
+Requires-Dist: pytest-xdist>=3.3.0; extra == 'dev'
+Requires-Dist: pytest>=7.4.0; extra == 'dev'
+Requires-Dist: ruff>=0.1.0; extra == 'dev'
+Requires-Dist: ty; extra == 'dev'
+Requires-Dist: xlsxwriter>=3.1.0; extra == 'dev'
+Provides-Extra: docs
+Requires-Dist: mkdocs-git-revision-date-localized-plugin>=1.2.0; extra == 'docs'
+Requires-Dist: mkdocs-material>=9.5.0; extra == 'docs'
+Requires-Dist: mkdocs<2,>=1.6; extra == 'docs'
+Requires-Dist: mkdocstrings[python]>=0.24.0; extra == 'docs'
+Provides-Extra: oanda
+Requires-Dist: oandapyv20>=0.7.0; extra == 'oanda'
+Requires-Dist: requests>=2.32.0; extra == 'oanda'
+Provides-Extra: yahoo
+Requires-Dist: yfinance>=0.2.0; extra == 'yahoo'
+Description-Content-Type: text/markdown
+
+# ml4t-data
+
+[![Python 3.12-3.14](https://img.shields.io/badge/python-3.12--3.14-blue.svg)](https://www.python.org/downloads/)
+[![PyPI](https://img.shields.io/pypi/v/ml4t-data)](https://pypi.org/project/ml4t-data/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+Unified market data acquisition and storage for quantitative research workflows.
+
+## Part of the ML4T Library Ecosystem
+
+This library is one of six interconnected libraries supporting the machine learning for trading workflow described in [Machine Learning for Trading](https://www.ml4trading.io):
+
+![ML4T Library Ecosystem](docs/images/ml4t_ecosystem_workflow_color.png)
+
+Together they cover data infrastructure, feature engineering, modeling, signal evaluation, strategy backtesting, and live deployment.
+
+## What This Library Does
+
+Quantitative research requires consistent, reproducible access to market data from multiple sources. ml4t-data provides:
+
+- `DataManager` as the unified OHLCV interface for registry providers with that capability
+- 20+ provider adapters covering equities, crypto, futures, forex, macro, prediction markets, and factors
+- Automated storage in Hive-partitioned Parquet format with metadata tracking
+- Incremental updates, gap detection, and backfill via CLI
+- Built-in data validation (OHLC invariants, deduplication, anomaly detection)
+- Futures module for CME/ICE bulk downloads with continuous contract construction
+- COT module for CFTC Commitment of Traders weekly reports
+- Resilience: rate limiting, retry with exponential backoff, gap detection
+
+The goal is to support an ongoing research workflow rather than one-off downloads. Data is stored locally, tracked for freshness, and queryable with tools like DuckDB or Polars.
+
+![ml4t-data Architecture](docs/images/ml4t_data_architecture_print.jpeg)
+
+## Installation
+
+ML4T Data supports stable CPython 3.12 through 3.14 on Linux, macOS, and Windows.
+Python 3.15 prereleases are tested as informational compatibility checks but are not supported
+until the required dependencies publish compatible distributions. The Databento SDK currently
+has no CPython 3.15 distribution, so the 3.15 prerelease lane tests the core package without the
+Databento extra.
+
+```bash
+pip install ml4t-data
+```
+
+## Quick Start
+
+### DataManager (Unified Interface)
+
+```python
+from ml4t.data import DataManager
+
+dm = DataManager()
+
+# Fetch and store
+dm.fetch("AAPL", "2020-01-01", "2024-12-31", provider="yahoo")
+
+# Load from local storage
+data = dm.load("AAPL", "2020-01-01", "2024-12-31")
+
+# Batch load multiple symbols
+prices = dm.batch_load(["AAPL", "MSFT", "GOOGL"], "2020-01-01", "2024-12-31")
+
+# Incremental update
+dm.update("AAPL")
+
+# List what's stored
+symbols = dm.list_symbols()
+metadata = dm.get_metadata("AAPL")
+```
+
+### Direct Provider Access
+
+Providers expose capability-specific methods. OHLCV providers use `fetch_ohlcv`, economic-series
+providers may also use `fetch_series`, and factor providers use `fetch`.
+
+```python
+from datetime import UTC, datetime, timedelta
+
+from ml4t.data.providers import YahooFinanceProvider, CoinGeckoProvider, FREDProvider
+
+# Equities
+provider = YahooFinanceProvider()
+data = provider.fetch_ohlcv("AAPL", "2020-01-01", "2024-12-31")
+
+# Crypto
+last_complete_day = datetime.now(UTC).date() - timedelta(days=1)
+crypto = CoinGeckoProvider().fetch_ohlcv(
+    "bitcoin",
+    str(last_complete_day - timedelta(days=6)),
+    str(last_complete_day),
+)
+
+# Economic data
+# Reads FRED_API_KEY from the environment
+fred = FREDProvider().fetch_series("GDP", "2020-01-01", "2024-12-31")
+```
+
+## Data Providers
+
+### No API Key Required
+
+| Provider | Coverage |
+|----------|----------|
+| Yahoo Finance | US/global equities, ETFs, crypto, forex |
+| CoinGecko | 10,000+ cryptocurrencies; daily OHLCV for 29 completed UTC days |
+| FXMacroData | FX macro releases, calendars, COT, commodities, sentiment |
+| Fama-French | Academic factor data |
+| AQR | Research factors (QMJ, BAB, HML Devil, VME, more) |
+| Wiki Prices | Frozen US equities history (1962-2018) |
+| Kalshi | Prediction market contracts |
+| Polymarket | Prediction market history/order book snapshots |
+| Binance Public | Bulk crypto data downloads |
+| Binance | Crypto exchange data |
+| OKX | Crypto perpetuals and funding rates |
+| NASDAQ ITCH Sample | Tick-level sample data |
+
+### Authenticated or Metered APIs
+
+| Provider | Coverage |
+|----------|----------|
+| FRED | 850,000 economic series |
+| Alpaca | US equities + crypto (free IEX feed) |
+| EODHD | 60+ global exchanges |
+| Tiingo | US equities with quality focus |
+| Twelve Data | Multi-asset coverage |
+| Databento | CME/ICE futures; OPRA options |
+| Massive | US equities, options, futures, forex, crypto |
+| Finnhub | 70+ global exchanges |
+| OANDA | Forex broker data |
+| CryptoCompare | Included adapter; not release-qualified for 0.1.0 |
+
+CryptoCompare account registration was unavailable during the 0.1.0 release review, so no live
+contract evidence was obtained. The adapter remains available for evaluation, but CryptoCompare is
+not part of the release-qualified provider set until its contract passes on a release commit.
+
+## Specialized Modules
+
+### Futures
+
+Bulk download and continuous contract construction for CME/ICE products:
+
+```python
+from ml4t.data.futures import FuturesDownloader, ContinuousContractBuilder
+
+# Bulk download via Databento (parent symbology)
+downloader = FuturesDownloader(config)
+downloader.download()  # Downloads ES, NQ, CL, GC, etc.
+
+# Build continuous contracts with configurable roll logic
+builder = ContinuousContractBuilder()
+continuous = builder.build(contracts_df, roll_method="volume")
+```
+
+Book-focused interface with profiling:
+
+```python
+from ml4t.data.futures import FuturesDataManager
+
+fm = FuturesDataManager.from_config("config.yaml")
+fm.download_all()
+data = fm.load_ohlcv("ES")
+profile = fm.generate_profile("ES")
+```
+
+### COT (Commitment of Traders)
+
+CFTC weekly positioning data for futures markets:
+
+```python
+import polars as pl
+
+from ml4t.data.cot import COTFetcher, combine_cot_ohlcv, create_cot_features
+
+fetcher = COTFetcher()
+official_schedule = pl.read_parquet("cot-release-schedule.parquet")
+cot_data = fetcher.fetch_product(
+    "ES",
+    start_year=2015,
+    end_year=2024,
+    release_schedule=official_schedule,
+)
+
+# Point-in-time combination with OHLCV
+combined = combine_cot_ohlcv(ohlcv_data, cot_data)
+
+# Generate weekly features without counting forward-filled daily rows as new reports
+features = create_cot_features(combined)
+```
+
+The schedule must contain one `report_date` and timezone-aware `available_at` timestamp per
+report. CFTC publishes a tentative schedule for only the latest 13 months, so retain the exact
+release timestamps you use for historical research. Shutdown catch-up releases and other
+exceptions must use their actual publication timestamps. See the
+[official CFTC release schedule](https://www.cftc.gov/MarketReports/CommitmentsofTraders/ReleaseSchedule/index.htm).
+
+### Book Data Managers
+
+Simplified interfaces for the ML4T book workflow:
+
+```python
+from ml4t.data.etfs import ETFDataManager
+from ml4t.data.crypto import CryptoDataManager
+
+# 50 diversified ETFs via Yahoo Finance
+etf_dm = ETFDataManager.from_config("config.yaml")
+etf_dm.download_all()
+aapl = etf_dm.load_ohlcv("AAPL")
+
+# Crypto premium index via Binance Public
+crypto_dm = CryptoDataManager.from_config("config.yaml")
+crypto_dm.download_premium_index()
+```
+
+## CLI for Automated Updates
+
+```bash
+# Fetch specific symbols
+ml4t-data fetch -s AAPL -s MSFT -s GOOGL --provider yahoo --start 2020-01-01
+
+# Incremental update
+ml4t-data update --symbol AAPL
+
+# Validate data quality
+ml4t-data validate --symbol AAPL --anomalies
+
+# Check storage status
+ml4t-data status --detailed
+
+# List available data
+ml4t-data list-data
+
+# Export to CSV/JSON/Excel
+ml4t-data export --symbol AAPL --format-type csv --output aapl.csv
+
+# Get symbol info
+ml4t-data info --symbol AAPL
+```
+
+Configuration-driven batch updates:
+
+```yaml
+storage:
+  base_path: ~/data/market
+
+datasets:
+  - name: sp500_daily
+    provider: yahoo
+    symbols_file: symbols/sp500.txt
+    frequency: daily
+    start_date: 2015-01-01
+
+  - name: crypto
+    provider: coingecko
+    symbols: [bitcoin, ethereum, solana]
+    frequency: daily
+    initial_load_days: 29
+```
+
+## Storage Format
+
+Data is stored in Hive-partitioned Parquet:
+
+```
+~/data/market/
+├── yahoo/daily/symbol=AAPL/data.parquet
+├── yahoo/daily/symbol=MSFT/data.parquet
+└── coingecko/daily/symbol=bitcoin/data.parquet
+```
+
+Query with DuckDB or Polars:
+
+```python
+import duckdb
+
+result = duckdb.execute("""
+    SELECT * FROM read_parquet('~/data/market/yahoo/daily/**/*.parquet')
+    WHERE symbol IN ('AAPL', 'MSFT')
+    AND date >= '2024-01-01'
+""").pl()
+```
+
+## Data Validation
+
+```python
+from ml4t.data.validation import OHLCVValidator, ValidationReport
+
+validator = OHLCVValidator()
+report = validator.validate(data)
+# Checks: high >= low, high >= open/close, low <= open/close
+# Detects: duplicates, gaps, anomalies
+```
+
+Anomaly detection:
+
+```python
+from ml4t.data.anomaly import AnomalyManager, ReturnOutlierDetector, VolumeSpikeDetector
+
+manager = AnomalyManager([
+    ReturnOutlierDetector(),
+    VolumeSpikeDetector(),
+])
+report = manager.detect(data)
+```
+
+## Documentation
+
+- [Getting Started](docs/user-guide/getting-started.md) - quick start guide
+- [Configuration](docs/user-guide/configuration.md) - YAML config reference
+- [Storage](docs/user-guide/storage.md) - Hive partitioning and backends
+- [Incremental Updates](docs/user-guide/incremental-updates.md) - update strategies and gap detection
+- [Data Quality](docs/user-guide/data-quality.md) - validation and anomaly detection
+- [CLI Reference](docs/user-guide/cli-reference.md) - command-line interface
+- [Provider Selection Guide](docs/getting-started/provider-selection.md) - choosing providers
+- [Creating a Provider](docs/contributing/creating-a-provider.md) - extending with new sources
+
+## Technical Characteristics
+
+- **Polars-based**: Native Polars DataFrames throughout
+- **Capability-specific schemas**: OHLCV, economic-series, factor, and specialized providers
+  expose contracts suited to their data
+- **Async support**: OHLCV providers implementing the async protocol can use parallel batch operations
+- **Metadata tracking**: Last update timestamps, row counts, date ranges
+- **Resilience**: Rate limiting, retry with exponential backoff, gap detection
+- **Storage layouts**: Partitioned Hive and single-file Parquet storage
+- **Type-safe**: Full type annotations throughout
+
+## Related Libraries
+
+- **ml4t-engineer**: Feature engineering and technical indicators
+- **ml4t-diagnostic**: Signal evaluation and statistical validation
+- **ml4t-backtest**: Event-driven backtesting
+- **ml4t-live**: Live trading with broker integration
+
+## Development
+
+```bash
+git clone https://github.com/ml4t/data.git ml4t-data
+cd ml4t-data
+uv sync
+uv run pytest tests/ -q
+uv run ty check
+```
+
+## License
+
+MIT License - see [LICENSE](LICENSE) for details.