ohmydata 0.2.0__tar.gz → 0.2.2__tar.gz

{ohmydata-0.2.0 → ohmydata-0.2.2}/CHANGELOG.md

@@ -1,5 +1,34 @@
 # Changelog
 
+## 0.2.2 — 2026-09-06
+
+- Enhanced `yfinance` fundamentals pipeline with institutional Forward P/E calibration and GAAP distortion detection:
+  - **Institutional FY1 Forward P/E Calibration**:
+    - Calibrated `forward_pe` anchored to current fiscal year (`0y`) sell-side consensus instead of out-year (+1y).
+    - Preserved raw provider-native values as `raw_forward_pe` and `raw_forward_eps` alongside `forward_pe_source` (`"FY1_CONSENSUS"` or `"RAW_FALLBACK"`).
+    - Implemented defensive price derivation and currency/share-class unit mismatch protection for ADRs (ratio guard between 0.25 and 3.0) to safely fall back to raw values.
+  - **GAAP vs. Non-GAAP Distortion Detection**:
+    - Integrated Non-GAAP core operating consensus from `info.get("epsCurrentYear")` (`eps_current_year`).
+    - Added algorithmic relative deviation calculation (`gaap_diff_pct = |eps_0y - eps_current_year| / min`) and distortion flagging (`has_gaap_distortion` when `gaap_diff_pct > 0.25`).
+    - Protects algorithmic screeners and downstream valuation pipelines from spin-off, M&A, and accounting windfall false-value traps (e.g. `GEV`).
+
+## 0.2.1 — 2026-09-04
+
+- Upgraded official `yfinance` provider lock from `1.5.1` to `1.7.0` (`ohmydata[yfinance]`):
+  - Solved `curl_cffi >= 0.16` compatibility issues and improved session robustness.
+  - Runtime version assertion updated to `EXPECTED_YFINANCE_VERSION = "1.7.0"`.
+- Built permanent, reusable **Zero-Drift Audit Engine** (`ohmydata.tools.drift_audit` and CLI command `omd audit-drift`):
+  - Exact calendar date-set alignment, missing/extra session detection, and column-by-column numeric delta auditing (`open`, `high`, `low`, `close`, `adj_close`, `volume`).
+  - Automated Markdown summary and JSON machine-readable audit receipt generation.
+  - Supports `--raw-only` for bit-exact unadjusted bar verification and configurable tolerances (`--abs-tolerance`, `--rel-tolerance`).
+- Registered canonical **`r10a0` Multi-Asset ETF Benchmark Universe** (`ohmydata.tools.universes`):
+  - 13 unique US ETFs across `equity_risk`, `sector_cyclicals`, and `defensive` clusters (Cluster Variant v3) and macro regime states (`risk_on`, `risk_off`).
+  - Audited 10+ years of history (2015-01-01 to 2026-09-01, 38,116 daily bars): achieved **100% bit-exact zero-drift on raw OHLCV** (0 diffs across all columns).
+- Established provider governance policy in `AGENTS.md`:
+  - Strict `repair=False, auto_adjust=False, actions=True, keepna=True` invariant for canonical ingestion and backtesting pipelines.
+  - Bounded domain QC anomaly checks in `quality.py` (`check_ohlc_anomalies`, `check_price_jump_anomalies`, `check_volume_anomalies`).
+  - Mandated `omd audit-drift` gate for all future provider upgrades.
+
 ## 0.2.0 — 2026-09-04
 
 - Added official `yfinance` market data provider locked against `yfinance==1.5.1` baseline (`ohmydata[yfinance]`):

{ohmydata-0.2.0 → ohmydata-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: ohmydata
-Version: 0.2.0
+Version: 0.2.2
 Summary: Financial and alternative market data ingestion SDK.
 Author: OMD contributors
 Requires-Python: <3.13,>=3.11
@@ -21,11 +21,13 @@ Requires-Dist: pandas<3.0,>=2.0; extra == 'vintage-plane'
 Requires-Dist: pyarrow<24.0,>=23.0.1; extra == 'vintage-plane'
 Provides-Extra: yfinance
 Requires-Dist: pandas<3.0,>=2.0; extra == 'yfinance'
-Requires-Dist: yfinance==1.5.1; extra == 'yfinance'
+Requires-Dist: yfinance==1.7.0; extra == 'yfinance'
 Description-Content-Type: text/markdown
 
 # Oh My Data (OMD)
 
+[English](README.md) | [中文说明](README_zh.md)
+
 `ohmydata` is an offline-first market-data ingestion SDK and CLI. Provider endpoint
 adapters accept already initialized official-compatible clients; credentials
 are never loaded by this library.
@@ -40,8 +42,9 @@ uv run python -c "import ohmydata; print(ohmydata.__version__)"
 ```
 
 The core has no runtime dependencies. Install `ohmydata[tushare]` for the
-Pandas-backed Tushare adapter, `ohmydata[sec-cli]` for the SEC N-PORT batch CLI,
-or `ohmydata[sec-financials]` for company 10-K/10-Q financial statements and Parquet
+Pandas-backed Tushare adapter, `ohmydata[yfinance]` for US/global market data and
+fundamentals, `ohmydata[sec-cli]` for the SEC N-PORT batch CLI, or
+`ohmydata[sec-financials]` for company 10-K/10-Q financial statements and Parquet
 dataset writer. Provider tests use fake clients and never call a network.
 
 ## Core Architecture (offline & immutable)
@@ -503,6 +506,65 @@ preserve native line item labels and concepts (`concept`, `label`, `value_native
 beside standardized XBRL categories (`standard_concept`) for cross-company
 quantitative comparisons.
 
+### yfinance (US & Global Market Data, Fundamentals, and Zero-Drift Audit)
+
+Install `ohmydata[yfinance]` to access normalized market data, valuation ratios,
+and financial statements with strict version pinning (`yfinance==1.7.0`):
+
+```python
+from ohmydata.providers.yfinance import (
+    YFinanceAdjustmentMode,
+    YFinanceBatchPolicy,
+    YFinanceClient,
+    YFinanceDailyBarsRequest,
+    YFinanceFundamentalsRequest,
+    YFinanceRepairPolicy,
+)
+
+client = YFinanceClient()
+
+# 1. Fetch normalized daily bars (OHLCV) with repair isolation
+bars_req = YFinanceDailyBarsRequest(
+    symbols=("SPY", "QQQ", "^VIX"),
+    start_date="2024-01-01",
+    end_date_exclusive="2024-02-01",
+    adjustment_mode=YFinanceAdjustmentMode.RAW_WITH_ADJ_CLOSE,
+    batch_policy=YFinanceBatchPolicy.STRICT,
+    repair_policy=YFinanceRepairPolicy.PER_SYMBOL,
+)
+bars_result = client.fetch_daily_bars(bars_req)
+df = bars_result.dataframe
+
+# 2. Fetch fundamentals with institutional FY1 Forward P/E calibration
+fund_req = YFinanceFundamentalsRequest(
+    symbols=("NVDA", "GEV"),
+    include_financials=True,
+    include_valuation=True,
+    include_estimates=True,
+)
+fund_result = client.fetch_fundamentals(fund_req)
+
+nvda = fund_result.records["NVDA"]
+# Forward P/E is calibrated to current year consensus (FY1 0y.avg) rather than out-year (+1y)
+print("NVDA Calibrated FPE:", nvda.valuation.forward_pe, nvda.valuation.forward_pe_source)
+print("NVDA Raw Yahoo FPE:", nvda.valuation.raw_forward_pe)
+
+gev = fund_result.records["GEV"]
+# GAAP vs Non-GAAP accounting distortion detection (flags one-off windfalls > 25%)
+if gev.estimates.has_gaap_distortion:
+    print(f"GEV GAAP distortion flagged! Gap: {gev.estimates.gaap_diff_pct * 100:.1f}%")
+```
+
+#### Zero-Drift Audit CLI (`omd audit-drift`)
+
+Audit 10+ years of historical data against the 13-ETF `r10a0` benchmark universe before
+any provider upgrade:
+
+```bash
+# Strict unadjusted market bar zero-drift gate (must be bit-exact 0.0 error)
+uv run omd audit-drift --universe r10a0 --baseline-dir <old_version_dir> --target-dir <new_version_dir> --raw-only
+```
+
 ## Dataframe Adapters (Polars & Pandas)
 
 The optional `ohmydata[polars]` extra provides explicit, eager representation

{ohmydata-0.2.0 → ohmydata-0.2.2}/README.md

@@ -1,5 +1,7 @@
 # Oh My Data (OMD)
 
+[English](README.md) | [中文说明](README_zh.md)
+
 `ohmydata` is an offline-first market-data ingestion SDK and CLI. Provider endpoint
 adapters accept already initialized official-compatible clients; credentials
 are never loaded by this library.
@@ -14,8 +16,9 @@ uv run python -c "import ohmydata; print(ohmydata.__version__)"
 ```
 
 The core has no runtime dependencies. Install `ohmydata[tushare]` for the
-Pandas-backed Tushare adapter, `ohmydata[sec-cli]` for the SEC N-PORT batch CLI,
-or `ohmydata[sec-financials]` for company 10-K/10-Q financial statements and Parquet
+Pandas-backed Tushare adapter, `ohmydata[yfinance]` for US/global market data and
+fundamentals, `ohmydata[sec-cli]` for the SEC N-PORT batch CLI, or
+`ohmydata[sec-financials]` for company 10-K/10-Q financial statements and Parquet
 dataset writer. Provider tests use fake clients and never call a network.
 
 ## Core Architecture (offline & immutable)
@@ -477,6 +480,65 @@ preserve native line item labels and concepts (`concept`, `label`, `value_native
 beside standardized XBRL categories (`standard_concept`) for cross-company
 quantitative comparisons.
 
+### yfinance (US & Global Market Data, Fundamentals, and Zero-Drift Audit)
+
+Install `ohmydata[yfinance]` to access normalized market data, valuation ratios,
+and financial statements with strict version pinning (`yfinance==1.7.0`):
+
+```python
+from ohmydata.providers.yfinance import (
+    YFinanceAdjustmentMode,
+    YFinanceBatchPolicy,
+    YFinanceClient,
+    YFinanceDailyBarsRequest,
+    YFinanceFundamentalsRequest,
+    YFinanceRepairPolicy,
+)
+
+client = YFinanceClient()
+
+# 1. Fetch normalized daily bars (OHLCV) with repair isolation
+bars_req = YFinanceDailyBarsRequest(
+    symbols=("SPY", "QQQ", "^VIX"),
+    start_date="2024-01-01",
+    end_date_exclusive="2024-02-01",
+    adjustment_mode=YFinanceAdjustmentMode.RAW_WITH_ADJ_CLOSE,
+    batch_policy=YFinanceBatchPolicy.STRICT,
+    repair_policy=YFinanceRepairPolicy.PER_SYMBOL,
+)
+bars_result = client.fetch_daily_bars(bars_req)
+df = bars_result.dataframe
+
+# 2. Fetch fundamentals with institutional FY1 Forward P/E calibration
+fund_req = YFinanceFundamentalsRequest(
+    symbols=("NVDA", "GEV"),
+    include_financials=True,
+    include_valuation=True,
+    include_estimates=True,
+)
+fund_result = client.fetch_fundamentals(fund_req)
+
+nvda = fund_result.records["NVDA"]
+# Forward P/E is calibrated to current year consensus (FY1 0y.avg) rather than out-year (+1y)
+print("NVDA Calibrated FPE:", nvda.valuation.forward_pe, nvda.valuation.forward_pe_source)
+print("NVDA Raw Yahoo FPE:", nvda.valuation.raw_forward_pe)
+
+gev = fund_result.records["GEV"]
+# GAAP vs Non-GAAP accounting distortion detection (flags one-off windfalls > 25%)
+if gev.estimates.has_gaap_distortion:
+    print(f"GEV GAAP distortion flagged! Gap: {gev.estimates.gaap_diff_pct * 100:.1f}%")
+```
+
+#### Zero-Drift Audit CLI (`omd audit-drift`)
+
+Audit 10+ years of historical data against the 13-ETF `r10a0` benchmark universe before
+any provider upgrade:
+
+```bash
+# Strict unadjusted market bar zero-drift gate (must be bit-exact 0.0 error)
+uv run omd audit-drift --universe r10a0 --baseline-dir <old_version_dir> --target-dir <new_version_dir> --raw-only
+```
+
 ## Dataframe Adapters (Polars & Pandas)
 
 The optional `ohmydata[polars]` extra provides explicit, eager representation

{ohmydata-0.2.0 → ohmydata-0.2.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "ohmydata"
-version = "0.2.0"
+version = "0.2.2"
 description = "Financial and alternative market data ingestion SDK."
 readme = "README.md"
 requires-python = ">=3.11,<3.13"
@@ -16,7 +16,7 @@ omd = "ohmydata.cli:main"
 
 [project.optional-dependencies]
 tushare = ["pandas>=2.0,<3.0", "tushare>=1.4,<2.0"]
-yfinance = ["pandas>=2.0,<3.0", "yfinance==1.5.1"]
+yfinance = ["pandas>=2.0,<3.0", "yfinance==1.7.0"]
 polars = [
   "pandas>=2.0,<3.0",
   "polars>=1.38.1,<2.0",

{ohmydata-0.2.0 → ohmydata-0.2.2}/src/ohmydata/__init__.py

@@ -1,5 +1,5 @@
 """Provisional offline-only Oh My Data package scaffold."""
 
-__version__ = "0.2.0"
+__version__ = "0.2.2"
 
 __all__ = ["__version__"]

{ohmydata-0.2.0 → ohmydata-0.2.2}/src/ohmydata/cli.py

@@ -7,18 +7,22 @@ import sys
 
 from .core.errors import OhMyDataError
 from .providers.sec.cli import add_nport_commands, run
+from .tools.cli import add_audit_commands, run_audit
 
 
 def main(argv: list[str] | None = None) -> int:
     parser = argparse.ArgumentParser(prog="omd")
     commands = parser.add_subparsers(dest="command", required=True)
     add_nport_commands(commands)
+    add_audit_commands(commands)
     args = parser.parse_args(argv)
     try:
         if args.command == "sec":
             return run(args)
+        if args.command == "audit-drift":
+            return run_audit(args)
     except OhMyDataError:
-        print("SEC operation failed", file=sys.stderr)
+        print("OMD operation failed", file=sys.stderr)
         return 2
     except ValueError as exc:
         print(str(exc), file=sys.stderr)

ohmydata-0.2.2/src/ohmydata/providers/yfinance/_fundamentals_parsers.py

@@ -0,0 +1,199 @@
+"""Internal helper functions for parsing financial statements, valuation, and analyst estimates."""
+
+from __future__ import annotations
+
+import datetime
+from typing import Any
+
+import pandas as pd
+
+
+def _safe_float(val: Any) -> float | None:
+    """Convert arbitrary numeric value to float or None."""
+    try:
+        if val is None or pd.isna(val):
+            return None
+        f = float(val)
+        return f if pd.notna(f) else None
+    except (ValueError, TypeError):
+        return None
+
+
+def extract_quarterly_pair(series: pd.Series | None) -> tuple[float | None, float | None]:
+    """Return (latest_quarter, same_quarter_last_year) from a quarterly series."""
+    if series is None or series.empty:
+        return (None, None)
+    cleaned = pd.to_numeric(series, errors="coerce").dropna()
+    if len(cleaned) < 5:
+        # If at least 1 quarter exists, return latest, but prev_year is None
+        latest = _safe_float(cleaned.iloc[0]) if len(cleaned) >= 1 else None
+        return (latest, None)
+    return (_safe_float(cleaned.iloc[0]), _safe_float(cleaned.iloc[4]))
+
+
+def extract_metric_pair(
+    stmt: pd.DataFrame | None, candidate_keys: list[str]
+) -> tuple[float | None, float | None]:
+    """Search for metric rows across aliases and extract (latest, prev_year)."""
+    if stmt is None or stmt.empty:
+        return (None, None)
+    # Search index names ignoring case and whitespace
+    norm_index = {str(idx).strip().lower(): idx for idx in stmt.index}
+    for candidate in candidate_keys:
+        cand_key = candidate.strip().lower()
+        if cand_key in norm_index:
+            row_key = norm_index[cand_key]
+            return extract_quarterly_pair(stmt.loc[row_key])
+    return (None, None)
+
+
+def extract_report_date(
+    info: dict[str, Any] | None, income_stmt: pd.DataFrame | None
+) -> datetime.date | None:
+    """Extract report date from mostRecentQuarter or statement columns, rejecting epoch 0."""
+    info = info or {}
+    candidates: list[datetime.date] = []
+
+    mrq = info.get("mostRecentQuarter")
+    if mrq is not None:
+        try:
+            if isinstance(mrq, (int, float)) and mrq > 0:
+                dt = datetime.datetime.fromtimestamp(mrq, tz=datetime.UTC).date()
+                candidates.append(dt)
+            elif isinstance(mrq, str):
+                dt = pd.to_datetime(mrq).date()
+                candidates.append(dt)
+        except (ValueError, TypeError, OSError):
+            pass
+
+    if income_stmt is not None and not income_stmt.empty:
+        for col in income_stmt.columns:
+            try:
+                dt = pd.to_datetime(col).date()
+                candidates.append(dt)
+                break
+            except (ValueError, TypeError, OSError):
+                pass
+
+    for dt in candidates:
+        # Must be after 1990-01-01 and not epoch 1970-01-01
+        if dt.year >= 1990:
+            return dt
+
+    return None
+
+
+def extract_estimates_horizons(
+    df_est: pd.DataFrame | None,
+) -> tuple[float | None, float | None, float | None, float | None]:
+    """Extract (current_q, next_q, current_y, next_y) from estimate DataFrame."""
+    if df_est is None or df_est.empty:
+        return (None, None, None, None)
+
+    # Clean index
+    df_clean = df_est.copy()
+    df_clean.index = [str(idx).strip().lower() for idx in df_clean.index]
+
+    # Find avg / mean column
+    target_col = None
+    for col in df_clean.columns:
+        c_low = str(col).lower()
+        if "avg" in c_low or "mean" in c_low:
+            target_col = col
+            break
+    if target_col is None and len(df_clean.columns) > 0:
+        target_col = df_clean.columns[0]
+
+    if target_col is None:
+        return (None, None, None, None)
+
+    series = pd.to_numeric(df_clean[target_col], errors="coerce")
+
+    def _get_val(keys: list[str]) -> float | None:
+        for k in keys:
+            if k in series.index:
+                val = series.loc[k]
+                return _safe_float(val)
+        return None
+
+    cq = _get_val(["0q", "current quarter", "currentq"])
+    nq = _get_val(["+1q", "next quarter", "nextq", "1q"])
+    cy = _get_val(["0y", "current year", "currenty"])
+    ny = _get_val(["+1y", "next year", "nexty", "1y"])
+    return (cq, nq, cy, ny)
+
+
+def derive_implied_price(
+    raw_forward_pe: float | None,
+    raw_forward_eps: float | None,
+    trailing_pe: float | None,
+    deps_latest: float | None,
+    market_cap: float | None,
+    shares_outstanding: float | None,
+) -> float | None:
+    """Derive implied market price defensively across priority cascade."""
+    if (
+        raw_forward_pe is not None
+        and raw_forward_eps is not None
+        and raw_forward_pe > 0
+        and raw_forward_eps > 0
+    ):
+        return raw_forward_pe * raw_forward_eps
+    if trailing_pe is not None and deps_latest is not None and trailing_pe > 0 and deps_latest > 0:
+        return trailing_pe * deps_latest
+    if market_cap is not None and shares_outstanding is not None and shares_outstanding > 0:
+        return market_cap / shares_outstanding
+    return None
+
+
+def calibrate_forward_pe(
+    implied_price: float | None,
+    raw_forward_pe: float | None,
+    raw_forward_eps: float | None,
+    eps_0y: float | None,
+) -> tuple[float | None, float | None, str | None]:
+    """Calibrate institutional FY1 Forward P/E against sell-side consensus with ADR protection.
+
+    Returns:
+        tuple of (calibrated_forward_pe, calibrated_forward_eps, forward_pe_source)
+    """
+    calibrated_fpe = raw_forward_pe
+    calibrated_feps = raw_forward_eps
+    fpe_source = "RAW_FALLBACK" if raw_forward_pe is not None else None
+
+    if implied_price is not None and eps_0y is not None and eps_0y > 0:
+        is_mismatched = False
+        if raw_forward_eps is not None and raw_forward_eps > 0:
+            ratio = eps_0y / raw_forward_eps
+            if ratio > 3.0 or ratio < 0.25:
+                is_mismatched = True
+
+        if not is_mismatched:
+            calibrated_fpe = round(implied_price / eps_0y, 4)
+            calibrated_feps = round(eps_0y, 4)
+            fpe_source = "FY1_CONSENSUS"
+
+    return (calibrated_fpe, calibrated_feps, fpe_source)
+
+
+def detect_gaap_distortion(
+    eps_0y: float | None,
+    eps_current_year: float | None,
+) -> tuple[float | None, bool]:
+    """Detect relative GAAP vs Non-GAAP consensus divergence (e.g. spin-offs, M&A windfalls).
+
+    Returns:
+        tuple of (gaap_diff_pct, has_gaap_distortion)
+    """
+    gaap_diff_pct = None
+    has_gaap_distortion = False
+
+    if eps_0y is not None and eps_current_year is not None and eps_0y > 0 and eps_current_year > 0:
+        denom = min(eps_0y, eps_current_year)
+        if denom > 0:
+            gap = abs(eps_0y - eps_current_year) / denom
+            gaap_diff_pct = round(gap, 4)
+            if gap > 0.25:
+                has_gaap_distortion = True
+
+    return (gaap_diff_pct, has_gaap_distortion)

{ohmydata-0.2.0 → ohmydata-0.2.2}/src/ohmydata/providers/yfinance/client.py

@@ -37,11 +37,11 @@ from .fundamentals import (
 )
 from .quality import evaluate_symbol_outcomes, validate_daily_bars_dataframe
 
-EXPECTED_YFINANCE_VERSION = "1.5.1"
+EXPECTED_YFINANCE_VERSION = "1.7.0"
 
 
 def assert_yfinance_version(module: Any = None) -> None:
-    """Assert that yfinance is installed and matches the reviewed 1.5.1 baseline."""
+    """Assert that yfinance is installed and matches the reviewed 1.7.0 baseline."""
     if module is None:
         try:
             module = importlib.import_module("yfinance")
@@ -54,7 +54,7 @@ def assert_yfinance_version(module: Any = None) -> None:
     if actual_version != EXPECTED_YFINANCE_VERSION:
         raise YFinanceVersionMismatchError(
             f"yfinance version mismatch: expected {EXPECTED_YFINANCE_VERSION!r}, got {actual_version!r}. "
-            "v0.2.0 is locked against yfinance==1.5.1."
+            f"OMD is locked against yfinance=={EXPECTED_YFINANCE_VERSION}."
         )