ohmydata 0.0.1__tar.gz

ohmydata-0.0.1/.gitignore

@@ -0,0 +1,50 @@
+# Credentials and local configuration
+.env
+.env.*
+!.env.example
+*.pem
+*.key
+*.p12
+*.pfx
+credentials.json
+secrets.json
+token.json
+
+# Provider responses and local data
+data/
+downloads/
+snapshots/
+raw/
+artifacts/
+*.parquet
+*.feather
+*.arrow
+*.duckdb
+*.db
+
+# Python environments and caches
+.venv/
+venv/
+__pycache__/
+*.py[cod]
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+.pyright/
+.coverage
+htmlcov/
+
+# Build and packaging output
+build/
+dist/
+*.egg-info/
+
+# Local tooling and OS files
+.agents/
+.codex/
+.DS_Store
+.idea/
+.vscode/
+*.log
+tmp/
+temp/

ohmydata-0.0.1/CHANGELOG.md

@@ -0,0 +1,20 @@
+# Changelog
+
+## Unreleased — 0.0.1
+
+- Added the offline-composable adjusted ETF bars recipe with explicit factor
+  coverage policy, raw-value preservation, provenance, and validation.
+
+- Added an injected-client Tushare adapter for trade calendar and fund endpoint
+  requests with explicit validation, retry, provenance, pagination, and empty
+  result policies. Tushare/Pandas remain optional.
+
+- Added the offline-only `ohmydata` package scaffold and Python 3.11/3.12 CI.
+- Added synthetic adjusted-ETF bar characterization tests and fixtures.
+- Added the initial consumers' Tushare behavioral inventory and conflict log.
+
+- Added provider-independent request identity, stable errors, classified retry,
+  rate limiting, provenance, and atomic snapshot/replay primitives.
+- Added typed Tushare Phase 2b endpoints for fund dividends, fund portfolios,
+  daily basics, and index weights with bounded selectors, explicit fields,
+  provenance, duplicate/cap validation, and provider-native units.

ohmydata-0.0.1/PKG-INFO

@@ -0,0 +1,160 @@
+Metadata-Version: 2.4
+Name: ohmydata
+Version: 0.0.1
+Summary: Offline-first market-data SDK scaffold
+Author: OMD contributors
+Requires-Python: <3.13,>=3.11
+Provides-Extra: tushare
+Requires-Dist: pandas<3.0,>=2.0; extra == 'tushare'
+Requires-Dist: tushare<2.0,>=1.4; extra == 'tushare'
+Description-Content-Type: text/markdown
+
+# Oh My Data (OMD)
+
+`ohmydata` is a provisional, offline-first market-data SDK. Tushare endpoint
+adapters accept an already initialized official-compatible client; credentials
+are never loaded by this library.
+
+## Supported Python and installation
+
+Python 3.11 and 3.12 are supported (`>=3.11,<3.13`). From a source checkout:
+
+```bash
+uv sync
+uv run python -c "import ohmydata; print(ohmydata.__version__)"
+```
+
+The core has no runtime dependencies. Install `ohmydata[tushare]` for the
+Pandas-backed adapter. Provider tests use fake clients and never call a network.
+
+## Phase 2 Tushare adapter (offline and injected)
+
+Pass an already initialized official-client-compatible object. The adapter
+does not create clients or read credentials; this fake-client example is safe
+to run offline:
+
+```python
+import pandas as pd
+from ohmydata.providers.tushare import EmptyPolicy, FundDailyRequest, TushareClient
+
+
+class FakeClient:
+    def fund_daily(self, **kwargs):
+        return pd.DataFrame(
+            {
+                "ts_code": ["FAKE.ETF"],
+                "trade_date": ["20240102"],
+                "open": [1.0],
+                "high": [1.1],
+                "low": [0.9],
+                "close": [1.05],
+                "pre_close": [1.0],
+                "change": [0.05],
+                "pct_chg": [5.0],
+                "vol": [100],
+                "amount": [250.0],
+            }
+        )
+
+
+request = FundDailyRequest(
+    empty_policy=EmptyPolicy.ERROR, ts_code="FAKE.ETF", start_date="20240101", end_date="20240102"
+)
+result = TushareClient(FakeClient()).fetch_fund_daily(request)
+```
+
+Values and nulls retain Tushare's native semantics: fund daily OHLC and
+`change`/`pct_chg` are provider values, `vol` is in hands, and `amount` is in
+thousand yuan. Empty responses must be selected explicitly with
+`EmptyPolicy.ALLOW` or `EmptyPolicy.ERROR`.
+`fund_share.fd_share` remains provider-native in ten-thousand shares (万份);
+`fund_adj` and `fund_nav` values are likewise preserved without adjustment or
+imputation.
+
+### Adjusted ETF bars recipe
+
+`fetch_adjusted_etf_bars` composes `fund_daily` with provider-native
+`fund_adj` factors. Choose `AdjustmentCoveragePolicy.STRICT` (the default) or
+`PRESERVE_MISSING_FACTOR`; raw OHLC and `adj_factor` remain available beside
+the explicitly derived adjusted OHLC columns. The recipe is offline-testable
+when supplied an injected `TushareClient` and does not claim point-in-time
+availability.
+
+```python
+from ohmydata.providers.tushare import (
+    AdjustmentCoveragePolicy,
+    AdjustedEtfBarsRequest,
+    EmptyPolicy,
+)
+
+request = AdjustedEtfBarsRequest(
+    "FAKE.ETF",
+    EmptyPolicy.ERROR,
+    AdjustmentCoveragePolicy.STRICT,
+    start_date="20240101",
+    end_date="20240131",
+)
+```
+
+## Local checks
+
+```bash
+uv lock
+uv run pytest
+uv run ruff check .
+uv run ruff format --check .
+uv run pyright
+uv build
+git diff --check
+```
+
+Behavioral evidence for the initial consumers is in
+[`docs/behavioral-inventory.md`](docs/behavioral-inventory.md). The adjusted
+ETF characterization is test-only and uses synthetic JSON fixtures.
+
+Phase 1 core is offline and explicit:
+
+```python
+from datetime import UTC, datetime
+from pathlib import Path
+from ohmydata.core import RequestSpec, RetryPolicy, RateLimitPolicy, RateLimiter, execute_with_retry
+from ohmydata.core import SnapshotMode, SnapshotStore
+
+try:
+    RequestSpec("demo", "bars", {"api_token": "never-serialize"})
+except ValueError:
+    pass
+limiter = RateLimiter(RateLimitPolicy(0.1))
+limiter.acquire()
+result = execute_with_retry(lambda: "ok", RetryPolicy(max_attempts=1))
+store = SnapshotStore(Path("snapshots"))
+store.write(
+    RequestSpec("demo", "bars", {}), b"[]", datetime.now(UTC), "json-v1", SnapshotMode.APPEND
+)
+store.write(
+    RequestSpec("demo", "bars", {}), b"[]", datetime.now(UTC), "json-v1", SnapshotMode.FROZEN
+)
+```
+
+`RetryPolicy(max_attempts=3)` counts the first call. APPEND preserves distinct
+observations; FROZEN permits one response identity. Limiter state is per instance.
+
+## Phase 1 core (offline)
+
+`ohmydata.core` provides canonical request identities, classified retry with
+total-attempt semantics, explicit instance-scoped rate limiters, dataframe-free
+provenance, and immutable APPEND/FROZEN snapshots. Request parameters reject
+secret-bearing keys before serialization. Snapshot callers provide exact bytes;
+the core never contacts providers or loads credentials.
+
+## Phase 2b Tushare endpoints
+
+The Tushare adapter exposes typed requests for fund dividends, fund portfolios,
+daily basics, and index weights through injected clients. Requests always send
+an explicit ordered field list and preserve provider-native values and missing
+data. `fund_portfolio` requires a bounded report selector (`ann_date`, exact
+`period`, or a same-year `start_date`/`end_date` range); unbounded holdings are
+rejected. Native units remain unchanged: dividend cash is yuan per share,
+portfolio market value is yuan and amount is shares, daily-basic share and
+market-value fields use Tushare's ten-thousand units, and index weights remain
+provider percentages.

ohmydata-0.0.1/README.md

@@ -0,0 +1,149 @@
+# Oh My Data (OMD)
+
+`ohmydata` is a provisional, offline-first market-data SDK. Tushare endpoint
+adapters accept an already initialized official-compatible client; credentials
+are never loaded by this library.
+
+## Supported Python and installation
+
+Python 3.11 and 3.12 are supported (`>=3.11,<3.13`). From a source checkout:
+
+```bash
+uv sync
+uv run python -c "import ohmydata; print(ohmydata.__version__)"
+```
+
+The core has no runtime dependencies. Install `ohmydata[tushare]` for the
+Pandas-backed adapter. Provider tests use fake clients and never call a network.
+
+## Phase 2 Tushare adapter (offline and injected)
+
+Pass an already initialized official-client-compatible object. The adapter
+does not create clients or read credentials; this fake-client example is safe
+to run offline:
+
+```python
+import pandas as pd
+from ohmydata.providers.tushare import EmptyPolicy, FundDailyRequest, TushareClient
+
+
+class FakeClient:
+    def fund_daily(self, **kwargs):
+        return pd.DataFrame(
+            {
+                "ts_code": ["FAKE.ETF"],
+                "trade_date": ["20240102"],
+                "open": [1.0],
+                "high": [1.1],
+                "low": [0.9],
+                "close": [1.05],
+                "pre_close": [1.0],
+                "change": [0.05],
+                "pct_chg": [5.0],
+                "vol": [100],
+                "amount": [250.0],
+            }
+        )
+
+
+request = FundDailyRequest(
+    empty_policy=EmptyPolicy.ERROR, ts_code="FAKE.ETF", start_date="20240101", end_date="20240102"
+)
+result = TushareClient(FakeClient()).fetch_fund_daily(request)
+```
+
+Values and nulls retain Tushare's native semantics: fund daily OHLC and
+`change`/`pct_chg` are provider values, `vol` is in hands, and `amount` is in
+thousand yuan. Empty responses must be selected explicitly with
+`EmptyPolicy.ALLOW` or `EmptyPolicy.ERROR`.
+`fund_share.fd_share` remains provider-native in ten-thousand shares (万份);
+`fund_adj` and `fund_nav` values are likewise preserved without adjustment or
+imputation.
+
+### Adjusted ETF bars recipe
+
+`fetch_adjusted_etf_bars` composes `fund_daily` with provider-native
+`fund_adj` factors. Choose `AdjustmentCoveragePolicy.STRICT` (the default) or
+`PRESERVE_MISSING_FACTOR`; raw OHLC and `adj_factor` remain available beside
+the explicitly derived adjusted OHLC columns. The recipe is offline-testable
+when supplied an injected `TushareClient` and does not claim point-in-time
+availability.
+
+```python
+from ohmydata.providers.tushare import (
+    AdjustmentCoveragePolicy,
+    AdjustedEtfBarsRequest,
+    EmptyPolicy,
+)
+
+request = AdjustedEtfBarsRequest(
+    "FAKE.ETF",
+    EmptyPolicy.ERROR,
+    AdjustmentCoveragePolicy.STRICT,
+    start_date="20240101",
+    end_date="20240131",
+)
+```
+
+## Local checks
+
+```bash
+uv lock
+uv run pytest
+uv run ruff check .
+uv run ruff format --check .
+uv run pyright
+uv build
+git diff --check
+```
+
+Behavioral evidence for the initial consumers is in
+[`docs/behavioral-inventory.md`](docs/behavioral-inventory.md). The adjusted
+ETF characterization is test-only and uses synthetic JSON fixtures.
+
+Phase 1 core is offline and explicit:
+
+```python
+from datetime import UTC, datetime
+from pathlib import Path
+from ohmydata.core import RequestSpec, RetryPolicy, RateLimitPolicy, RateLimiter, execute_with_retry
+from ohmydata.core import SnapshotMode, SnapshotStore
+
+try:
+    RequestSpec("demo", "bars", {"api_token": "never-serialize"})
+except ValueError:
+    pass
+limiter = RateLimiter(RateLimitPolicy(0.1))
+limiter.acquire()
+result = execute_with_retry(lambda: "ok", RetryPolicy(max_attempts=1))
+store = SnapshotStore(Path("snapshots"))
+store.write(
+    RequestSpec("demo", "bars", {}), b"[]", datetime.now(UTC), "json-v1", SnapshotMode.APPEND
+)
+store.write(
+    RequestSpec("demo", "bars", {}), b"[]", datetime.now(UTC), "json-v1", SnapshotMode.FROZEN
+)
+```
+
+`RetryPolicy(max_attempts=3)` counts the first call. APPEND preserves distinct
+observations; FROZEN permits one response identity. Limiter state is per instance.
+
+## Phase 1 core (offline)
+
+`ohmydata.core` provides canonical request identities, classified retry with
+total-attempt semantics, explicit instance-scoped rate limiters, dataframe-free
+provenance, and immutable APPEND/FROZEN snapshots. Request parameters reject
+secret-bearing keys before serialization. Snapshot callers provide exact bytes;
+the core never contacts providers or loads credentials.
+
+## Phase 2b Tushare endpoints
+
+The Tushare adapter exposes typed requests for fund dividends, fund portfolios,
+daily basics, and index weights through injected clients. Requests always send
+an explicit ordered field list and preserve provider-native values and missing
+data. `fund_portfolio` requires a bounded report selector (`ann_date`, exact
+`period`, or a same-year `start_date`/`end_date` range); unbounded holdings are
+rejected. Native units remain unchanged: dividend cash is yuan per share,
+portfolio market value is yuan and amount is shares, daily-basic share and
+market-value fields use Tushare's ten-thousand units, and index weights remain
+provider percentages.

ohmydata-0.0.1/pyproject.toml

@@ -0,0 +1,49 @@
+[build-system]
+requires = ["hatchling>=1.25"]
+build-backend = "hatchling.build"
+
+[project]
+name = "ohmydata"
+version = "0.0.1"
+description = "Offline-first market-data SDK scaffold"
+readme = "README.md"
+requires-python = ">=3.11,<3.13"
+authors = [{ name = "OMD contributors" }]
+dependencies = []
+
+[project.optional-dependencies]
+tushare = ["pandas>=2.0,<3.0", "tushare>=1.4,<2.0"]
+
+[dependency-groups]
+dev = [
+  "pandas>=2.0",
+  "pandas-stubs>=2.2",
+  "pyright>=1.1.390",
+  "pytest>=8.3",
+  "ruff>=0.8",
+]
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/ohmydata"]
+
+[tool.hatch.build.targets.sdist]
+include = [
+  "src/ohmydata",
+  "README.md",
+  "CHANGELOG.md",
+  "pyproject.toml",
+  "uv.lock",
+]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = ["--import-mode=importlib"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py311"
+
+[tool.pyright]
+include = ["src", "tests"]
+pythonVersion = "3.11"
+typeCheckingMode = "strict"

ohmydata-0.0.1/src/ohmydata/__init__.py

@@ -0,0 +1,5 @@
+"""Provisional offline-only Oh My Data package scaffold."""
+
+__version__ = "0.0.0"
+
+__all__ = ["__version__"]

ohmydata-0.0.1/src/ohmydata/core/__init__.py

@@ -0,0 +1,52 @@
+from .errors import (
+    AuthenticationError,
+    CoverageError,
+    EmptyResponseError,
+    OhMyDataError,
+    PaginationError,
+    PermanentProviderError,
+    PermissionDeniedError,
+    ProviderError,
+    RateLimitError,
+    RetryExhaustedError,
+    SchemaMismatchError,
+    SnapshotConflictError,
+    SnapshotIntegrityError,
+    TransientProviderError,
+)
+from .policy import AttemptRecord, RetryPolicy, RetryResult, execute_with_retry
+from .provenance import EmptyDisposition, FetchProvenance
+from .rate_limit import RateLimitDecision, RateLimiter, RateLimitPolicy
+from .snapshot import SnapshotMode, SnapshotRef, SnapshotReplay, SnapshotStore
+from .specs import RequestSpec
+
+__all__ = [
+    "AttemptRecord",
+    "AuthenticationError",
+    "CoverageError",
+    "EmptyDisposition",
+    "EmptyResponseError",
+    "FetchProvenance",
+    "OhMyDataError",
+    "PaginationError",
+    "PermanentProviderError",
+    "PermissionDeniedError",
+    "ProviderError",
+    "RateLimitDecision",
+    "RateLimitError",
+    "RateLimitPolicy",
+    "RateLimiter",
+    "RequestSpec",
+    "RetryExhaustedError",
+    "RetryPolicy",
+    "RetryResult",
+    "SchemaMismatchError",
+    "SnapshotConflictError",
+    "SnapshotIntegrityError",
+    "SnapshotMode",
+    "SnapshotRef",
+    "SnapshotReplay",
+    "SnapshotStore",
+    "TransientProviderError",
+    "execute_with_retry",
+]

ohmydata-0.0.1/src/ohmydata/core/errors.py

@@ -0,0 +1,66 @@
+from dataclasses import dataclass
+
+
+class OhMyDataError(Exception):
+    pass
+
+
+class ProviderError(OhMyDataError):
+    pass
+
+
+class PermanentProviderError(ProviderError):
+    pass
+
+
+class AuthenticationError(PermanentProviderError):
+    pass
+
+
+class PermissionDeniedError(PermanentProviderError):
+    pass
+
+
+class EmptyResponseError(PermanentProviderError):
+    pass
+
+
+class SchemaMismatchError(PermanentProviderError):
+    pass
+
+
+class PaginationError(PermanentProviderError):
+    pass
+
+
+class TransientProviderError(ProviderError):
+    pass
+
+
+class RateLimitError(TransientProviderError):
+    pass
+
+
+class SnapshotIntegrityError(OhMyDataError):
+    pass
+
+
+class SnapshotConflictError(SnapshotIntegrityError):
+    pass
+
+
+class CoverageError(OhMyDataError):
+    pass
+
+
+@dataclass(frozen=True)
+class AttemptRecord:
+    attempt: int
+    exception_type: str | None
+    retry_delay_seconds: float | None
+
+
+class RetryExhaustedError(TransientProviderError):
+    def __init__(self, attempts: tuple[AttemptRecord, ...]):
+        self.attempts = attempts
+        super().__init__(f"retry exhausted after {len(attempts)} attempts")

ohmydata-0.0.1/src/ohmydata/core/policy.py

@@ -0,0 +1,101 @@
+import math
+import random
+import time
+from collections.abc import Callable
+from dataclasses import dataclass
+from typing import Generic, TypeVar
+
+from .errors import AttemptRecord, RetryExhaustedError, TransientProviderError
+
+__all__ = ["AttemptRecord", "RetryPolicy", "RetryResult", "execute_with_retry"]
+
+T = TypeVar("T")
+
+
+def _finite_number(value: object) -> bool:
+    return (
+        not isinstance(value, bool)
+        and isinstance(value, (int, float))
+        and math.isfinite(float(value))
+    )
+
+
+def _valid_attempts(value: object) -> bool:
+    return type(value) is int and value >= 1
+
+
+@dataclass(frozen=True)
+class RetryPolicy:
+    max_attempts: int = 3
+    base_delay_seconds: float = 1.0
+    backoff_multiplier: float = 2.0
+    max_delay_seconds: float = 60.0
+    jitter_ratio: float = 0.0
+
+    def __post_init__(self):
+        if (
+            not _valid_attempts(self.max_attempts)
+            or any(
+                not _finite_number(x)
+                for x in (
+                    self.base_delay_seconds,
+                    self.backoff_multiplier,
+                    self.max_delay_seconds,
+                    self.jitter_ratio,
+                )
+            )
+            or self.max_attempts < 1
+            or self.base_delay_seconds < 0
+            or self.backoff_multiplier < 1
+            or self.max_delay_seconds < 0
+            or not 0 <= self.jitter_ratio <= 1
+        ):
+            raise ValueError("invalid retry policy")
+
+
+@dataclass(frozen=True)
+class RetryResult(Generic[T]):
+    value: T
+    attempts: tuple[AttemptRecord, ...]
+
+
+def execute_with_retry(
+    fn: Callable[[], T],
+    policy: RetryPolicy | None = None,
+    *,
+    sleep: Callable[[float], None] = time.sleep,
+    random_value: Callable[[], float] = random.random,
+    classifier: Callable[[Exception], bool] = lambda e: isinstance(e, TransientProviderError),
+) -> RetryResult[T]:
+    policy = policy or RetryPolicy()
+    records: list[AttemptRecord] = []
+    for i in range(policy.max_attempts):
+        try:
+            value = fn()
+            records.append(AttemptRecord(i + 1, None, None))
+            return RetryResult(value, tuple(records))
+        except Exception as exc:
+            if not isinstance(exc, TransientProviderError) or not classifier(exc):
+                raise
+            delay = None
+            if i + 1 < policy.max_attempts:
+                r = random_value()
+                if not 0 <= r <= 1:
+                    raise ValueError("random value out of range")
+                bounded = min(
+                    policy.max_delay_seconds,
+                    policy.base_delay_seconds * policy.backoff_multiplier**i,
+                )
+                delay = bounded * (1 + policy.jitter_ratio * (2 * r - 1))
+                sleep(delay)
+            records.append(AttemptRecord(i + 1, type(exc).__name__, delay))
+            if i + 1 == policy.max_attempts:
+                err = RetryExhaustedError(
+                    tuple(
+                        AttemptRecord(x.attempt, x.exception_type, x.retry_delay_seconds)
+                        for x in records
+                    )
+                )
+                err.__cause__ = exc
+                raise err from exc
+    raise AssertionError