tradernick-data-provider 0.2.0__tar.gz

tradernick_data_provider-0.2.0/.gitignore

@@ -0,0 +1,14 @@
+.env
+.venv/
+__pycache__/
+*.pyc
+.pytest_cache/
+
+node_modules/
+.svelte-kit/
+build/
+.DS_Store
+
+data/
+
+.claude/

tradernick_data_provider-0.2.0/PKG-INFO

@@ -0,0 +1,114 @@
+Metadata-Version: 2.4
+Name: tradernick-data-provider
+Version: 0.2.0
+Summary: Python client for the TraderNick data_provider service. Drop-in compatible with horatio-data-provider (same DataProviderClient API; only the import path changes).
+Author: TraderNick
+License: MIT
+Keywords: binance,clickhouse,data-provider,defi,evm,hyperliquid
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.10
+Requires-Dist: httpx>=0.25
+Requires-Dist: pandas>=2.0
+Requires-Dist: polars>=1.0
+Requires-Dist: pyarrow>=14.0
+Requires-Dist: pytz>=2023.3
+Provides-Extra: dev
+Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: respx>=0.21; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# tradernick-data-provider
+
+Python client for the TraderNick `data_provider` service. Drop-in
+compatible with [`horatio-data-provider`](https://pypi.org/project/horatio-data-provider/):
+same `DataProviderClient` class, same namespaces (`evm`, `tron`, `btc`,
+`binance`, `hyperliquid`, `wallets`, `cache`, `jobs`), same chainable
+builder methods, same `as_pandas()` / `as_polars()` / `as_parquet()`
+terminators. The only visible difference is the import path.
+
+```python
+# Before
+from horatio_data_provider import DataProviderClient
+
+# After
+from tradernick_data_provider import DataProviderClient
+```
+
+The server URL passed to the constructor is the only thing you need to
+change at the call site.
+
+## Install
+
+```sh
+pip install tradernick-data-provider
+```
+
+## Usage
+
+```python
+import asyncio
+from tradernick_data_provider import DataProviderClient
+
+async def main():
+    async with DataProviderClient("http://localhost:10005") as client:
+        df = await client.binance.ohlcv("BTC", "1h") \
+            .time_range("2026-06-01T00:00:00Z", "2026-06-08T00:00:00Z") \
+            .as_polars()
+        print(df)
+
+asyncio.run(main())
+```
+
+All Horatio query builders work unchanged. The server delegates to
+ClickHouse instead of DeFiStream, so reads stay sub-second on tables
+where Horatio has to pay a fresh upstream fetch.
+
+## Status
+
+**0.2.0 — Phase 1+2 read parity.** Validated against horatio-data-provider
+on 2026-06-07:
+- Column-shape parity: 100% across binance / aave / lido / uniswap /
+  hyperliquid / transfers.
+- Row-count + value parity: exact match on stable tables
+  (binance.ohlcv 1m, raw_trades, lido.deposit, aave.borrows/repays,
+  evm.native with `min_amount`); within ±2 rows on AAVE event tables
+  where DeFiStream's sweep loop occasionally re-fetches windows with
+  slightly different cuts.
+
+What works:
+- `binance.{ohlcv, raw_trades, book_depth, open_interest, funding_rate,
+  long_short_ratios}` (with `with_id`, `add_symbol`)
+- `evm.aave.{deposit, withdraw, borrow, repay, flashloan, liquidation}`
+  with `involving`, `exclude_involving`, `eth_market_type`
+- `evm.uniswap.{swap, deposit, withdraw, collect}` — V3
+- `evm.lido.{deposit, withdrawal_request, withdrawal_claimed,
+  l2_deposit, l2_withdrawal_request}`
+- `evm.erc20.transfers`, `evm.native_transfers` with full filter set
+  (sender / receiver / involving / exclude_* / min_amount / max_amount)
+- `tron.{native, trc20}.transfers`, `btc.native.transfers`
+- `hyperliquid.{ohlcv, trades, fills, funding, transfers, vaults,
+  trade_history, position_history}`
+- `client.{wallets.list, wallets.get, wallets.upsert, wallets.delete}`
+- `client.{load_parquet, scan_parquet, list_snapshots, delete_snapshot,
+  as_parquet}` — server-side parquet snapshots with `local_*` filters
+- `client.jobs.{list, get, cancel}` — proxies to the ingestion job queue
+
+Not yet exposed:
+- `evm.{stader, threshold}` — return empty schemas (TN doesn't ingest
+  these networks yet)
+- `hyperliquid.{sends, spot_transfers}` — return empty schemas
+- `client.scan_parquet` only honors address-based `local_*` filters;
+  label/category/entity variants are wired but no-op without
+  wallet_labels co-mounted on the snapshot.
+
+## Compatibility
+
+Drop-in compatible with horatio-data-provider 4.x.

tradernick_data_provider-0.2.0/README.md

@@ -0,0 +1,87 @@
+# tradernick-data-provider
+
+Python client for the TraderNick `data_provider` service. Drop-in
+compatible with [`horatio-data-provider`](https://pypi.org/project/horatio-data-provider/):
+same `DataProviderClient` class, same namespaces (`evm`, `tron`, `btc`,
+`binance`, `hyperliquid`, `wallets`, `cache`, `jobs`), same chainable
+builder methods, same `as_pandas()` / `as_polars()` / `as_parquet()`
+terminators. The only visible difference is the import path.
+
+```python
+# Before
+from horatio_data_provider import DataProviderClient
+
+# After
+from tradernick_data_provider import DataProviderClient
+```
+
+The server URL passed to the constructor is the only thing you need to
+change at the call site.
+
+## Install
+
+```sh
+pip install tradernick-data-provider
+```
+
+## Usage
+
+```python
+import asyncio
+from tradernick_data_provider import DataProviderClient
+
+async def main():
+    async with DataProviderClient("http://localhost:10005") as client:
+        df = await client.binance.ohlcv("BTC", "1h") \
+            .time_range("2026-06-01T00:00:00Z", "2026-06-08T00:00:00Z") \
+            .as_polars()
+        print(df)
+
+asyncio.run(main())
+```
+
+All Horatio query builders work unchanged. The server delegates to
+ClickHouse instead of DeFiStream, so reads stay sub-second on tables
+where Horatio has to pay a fresh upstream fetch.
+
+## Status
+
+**0.2.0 — Phase 1+2 read parity.** Validated against horatio-data-provider
+on 2026-06-07:
+- Column-shape parity: 100% across binance / aave / lido / uniswap /
+  hyperliquid / transfers.
+- Row-count + value parity: exact match on stable tables
+  (binance.ohlcv 1m, raw_trades, lido.deposit, aave.borrows/repays,
+  evm.native with `min_amount`); within ±2 rows on AAVE event tables
+  where DeFiStream's sweep loop occasionally re-fetches windows with
+  slightly different cuts.
+
+What works:
+- `binance.{ohlcv, raw_trades, book_depth, open_interest, funding_rate,
+  long_short_ratios}` (with `with_id`, `add_symbol`)
+- `evm.aave.{deposit, withdraw, borrow, repay, flashloan, liquidation}`
+  with `involving`, `exclude_involving`, `eth_market_type`
+- `evm.uniswap.{swap, deposit, withdraw, collect}` — V3
+- `evm.lido.{deposit, withdrawal_request, withdrawal_claimed,
+  l2_deposit, l2_withdrawal_request}`
+- `evm.erc20.transfers`, `evm.native_transfers` with full filter set
+  (sender / receiver / involving / exclude_* / min_amount / max_amount)
+- `tron.{native, trc20}.transfers`, `btc.native.transfers`
+- `hyperliquid.{ohlcv, trades, fills, funding, transfers, vaults,
+  trade_history, position_history}`
+- `client.{wallets.list, wallets.get, wallets.upsert, wallets.delete}`
+- `client.{load_parquet, scan_parquet, list_snapshots, delete_snapshot,
+  as_parquet}` — server-side parquet snapshots with `local_*` filters
+- `client.jobs.{list, get, cancel}` — proxies to the ingestion job queue
+
+Not yet exposed:
+- `evm.{stader, threshold}` — return empty schemas (TN doesn't ingest
+  these networks yet)
+- `hyperliquid.{sends, spot_transfers}` — return empty schemas
+- `client.scan_parquet` only honors address-based `local_*` filters;
+  label/category/entity variants are wired but no-op without
+  wallet_labels co-mounted on the snapshot.
+
+## Compatibility
+
+Drop-in compatible with horatio-data-provider 4.x.

tradernick_data_provider-0.2.0/pyproject.toml

@@ -0,0 +1,42 @@
+[project]
+name = "tradernick-data-provider"
+version = "0.2.0"
+description = "Python client for the TraderNick data_provider service. Drop-in compatible with horatio-data-provider (same DataProviderClient API; only the import path changes)."
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+authors = [{ name = "TraderNick" }]
+keywords = ["clickhouse", "binance", "hyperliquid", "evm", "defi", "data-provider"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Software Development :: Libraries :: Python Modules",
+]
+dependencies = [
+    "httpx>=0.25",
+    "pyarrow>=14.0",
+    "pandas>=2.0",
+    "polars>=1.0",
+    "pytz>=2023.3",
+]
+
+[project.optional-dependencies]
+dev = ["pytest>=8.0", "pytest-asyncio>=0.23", "respx>=0.21"]
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+
+[tool.hatch.build.targets.wheel]
+packages = ["tradernick_data_provider"]
+
+[tool.hatch.build.targets.sdist]
+include = ["tradernick_data_provider", "README.md", "pyproject.toml"]
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"

tradernick_data_provider-0.2.0/tradernick_data_provider/__init__.py

@@ -0,0 +1,10 @@
+from ._client import DataProviderClient
+from .exceptions import DataProviderError, DataProviderHTTPError
+from .wallets import WalletsNamespace
+
+__all__ = [
+    "DataProviderClient",
+    "DataProviderError",
+    "DataProviderHTTPError",
+    "WalletsNamespace",
+]

tradernick_data_provider-0.2.0/tradernick_data_provider/_client.py

@@ -0,0 +1,148 @@
+import io
+from datetime import datetime
+from typing import Literal, Optional, Union
+
+import httpx
+import polars as pl
+
+from ._http import load_parquet_bytes, list_snapshots, delete_snapshot
+from ._query import _to_timestamp
+
+
+def _to_datetime(date: datetime | str | int) -> datetime:
+    """Convert any date input to a timezone-aware datetime."""
+    ts = _to_timestamp(date)  # returns 'YYYY-MM-DDTHH:MM:SSZ'
+    return datetime.fromisoformat(ts.replace("Z", "+00:00"))
+
+
+def _cast_time_ms_utc(df: pl.DataFrame) -> pl.DataFrame:
+    """Normalize the ``time`` column to ``Datetime('ms', 'UTC')``.
+
+    DuckDB-written snapshots come back as μs+UTC; cache reads come back
+    as ms+UTC. The cast keeps everything joinable on the polars side.
+    """
+    if 'time' in df.columns:
+        dt = df.schema['time']
+        if isinstance(dt, pl.Datetime) and (
+            dt.time_unit != 'ms' or dt.time_zone != 'UTC'
+        ):
+            df = df.with_columns(pl.col('time').cast(pl.Datetime('ms', 'UTC')))
+    return df
+
+from .binance import BinanceNamespace, HyperliquidNamespace
+from .btc import BtcNamespace
+from .evm import EvmNamespace
+from .jobs import JobsNamespace
+from .protocols import CacheNamespace
+from .tron import TronNamespace
+from .wallets import WalletsNamespace
+
+
+class DataProviderClient:
+    evm: EvmNamespace
+    tron: TronNamespace
+    btc: BtcNamespace
+    binance: BinanceNamespace
+    hyperliquid: HyperliquidNamespace
+    wallets: WalletsNamespace
+    cache: CacheNamespace
+    jobs: JobsNamespace
+
+    def __init__(self, url: str):
+        self._url = url.rstrip("/")
+        self._session = httpx.AsyncClient(timeout=86400)
+        self.evm = EvmNamespace(self._session, self._url)
+        self.tron = TronNamespace(self._session, self._url)
+        self.btc = BtcNamespace(self._session, self._url)
+        self.binance = BinanceNamespace(self._session, self._url)
+        self.hyperliquid = HyperliquidNamespace(self._session, self._url)
+        self.wallets = WalletsNamespace(self._session, self._url)
+        self.cache = CacheNamespace(self._session, self._url)
+        self.jobs = JobsNamespace(self._session, self._url)
+
+    async def health(self) -> bool:
+        response = await self._session.get(self._url + "/health")
+        response.raise_for_status()
+        return True
+
+    async def load_parquet(
+        self,
+        key: str,
+        since: Optional[Union[datetime, str, int]] = None,
+        until: Optional[Union[datetime, str, int]] = None,
+    ) -> pl.DataFrame:
+        """Load a saved snapshot as a polars DataFrame.
+
+        ``time`` is normalized to ``Datetime('ms', UTC)`` so joins with
+        transfer-read DataFrames (which the cache layer also returns at
+        ms+UTC) don't trip the polars 'datatypes of join keys don't
+        match' check. Snapshots saved via DuckDB COPY are stored at
+        μs+UTC internally; we cast on read.
+
+        For pandas, call ``(await client.load_parquet(key)).to_pandas()``.
+        """
+        raw = await load_parquet_bytes(self._session, self._url, key)
+        df = pl.read_parquet(io.BytesIO(raw))
+        df = _cast_time_ms_utc(df)
+        if since is not None or until is not None:
+            time_col = "timestamp" if "timestamp" in df.columns else "time"
+            if time_col in df.columns:
+                if since is not None:
+                    df = df.filter(pl.col(time_col) >= _to_datetime(since))
+                if until is not None:
+                    df = df.filter(pl.col(time_col) <= _to_datetime(until))
+        return df
+
+    def scan_parquet(self, key: str, *,
+                     since: Optional[Union[datetime, str, int]] = None,
+                     until: Optional[Union[datetime, str, int]] = None,
+                     engine: Literal['polars', 'duckdb'] = 'duckdb',
+                     normalize_addresses: Optional[bool] = None):
+        """Lazy-scan a saved snapshot with ``local_*`` filters applied
+        server-side. Returns a ``ScanParquetQuery`` builder. Chain
+        ``local_*`` filter methods then call a terminal ``as_polars()`` /
+        ``as_pandas()`` / ``as_parquet(new_key)``.
+
+        ``engine``:
+          - ``'duckdb'`` (default): server mounts the snapshot + wallets
+            parquets as DuckDB views and runs the filter as SQL.
+            Streams via ``COPY ... TO PARQUET``. Best optimizer for
+            large ``IN`` filters; ~3-50× faster than polars on big
+            wallet-set queries.
+          - ``'polars'``: server uses ``pl.scan_parquet`` and a polars
+            lazy filter pipeline. Streams via ``sink_parquet``.
+
+        ``normalize_addresses``: default ``None`` (auto). Set to ``False``
+        only when you know the snapshot is canonical and the file lacks
+        the metadata flag — auto-detect already handles canonical files.
+
+        Example::
+
+            df = await client.scan_parquet('huge_snapshot') \\
+                .local_exclude_sender_categories(['Hot-Wallet','Cold-Wallet']) \\
+                .local_involving_entities(['Binance']) \\
+                .as_polars()
+        """
+        from .snapshots import ScanParquetQuery
+        return ScanParquetQuery(
+            self._session, self._url, key,
+            since=since, until=until,
+            engine=engine, normalize_addresses=normalize_addresses,
+        )
+
+    async def list_snapshots(self) -> list[str]:
+        """List all saved snapshot keys."""
+        return await list_snapshots(self._session, self._url)
+
+    async def delete_snapshot(self, key: str) -> None:
+        """Delete a saved snapshot."""
+        await delete_snapshot(self._session, self._url, key)
+
+    async def close(self) -> None:
+        await self._session.aclose()
+
+    async def __aenter__(self) -> "DataProviderClient":
+        return self
+
+    async def __aexit__(self, *_) -> None:
+        await self.close()

tradernick_data_provider-0.2.0/tradernick_data_provider/_http.py

@@ -0,0 +1,55 @@
+import io
+
+import httpx
+import pyarrow as pa
+import pyarrow.parquet as pq
+
+from .exceptions import DataProviderHTTPError
+
+
+async def fetch_table(session: httpx.AsyncClient, url: str, body: dict) -> pa.Table | None:
+    response = await session.post(url, json=body)
+    content_type = response.headers.get("content-type", "")
+    if "application/json" in content_type:
+        data = response.json()
+        if response.is_success and data.get("saved"):
+            return None
+        raise DataProviderHTTPError(response.status_code, data.get("error", str(data)))
+    response.raise_for_status()
+    return pq.read_table(io.BytesIO(response.content))
+
+
+async def save_parquet(session: httpx.AsyncClient, url: str, body: dict, key: str) -> None:
+    """Send a query with save_key to save the result as a named snapshot."""
+    resp = await session.post(url, json={**body, "save_key": key})
+    resp.raise_for_status()
+
+
+async def load_parquet_bytes(session: httpx.AsyncClient, base_url: str, key: str) -> bytes:
+    """Load a previously saved snapshot as raw parquet bytes."""
+    resp = await session.post(f"{base_url}/snapshots/load", json={"key": key})
+    content_type = resp.headers.get("content-type", "")
+    if "application/json" in content_type:
+        data = resp.json()
+        raise DataProviderHTTPError(resp.status_code, data.get("error", str(data)))
+    resp.raise_for_status()
+    return resp.content
+
+
+async def load_parquet(session: httpx.AsyncClient, base_url: str, key: str) -> pa.Table:
+    """Load a previously saved snapshot as a pyarrow Table."""
+    raw = await load_parquet_bytes(session, base_url, key)
+    return pq.read_table(io.BytesIO(raw))
+
+
+async def delete_snapshot(session: httpx.AsyncClient, base_url: str, key: str) -> None:
+    """Delete a snapshot by key."""
+    resp = await session.post(f"{base_url}/snapshots/delete", json={"key": key})
+    resp.raise_for_status()
+
+
+async def list_snapshots(session: httpx.AsyncClient, base_url: str) -> list[str]:
+    """List all saved snapshot keys."""
+    resp = await session.get(f"{base_url}/snapshots/list")
+    resp.raise_for_status()
+    return resp.json()["keys"]