convexlake 0.2.0__tar.gz

convexlake-0.2.0/PKG-INFO

@@ -0,0 +1,544 @@
+Metadata-Version: 2.4
+Name: convexlake
+Version: 0.2.0
+Summary: Python SDK for the Convex Lake market data API
+License-Expression: MIT
+Project-URL: Homepage, https://convexlake.com
+Project-URL: Repository, https://github.com/convexlake/convexlake-sdk
+Project-URL: Documentation, https://github.com/convexlake/convexlake-sdk#readme
+Keywords: polymarket,kalshi,deribit,market-data,prediction-markets,options
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Financial and Insurance Industry
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Office/Business :: Financial
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: requests>=2.31
+Requires-Dist: pandas>=2.0
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: responses; extra == "dev"
+
+# convexlake
+
+Python SDK for the [Convex Lake](https://convexlake.com) market data API.
+
+Convex Lake provides historical and live orderbook, trade, candle, volatility surface, and leaderboard data from prediction markets (Polymarket, Kalshi, Limitless, Predict.fun) and derivatives exchanges (Deribit, Hyperliquid).
+
+---
+
+## Installation
+
+```bash
+pip install convexlake
+```
+
+Or from source:
+
+```bash
+git clone https://github.com/convexlake/convexlake-sdk.git
+cd convexlake-sdk
+pip install -e .
+```
+
+**Requirements:** Python 3.10+, `requests>=2.31`, `pandas>=2.0`
+
+---
+
+## Authentication
+
+All requests require an API key passed at construction time. Keys start with `do_`.
+
+```python
+from convexlake import ConvexLake
+
+client = ConvexLake(api_key="do_XXXX...")
+```
+
+The key is sent as the `x-api-key` request header on every call.
+
+---
+
+## Quick start
+
+```python
+from convexlake import ConvexLake
+
+client = ConvexLake(api_key="do_XXXX...")
+
+# 1. Discover what's available
+print(client.info())                              # list exchanges
+print(client.info(exchange="kalshi"))             # list tickers
+print(client.info(exchange="kalshi", ticker="btc"))            # list dates
+print(client.info(exchange="kalshi", ticker="btc", date="2026-07-30"))  # list files
+
+# 2. Download data
+df = client.candles(exchange="kalshi", ticker="btc", date="2026-07-30",
+                    slug="KXBTC-2601-prices.csv")
+
+df = client.trades(exchange="deribit", ticker="BTC", date="2026-08-07",
+                   slug="BTC-28AUG26-80000-C-trades.csv")
+
+msgs = client.orderbook(exchange="polymarket", ticker="btc", timeframe="15m",
+                        date="2026-04-23", slug="btc-updown-15m-1776943800-bbo.gz")
+```
+
+---
+
+## Navigation: `info()`
+
+Use `info()` to browse the data catalog. Call it progressively to drill down to individual files, then pass the `slug` you find to a data method.
+
+```python
+# Top level — list exchanges and top-level folders
+client.info()
+# {"tier": "pro", "allowed_exchanges": ["kalshi", "polymarket", ...],
+#  "folders": ["candles/", "orderbook/", "trades/", "surface/"]}
+
+# List tickers available for an exchange
+client.info(exchange="polymarket")
+
+# List dates that have data for a ticker
+client.info(exchange="polymarket", ticker="btc")
+
+# List individual files for a date
+client.info(exchange="polymarket", ticker="btc", date="2026-07-30")
+# {"files": ["btc-updown-5m-1785343200-bbo.gz", "btc-updown-5m-1785343200-book.gz", ...]}
+```
+
+---
+
+## Methods
+
+### `candles()`
+
+Download OHLCV or price data as a `pd.DataFrame`.
+
+```python
+df = client.candles(
+    exchange="kalshi",
+    ticker="btc",
+    date="2026-07-30",
+    slug="KXBTC-2601-prices.csv",
+)
+```
+
+**Column schemas by exchange:**
+
+| Exchange | Columns |
+|---|---|
+| `kalshi` | `timestamp`, `yes_bid`, `yes_ask`, `price`, `volume` |
+| `polymarket` | `timestamp`, `Yes`, `No` |
+| `predict` | `slug`, `starts_at`, `ends_at`, `market_variant`, `title`, `winning_outcome`, `open`, `close`, `feed_symbol`, `outcomes` |
+| `deribit` | `timestamp_ms`, `open`, `high`, `low`, `close`, `volume`, `cost` |
+| `hyperliquid` | `timestamp_ms`, `open`, `high`, `low`, `close`, `volume`, `cost` |
+
+**Deribit and Hyperliquid** use rolling per-instrument files (not partitioned by date). Pass `timeframe` and `slug` only — `ticker` and `date` are not used:
+
+```python
+df = client.candles(exchange="deribit", timeframe="15m", slug="BTC-PERPETUAL.csv")
+df = client.candles(exchange="deribit", timeframe="1m",  slug="ETH-PERPETUAL.csv")
+```
+
+Available timeframes: `1m`, `5m`, `15m`, `30m`, `1h`
+
+---
+
+### `trades()`
+
+Download trade history as a `pd.DataFrame`.
+
+```python
+df = client.trades(
+    exchange="deribit",
+    ticker="BTC",
+    date="2026-08-07",
+    slug="BTC-28AUG26-80000-C-trades.csv",
+)
+```
+
+**Column schemas by exchange:**
+
+| Exchange | Columns |
+|---|---|
+| `kalshi` | `created_time`, `yes_price`, `no_price`, `count`, `taker_side`, `taker_outcome_side`, `is_block_trade` |
+| `polymarket` | `timestamp`, `price`, `size`, `side`, `outcome`, `transaction_hash` |
+| `deribit` | `timestamp_ms`, `trade_id`, `trade_seq`, `instrument_name`, `direction`, `amount`, `price`, `mark_price`, `index_price`, `iv`, `liquidation` |
+
+---
+
+### `orderbook()`
+
+Download orderbook messages as a `list[dict]`. Each dict is one message or snapshot event.
+
+```python
+msgs = client.orderbook(
+    exchange="polymarket",
+    ticker="btc",
+    timeframe="15m",
+    date="2026-04-23",
+    slug="btc-updown-15m-1776943800-bbo.gz",
+)
+
+# msgs[0] → {"event_type": "best_bid_ask", "best_bid": "0.52", "best_ask": "0.54",
+#             "timestamp": 1745416812000, "direction": "up", ...}
+```
+
+**Message formats by exchange:**
+
+| Exchange | Format | Message types |
+|---|---|---|
+| `kalshi` | Concatenated Python dict literals | `orderbook_snapshot`, `orderbook_delta` |
+| `polymarket` | Newline-delimited Python dict literals | `best_bid_ask`, `book`, `last_trade_price` |
+| `predict` | Newline-delimited Python dict literals | Full book snapshots with `bids`/`asks` arrays |
+| `limitless` | Newline-delimited JSON | Full book snapshots |
+
+Format is auto-detected by default. You can override with `fmt="json"` or `fmt="python"`.
+
+> **Deribit** orderbooks are CSV, not message-based. Use `orderbook_df()` instead — calling `orderbook()` with `exchange="deribit"` raises a helpful error.
+
+---
+
+### `orderbook_df()`
+
+Download orderbook data as a `pd.DataFrame`. Use this for Deribit, which stores orderbook snapshots in CSV format.
+
+```python
+df = client.orderbook_df(
+    exchange="deribit",
+    ticker="BTC",
+    date="2026-08-07",
+    slug="BTC-28AUG26-80000-C-book.csv",
+)
+```
+
+---
+
+### `surface()`
+
+Download a volatility surface snapshot as a `pd.DataFrame`. Currently supported for Deribit.
+
+```python
+df = client.surface(
+    exchange="deribit",
+    date="2026-09-14",
+    slug="BTC-2026-09-14",
+)
+```
+
+`slug` format: `"{CURRENCY}-{YYYY-MM-DD}"` — e.g. `"BTC-2026-09-14"`, `"ETH-2026-09-14"`.
+
+**Columns:** `snapshot_ts`, `instrument_name`, `expiry`, `strike`, `option_type`, `underlying_price`, `mark_price`, `mark_iv`, `bid_price`, `ask_price`, `mid_price`, `volume`, `volume_usd`, `open_interest`, `estimated_delivery_price`
+
+---
+
+### `leaderboard()`
+
+Download a daily leaderboard snapshot as a `pd.DataFrame`.
+
+```python
+df = client.leaderboard(exchange="polymarket", date="2026-09-14")
+df = client.leaderboard(exchange="limitless",  date="2026-09-14")
+```
+
+**Column schemas by exchange:**
+
+| Exchange | Columns |
+|---|---|
+| `polymarket` | `fetched_at`, `time_period`, `order_by`, `category`, `rank`, `proxyWallet`, `userName`, `xUsername`, `verifiedBadge`, `vol`, `pnl` |
+| `limitless` | `fetched_at`, `metric`, `rank`, `account`, `username`, `displayName`, `positionSize_usd`, `unrealizedPnl_usd`, `heldShares`, `toWin_usd`, `mark_price`, `mark_source`, `side`, `outcomeLabel`, `market_id`, `market_title`, `market_category` |
+
+---
+
+### `leaderboard_trades()`
+
+Two-step access to per-trader fill history.
+
+**Step 1** — list available trader addresses:
+
+```python
+result = client.leaderboard_trades(exchange="polymarket")
+# {"addresses": ["0xabc...", "0xdef...", ...]}  (up to 1000 addresses)
+addresses = result["addresses"]
+```
+
+**Step 2** — fetch one trader's fills:
+
+```python
+df = client.leaderboard_trades(exchange="polymarket", address="0xabc...")
+```
+
+**Columns (Polymarket):** `proxyWallet`, `side`, `asset`, `conditionId`, `size`, `price`, `timestamp`, `title`, `slug`, `outcome`, `outcomeIndex`, `name`, `transactionHash`
+
+Supported exchanges: `polymarket`, `limitless`
+
+---
+
+## NautilusTrader integration
+
+The SDK ships a complete conversion pipeline for [NautilusTrader](https://nautilustrader.io)
+backtesting. Data flows through three stages:
+
+```
+orderbook() → list[dict]      raw messages from S3
+to_nautilus() → pd.DataFrame  NautilusTrader schema, int64 fixed-point encoding
+to_orderbook_deltas() → list  typed OrderBookDelta Cython objects for BacktestEngine
+```
+
+### All-in-one: `download_orderbook_deltas()`
+
+The simplest path — one call streams the file, converts every message, and returns
+`list[OrderBookDelta]` ready for `BacktestEngine.add_data()`:
+
+```python
+from convexlake import ConvexLake
+
+client = ConvexLake(api_key="do_XXXX...")
+
+deltas = client.download_orderbook_deltas(
+    exchange="binance",
+    instrument_id="BTCPERP.BINANCE",
+    slug="BTCPERP-book",
+    ticker="BTCPERP",
+    date="2026-11-04",
+    price_precision=1,
+    size_precision=3,
+)
+
+engine.add_data(deltas)
+engine.run(start=START, end=END)
+```
+
+Internally this runs `orderbook()` → `to_nautilus()` → `to_orderbook_deltas()`.
+
+---
+
+### Efficient path: filter at the DataFrame stage
+
+For large files (Binance full-day L2 book can produce 30 M+ delta objects), converting
+everything before filtering is expensive. Stay in DataFrame land as long as possible —
+pandas vectorized operations on int64 columns take milliseconds regardless of row count:
+
+```python
+from convexlake import ConvexLake
+from datetime import datetime, timezone
+
+WIN_START_NS = int(datetime(2026, 11, 4, 10, 0, tzinfo=timezone.utc).timestamp() * 1e9)
+WIN_END_NS   = int(datetime(2026, 11, 4, 14, 0, tzinfo=timezone.utc).timestamp() * 1e9)
+
+# Step 1 — download as DataFrame (raw messages → int64 schema)
+df = client.download_nautilus(
+    exchange="binance",
+    instrument_id="BTCPERP.BINANCE",
+    slug="BTCPERP-book",
+    ticker="BTCPERP",
+    date="2026-11-04",
+    price_precision=1,
+    size_precision=3,
+)
+
+# Step 2 — filter at the DataFrame stage (vectorized, instant)
+df = df[df["ts_event"].between(WIN_START_NS, WIN_END_NS)]
+
+# Step 3 — convert only the rows you need
+deltas = ConvexLake.to_orderbook_deltas(
+    df,
+    instrument_id="BTCPERP.BINANCE",
+    price_precision=1,
+    size_precision=3,
+)
+
+engine.add_data(deltas)
+```
+
+This produces the same result as `download_orderbook_deltas()` followed by a list
+comprehension filter, but avoids creating the Cython objects you will immediately discard.
+
+---
+
+### DataFrame schema
+
+`to_nautilus()` and `download_nautilus()` return a DataFrame with these columns:
+
+| Column | dtype | Description |
+|---|---|---|
+| `instrument_id` | `object` | NautilusTrader instrument identifier |
+| `action` | `uint8` | 1=ADD, 2=UPDATE, 3=DELETE, 4=CLEAR |
+| `side` | `uint8` | 0=NONE, 1=BUY, 2=SELL |
+| `price` | `int64` | price × 10^price_precision |
+| `size` | `uint64` | size × 10^size_precision |
+| `order_id` | `uint64` | always 0 (not tracked) |
+| `flags` | `uint8` | 1 = F_LAST (last delta in a snapshot sequence) |
+| `sequence` | `uint64` | monotonically increasing row counter |
+| `ts_event` | `uint64` | event timestamp, nanoseconds since epoch |
+| `ts_init` | `uint64` | same as ts_event |
+
+---
+
+### Persistent storage: `ParquetDataCatalog`
+
+For repeated backtests on the same data, write once to a NautilusTrader catalog instead
+of re-downloading or loading large pickle files:
+
+```python
+from nautilus_trader.persistence.catalog import ParquetDataCatalog
+
+catalog = ParquetDataCatalog("./nautilus_catalog")
+catalog.write_data(deltas)   # Arrow columnar — smaller and faster to reload than pickle
+```
+
+Read back for a later backtest:
+
+```python
+from nautilus_trader.backtest.config import BacktestDataConfig
+from nautilus_trader.model.data import OrderBookDelta
+
+data_config = BacktestDataConfig(
+    catalog_path="./nautilus_catalog",
+    data_cls=OrderBookDelta,
+    instrument_id=InstrumentId.from_str("BTCPERP.BINANCE"),
+    start_time=START,
+    end_time=END,
+)
+```
+
+---
+
+### L1/BBO: `download_quote_ticks()`
+
+For BBO files or when `BookType.L1_MBP` is sufficient, use `to_quote_ticks()` /
+`download_quote_ticks()`. This returns a lightweight DataFrame (`ts_ns`, `bid`, `ask`)
+that converts to `QuoteTick` objects — one object per BBO update instead of a CLEAR + two
+ADDs per snapshot:
+
+```python
+df = client.download_quote_ticks(
+    exchange="hyperliquid",
+    slug="BTCPERP-bbo",
+    ticker="BTCPERP",
+    date="2026-11-04",
+    price_precision=1,
+)
+
+from nautilus_trader.model.data import QuoteTick
+from nautilus_trader.model.objects import Price, Quantity
+
+ticks = [
+    QuoteTick(
+        instrument_id=iid,
+        bid_price=Price(row.bid, 1),
+        ask_price=Price(row.ask, 1),
+        bid_size=Quantity(1, 3),
+        ask_size=Quantity(1, 3),
+        ts_event=int(row.ts_ns),
+        ts_init=int(row.ts_ns),
+    )
+    for row in df.itertuples()
+]
+engine.add_data(ticks)
+```
+
+---
+
+### Choosing between `OrderBookDelta` and `QuoteTick`
+
+| | `OrderBookDelta` (L2_MBP) | `QuoteTick` (L1_MBP) |
+|---|---|---|
+| Data depth | Full depth, all levels | Top of book only |
+| Fill simulation | Passive limit fills at quoted price | Passive limit fills when market crosses |
+| Objects per snapshot | CLEAR + N×ADD (N levels) | 1 |
+| Use for | Market-making, depth-sensitive strategies | Directional, simple spread strategies |
+| SDK method | `download_orderbook_deltas()` | `download_quote_ticks()` |
+
+---
+
+## Error handling
+
+All exceptions inherit from `ConvexLakeError`.
+
+```python
+from convexlake import (
+    ConvexLakeError,
+    AuthError,       # 401 — invalid or missing API key
+    PlanError,       # 403 — endpoint requires a higher subscription tier
+    NotFoundError,   # 404 — no data for the requested parameters
+    RateLimitError,  # 429 — rate limit exceeded after all retries
+)
+
+try:
+    df = client.candles(exchange="kalshi", ticker="btc", date="2026-07-30",
+                        slug="KXBTC-2601-prices.csv")
+except NotFoundError:
+    print("No data for that date/slug")
+except PlanError:
+    print("Upgrade your plan to access this endpoint")
+except AuthError:
+    print("Check your API key")
+except RateLimitError:
+    print("Slow down — rate limit hit")
+except ConvexLakeError as e:
+    print(f"API error: {e}")
+```
+
+---
+
+## Configuration
+
+```python
+client = ConvexLake(
+    api_key="do_XXXX...",
+    timeout=60,       # request timeout in seconds (default: 60)
+    max_retries=3,    # number of retries on 429 rate-limit responses (default: 3)
+)
+```
+
+On a `429` response the client waits for the `Retry-After` header value (defaulting to 2 s) and retries with exponential backoff. If all retries are exhausted, `RateLimitError` is raised.
+
+---
+
+## Supported exchanges
+
+| Exchange | candles | trades | orderbook | surface | leaderboard |
+|---|:---:|:---:|:---:|:---:|:---:|
+| `kalshi` | ✓ | ✓ | ✓ | — | — |
+| `polymarket` | ✓ | ✓ | ✓ | — | ✓ |
+| `deribit` | ✓ | ✓ | ✓ | ✓ | — |
+| `predict` | ✓ | — | ✓ | — | — |
+| `limitless` | — | — | ✓ | — | ✓ |
+| `hyperliquid` | ✓ | ✓ | ✓ | — | — |
+| `binance` | — | — | ✓ | — | — |
+| `okx` | — | — | ✓ | — | — |
+| `bitfinex` | — | — | ✓ | — | — |
+
+> **Deribit options** orderbooks are CSV format — use `orderbook_df()` instead of `orderbook()`.  
+> **Bybit** stores perpetual options as individual CSV files; there is no single perp orderbook file.
+
+---
+
+## Development
+
+```bash
+git clone https://github.com/convexlake/convexlake-sdk.git
+cd convexlake-sdk
+pip install -e ".[dev]"
+```
+
+**Run unit tests** (no API key required — HTTP layer is mocked):
+
+```bash
+pytest tests/ -v -k "not (TestInfo or TestCandles or TestTrades or TestOrderbook or TestSurface or TestLeaderboard or TestDeribit)"
+```
+
+**Run integration tests** (requires a real API key in the environment):
+
+```bash
+export CONVEXLAKE_API_KEY="do_XXXX..."
+pytest tests/ -v
+```
+
+The test suite covers decompression logic, orderbook format auto-detection, HTTP error mapping, retry behavior, and live API responses for each exchange and data type.