dataquery-sdk 0.1.5__tar.gz → 0.2.0__tar.gz

{dataquery_sdk-0.1.5 → dataquery_sdk-0.2.0}/CHANGELOG.md

@@ -26,4 +26,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [0.1.4] - 2026-03-23
 - Introduced circuit breaker environment variable (`DATAQUERY_CIRCUIT_BREAKER_THRESHOLD`) 
 ## [0.1.5] - 2026-04-16
-- Introduced file-group-id to the group downloads 
+- Introduced file-group-id to the group downloads
+## [0.2.0] - 2026-05-06
+- SSE auto-download with cross-process event replay and multi-group support
+- Python 3.11+ only

{dataquery_sdk-0.1.5 → dataquery_sdk-0.2.0}/CONTRIBUTING.md

@@ -1,4 +1,4 @@
-# Contributing to DataQuery SDK
+ # Contributing to DataQuery SDK
 
 Thank you for your interest in contributing to the DataQuery SDK! This document provides guidelines and information for contributors.
 
@@ -6,7 +6,7 @@ Thank you for your interest in contributing to the DataQuery SDK! This document
 
 ### Prerequisites
 
-- Python 3.10 or higher
+- Python 3.11 or higher
 - [uv](https://github.com/astral-sh/uv) package manager
 - Git
 

{dataquery_sdk-0.1.5 → dataquery_sdk-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dataquery-sdk
-Version: 0.1.5
+Version: 0.2.0
 Summary: Python SDK for DATAQUERY Data API - Query, download, and check availability of economic data files
 Author-email: DATAQUERY SDK Team <dataquery_support@jpmorgan.com>
 Project-URL: Homepage, https://github.com/dataquery/dataquery-sdk
@@ -15,7 +15,6 @@ Classifier: Intended Audience :: Science/Research
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
@@ -23,7 +22,7 @@ Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Classifier: Topic :: Office/Business :: Financial
 Classifier: Topic :: Scientific/Engineering :: Information Analysis
 Classifier: Topic :: Internet :: WWW/HTTP :: Dynamic Content
-Requires-Python: >=3.10
+Requires-Python: >=3.11
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: aiohttp<4.0.0,>=3.8.0
@@ -67,15 +66,33 @@ Dynamic: license-file
 
 Python SDK for the J.P. Morgan DataQuery API — authenticated file downloads, time-series queries, and real-time notification-driven downloads with OAuth 2.0, rate limiting, and automatic retries built in.
 
-[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
+[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![Linting: Ruff](https://img.shields.io/badge/linting-ruff-261230.svg)](https://github.com/astral-sh/ruff)
 
+## Contents
+
+- [Features](#features)
+- [New here? Three steps to your first download](#new-here-three-steps-to-your-first-download)
+- [Installation](#installation)
+- [Configure credentials](#configure-credentials)
+- [Quick start](#quick-start)
+- [Auto-download (coming soon)](#auto-download-coming-soon)
+- [CLI](#cli)
+- [Configuration](#configuration)
+- [Logging](#logging)
+- [Error handling](#error-handling)
+- [Troubleshooting](#troubleshooting)
+- [Date formats](#date-formats)
+- [Performance tuning](#performance-tuning)
+- [API reference (most-used methods)](#api-reference-most-used-methods)
+- [Examples](#examples) · [Development](#development) · [Requirements](#requirements) · [Support](#support)
+
 ## Features
 
-- **Parallel file downloads** — when `num_parts > 1`, HTTP range requests split the file into parallel chunks with bounded concurrency; `num_parts=1` (default) uses a simple streaming GET
+- **Streaming file downloads** — single streaming GET per file
 - **Historical and date-range downloads** — fetch every file in a group (optionally filtered to one or many `file-group-id`s) between two dates
-- **Notification-driven downloads (SSE)** — subscribe to the `/notification` stream and auto-download files as soon as they are published
+- **Notification-driven downloads (SSE)** — subscribe to the `/events/notification` stream and auto-download files as soon as they are published
 - **Time-series queries** — by expression, by instrument, or by group with attribute / filter projections
 - **OAuth 2.0 with token caching and refresh** — or supply a bearer token directly
 - **Token-bucket rate limiter** — 300 rpm / 5 tps defaults (configurable up to API limits)
@@ -84,6 +101,28 @@ Python SDK for the J.P. Morgan DataQuery API — authenticated file downloads, t
 - **Optional pandas integration** — `to_dataframe(...)` on any response
 - **CLI** — `dataquery groups | files | availability | download | download-group | auth | config`
 
+## New here? Three steps to your first download
+
+1. `pip install dataquery-sdk`
+2. Put your OAuth credentials in a `.env` file (see [Configure credentials](#configure-credentials))
+3. Run a one-liner to confirm everything works:
+
+   ```python
+   import asyncio
+   from dataquery import DataQuery
+
+   async def main():
+       async with DataQuery() as dq:
+           groups = await dq.list_groups_async(limit=5)
+           for g in groups:
+               print(g.group_id, "—", g.group_name)
+
+   asyncio.run(main())
+   ```
+
+If that prints groups, auth and networking are working. From there, jump to
+[Quick start](#quick-start) (date-range download) or the [CLI](#cli).
+
 ## Installation
 
 ```bash
@@ -97,7 +136,7 @@ pip install "dataquery-sdk[pandas]"
 pip install "dataquery-sdk[dev]"
 ```
 
-Python 3.10+ is required.
+Python 3.11+ is required.
 
 ## Configure credentials
 
@@ -140,9 +179,10 @@ async with DataQuery() as dq:
         end_date="20250131",
         destination_dir="./data",
     )
-    print(f"{result['successful_downloads']}/{result['total_files']} files downloaded")
+    # OperationReport (Pydantic model) — counts/timing/data/details are dicts on it.
+    print(f"{result.counts['successful_downloads']}/{result.counts['total_files']} files downloaded")
 
-# sync
+# sync — same arguments, drop the _async suffix
 with DataQuery() as dq:
     result = dq.run_group_download(
         group_id="JPMAQS_GENERIC_RETURNS",
@@ -179,7 +219,6 @@ async with DataQuery() as dq:
         file_group_id="JPMAQS_GENERIC_RETURNS",
         file_datetime="20250115",
         destination_path=Path("./downloads"),
-        num_parts=5,  # parallel chunks
     )
     print(f"Downloaded: {result.local_path} ({result.file_size} bytes)")
 ```
@@ -227,8 +266,8 @@ async with DataQuery() as dq:
 
 Real-time notification-driven downloads via the DataQuery SSE stream are under
 active development and will be available in a future release. Once released,
-`auto_download_async` will subscribe to the `/notification` endpoint and
-download files automatically as they are published — no polling required.
+`auto_download_async` will subscribe to the notification endpoint and download
+files automatically as they are published — no polling required.
 
 ## CLI
 
@@ -248,17 +287,16 @@ dataquery availability --file-group-id JPMAQS_GENERIC_RETURNS --file-datetime 20
 # Download a single file
 dataquery download --file-group-id JPMAQS_GENERIC_RETURNS \
                    --file-datetime 20250115 \
-                   --destination ./downloads \
-                   --num-parts 5
+                   --destination ./downloads
 
-# Watch a group and download as new files arrive
+# Watch a group and download as new files arrive (calls auto_download_async under the hood —
+# same SSE subscription, same event-replay state files)
 dataquery download --watch --group-id JPMAQS_GENERIC_RETURNS --destination ./downloads
 
 # Download everything in a date range
 dataquery download-group --group-id JPMAQS_GENERIC_RETURNS \
                          --start-date 20250101 --end-date 20250131 \
-                         --destination ./data \
-                         --max-concurrent 5 --num-parts 4
+                         --destination ./data
 
 # Restrict to one or more file-group-ids
 dataquery download-group --group-id JPMAQS_GENERIC_RETURNS \
@@ -291,8 +329,6 @@ All environment variables use the `DATAQUERY_` prefix.
 | `DATAQUERY_CLIENT_SECRET` | _(none)_ | Required for OAuth |
 | `DATAQUERY_BEARER_TOKEN` | _(none)_ | Alternative to OAuth |
 | `DATAQUERY_OAUTH_ENABLED` | `true` | Set `false` to use bearer-token mode |
-| `DATAQUERY_OAUTH_TOKEN_URL` | `https://authe.jpmorgan.com/as/token.oauth2` | |
-| `DATAQUERY_OAUTH_AUD` | `JPMC:URI:RS-06785-DataQueryExternalApi-PROD` | |
 
 **API endpoints**
 
@@ -342,13 +378,57 @@ async with DataQuery(client_id="...", client_secret="...", timeout=60.0) as dq:
     ...
 ```
 
+## Logging
+
+The SDK logs through [structlog](https://www.structlog.org/) and emits
+structured events for requests, retries, rate-limit waits, SSE reconnects, and
+download progress. Two ways to drive it:
+
+**Standard Python logging** — works without extra setup:
+
+```python
+import logging
+logging.basicConfig(level=logging.INFO)
+logging.getLogger("dataquery").setLevel(logging.DEBUG)   # SDK-only DEBUG
+```
+
+DEBUG includes per-chunk download progress and SSE keepalives — useful while
+debugging, noisy in production.
+
+**Structured (JSON) output** — recommended for long-running `auto_download`
+services so a log shipper can parse the events:
+
+```python
+from pathlib import Path
+from dataquery.config.logging import (
+    LogFormat, LogLevel, create_logging_config, create_logging_manager,
+)
+
+cfg = create_logging_config(
+    level=LogLevel.INFO,
+    format=LogFormat.JSON,           # or LogFormat.CONSOLE for humans
+    enable_file=True,
+    log_file=Path("./dataquery.log"),
+    enable_request_logging=False,    # set True to log every HTTP request/response
+)
+create_logging_manager(cfg)          # installs handlers on the root logger
+```
+
+`examples/system/enable_request_logging.py` is a runnable version showing
+request/response logging for traffic debugging.
+
+**Health snapshots for `auto_download`** — `manager.get_stats()` returns
+notifications received, files downloaded / skipped / failed, the last event
+id, and a bounded ring of recent errors. Wire it into a `/healthz` endpoint
+for daemon-style deployments.
+
 ## Error handling
 
 All errors inherit from `DataQueryError`:
 
 ```python
 from dataquery import DataQuery
-from dataquery.exceptions import (
+from dataquery.types.exceptions import (
     DataQueryError,
     AuthenticationError,
     NotFoundError,
@@ -375,6 +455,54 @@ async with DataQuery() as dq:
         ...  # any other SDK-level failure
 ```
 
+## Troubleshooting
+
+**`AuthenticationError` / HTTP 401 on the first call**
+
+- Verify both `DATAQUERY_CLIENT_ID` and `DATAQUERY_CLIENT_SECRET` are set:
+  `dataquery config show` will print the resolved config (secrets masked).
+- Confirm OAuth is reaching the right endpoint:
+  `dataquery auth test` performs a token exchange and reports the failure mode.
+- If the credentials are correct but the audience is wrong, set
+  `DATAQUERY_OAUTH_AUD` to the value provisioned for your client.
+
+**`.env` file isn't picked up**
+
+- The SDK looks for `.env` in the current working directory at instantiation.
+  Either `cd` to the directory containing `.env` before running, or pass the
+  path explicitly: `DataQuery(env_file=".env.production")`.
+- Variables already set in the shell environment win over the `.env` file —
+  unset them (`unset DATAQUERY_CLIENT_ID`) if you want the file to take effect.
+- The CLI accepts `--env-file PATH` on every subcommand for the same reason.
+
+**Connection / proxy / SSL failures**
+
+- Behind a corporate proxy, set `DATAQUERY_PROXY_ENABLED=true` and
+  `DATAQUERY_PROXY_URL=http://proxy.host:port`. Add `DATAQUERY_PROXY_USERNAME`
+  / `DATAQUERY_PROXY_PASSWORD` if auth is required.
+- For self-signed proxy CAs, set `DATAQUERY_PROXY_VERIFY_SSL=false` (insecure;
+  prefer pointing `SSL_CERT_FILE` at the corporate root CA bundle).
+- Sporadic `NetworkError` after long idle periods usually means a stateful
+  middlebox is dropping the SSE socket — set
+  `heartbeat_timeout=90.0` on `auto_download_async` to force a reconnect when
+  no bytes arrive within the window.
+
+**Rate-limit pauses**
+
+- Default is 300 rpm / 5 tps. The SDK self-throttles via the token-bucket
+  limiter; if you see long sleeps before requests, lower
+  `DATAQUERY_REQUESTS_PER_MINUTE` is not the cure — it's likely working as
+  designed. Raise it (up to your provisioned limit) to go faster.
+- `dq.get_rate_limit_info()` shows the current bucket state.
+
+**SSE auto-download "missed" events after a restart**
+
+- Confirm `enable_event_replay=True` (the default).
+- Replay state lives under `<destination>/.sse_state/sse_<fingerprint>.json` —
+  if that directory was wiped, the next start has nothing to resume from. Use
+  `manager.clear_event_id()` (or `dataquery download --watch --reset-event-id`)
+  only when you intentionally want a clean slate.
+
 ## Date formats
 
 ```python
@@ -388,13 +516,9 @@ start_date="TODAY-1Y"
 
 ## Performance tuning
 
-`run_group_download_async` uses a flattened concurrency model — total concurrent
-HTTP requests = `max_concurrent × num_parts`.
-
-With `num_parts=1` (the default) each file downloads as a single streaming GET.
-Set `num_parts > 1` to enable parallel HTTP range requests per file — the SDK
-probes the file size first, then downloads byte ranges concurrently. Files under
-10 MB always use a single stream regardless of `num_parts`.
+`run_group_download_async` streams each file as a single GET. The SDK
+automatically inserts delays between file starts so the configured
+`requests_per_minute` is not exceeded.
 
 ```python
 await dq.run_group_download_async(
@@ -402,15 +526,13 @@ await dq.run_group_download_async(
     start_date="20250101",
     end_date="20250131",
     destination_dir="./data",
-    max_concurrent=5,   # parallel files
-    num_parts=4,        # parallel chunks per file (1 = single stream)
     max_retries=3,
 )
 ```
 
-Typical settings: `max_concurrent` 3–5, `num_parts` 2–8. The SDK automatically
-inserts delays between file starts so the configured `requests_per_minute` is not
-exceeded.
+Tune throughput via `DATAQUERY_REQUESTS_PER_MINUTE` and `DATAQUERY_BURST_CAPACITY`
+(see [Configuration](#environment-variables)) rather than per-call concurrency
+flags.
 
 ## API reference (most-used methods)
 
@@ -437,8 +559,10 @@ exceeded.
 | | `to_dataframe(response)` | requires `pandas` extra |
 | | `get_stats()` / `get_pool_stats()` / `get_rate_limit_info()` | diagnostics |
 
-Every async method above has a sync counterpart (without the `_async` suffix, or
-with an explicit `_sync` suffix) that runs the coroutine via `asyncio.run`.
+Every async method has a sync counterpart with the same name minus the
+`_async` suffix — `list_groups_async` ↔ `list_groups`, `download_file_async` ↔
+`download_file`, etc. Sync calls run the coroutine internally via
+`asyncio.run`, so do not call them from inside an existing event loop.
 
 ## Examples
 
@@ -449,13 +573,14 @@ The `examples/` directory is organised by feature:
 - `examples/instruments/` — instrument discovery + time series
 - `examples/groups/` and `examples/groups_advanced/` — group discovery and time series
 - `examples/grid/` — grid data
-- `examples/system/` — SSE notification subscriber, diagnostics
+- `examples/system/` — SSE notification subscriber (single + multi-group), diagnostics
 
 Run any example directly:
 
 ```bash
 python examples/files/download_file.py
-python examples/system/sse_local_server_example.py
+python examples/system/auto_download_example.py            # single group
+python examples/system/auto_download_multi_group_example.py  # several groups in parallel
 ```
 
 ## Development
@@ -484,7 +609,7 @@ Pytest markers: `slow`, `integration`, `unit`, `asyncio`.
 
 ## Requirements
 
-- Python 3.10+
+- Python 3.11+
 - `aiohttp>=3.8,<4`, `pydantic>=2,<3`, `structlog>=23`, `python-dotenv>=1`
 - Optional: `pandas>=2` (for `to_dataframe`)