dataquery-sdk 0.1.4__tar.gz → 0.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (51) hide show
  1. {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/CHANGELOG.md +6 -1
  2. {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/CONTRIBUTING.md +5 -8
  3. dataquery_sdk-0.2.0/PKG-INFO +627 -0
  4. dataquery_sdk-0.2.0/README.md +563 -0
  5. {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/dataquery/__init__.py +14 -116
  6. {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/dataquery/cli.py +119 -32
  7. dataquery_sdk-0.2.0/dataquery/config/__init__.py +27 -0
  8. dataquery_sdk-0.2.0/dataquery/config/env.py +364 -0
  9. dataquery_sdk-0.2.0/dataquery/constants/__init__.py +85 -0
  10. dataquery_sdk-0.2.0/dataquery/constants/api.py +35 -0
  11. dataquery_sdk-0.2.0/dataquery/constants/download.py +51 -0
  12. dataquery_sdk-0.2.0/dataquery/constants/rate_limit.py +6 -0
  13. dataquery_sdk-0.2.0/dataquery/constants/sse.py +9 -0
  14. dataquery_sdk-0.2.0/dataquery/core/__init__.py +5 -0
  15. dataquery_sdk-0.2.0/dataquery/core/_mixins.py +954 -0
  16. {dataquery_sdk-0.1.4/dataquery → dataquery_sdk-0.2.0/dataquery/core}/client.py +454 -1161
  17. {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/dataquery/dataquery.py +468 -786
  18. dataquery_sdk-0.2.0/dataquery/download/__init__.py +14 -0
  19. dataquery_sdk-0.2.0/dataquery/download/parallel.py +701 -0
  20. dataquery_sdk-0.2.0/dataquery/download/utils.py +137 -0
  21. dataquery_sdk-0.2.0/dataquery/sse/__init__.py +23 -0
  22. dataquery_sdk-0.2.0/dataquery/sse/client.py +473 -0
  23. dataquery_sdk-0.2.0/dataquery/sse/event_store.py +241 -0
  24. dataquery_sdk-0.2.0/dataquery/sse/subscriber.py +653 -0
  25. dataquery_sdk-0.2.0/dataquery/transport/__init__.py +61 -0
  26. {dataquery_sdk-0.1.4/dataquery → dataquery_sdk-0.2.0/dataquery/transport}/auth.py +33 -16
  27. {dataquery_sdk-0.1.4/dataquery → dataquery_sdk-0.2.0/dataquery/transport}/connection_pool.py +47 -103
  28. {dataquery_sdk-0.1.4/dataquery → dataquery_sdk-0.2.0/dataquery/transport}/rate_limiter.py +22 -55
  29. {dataquery_sdk-0.1.4/dataquery → dataquery_sdk-0.2.0/dataquery/transport}/retry.py +2 -3
  30. dataquery_sdk-0.2.0/dataquery/types/__init__.py +79 -0
  31. {dataquery_sdk-0.1.4/dataquery → dataquery_sdk-0.2.0/dataquery/types}/exceptions.py +37 -2
  32. {dataquery_sdk-0.1.4/dataquery → dataquery_sdk-0.2.0/dataquery/types}/models.py +75 -34
  33. {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/dataquery/utils.py +7 -10
  34. dataquery_sdk-0.2.0/dataquery_sdk.egg-info/PKG-INFO +627 -0
  35. dataquery_sdk-0.2.0/dataquery_sdk.egg-info/SOURCES.txt +43 -0
  36. {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/pyproject.toml +4 -5
  37. dataquery_sdk-0.1.4/PKG-INFO +0 -503
  38. dataquery_sdk-0.1.4/README.md +0 -438
  39. dataquery_sdk-0.1.4/dataquery/auto_download.py +0 -443
  40. dataquery_sdk-0.1.4/dataquery/config.py +0 -564
  41. dataquery_sdk-0.1.4/dataquery_sdk.egg-info/PKG-INFO +0 -503
  42. dataquery_sdk-0.1.4/dataquery_sdk.egg-info/SOURCES.txt +0 -27
  43. {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/LICENSE +0 -0
  44. {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/MANIFEST.in +0 -0
  45. /dataquery_sdk-0.1.4/dataquery/logging_config.py → /dataquery_sdk-0.2.0/dataquery/config/logging.py +0 -0
  46. {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/dataquery/py.typed +0 -0
  47. {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/dataquery_sdk.egg-info/dependency_links.txt +0 -0
  48. {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/dataquery_sdk.egg-info/entry_points.txt +0 -0
  49. {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/dataquery_sdk.egg-info/requires.txt +0 -0
  50. {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/dataquery_sdk.egg-info/top_level.txt +0 -0
  51. {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/setup.cfg +0 -0
@@ -24,4 +24,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
24
24
  ## [0.1.3] - 2026-02-21
25
25
  - Historical file download added
26
26
  ## [0.1.4] - 2026-03-23
27
- - Introduced circuit breaker environment variable (`DATAQUERY_CIRCUIT_BREAKER_THRESHOLD`)
27
+ - Introduced circuit breaker environment variable (`DATAQUERY_CIRCUIT_BREAKER_THRESHOLD`)
28
+ ## [0.1.5] - 2026-04-16
29
+ - Introduced file-group-id to the group downloads
30
+ ## [0.2.0] - 2026-05-06
31
+ - SSE auto-download with cross-process event replay and multi-group support
32
+ - Python 3.11+ only
@@ -1,4 +1,4 @@
1
- # Contributing to DataQuery SDK
1
+ # Contributing to DataQuery SDK
2
2
 
3
3
  Thank you for your interest in contributing to the DataQuery SDK! This document provides guidelines and information for contributors.
4
4
 
@@ -6,7 +6,7 @@ Thank you for your interest in contributing to the DataQuery SDK! This document
6
6
 
7
7
  ### Prerequisites
8
8
 
9
- - Python 3.10 or higher
9
+ - Python 3.11 or higher
10
10
  - [uv](https://github.com/astral-sh/uv) package manager
11
11
  - Git
12
12
 
@@ -54,9 +54,8 @@ Thank you for your interest in contributing to the DataQuery SDK! This document
54
54
 
55
55
  5. Run linting and formatting:
56
56
  ```bash
57
- uv run black .
58
- uv run isort .
59
- uv run flake8 dataquery/ tests/
57
+ uv run ruff check dataquery/ tests/ examples/
58
+ uv run ruff format dataquery/ tests/ examples/
60
59
  uv run mypy dataquery/
61
60
  ```
62
61
 
@@ -71,9 +70,7 @@ Thank you for your interest in contributing to the DataQuery SDK! This document
71
70
  ### Coding Standards
72
71
 
73
72
  - **Python Style**: Follow PEP 8
74
- - **Formatting**: Use Black for code formatting
75
- - **Import Sorting**: Use isort for import organization
76
- - **Linting**: Use flake8 for code quality checks
73
+ - **Linting & Formatting**: Use [Ruff](https://github.com/astral-sh/ruff) for linting, formatting, and import sorting
77
74
  - **Type Hints**: Use mypy for type checking
78
75
  - **Docstrings**: Follow Google docstring format
79
76
 
@@ -0,0 +1,627 @@
1
+ Metadata-Version: 2.4
2
+ Name: dataquery-sdk
3
+ Version: 0.2.0
4
+ Summary: Python SDK for DATAQUERY Data API - Query, download, and check availability of economic data files
5
+ Author-email: DATAQUERY SDK Team <dataquery_support@jpmorgan.com>
6
+ Project-URL: Homepage, https://github.com/dataquery/dataquery-sdk
7
+ Project-URL: Bug Tracker, https://github.com/dataquery/dataquery-sdk/issues
8
+ Project-URL: Documentation, https://github.com/dataquery/dataquery-sdk/wiki
9
+ Project-URL: Source Code, https://github.com/dataquery/dataquery-sdk
10
+ Keywords: dataquery,data,api,economic,financial,download,async,oauth,rate-limiting,sdk
11
+ Classifier: Development Status :: 5 - Production/Stable
12
+ Classifier: Intended Audience :: Developers
13
+ Classifier: Intended Audience :: Financial and Insurance Industry
14
+ Classifier: Intended Audience :: Science/Research
15
+ Classifier: License :: OSI Approved :: MIT License
16
+ Classifier: Operating System :: OS Independent
17
+ Classifier: Programming Language :: Python :: 3
18
+ Classifier: Programming Language :: Python :: 3.11
19
+ Classifier: Programming Language :: Python :: 3.12
20
+ Classifier: Programming Language :: Python :: 3.13
21
+ Classifier: Topic :: Software Development :: Libraries :: Python Modules
22
+ Classifier: Topic :: Office/Business :: Financial
23
+ Classifier: Topic :: Scientific/Engineering :: Information Analysis
24
+ Classifier: Topic :: Internet :: WWW/HTTP :: Dynamic Content
25
+ Requires-Python: >=3.11
26
+ Description-Content-Type: text/markdown
27
+ License-File: LICENSE
28
+ Requires-Dist: aiohttp<4.0.0,>=3.8.0
29
+ Requires-Dist: pydantic<3.0.0,>=2.0.0
30
+ Requires-Dist: structlog>=23.0.0
31
+ Requires-Dist: python-dotenv>=1.0.0
32
+ Provides-Extra: dev
33
+ Requires-Dist: pytest>=7.0.0; extra == "dev"
34
+ Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
35
+ Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
36
+ Requires-Dist: ruff>=0.4.0; extra == "dev"
37
+ Requires-Dist: mypy>=1.5.0; extra == "dev"
38
+ Requires-Dist: pre-commit>=3.0.0; extra == "dev"
39
+ Requires-Dist: pip-audit>=2.7.0; extra == "dev"
40
+ Requires-Dist: build>=1.3.0; extra == "dev"
41
+ Requires-Dist: twine>=4.0.0; extra == "dev"
42
+ Provides-Extra: docs
43
+ Requires-Dist: mkdocs>=1.5.0; extra == "docs"
44
+ Requires-Dist: mkdocs-material>=9.0.0; extra == "docs"
45
+ Requires-Dist: mkdocstrings[python]>=0.24.0; extra == "docs"
46
+ Requires-Dist: mike>=1.1.0; extra == "docs"
47
+ Provides-Extra: pandas
48
+ Requires-Dist: pandas>=2.0.0; extra == "pandas"
49
+ Provides-Extra: all
50
+ Requires-Dist: pytest>=7.0.0; extra == "all"
51
+ Requires-Dist: pytest-asyncio>=0.21.0; extra == "all"
52
+ Requires-Dist: pytest-cov>=4.0.0; extra == "all"
53
+ Requires-Dist: ruff>=0.4.0; extra == "all"
54
+ Requires-Dist: mypy>=1.5.0; extra == "all"
55
+ Requires-Dist: pre-commit>=3.0.0; extra == "all"
56
+ Requires-Dist: mkdocs>=1.5.0; extra == "all"
57
+ Requires-Dist: mkdocs-material>=9.0.0; extra == "all"
58
+ Requires-Dist: mkdocstrings[python]>=0.24.0; extra == "all"
59
+ Requires-Dist: mike>=1.1.0; extra == "all"
60
+ Requires-Dist: pandas>=2.0.0; extra == "all"
61
+ Requires-Dist: pip-audit>=2.7.0; extra == "all"
62
+ Requires-Dist: build>=1.3.0; extra == "all"
63
+ Dynamic: license-file
64
+
65
+ # DataQuery SDK
66
+
67
+ Python SDK for the J.P. Morgan DataQuery API — authenticated file downloads, time-series queries, and real-time notification-driven downloads with OAuth 2.0, rate limiting, and automatic retries built in.
68
+
69
+ [![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
70
+ [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
71
+ [![Linting: Ruff](https://img.shields.io/badge/linting-ruff-261230.svg)](https://github.com/astral-sh/ruff)
72
+
73
+ ## Contents
74
+
75
+ - [Features](#features)
76
+ - [New here? Three steps to your first download](#new-here-three-steps-to-your-first-download)
77
+ - [Installation](#installation)
78
+ - [Configure credentials](#configure-credentials)
79
+ - [Quick start](#quick-start)
80
+ - [Auto-download (coming soon)](#auto-download-coming-soon)
81
+ - [CLI](#cli)
82
+ - [Configuration](#configuration)
83
+ - [Logging](#logging)
84
+ - [Error handling](#error-handling)
85
+ - [Troubleshooting](#troubleshooting)
86
+ - [Date formats](#date-formats)
87
+ - [Performance tuning](#performance-tuning)
88
+ - [API reference (most-used methods)](#api-reference-most-used-methods)
89
+ - [Examples](#examples) · [Development](#development) · [Requirements](#requirements) · [Support](#support)
90
+
91
+ ## Features
92
+
93
+ - **Streaming file downloads** — single streaming GET per file
94
+ - **Historical and date-range downloads** — fetch every file in a group (optionally filtered to one or many `file-group-id`s) between two dates
95
+ - **Notification-driven downloads (SSE)** — subscribe to the `/events/notification` stream and auto-download files as soon as they are published
96
+ - **Time-series queries** — by expression, by instrument, or by group with attribute / filter projections
97
+ - **OAuth 2.0 with token caching and refresh** — or supply a bearer token directly
98
+ - **Token-bucket rate limiter** — 300 rpm / 5 tps defaults (configurable up to API limits)
99
+ - **Retry + circuit breaker** — exponential backoff, configurable failure threshold
100
+ - **Sync and async APIs** — every operation has `_async` and sync variants
101
+ - **Optional pandas integration** — `to_dataframe(...)` on any response
102
+ - **CLI** — `dataquery groups | files | availability | download | download-group | auth | config`
103
+
104
+ ## New here? Three steps to your first download
105
+
106
+ 1. `pip install dataquery-sdk`
107
+ 2. Put your OAuth credentials in a `.env` file (see [Configure credentials](#configure-credentials))
108
+ 3. Run a one-liner to confirm everything works:
109
+
110
+ ```python
111
+ import asyncio
112
+ from dataquery import DataQuery
113
+
114
+ async def main():
115
+ async with DataQuery() as dq:
116
+ groups = await dq.list_groups_async(limit=5)
117
+ for g in groups:
118
+ print(g.group_id, "—", g.group_name)
119
+
120
+ asyncio.run(main())
121
+ ```
122
+
123
+ If that prints groups, auth and networking are working. From there, jump to
124
+ [Quick start](#quick-start) (date-range download) or the [CLI](#cli).
125
+
126
+ ## Installation
127
+
128
+ ```bash
129
+ # Core install
130
+ pip install dataquery-sdk
131
+
132
+ # With pandas DataFrame conversion
133
+ pip install "dataquery-sdk[pandas]"
134
+
135
+ # With dev tooling (ruff, mypy, pytest)
136
+ pip install "dataquery-sdk[dev]"
137
+ ```
138
+
139
+ Python 3.11+ is required.
140
+
141
+ ## Configure credentials
142
+
143
+ Set OAuth client credentials via environment variables:
144
+
145
+ ```bash
146
+ export DATAQUERY_CLIENT_ID="your_client_id"
147
+ export DATAQUERY_CLIENT_SECRET="your_client_secret"
148
+ ```
149
+
150
+ Or create a `.env` file in the working directory:
151
+
152
+ ```env
153
+ DATAQUERY_CLIENT_ID=your_client_id
154
+ DATAQUERY_CLIENT_SECRET=your_client_secret
155
+ ```
156
+
157
+ Or pass them directly to the constructor:
158
+
159
+ ```python
160
+ from dataquery import DataQuery
161
+
162
+ dq = DataQuery(client_id="...", client_secret="...")
163
+ ```
164
+
165
+ A starter `.env` can be generated with `dataquery config template --output .env`.
166
+
167
+ ## Quick start
168
+
169
+ ### Download files for a date range
170
+
171
+ ```python
172
+ from dataquery import DataQuery
173
+
174
+ # async
175
+ async with DataQuery() as dq:
176
+ result = await dq.run_group_download_async(
177
+ group_id="JPMAQS_GENERIC_RETURNS",
178
+ start_date="20250101",
179
+ end_date="20250131",
180
+ destination_dir="./data",
181
+ )
182
+ # OperationReport (Pydantic model) — counts/timing/data/details are dicts on it.
183
+ print(f"{result.counts['successful_downloads']}/{result.counts['total_files']} files downloaded")
184
+
185
+ # sync — same arguments, drop the _async suffix
186
+ with DataQuery() as dq:
187
+ result = dq.run_group_download(
188
+ group_id="JPMAQS_GENERIC_RETURNS",
189
+ start_date="20250101",
190
+ end_date="20250131",
191
+ destination_dir="./data",
192
+ )
193
+ ```
194
+
195
+ ### Restrict a date-range download to specific file-group-ids
196
+
197
+ `file_group_id` accepts a single id or a list. When a list is supplied, availability
198
+ queries run in parallel per id and the union of dates is downloaded.
199
+
200
+ ```python
201
+ async with DataQuery() as dq:
202
+ result = await dq.run_group_download_async(
203
+ group_id="JPMAQS_GENERIC_RETURNS",
204
+ start_date="20250101",
205
+ end_date="20250131",
206
+ destination_dir="./data",
207
+ file_group_id=["FG_ABC", "FG_DEF", "FG_XYZ"],
208
+ )
209
+ ```
210
+
211
+ ### Download a single file
212
+
213
+ ```python
214
+ from pathlib import Path
215
+ from dataquery import DataQuery
216
+
217
+ async with DataQuery() as dq:
218
+ result = await dq.download_file_async(
219
+ file_group_id="JPMAQS_GENERIC_RETURNS",
220
+ file_datetime="20250115",
221
+ destination_path=Path("./downloads"),
222
+ )
223
+ print(f"Downloaded: {result.local_path} ({result.file_size} bytes)")
224
+ ```
225
+
226
+ ### Time-series queries
227
+
228
+ ```python
229
+ async with DataQuery() as dq:
230
+ # By expression
231
+ ts = await dq.get_expressions_time_series_async(
232
+ expressions=["DB(MTE,IRISH EUR 1.100 15-May-2029 LON,,IE00BH3SQ895,MIDPRC)"],
233
+ start_date="20240101",
234
+ end_date="20240131",
235
+ )
236
+
237
+ # By group with attributes + filter
238
+ ts = await dq.get_group_time_series_async(
239
+ group_id="FI_GO_BO_EA",
240
+ attributes=["MIDPRC", "REPO_1M"],
241
+ filter="country(IRL)",
242
+ start_date="20240101",
243
+ end_date="20240131",
244
+ )
245
+
246
+ df = dq.to_dataframe(ts) # requires pandas extra
247
+ ```
248
+
249
+ ### Discover available data
250
+
251
+ ```python
252
+ async with DataQuery() as dq:
253
+ groups = await dq.list_groups_async(limit=100)
254
+ files = await dq.list_files_async(group_id="JPMAQS_GENERIC_RETURNS")
255
+ available = await dq.list_available_files_async(
256
+ group_id="JPMAQS_GENERIC_RETURNS",
257
+ start_date="20250101",
258
+ end_date="20250131",
259
+ )
260
+ instruments = await dq.search_instruments_async(
261
+ group_id="FI_GO_BO_EA", keywords="irish"
262
+ )
263
+ ```
264
+
265
+ ## Auto-download (coming soon)
266
+
267
+ Real-time notification-driven downloads via the DataQuery SSE stream are under
268
+ active development and will be available in a future release. Once released,
269
+ `auto_download_async` will subscribe to the notification endpoint and download
270
+ files automatically as they are published — no polling required.
271
+
272
+ ## CLI
273
+
274
+ The installer registers a `dataquery` script:
275
+
276
+ ```bash
277
+ # List / search groups
278
+ dataquery groups --limit 100
279
+ dataquery groups --search "fixed income" --json
280
+
281
+ # List files in a group
282
+ dataquery files --group-id JPMAQS_GENERIC_RETURNS --json
283
+
284
+ # Check availability for a single file
285
+ dataquery availability --file-group-id JPMAQS_GENERIC_RETURNS --file-datetime 20250115
286
+
287
+ # Download a single file
288
+ dataquery download --file-group-id JPMAQS_GENERIC_RETURNS \
289
+ --file-datetime 20250115 \
290
+ --destination ./downloads
291
+
292
+ # Watch a group and download as new files arrive (calls auto_download_async under the hood —
293
+ # same SSE subscription, same event-replay state files)
294
+ dataquery download --watch --group-id JPMAQS_GENERIC_RETURNS --destination ./downloads
295
+
296
+ # Download everything in a date range
297
+ dataquery download-group --group-id JPMAQS_GENERIC_RETURNS \
298
+ --start-date 20250101 --end-date 20250131 \
299
+ --destination ./data
300
+
301
+ # Restrict to one or more file-group-ids
302
+ dataquery download-group --group-id JPMAQS_GENERIC_RETURNS \
303
+ --file-group-id FG_ABC FG_DEF \
304
+ --start-date 20250101 --end-date 20250131
305
+
306
+ # Config utilities
307
+ dataquery config show
308
+ dataquery config validate
309
+ dataquery config template --output .env
310
+
311
+ # Verify auth
312
+ dataquery auth test
313
+ ```
314
+
315
+ Every subcommand accepts `--env-file PATH` (to point at a non-default `.env`) and
316
+ most accept `--json` for machine-readable output.
317
+
318
+ ## Configuration
319
+
320
+ ### Environment variables
321
+
322
+ All environment variables use the `DATAQUERY_` prefix.
323
+
324
+ **Credentials**
325
+
326
+ | Variable | Default | Notes |
327
+ |---|---|---|
328
+ | `DATAQUERY_CLIENT_ID` | _(none)_ | Required for OAuth |
329
+ | `DATAQUERY_CLIENT_SECRET` | _(none)_ | Required for OAuth |
330
+ | `DATAQUERY_BEARER_TOKEN` | _(none)_ | Alternative to OAuth |
331
+ | `DATAQUERY_OAUTH_ENABLED` | `true` | Set `false` to use bearer-token mode |
332
+
333
+ **API endpoints**
334
+
335
+ | Variable | Default |
336
+ |---|---|
337
+ | `DATAQUERY_BASE_URL` | `https://api-developer.jpmorgan.com` |
338
+ | `DATAQUERY_FILES_BASE_URL` | `https://api-dataquery.jpmchase.com` |
339
+ | `DATAQUERY_CONTEXT_PATH` | `/research/dataquery-authe/api/v2` |
340
+
341
+ **HTTP / retry / rate limit**
342
+
343
+ | Variable | Default |
344
+ |---|---|
345
+ | `DATAQUERY_TIMEOUT` | `600.0` |
346
+ | `DATAQUERY_MAX_RETRIES` | `3` |
347
+ | `DATAQUERY_RETRY_DELAY` | `1.0` |
348
+ | `DATAQUERY_CIRCUIT_BREAKER_THRESHOLD` | `5` |
349
+ | `DATAQUERY_REQUESTS_PER_MINUTE` | `300` |
350
+ | `DATAQUERY_BURST_CAPACITY` | `5` |
351
+ | `DATAQUERY_POOL_CONNECTIONS` | `10` |
352
+ | `DATAQUERY_POOL_MAXSIZE` | `20` |
353
+
354
+ **Proxy** (optional)
355
+
356
+ `DATAQUERY_PROXY_ENABLED`, `DATAQUERY_PROXY_URL`, `DATAQUERY_PROXY_USERNAME`,
357
+ `DATAQUERY_PROXY_PASSWORD`, `DATAQUERY_PROXY_VERIFY_SSL`.
358
+
359
+ ### Programmatic configuration
360
+
361
+ ```python
362
+ from dataquery import ClientConfig, DataQuery
363
+
364
+ config = ClientConfig(
365
+ client_id="...",
366
+ client_secret="...",
367
+ base_url="https://api-developer.jpmorgan.com",
368
+ timeout=60.0,
369
+ max_retries=3,
370
+ requests_per_minute=300,
371
+ )
372
+
373
+ async with DataQuery(config) as dq:
374
+ ...
375
+
376
+ # Or pass overrides as kwargs on top of env/.env resolution:
377
+ async with DataQuery(client_id="...", client_secret="...", timeout=60.0) as dq:
378
+ ...
379
+ ```
380
+
381
+ ## Logging
382
+
383
+ The SDK logs through [structlog](https://www.structlog.org/) and emits
384
+ structured events for requests, retries, rate-limit waits, SSE reconnects, and
385
+ download progress. Two ways to drive it:
386
+
387
+ **Standard Python logging** — works without extra setup:
388
+
389
+ ```python
390
+ import logging
391
+ logging.basicConfig(level=logging.INFO)
392
+ logging.getLogger("dataquery").setLevel(logging.DEBUG) # SDK-only DEBUG
393
+ ```
394
+
395
+ DEBUG includes per-chunk download progress and SSE keepalives — useful while
396
+ debugging, noisy in production.
397
+
398
+ **Structured (JSON) output** — recommended for long-running `auto_download`
399
+ services so a log shipper can parse the events:
400
+
401
+ ```python
402
+ from pathlib import Path
403
+ from dataquery.config.logging import (
404
+ LogFormat, LogLevel, create_logging_config, create_logging_manager,
405
+ )
406
+
407
+ cfg = create_logging_config(
408
+ level=LogLevel.INFO,
409
+ format=LogFormat.JSON, # or LogFormat.CONSOLE for humans
410
+ enable_file=True,
411
+ log_file=Path("./dataquery.log"),
412
+ enable_request_logging=False, # set True to log every HTTP request/response
413
+ )
414
+ create_logging_manager(cfg) # installs handlers on the root logger
415
+ ```
416
+
417
+ `examples/system/enable_request_logging.py` is a runnable version showing
418
+ request/response logging for traffic debugging.
419
+
420
+ **Health snapshots for `auto_download`** — `manager.get_stats()` returns
421
+ notifications received, files downloaded / skipped / failed, the last event
422
+ id, and a bounded ring of recent errors. Wire it into a `/healthz` endpoint
423
+ for daemon-style deployments.
424
+
425
+ ## Error handling
426
+
427
+ All errors inherit from `DataQueryError`:
428
+
429
+ ```python
430
+ from dataquery import DataQuery
431
+ from dataquery.types.exceptions import (
432
+ DataQueryError,
433
+ AuthenticationError,
434
+ NotFoundError,
435
+ RateLimitError,
436
+ NetworkError,
437
+ DownloadError,
438
+ ConfigurationError,
439
+ )
440
+
441
+ async with DataQuery() as dq:
442
+ try:
443
+ ts = await dq.get_expressions_time_series_async(
444
+ expressions=["DB(...)"], start_date="20240101", end_date="20240131"
445
+ )
446
+ except AuthenticationError:
447
+ ... # check credentials
448
+ except RateLimitError:
449
+ ... # back off — SDK already retried
450
+ except NotFoundError:
451
+ ... # group / file / instrument not found
452
+ except NetworkError:
453
+ ... # transient; SDK already retried
454
+ except DataQueryError:
455
+ ... # any other SDK-level failure
456
+ ```
457
+
458
+ ## Troubleshooting
459
+
460
+ **`AuthenticationError` / HTTP 401 on the first call**
461
+
462
+ - Verify both `DATAQUERY_CLIENT_ID` and `DATAQUERY_CLIENT_SECRET` are set:
463
+ `dataquery config show` will print the resolved config (secrets masked).
464
+ - Confirm OAuth is reaching the right endpoint:
465
+ `dataquery auth test` performs a token exchange and reports the failure mode.
466
+ - If the credentials are correct but the audience is wrong, set
467
+ `DATAQUERY_OAUTH_AUD` to the value provisioned for your client.
468
+
469
+ **`.env` file isn't picked up**
470
+
471
+ - The SDK looks for `.env` in the current working directory at instantiation.
472
+ Either `cd` to the directory containing `.env` before running, or pass the
473
+ path explicitly: `DataQuery(env_file=".env.production")`.
474
+ - Variables already set in the shell environment win over the `.env` file —
475
+ unset them (`unset DATAQUERY_CLIENT_ID`) if you want the file to take effect.
476
+ - The CLI accepts `--env-file PATH` on every subcommand for the same reason.
477
+
478
+ **Connection / proxy / SSL failures**
479
+
480
+ - Behind a corporate proxy, set `DATAQUERY_PROXY_ENABLED=true` and
481
+ `DATAQUERY_PROXY_URL=http://proxy.host:port`. Add `DATAQUERY_PROXY_USERNAME`
482
+ / `DATAQUERY_PROXY_PASSWORD` if auth is required.
483
+ - For self-signed proxy CAs, set `DATAQUERY_PROXY_VERIFY_SSL=false` (insecure;
484
+ prefer pointing `SSL_CERT_FILE` at the corporate root CA bundle).
485
+ - Sporadic `NetworkError` after long idle periods usually means a stateful
486
+ middlebox is dropping the SSE socket — set
487
+ `heartbeat_timeout=90.0` on `auto_download_async` to force a reconnect when
488
+ no bytes arrive within the window.
489
+
490
+ **Rate-limit pauses**
491
+
492
+ - Default is 300 rpm / 5 tps. The SDK self-throttles via the token-bucket
493
+ limiter; if you see long sleeps before requests, lower
494
+ `DATAQUERY_REQUESTS_PER_MINUTE` is not the cure — it's likely working as
495
+ designed. Raise it (up to your provisioned limit) to go faster.
496
+ - `dq.get_rate_limit_info()` shows the current bucket state.
497
+
498
+ **SSE auto-download "missed" events after a restart**
499
+
500
+ - Confirm `enable_event_replay=True` (the default).
501
+ - Replay state lives under `<destination>/.sse_state/sse_<fingerprint>.json` —
502
+ if that directory was wiped, the next start has nothing to resume from. Use
503
+ `manager.clear_event_id()` (or `dataquery download --watch --reset-event-id`)
504
+ only when you intentionally want a clean slate.
505
+
506
+ ## Date formats
507
+
508
+ ```python
509
+ start_date="20240101" # absolute, YYYYMMDD
510
+ start_date="TODAY"
511
+ start_date="TODAY-1D" # yesterday
512
+ start_date="TODAY-1W"
513
+ start_date="TODAY-1M"
514
+ start_date="TODAY-1Y"
515
+ ```
516
+
517
+ ## Performance tuning
518
+
519
+ `run_group_download_async` streams each file as a single GET. The SDK
520
+ automatically inserts delays between file starts so the configured
521
+ `requests_per_minute` is not exceeded.
522
+
523
+ ```python
524
+ await dq.run_group_download_async(
525
+ group_id="JPMAQS_GENERIC_RETURNS",
526
+ start_date="20250101",
527
+ end_date="20250131",
528
+ destination_dir="./data",
529
+ max_retries=3,
530
+ )
531
+ ```
532
+
533
+ Tune throughput via `DATAQUERY_REQUESTS_PER_MINUTE` and `DATAQUERY_BURST_CAPACITY`
534
+ (see [Configuration](#environment-variables)) rather than per-call concurrency
535
+ flags.
536
+
537
+ ## API reference (most-used methods)
538
+
539
+ | Area | Method | Notes |
540
+ |---|---|---|
541
+ | Discovery | `list_groups_async(limit)` | |
542
+ | | `search_groups_async(keywords, limit, offset)` | |
543
+ | | `list_files_async(group_id, file_group_id=None)` | |
544
+ | | `list_available_files_async(group_id, file_group_id, start_date, end_date)` | |
545
+ | | `list_instruments_async(group_id, instrument_id=None, page=None)` | |
546
+ | | `search_instruments_async(group_id, keywords, page=None)` | |
547
+ | | `get_group_attributes_async(group_id, ...)` | |
548
+ | | `get_group_filters_async(group_id, page=None)` | |
549
+ | Downloads | `download_file_async(file_group_id, file_datetime, ...)` | single file |
550
+ | | `run_group_download_async(group_id, start_date, end_date, file_group_id=None, ...)` | date range, single or list of ids |
551
+ | | `download_historical_async(...)` | chunked historical backfill |
552
+ | | `auto_download_async(group_id, ...)` | SSE notifications (the only watch path) |
553
+ | Time series | `get_expressions_time_series_async(expressions, start_date, end_date)` | |
554
+ | | `get_instrument_time_series_async(instruments, attributes, start_date, end_date)` | |
555
+ | | `get_group_time_series_async(group_id, attributes, filter, start_date, end_date)` | |
556
+ | Grid data | `get_grid_data_async(...)` | |
557
+ | Utilities | `check_availability_async(file_group_id, file_datetime)` | |
558
+ | | `health_check_async()` | |
559
+ | | `to_dataframe(response)` | requires `pandas` extra |
560
+ | | `get_stats()` / `get_pool_stats()` / `get_rate_limit_info()` | diagnostics |
561
+
562
+ Every async method has a sync counterpart with the same name minus the
563
+ `_async` suffix — `list_groups_async` ↔ `list_groups`, `download_file_async` ↔
564
+ `download_file`, etc. Sync calls run the coroutine internally via
565
+ `asyncio.run`, so do not call them from inside an existing event loop.
566
+
567
+ ## Examples
568
+
569
+ The `examples/` directory is organised by feature:
570
+
571
+ - `examples/files/` — single-file and date-range downloads
572
+ - `examples/expressions/` — expression time series
573
+ - `examples/instruments/` — instrument discovery + time series
574
+ - `examples/groups/` and `examples/groups_advanced/` — group discovery and time series
575
+ - `examples/grid/` — grid data
576
+ - `examples/system/` — SSE notification subscriber (single + multi-group), diagnostics
577
+
578
+ Run any example directly:
579
+
580
+ ```bash
581
+ python examples/files/download_file.py
582
+ python examples/system/auto_download_example.py # single group
583
+ python examples/system/auto_download_multi_group_example.py # several groups in parallel
584
+ ```
585
+
586
+ ## Development
587
+
588
+ ```bash
589
+ # Clone
590
+ git clone https://github.com/jpmorganchase/dataquery-sdk.git
591
+ cd dataquery-sdk
592
+
593
+ # Install with dev + all extras
594
+ uv sync --all-extras --dev # using uv
595
+ # or
596
+ pip install -e ".[dev,pandas]"
597
+
598
+ # Run tests
599
+ pytest tests/ -v
600
+ pytest tests/ --cov=dataquery --cov-report=term-missing
601
+
602
+ # Lint / format / type-check
603
+ ruff check dataquery/ tests/ examples/
604
+ ruff format dataquery/ tests/ examples/
605
+ mypy dataquery/
606
+ ```
607
+
608
+ Pytest markers: `slow`, `integration`, `unit`, `asyncio`.
609
+
610
+ ## Requirements
611
+
612
+ - Python 3.11+
613
+ - `aiohttp>=3.8,<4`, `pydantic>=2,<3`, `structlog>=23`, `python-dotenv>=1`
614
+ - Optional: `pandas>=2` (for `to_dataframe`)
615
+
616
+ ## Support
617
+
618
+ - GitHub Issues: <https://github.com/jpmorganchase/dataquery-sdk/issues>
619
+ - Email: dataquery_support@jpmorgan.com
620
+
621
+ ## License
622
+
623
+ MIT — see [LICENSE](LICENSE).
624
+
625
+ ## Changelog
626
+
627
+ See [docs/changelog.md](docs/changelog.md).