dataquery-sdk 0.1.4__tar.gz → 0.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/CHANGELOG.md +6 -1
- {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/CONTRIBUTING.md +5 -8
- dataquery_sdk-0.2.0/PKG-INFO +627 -0
- dataquery_sdk-0.2.0/README.md +563 -0
- {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/dataquery/__init__.py +14 -116
- {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/dataquery/cli.py +119 -32
- dataquery_sdk-0.2.0/dataquery/config/__init__.py +27 -0
- dataquery_sdk-0.2.0/dataquery/config/env.py +364 -0
- dataquery_sdk-0.2.0/dataquery/constants/__init__.py +85 -0
- dataquery_sdk-0.2.0/dataquery/constants/api.py +35 -0
- dataquery_sdk-0.2.0/dataquery/constants/download.py +51 -0
- dataquery_sdk-0.2.0/dataquery/constants/rate_limit.py +6 -0
- dataquery_sdk-0.2.0/dataquery/constants/sse.py +9 -0
- dataquery_sdk-0.2.0/dataquery/core/__init__.py +5 -0
- dataquery_sdk-0.2.0/dataquery/core/_mixins.py +954 -0
- {dataquery_sdk-0.1.4/dataquery → dataquery_sdk-0.2.0/dataquery/core}/client.py +454 -1161
- {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/dataquery/dataquery.py +468 -786
- dataquery_sdk-0.2.0/dataquery/download/__init__.py +14 -0
- dataquery_sdk-0.2.0/dataquery/download/parallel.py +701 -0
- dataquery_sdk-0.2.0/dataquery/download/utils.py +137 -0
- dataquery_sdk-0.2.0/dataquery/sse/__init__.py +23 -0
- dataquery_sdk-0.2.0/dataquery/sse/client.py +473 -0
- dataquery_sdk-0.2.0/dataquery/sse/event_store.py +241 -0
- dataquery_sdk-0.2.0/dataquery/sse/subscriber.py +653 -0
- dataquery_sdk-0.2.0/dataquery/transport/__init__.py +61 -0
- {dataquery_sdk-0.1.4/dataquery → dataquery_sdk-0.2.0/dataquery/transport}/auth.py +33 -16
- {dataquery_sdk-0.1.4/dataquery → dataquery_sdk-0.2.0/dataquery/transport}/connection_pool.py +47 -103
- {dataquery_sdk-0.1.4/dataquery → dataquery_sdk-0.2.0/dataquery/transport}/rate_limiter.py +22 -55
- {dataquery_sdk-0.1.4/dataquery → dataquery_sdk-0.2.0/dataquery/transport}/retry.py +2 -3
- dataquery_sdk-0.2.0/dataquery/types/__init__.py +79 -0
- {dataquery_sdk-0.1.4/dataquery → dataquery_sdk-0.2.0/dataquery/types}/exceptions.py +37 -2
- {dataquery_sdk-0.1.4/dataquery → dataquery_sdk-0.2.0/dataquery/types}/models.py +75 -34
- {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/dataquery/utils.py +7 -10
- dataquery_sdk-0.2.0/dataquery_sdk.egg-info/PKG-INFO +627 -0
- dataquery_sdk-0.2.0/dataquery_sdk.egg-info/SOURCES.txt +43 -0
- {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/pyproject.toml +4 -5
- dataquery_sdk-0.1.4/PKG-INFO +0 -503
- dataquery_sdk-0.1.4/README.md +0 -438
- dataquery_sdk-0.1.4/dataquery/auto_download.py +0 -443
- dataquery_sdk-0.1.4/dataquery/config.py +0 -564
- dataquery_sdk-0.1.4/dataquery_sdk.egg-info/PKG-INFO +0 -503
- dataquery_sdk-0.1.4/dataquery_sdk.egg-info/SOURCES.txt +0 -27
- {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/LICENSE +0 -0
- {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/MANIFEST.in +0 -0
- /dataquery_sdk-0.1.4/dataquery/logging_config.py → /dataquery_sdk-0.2.0/dataquery/config/logging.py +0 -0
- {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/dataquery/py.typed +0 -0
- {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/dataquery_sdk.egg-info/dependency_links.txt +0 -0
- {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/dataquery_sdk.egg-info/entry_points.txt +0 -0
- {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/dataquery_sdk.egg-info/requires.txt +0 -0
- {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/dataquery_sdk.egg-info/top_level.txt +0 -0
- {dataquery_sdk-0.1.4 → dataquery_sdk-0.2.0}/setup.cfg +0 -0
|
@@ -24,4 +24,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
24
24
|
## [0.1.3] - 2026-02-21
|
|
25
25
|
- Historical file download added
|
|
26
26
|
## [0.1.4] - 2026-03-23
|
|
27
|
-
- Introduced circuit breaker environment variable (`DATAQUERY_CIRCUIT_BREAKER_THRESHOLD`)
|
|
27
|
+
- Introduced circuit breaker environment variable (`DATAQUERY_CIRCUIT_BREAKER_THRESHOLD`)
|
|
28
|
+
## [0.1.5] - 2026-04-16
|
|
29
|
+
- Introduced file-group-id to the group downloads
|
|
30
|
+
## [0.2.0] - 2026-05-06
|
|
31
|
+
- SSE auto-download with cross-process event replay and multi-group support
|
|
32
|
+
- Python 3.11+ only
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
# Contributing to DataQuery SDK
|
|
1
|
+
# Contributing to DataQuery SDK
|
|
2
2
|
|
|
3
3
|
Thank you for your interest in contributing to the DataQuery SDK! This document provides guidelines and information for contributors.
|
|
4
4
|
|
|
@@ -6,7 +6,7 @@ Thank you for your interest in contributing to the DataQuery SDK! This document
|
|
|
6
6
|
|
|
7
7
|
### Prerequisites
|
|
8
8
|
|
|
9
|
-
- Python 3.
|
|
9
|
+
- Python 3.11 or higher
|
|
10
10
|
- [uv](https://github.com/astral-sh/uv) package manager
|
|
11
11
|
- Git
|
|
12
12
|
|
|
@@ -54,9 +54,8 @@ Thank you for your interest in contributing to the DataQuery SDK! This document
|
|
|
54
54
|
|
|
55
55
|
5. Run linting and formatting:
|
|
56
56
|
```bash
|
|
57
|
-
uv run
|
|
58
|
-
uv run
|
|
59
|
-
uv run flake8 dataquery/ tests/
|
|
57
|
+
uv run ruff check dataquery/ tests/ examples/
|
|
58
|
+
uv run ruff format dataquery/ tests/ examples/
|
|
60
59
|
uv run mypy dataquery/
|
|
61
60
|
```
|
|
62
61
|
|
|
@@ -71,9 +70,7 @@ Thank you for your interest in contributing to the DataQuery SDK! This document
|
|
|
71
70
|
### Coding Standards
|
|
72
71
|
|
|
73
72
|
- **Python Style**: Follow PEP 8
|
|
74
|
-
- **Formatting**: Use
|
|
75
|
-
- **Import Sorting**: Use isort for import organization
|
|
76
|
-
- **Linting**: Use flake8 for code quality checks
|
|
73
|
+
- **Linting & Formatting**: Use [Ruff](https://github.com/astral-sh/ruff) for linting, formatting, and import sorting
|
|
77
74
|
- **Type Hints**: Use mypy for type checking
|
|
78
75
|
- **Docstrings**: Follow Google docstring format
|
|
79
76
|
|
|
@@ -0,0 +1,627 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: dataquery-sdk
|
|
3
|
+
Version: 0.2.0
|
|
4
|
+
Summary: Python SDK for DATAQUERY Data API - Query, download, and check availability of economic data files
|
|
5
|
+
Author-email: DATAQUERY SDK Team <dataquery_support@jpmorgan.com>
|
|
6
|
+
Project-URL: Homepage, https://github.com/dataquery/dataquery-sdk
|
|
7
|
+
Project-URL: Bug Tracker, https://github.com/dataquery/dataquery-sdk/issues
|
|
8
|
+
Project-URL: Documentation, https://github.com/dataquery/dataquery-sdk/wiki
|
|
9
|
+
Project-URL: Source Code, https://github.com/dataquery/dataquery-sdk
|
|
10
|
+
Keywords: dataquery,data,api,economic,financial,download,async,oauth,rate-limiting,sdk
|
|
11
|
+
Classifier: Development Status :: 5 - Production/Stable
|
|
12
|
+
Classifier: Intended Audience :: Developers
|
|
13
|
+
Classifier: Intended Audience :: Financial and Insurance Industry
|
|
14
|
+
Classifier: Intended Audience :: Science/Research
|
|
15
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
16
|
+
Classifier: Operating System :: OS Independent
|
|
17
|
+
Classifier: Programming Language :: Python :: 3
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
21
|
+
Classifier: Topic :: Software Development :: Libraries :: Python Modules
|
|
22
|
+
Classifier: Topic :: Office/Business :: Financial
|
|
23
|
+
Classifier: Topic :: Scientific/Engineering :: Information Analysis
|
|
24
|
+
Classifier: Topic :: Internet :: WWW/HTTP :: Dynamic Content
|
|
25
|
+
Requires-Python: >=3.11
|
|
26
|
+
Description-Content-Type: text/markdown
|
|
27
|
+
License-File: LICENSE
|
|
28
|
+
Requires-Dist: aiohttp<4.0.0,>=3.8.0
|
|
29
|
+
Requires-Dist: pydantic<3.0.0,>=2.0.0
|
|
30
|
+
Requires-Dist: structlog>=23.0.0
|
|
31
|
+
Requires-Dist: python-dotenv>=1.0.0
|
|
32
|
+
Provides-Extra: dev
|
|
33
|
+
Requires-Dist: pytest>=7.0.0; extra == "dev"
|
|
34
|
+
Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
|
|
35
|
+
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
|
|
36
|
+
Requires-Dist: ruff>=0.4.0; extra == "dev"
|
|
37
|
+
Requires-Dist: mypy>=1.5.0; extra == "dev"
|
|
38
|
+
Requires-Dist: pre-commit>=3.0.0; extra == "dev"
|
|
39
|
+
Requires-Dist: pip-audit>=2.7.0; extra == "dev"
|
|
40
|
+
Requires-Dist: build>=1.3.0; extra == "dev"
|
|
41
|
+
Requires-Dist: twine>=4.0.0; extra == "dev"
|
|
42
|
+
Provides-Extra: docs
|
|
43
|
+
Requires-Dist: mkdocs>=1.5.0; extra == "docs"
|
|
44
|
+
Requires-Dist: mkdocs-material>=9.0.0; extra == "docs"
|
|
45
|
+
Requires-Dist: mkdocstrings[python]>=0.24.0; extra == "docs"
|
|
46
|
+
Requires-Dist: mike>=1.1.0; extra == "docs"
|
|
47
|
+
Provides-Extra: pandas
|
|
48
|
+
Requires-Dist: pandas>=2.0.0; extra == "pandas"
|
|
49
|
+
Provides-Extra: all
|
|
50
|
+
Requires-Dist: pytest>=7.0.0; extra == "all"
|
|
51
|
+
Requires-Dist: pytest-asyncio>=0.21.0; extra == "all"
|
|
52
|
+
Requires-Dist: pytest-cov>=4.0.0; extra == "all"
|
|
53
|
+
Requires-Dist: ruff>=0.4.0; extra == "all"
|
|
54
|
+
Requires-Dist: mypy>=1.5.0; extra == "all"
|
|
55
|
+
Requires-Dist: pre-commit>=3.0.0; extra == "all"
|
|
56
|
+
Requires-Dist: mkdocs>=1.5.0; extra == "all"
|
|
57
|
+
Requires-Dist: mkdocs-material>=9.0.0; extra == "all"
|
|
58
|
+
Requires-Dist: mkdocstrings[python]>=0.24.0; extra == "all"
|
|
59
|
+
Requires-Dist: mike>=1.1.0; extra == "all"
|
|
60
|
+
Requires-Dist: pandas>=2.0.0; extra == "all"
|
|
61
|
+
Requires-Dist: pip-audit>=2.7.0; extra == "all"
|
|
62
|
+
Requires-Dist: build>=1.3.0; extra == "all"
|
|
63
|
+
Dynamic: license-file
|
|
64
|
+
|
|
65
|
+
# DataQuery SDK
|
|
66
|
+
|
|
67
|
+
Python SDK for the J.P. Morgan DataQuery API — authenticated file downloads, time-series queries, and real-time notification-driven downloads with OAuth 2.0, rate limiting, and automatic retries built in.
|
|
68
|
+
|
|
69
|
+
[](https://www.python.org/downloads/)
|
|
70
|
+
[](https://opensource.org/licenses/MIT)
|
|
71
|
+
[](https://github.com/astral-sh/ruff)
|
|
72
|
+
|
|
73
|
+
## Contents
|
|
74
|
+
|
|
75
|
+
- [Features](#features)
|
|
76
|
+
- [New here? Three steps to your first download](#new-here-three-steps-to-your-first-download)
|
|
77
|
+
- [Installation](#installation)
|
|
78
|
+
- [Configure credentials](#configure-credentials)
|
|
79
|
+
- [Quick start](#quick-start)
|
|
80
|
+
- [Auto-download (coming soon)](#auto-download-coming-soon)
|
|
81
|
+
- [CLI](#cli)
|
|
82
|
+
- [Configuration](#configuration)
|
|
83
|
+
- [Logging](#logging)
|
|
84
|
+
- [Error handling](#error-handling)
|
|
85
|
+
- [Troubleshooting](#troubleshooting)
|
|
86
|
+
- [Date formats](#date-formats)
|
|
87
|
+
- [Performance tuning](#performance-tuning)
|
|
88
|
+
- [API reference (most-used methods)](#api-reference-most-used-methods)
|
|
89
|
+
- [Examples](#examples) · [Development](#development) · [Requirements](#requirements) · [Support](#support)
|
|
90
|
+
|
|
91
|
+
## Features
|
|
92
|
+
|
|
93
|
+
- **Streaming file downloads** — single streaming GET per file
|
|
94
|
+
- **Historical and date-range downloads** — fetch every file in a group (optionally filtered to one or many `file-group-id`s) between two dates
|
|
95
|
+
- **Notification-driven downloads (SSE)** — subscribe to the `/events/notification` stream and auto-download files as soon as they are published
|
|
96
|
+
- **Time-series queries** — by expression, by instrument, or by group with attribute / filter projections
|
|
97
|
+
- **OAuth 2.0 with token caching and refresh** — or supply a bearer token directly
|
|
98
|
+
- **Token-bucket rate limiter** — 300 rpm / 5 tps defaults (configurable up to API limits)
|
|
99
|
+
- **Retry + circuit breaker** — exponential backoff, configurable failure threshold
|
|
100
|
+
- **Sync and async APIs** — every operation has `_async` and sync variants
|
|
101
|
+
- **Optional pandas integration** — `to_dataframe(...)` on any response
|
|
102
|
+
- **CLI** — `dataquery groups | files | availability | download | download-group | auth | config`
|
|
103
|
+
|
|
104
|
+
## New here? Three steps to your first download
|
|
105
|
+
|
|
106
|
+
1. `pip install dataquery-sdk`
|
|
107
|
+
2. Put your OAuth credentials in a `.env` file (see [Configure credentials](#configure-credentials))
|
|
108
|
+
3. Run a one-liner to confirm everything works:
|
|
109
|
+
|
|
110
|
+
```python
|
|
111
|
+
import asyncio
|
|
112
|
+
from dataquery import DataQuery
|
|
113
|
+
|
|
114
|
+
async def main():
|
|
115
|
+
async with DataQuery() as dq:
|
|
116
|
+
groups = await dq.list_groups_async(limit=5)
|
|
117
|
+
for g in groups:
|
|
118
|
+
print(g.group_id, "—", g.group_name)
|
|
119
|
+
|
|
120
|
+
asyncio.run(main())
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
If that prints groups, auth and networking are working. From there, jump to
|
|
124
|
+
[Quick start](#quick-start) (date-range download) or the [CLI](#cli).
|
|
125
|
+
|
|
126
|
+
## Installation
|
|
127
|
+
|
|
128
|
+
```bash
|
|
129
|
+
# Core install
|
|
130
|
+
pip install dataquery-sdk
|
|
131
|
+
|
|
132
|
+
# With pandas DataFrame conversion
|
|
133
|
+
pip install "dataquery-sdk[pandas]"
|
|
134
|
+
|
|
135
|
+
# With dev tooling (ruff, mypy, pytest)
|
|
136
|
+
pip install "dataquery-sdk[dev]"
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
Python 3.11+ is required.
|
|
140
|
+
|
|
141
|
+
## Configure credentials
|
|
142
|
+
|
|
143
|
+
Set OAuth client credentials via environment variables:
|
|
144
|
+
|
|
145
|
+
```bash
|
|
146
|
+
export DATAQUERY_CLIENT_ID="your_client_id"
|
|
147
|
+
export DATAQUERY_CLIENT_SECRET="your_client_secret"
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
Or create a `.env` file in the working directory:
|
|
151
|
+
|
|
152
|
+
```env
|
|
153
|
+
DATAQUERY_CLIENT_ID=your_client_id
|
|
154
|
+
DATAQUERY_CLIENT_SECRET=your_client_secret
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
Or pass them directly to the constructor:
|
|
158
|
+
|
|
159
|
+
```python
|
|
160
|
+
from dataquery import DataQuery
|
|
161
|
+
|
|
162
|
+
dq = DataQuery(client_id="...", client_secret="...")
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
A starter `.env` can be generated with `dataquery config template --output .env`.
|
|
166
|
+
|
|
167
|
+
## Quick start
|
|
168
|
+
|
|
169
|
+
### Download files for a date range
|
|
170
|
+
|
|
171
|
+
```python
|
|
172
|
+
from dataquery import DataQuery
|
|
173
|
+
|
|
174
|
+
# async
|
|
175
|
+
async with DataQuery() as dq:
|
|
176
|
+
result = await dq.run_group_download_async(
|
|
177
|
+
group_id="JPMAQS_GENERIC_RETURNS",
|
|
178
|
+
start_date="20250101",
|
|
179
|
+
end_date="20250131",
|
|
180
|
+
destination_dir="./data",
|
|
181
|
+
)
|
|
182
|
+
# OperationReport (Pydantic model) — counts/timing/data/details are dicts on it.
|
|
183
|
+
print(f"{result.counts['successful_downloads']}/{result.counts['total_files']} files downloaded")
|
|
184
|
+
|
|
185
|
+
# sync — same arguments, drop the _async suffix
|
|
186
|
+
with DataQuery() as dq:
|
|
187
|
+
result = dq.run_group_download(
|
|
188
|
+
group_id="JPMAQS_GENERIC_RETURNS",
|
|
189
|
+
start_date="20250101",
|
|
190
|
+
end_date="20250131",
|
|
191
|
+
destination_dir="./data",
|
|
192
|
+
)
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
### Restrict a date-range download to specific file-group-ids
|
|
196
|
+
|
|
197
|
+
`file_group_id` accepts a single id or a list. When a list is supplied, availability
|
|
198
|
+
queries run in parallel per id and the union of dates is downloaded.
|
|
199
|
+
|
|
200
|
+
```python
|
|
201
|
+
async with DataQuery() as dq:
|
|
202
|
+
result = await dq.run_group_download_async(
|
|
203
|
+
group_id="JPMAQS_GENERIC_RETURNS",
|
|
204
|
+
start_date="20250101",
|
|
205
|
+
end_date="20250131",
|
|
206
|
+
destination_dir="./data",
|
|
207
|
+
file_group_id=["FG_ABC", "FG_DEF", "FG_XYZ"],
|
|
208
|
+
)
|
|
209
|
+
```
|
|
210
|
+
|
|
211
|
+
### Download a single file
|
|
212
|
+
|
|
213
|
+
```python
|
|
214
|
+
from pathlib import Path
|
|
215
|
+
from dataquery import DataQuery
|
|
216
|
+
|
|
217
|
+
async with DataQuery() as dq:
|
|
218
|
+
result = await dq.download_file_async(
|
|
219
|
+
file_group_id="JPMAQS_GENERIC_RETURNS",
|
|
220
|
+
file_datetime="20250115",
|
|
221
|
+
destination_path=Path("./downloads"),
|
|
222
|
+
)
|
|
223
|
+
print(f"Downloaded: {result.local_path} ({result.file_size} bytes)")
|
|
224
|
+
```
|
|
225
|
+
|
|
226
|
+
### Time-series queries
|
|
227
|
+
|
|
228
|
+
```python
|
|
229
|
+
async with DataQuery() as dq:
|
|
230
|
+
# By expression
|
|
231
|
+
ts = await dq.get_expressions_time_series_async(
|
|
232
|
+
expressions=["DB(MTE,IRISH EUR 1.100 15-May-2029 LON,,IE00BH3SQ895,MIDPRC)"],
|
|
233
|
+
start_date="20240101",
|
|
234
|
+
end_date="20240131",
|
|
235
|
+
)
|
|
236
|
+
|
|
237
|
+
# By group with attributes + filter
|
|
238
|
+
ts = await dq.get_group_time_series_async(
|
|
239
|
+
group_id="FI_GO_BO_EA",
|
|
240
|
+
attributes=["MIDPRC", "REPO_1M"],
|
|
241
|
+
filter="country(IRL)",
|
|
242
|
+
start_date="20240101",
|
|
243
|
+
end_date="20240131",
|
|
244
|
+
)
|
|
245
|
+
|
|
246
|
+
df = dq.to_dataframe(ts) # requires pandas extra
|
|
247
|
+
```
|
|
248
|
+
|
|
249
|
+
### Discover available data
|
|
250
|
+
|
|
251
|
+
```python
|
|
252
|
+
async with DataQuery() as dq:
|
|
253
|
+
groups = await dq.list_groups_async(limit=100)
|
|
254
|
+
files = await dq.list_files_async(group_id="JPMAQS_GENERIC_RETURNS")
|
|
255
|
+
available = await dq.list_available_files_async(
|
|
256
|
+
group_id="JPMAQS_GENERIC_RETURNS",
|
|
257
|
+
start_date="20250101",
|
|
258
|
+
end_date="20250131",
|
|
259
|
+
)
|
|
260
|
+
instruments = await dq.search_instruments_async(
|
|
261
|
+
group_id="FI_GO_BO_EA", keywords="irish"
|
|
262
|
+
)
|
|
263
|
+
```
|
|
264
|
+
|
|
265
|
+
## Auto-download (coming soon)
|
|
266
|
+
|
|
267
|
+
Real-time notification-driven downloads via the DataQuery SSE stream are under
|
|
268
|
+
active development and will be available in a future release. Once released,
|
|
269
|
+
`auto_download_async` will subscribe to the notification endpoint and download
|
|
270
|
+
files automatically as they are published — no polling required.
|
|
271
|
+
|
|
272
|
+
## CLI
|
|
273
|
+
|
|
274
|
+
The installer registers a `dataquery` script:
|
|
275
|
+
|
|
276
|
+
```bash
|
|
277
|
+
# List / search groups
|
|
278
|
+
dataquery groups --limit 100
|
|
279
|
+
dataquery groups --search "fixed income" --json
|
|
280
|
+
|
|
281
|
+
# List files in a group
|
|
282
|
+
dataquery files --group-id JPMAQS_GENERIC_RETURNS --json
|
|
283
|
+
|
|
284
|
+
# Check availability for a single file
|
|
285
|
+
dataquery availability --file-group-id JPMAQS_GENERIC_RETURNS --file-datetime 20250115
|
|
286
|
+
|
|
287
|
+
# Download a single file
|
|
288
|
+
dataquery download --file-group-id JPMAQS_GENERIC_RETURNS \
|
|
289
|
+
--file-datetime 20250115 \
|
|
290
|
+
--destination ./downloads
|
|
291
|
+
|
|
292
|
+
# Watch a group and download as new files arrive (calls auto_download_async under the hood —
|
|
293
|
+
# same SSE subscription, same event-replay state files)
|
|
294
|
+
dataquery download --watch --group-id JPMAQS_GENERIC_RETURNS --destination ./downloads
|
|
295
|
+
|
|
296
|
+
# Download everything in a date range
|
|
297
|
+
dataquery download-group --group-id JPMAQS_GENERIC_RETURNS \
|
|
298
|
+
--start-date 20250101 --end-date 20250131 \
|
|
299
|
+
--destination ./data
|
|
300
|
+
|
|
301
|
+
# Restrict to one or more file-group-ids
|
|
302
|
+
dataquery download-group --group-id JPMAQS_GENERIC_RETURNS \
|
|
303
|
+
--file-group-id FG_ABC FG_DEF \
|
|
304
|
+
--start-date 20250101 --end-date 20250131
|
|
305
|
+
|
|
306
|
+
# Config utilities
|
|
307
|
+
dataquery config show
|
|
308
|
+
dataquery config validate
|
|
309
|
+
dataquery config template --output .env
|
|
310
|
+
|
|
311
|
+
# Verify auth
|
|
312
|
+
dataquery auth test
|
|
313
|
+
```
|
|
314
|
+
|
|
315
|
+
Every subcommand accepts `--env-file PATH` (to point at a non-default `.env`) and
|
|
316
|
+
most accept `--json` for machine-readable output.
|
|
317
|
+
|
|
318
|
+
## Configuration
|
|
319
|
+
|
|
320
|
+
### Environment variables
|
|
321
|
+
|
|
322
|
+
All environment variables use the `DATAQUERY_` prefix.
|
|
323
|
+
|
|
324
|
+
**Credentials**
|
|
325
|
+
|
|
326
|
+
| Variable | Default | Notes |
|
|
327
|
+
|---|---|---|
|
|
328
|
+
| `DATAQUERY_CLIENT_ID` | _(none)_ | Required for OAuth |
|
|
329
|
+
| `DATAQUERY_CLIENT_SECRET` | _(none)_ | Required for OAuth |
|
|
330
|
+
| `DATAQUERY_BEARER_TOKEN` | _(none)_ | Alternative to OAuth |
|
|
331
|
+
| `DATAQUERY_OAUTH_ENABLED` | `true` | Set `false` to use bearer-token mode |
|
|
332
|
+
|
|
333
|
+
**API endpoints**
|
|
334
|
+
|
|
335
|
+
| Variable | Default |
|
|
336
|
+
|---|---|
|
|
337
|
+
| `DATAQUERY_BASE_URL` | `https://api-developer.jpmorgan.com` |
|
|
338
|
+
| `DATAQUERY_FILES_BASE_URL` | `https://api-dataquery.jpmchase.com` |
|
|
339
|
+
| `DATAQUERY_CONTEXT_PATH` | `/research/dataquery-authe/api/v2` |
|
|
340
|
+
|
|
341
|
+
**HTTP / retry / rate limit**
|
|
342
|
+
|
|
343
|
+
| Variable | Default |
|
|
344
|
+
|---|---|
|
|
345
|
+
| `DATAQUERY_TIMEOUT` | `600.0` |
|
|
346
|
+
| `DATAQUERY_MAX_RETRIES` | `3` |
|
|
347
|
+
| `DATAQUERY_RETRY_DELAY` | `1.0` |
|
|
348
|
+
| `DATAQUERY_CIRCUIT_BREAKER_THRESHOLD` | `5` |
|
|
349
|
+
| `DATAQUERY_REQUESTS_PER_MINUTE` | `300` |
|
|
350
|
+
| `DATAQUERY_BURST_CAPACITY` | `5` |
|
|
351
|
+
| `DATAQUERY_POOL_CONNECTIONS` | `10` |
|
|
352
|
+
| `DATAQUERY_POOL_MAXSIZE` | `20` |
|
|
353
|
+
|
|
354
|
+
**Proxy** (optional)
|
|
355
|
+
|
|
356
|
+
`DATAQUERY_PROXY_ENABLED`, `DATAQUERY_PROXY_URL`, `DATAQUERY_PROXY_USERNAME`,
|
|
357
|
+
`DATAQUERY_PROXY_PASSWORD`, `DATAQUERY_PROXY_VERIFY_SSL`.
|
|
358
|
+
|
|
359
|
+
### Programmatic configuration
|
|
360
|
+
|
|
361
|
+
```python
|
|
362
|
+
from dataquery import ClientConfig, DataQuery
|
|
363
|
+
|
|
364
|
+
config = ClientConfig(
|
|
365
|
+
client_id="...",
|
|
366
|
+
client_secret="...",
|
|
367
|
+
base_url="https://api-developer.jpmorgan.com",
|
|
368
|
+
timeout=60.0,
|
|
369
|
+
max_retries=3,
|
|
370
|
+
requests_per_minute=300,
|
|
371
|
+
)
|
|
372
|
+
|
|
373
|
+
async with DataQuery(config) as dq:
|
|
374
|
+
...
|
|
375
|
+
|
|
376
|
+
# Or pass overrides as kwargs on top of env/.env resolution:
|
|
377
|
+
async with DataQuery(client_id="...", client_secret="...", timeout=60.0) as dq:
|
|
378
|
+
...
|
|
379
|
+
```
|
|
380
|
+
|
|
381
|
+
## Logging
|
|
382
|
+
|
|
383
|
+
The SDK logs through [structlog](https://www.structlog.org/) and emits
|
|
384
|
+
structured events for requests, retries, rate-limit waits, SSE reconnects, and
|
|
385
|
+
download progress. Two ways to drive it:
|
|
386
|
+
|
|
387
|
+
**Standard Python logging** — works without extra setup:
|
|
388
|
+
|
|
389
|
+
```python
|
|
390
|
+
import logging
|
|
391
|
+
logging.basicConfig(level=logging.INFO)
|
|
392
|
+
logging.getLogger("dataquery").setLevel(logging.DEBUG) # SDK-only DEBUG
|
|
393
|
+
```
|
|
394
|
+
|
|
395
|
+
DEBUG includes per-chunk download progress and SSE keepalives — useful while
|
|
396
|
+
debugging, noisy in production.
|
|
397
|
+
|
|
398
|
+
**Structured (JSON) output** — recommended for long-running `auto_download`
|
|
399
|
+
services so a log shipper can parse the events:
|
|
400
|
+
|
|
401
|
+
```python
|
|
402
|
+
from pathlib import Path
|
|
403
|
+
from dataquery.config.logging import (
|
|
404
|
+
LogFormat, LogLevel, create_logging_config, create_logging_manager,
|
|
405
|
+
)
|
|
406
|
+
|
|
407
|
+
cfg = create_logging_config(
|
|
408
|
+
level=LogLevel.INFO,
|
|
409
|
+
format=LogFormat.JSON, # or LogFormat.CONSOLE for humans
|
|
410
|
+
enable_file=True,
|
|
411
|
+
log_file=Path("./dataquery.log"),
|
|
412
|
+
enable_request_logging=False, # set True to log every HTTP request/response
|
|
413
|
+
)
|
|
414
|
+
create_logging_manager(cfg) # installs handlers on the root logger
|
|
415
|
+
```
|
|
416
|
+
|
|
417
|
+
`examples/system/enable_request_logging.py` is a runnable version showing
|
|
418
|
+
request/response logging for traffic debugging.
|
|
419
|
+
|
|
420
|
+
**Health snapshots for `auto_download`** — `manager.get_stats()` returns
|
|
421
|
+
notifications received, files downloaded / skipped / failed, the last event
|
|
422
|
+
id, and a bounded ring of recent errors. Wire it into a `/healthz` endpoint
|
|
423
|
+
for daemon-style deployments.
|
|
424
|
+
|
|
425
|
+
## Error handling
|
|
426
|
+
|
|
427
|
+
All errors inherit from `DataQueryError`:
|
|
428
|
+
|
|
429
|
+
```python
|
|
430
|
+
from dataquery import DataQuery
|
|
431
|
+
from dataquery.types.exceptions import (
|
|
432
|
+
DataQueryError,
|
|
433
|
+
AuthenticationError,
|
|
434
|
+
NotFoundError,
|
|
435
|
+
RateLimitError,
|
|
436
|
+
NetworkError,
|
|
437
|
+
DownloadError,
|
|
438
|
+
ConfigurationError,
|
|
439
|
+
)
|
|
440
|
+
|
|
441
|
+
async with DataQuery() as dq:
|
|
442
|
+
try:
|
|
443
|
+
ts = await dq.get_expressions_time_series_async(
|
|
444
|
+
expressions=["DB(...)"], start_date="20240101", end_date="20240131"
|
|
445
|
+
)
|
|
446
|
+
except AuthenticationError:
|
|
447
|
+
... # check credentials
|
|
448
|
+
except RateLimitError:
|
|
449
|
+
... # back off — SDK already retried
|
|
450
|
+
except NotFoundError:
|
|
451
|
+
... # group / file / instrument not found
|
|
452
|
+
except NetworkError:
|
|
453
|
+
... # transient; SDK already retried
|
|
454
|
+
except DataQueryError:
|
|
455
|
+
... # any other SDK-level failure
|
|
456
|
+
```
|
|
457
|
+
|
|
458
|
+
## Troubleshooting
|
|
459
|
+
|
|
460
|
+
**`AuthenticationError` / HTTP 401 on the first call**
|
|
461
|
+
|
|
462
|
+
- Verify both `DATAQUERY_CLIENT_ID` and `DATAQUERY_CLIENT_SECRET` are set:
|
|
463
|
+
`dataquery config show` will print the resolved config (secrets masked).
|
|
464
|
+
- Confirm OAuth is reaching the right endpoint:
|
|
465
|
+
`dataquery auth test` performs a token exchange and reports the failure mode.
|
|
466
|
+
- If the credentials are correct but the audience is wrong, set
|
|
467
|
+
`DATAQUERY_OAUTH_AUD` to the value provisioned for your client.
|
|
468
|
+
|
|
469
|
+
**`.env` file isn't picked up**
|
|
470
|
+
|
|
471
|
+
- The SDK looks for `.env` in the current working directory at instantiation.
|
|
472
|
+
Either `cd` to the directory containing `.env` before running, or pass the
|
|
473
|
+
path explicitly: `DataQuery(env_file=".env.production")`.
|
|
474
|
+
- Variables already set in the shell environment win over the `.env` file —
|
|
475
|
+
unset them (`unset DATAQUERY_CLIENT_ID`) if you want the file to take effect.
|
|
476
|
+
- The CLI accepts `--env-file PATH` on every subcommand for the same reason.
|
|
477
|
+
|
|
478
|
+
**Connection / proxy / SSL failures**
|
|
479
|
+
|
|
480
|
+
- Behind a corporate proxy, set `DATAQUERY_PROXY_ENABLED=true` and
|
|
481
|
+
`DATAQUERY_PROXY_URL=http://proxy.host:port`. Add `DATAQUERY_PROXY_USERNAME`
|
|
482
|
+
/ `DATAQUERY_PROXY_PASSWORD` if auth is required.
|
|
483
|
+
- For self-signed proxy CAs, set `DATAQUERY_PROXY_VERIFY_SSL=false` (insecure;
|
|
484
|
+
prefer pointing `SSL_CERT_FILE` at the corporate root CA bundle).
|
|
485
|
+
- Sporadic `NetworkError` after long idle periods usually means a stateful
|
|
486
|
+
middlebox is dropping the SSE socket — set
|
|
487
|
+
`heartbeat_timeout=90.0` on `auto_download_async` to force a reconnect when
|
|
488
|
+
no bytes arrive within the window.
|
|
489
|
+
|
|
490
|
+
**Rate-limit pauses**
|
|
491
|
+
|
|
492
|
+
- Default is 300 rpm / 5 tps. The SDK self-throttles via the token-bucket
|
|
493
|
+
limiter; if you see long sleeps before requests, lower
|
|
494
|
+
`DATAQUERY_REQUESTS_PER_MINUTE` is not the cure — it's likely working as
|
|
495
|
+
designed. Raise it (up to your provisioned limit) to go faster.
|
|
496
|
+
- `dq.get_rate_limit_info()` shows the current bucket state.
|
|
497
|
+
|
|
498
|
+
**SSE auto-download "missed" events after a restart**
|
|
499
|
+
|
|
500
|
+
- Confirm `enable_event_replay=True` (the default).
|
|
501
|
+
- Replay state lives under `<destination>/.sse_state/sse_<fingerprint>.json` —
|
|
502
|
+
if that directory was wiped, the next start has nothing to resume from. Use
|
|
503
|
+
`manager.clear_event_id()` (or `dataquery download --watch --reset-event-id`)
|
|
504
|
+
only when you intentionally want a clean slate.
|
|
505
|
+
|
|
506
|
+
## Date formats
|
|
507
|
+
|
|
508
|
+
```python
|
|
509
|
+
start_date="20240101" # absolute, YYYYMMDD
|
|
510
|
+
start_date="TODAY"
|
|
511
|
+
start_date="TODAY-1D" # yesterday
|
|
512
|
+
start_date="TODAY-1W"
|
|
513
|
+
start_date="TODAY-1M"
|
|
514
|
+
start_date="TODAY-1Y"
|
|
515
|
+
```
|
|
516
|
+
|
|
517
|
+
## Performance tuning
|
|
518
|
+
|
|
519
|
+
`run_group_download_async` streams each file as a single GET. The SDK
|
|
520
|
+
automatically inserts delays between file starts so the configured
|
|
521
|
+
`requests_per_minute` is not exceeded.
|
|
522
|
+
|
|
523
|
+
```python
|
|
524
|
+
await dq.run_group_download_async(
|
|
525
|
+
group_id="JPMAQS_GENERIC_RETURNS",
|
|
526
|
+
start_date="20250101",
|
|
527
|
+
end_date="20250131",
|
|
528
|
+
destination_dir="./data",
|
|
529
|
+
max_retries=3,
|
|
530
|
+
)
|
|
531
|
+
```
|
|
532
|
+
|
|
533
|
+
Tune throughput via `DATAQUERY_REQUESTS_PER_MINUTE` and `DATAQUERY_BURST_CAPACITY`
|
|
534
|
+
(see [Configuration](#environment-variables)) rather than per-call concurrency
|
|
535
|
+
flags.
|
|
536
|
+
|
|
537
|
+
## API reference (most-used methods)
|
|
538
|
+
|
|
539
|
+
| Area | Method | Notes |
|
|
540
|
+
|---|---|---|
|
|
541
|
+
| Discovery | `list_groups_async(limit)` | |
|
|
542
|
+
| | `search_groups_async(keywords, limit, offset)` | |
|
|
543
|
+
| | `list_files_async(group_id, file_group_id=None)` | |
|
|
544
|
+
| | `list_available_files_async(group_id, file_group_id, start_date, end_date)` | |
|
|
545
|
+
| | `list_instruments_async(group_id, instrument_id=None, page=None)` | |
|
|
546
|
+
| | `search_instruments_async(group_id, keywords, page=None)` | |
|
|
547
|
+
| | `get_group_attributes_async(group_id, ...)` | |
|
|
548
|
+
| | `get_group_filters_async(group_id, page=None)` | |
|
|
549
|
+
| Downloads | `download_file_async(file_group_id, file_datetime, ...)` | single file |
|
|
550
|
+
| | `run_group_download_async(group_id, start_date, end_date, file_group_id=None, ...)` | date range, single or list of ids |
|
|
551
|
+
| | `download_historical_async(...)` | chunked historical backfill |
|
|
552
|
+
| | `auto_download_async(group_id, ...)` | SSE notifications (the only watch path) |
|
|
553
|
+
| Time series | `get_expressions_time_series_async(expressions, start_date, end_date)` | |
|
|
554
|
+
| | `get_instrument_time_series_async(instruments, attributes, start_date, end_date)` | |
|
|
555
|
+
| | `get_group_time_series_async(group_id, attributes, filter, start_date, end_date)` | |
|
|
556
|
+
| Grid data | `get_grid_data_async(...)` | |
|
|
557
|
+
| Utilities | `check_availability_async(file_group_id, file_datetime)` | |
|
|
558
|
+
| | `health_check_async()` | |
|
|
559
|
+
| | `to_dataframe(response)` | requires `pandas` extra |
|
|
560
|
+
| | `get_stats()` / `get_pool_stats()` / `get_rate_limit_info()` | diagnostics |
|
|
561
|
+
|
|
562
|
+
Every async method has a sync counterpart with the same name minus the
|
|
563
|
+
`_async` suffix — `list_groups_async` ↔ `list_groups`, `download_file_async` ↔
|
|
564
|
+
`download_file`, etc. Sync calls run the coroutine internally via
|
|
565
|
+
`asyncio.run`, so do not call them from inside an existing event loop.
|
|
566
|
+
|
|
567
|
+
## Examples
|
|
568
|
+
|
|
569
|
+
The `examples/` directory is organised by feature:
|
|
570
|
+
|
|
571
|
+
- `examples/files/` — single-file and date-range downloads
|
|
572
|
+
- `examples/expressions/` — expression time series
|
|
573
|
+
- `examples/instruments/` — instrument discovery + time series
|
|
574
|
+
- `examples/groups/` and `examples/groups_advanced/` — group discovery and time series
|
|
575
|
+
- `examples/grid/` — grid data
|
|
576
|
+
- `examples/system/` — SSE notification subscriber (single + multi-group), diagnostics
|
|
577
|
+
|
|
578
|
+
Run any example directly:
|
|
579
|
+
|
|
580
|
+
```bash
|
|
581
|
+
python examples/files/download_file.py
|
|
582
|
+
python examples/system/auto_download_example.py # single group
|
|
583
|
+
python examples/system/auto_download_multi_group_example.py # several groups in parallel
|
|
584
|
+
```
|
|
585
|
+
|
|
586
|
+
## Development
|
|
587
|
+
|
|
588
|
+
```bash
|
|
589
|
+
# Clone
|
|
590
|
+
git clone https://github.com/jpmorganchase/dataquery-sdk.git
|
|
591
|
+
cd dataquery-sdk
|
|
592
|
+
|
|
593
|
+
# Install with dev + all extras
|
|
594
|
+
uv sync --all-extras --dev # using uv
|
|
595
|
+
# or
|
|
596
|
+
pip install -e ".[dev,pandas]"
|
|
597
|
+
|
|
598
|
+
# Run tests
|
|
599
|
+
pytest tests/ -v
|
|
600
|
+
pytest tests/ --cov=dataquery --cov-report=term-missing
|
|
601
|
+
|
|
602
|
+
# Lint / format / type-check
|
|
603
|
+
ruff check dataquery/ tests/ examples/
|
|
604
|
+
ruff format dataquery/ tests/ examples/
|
|
605
|
+
mypy dataquery/
|
|
606
|
+
```
|
|
607
|
+
|
|
608
|
+
Pytest markers: `slow`, `integration`, `unit`, `asyncio`.
|
|
609
|
+
|
|
610
|
+
## Requirements
|
|
611
|
+
|
|
612
|
+
- Python 3.11+
|
|
613
|
+
- `aiohttp>=3.8,<4`, `pydantic>=2,<3`, `structlog>=23`, `python-dotenv>=1`
|
|
614
|
+
- Optional: `pandas>=2` (for `to_dataframe`)
|
|
615
|
+
|
|
616
|
+
## Support
|
|
617
|
+
|
|
618
|
+
- GitHub Issues: <https://github.com/jpmorganchase/dataquery-sdk/issues>
|
|
619
|
+
- Email: dataquery_support@jpmorgan.com
|
|
620
|
+
|
|
621
|
+
## License
|
|
622
|
+
|
|
623
|
+
MIT — see [LICENSE](LICENSE).
|
|
624
|
+
|
|
625
|
+
## Changelog
|
|
626
|
+
|
|
627
|
+
See [docs/changelog.md](docs/changelog.md).
|