binance-datatool 0.1.0__tar.gz → 0.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- binance_datatool-0.2.0/.github/workflows/ci.yml +38 -0
- binance_datatool-0.2.0/.github/workflows/integration.yml +29 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/.github/workflows/publish.yml +24 -4
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/.gitignore +0 -1
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/.pre-commit-config.yaml +1 -1
- binance_datatool-0.2.0/AGENTS.md +31 -0
- binance_datatool-0.2.0/CLAUDE.md +1 -0
- binance_datatool-0.2.0/GLOSSARY.md +53 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/PKG-INFO +75 -37
- binance_datatool-0.2.0/README.md +181 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/README.md +3 -1
- binance_datatool-0.2.0/docs/adr/0001-public-api-conveniences.md +11 -0
- binance_datatool-0.2.0/docs/agents/coding-standards.md +58 -0
- binance_datatool-0.2.0/docs/agents/domain.md +13 -0
- binance_datatool-0.2.0/docs/agents/issue-tracker.md +50 -0
- binance_datatool-0.2.0/docs/agents/triage-labels.md +15 -0
- binance_datatool-0.2.0/docs/architecture.md +37 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/extending.md +18 -29
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/README.md +5 -4
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/archive/README.md +4 -5
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/archive/checksum.md +4 -2
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/archive/client.md +10 -4
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/archive/downloader.md +31 -3
- binance_datatool-0.2.0/docs/reference/archive/path-structure.md +210 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/archive/s3-protocol.md +8 -4
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/archive/symbol_dir.md +35 -5
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/cli/README.md +1 -1
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/cli/archive.md +261 -54
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/common/constants.md +1 -1
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/common/enums.md +3 -3
- {binance_datatool-0.1.0/docs/reference/archive → binance_datatool-0.2.0/docs/reference/common}/filter.md +7 -7
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/common/logging.md +3 -1
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/common/path.md +40 -1
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/common/progress.md +16 -4
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/common/symbols.md +5 -2
- binance_datatool-0.2.0/docs/reference/performance.md +141 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/testing.md +29 -2
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/workflow/README.md +123 -43
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/pyproject.toml +1 -2
- binance_datatool-0.2.0/skills/binance-datatool/SKILL.md +424 -0
- binance_datatool-0.2.0/skills-lock.json +167 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/__init__.py +1 -1
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/archive/__init__.py +2 -12
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/archive/checksum.py +10 -5
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/archive/client.py +86 -87
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/archive/downloader.py +75 -8
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/archive/symbol_dir.py +124 -19
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/cli/archive.py +234 -108
- binance_datatool-0.2.0/src/binance_datatool/cli/output.py +32 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/common/__init__.py +12 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/common/constants.py +3 -3
- {binance_datatool-0.1.0/src/binance_datatool/archive → binance_datatool-0.2.0/src/binance_datatool/common}/filter.py +4 -3
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/common/logging.py +7 -1
- binance_datatool-0.2.0/src/binance_datatool/common/path.py +91 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/common/progress.py +9 -121
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/common/symbols.py +4 -4
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/workflow/_shared.py +1 -11
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/workflow/download.py +50 -8
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/workflow/list_files.py +1 -1
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/workflow/list_symbols.py +6 -4
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/workflow/results.py +3 -6
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/workflow/verify.py +93 -99
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/conftest.py +45 -0
- binance_datatool-0.2.0/tests/test_archive_client.py +618 -0
- binance_datatool-0.2.0/tests/test_archive_workflow.py +1554 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/test_checksum.py +24 -1
- binance_datatool-0.2.0/tests/test_cli.py +2706 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/test_downloader.py +136 -1
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/test_filter.py +84 -3
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/test_path.py +1 -3
- binance_datatool-0.2.0/tests/test_progress.py +93 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/test_symbol_dir.py +40 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/test_symbols.py +186 -3
- binance_datatool-0.2.0/tests/test_verify_interrupt.py +396 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/uv.lock +3 -14
- binance_datatool-0.1.0/AGENTS.md +0 -85
- binance_datatool-0.1.0/CLAUDE.md +0 -1
- binance_datatool-0.1.0/README.md +0 -141
- binance_datatool-0.1.0/docs/architecture.md +0 -83
- binance_datatool-0.1.0/src/binance_datatool/common/path.py +0 -40
- binance_datatool-0.1.0/tests/test_archive_client.py +0 -404
- binance_datatool-0.1.0/tests/test_archive_workflow.py +0 -796
- binance_datatool-0.1.0/tests/test_cli.py +0 -1024
- binance_datatool-0.1.0/tests/test_progress.py +0 -28
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/LICENSE +0 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/common/types.md +0 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/examples/download_spot_klines_usdt_1m.sh +0 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/examples/download_um_funding_usdt_perpetual.sh +0 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/examples/download_um_klines_usdt_1m.sh +0 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/cli/__init__.py +0 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/common/enums.py +0 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/common/types.py +0 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/py.typed +0 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/workflow/__init__.py +0 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/data/BTCUSDT-1m-2020-01-01.zip +0 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/data/BTCUSDT-1m-2020-01-01.zip.CHECKSUM +0 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/data/BTCUSDT-1m-2020-01-02.zip +0 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/data/BTCUSDT-1m-2020-01-02.zip.CHECKSUM +0 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/data/BTCUSDT-1m-2020-01-03.zip +0 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/data/BTCUSDT-1m-2020-01-03.zip.CHECKSUM +0 -0
- {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/test_enums.py +0 -0
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
name: CI
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
pull_request:
|
|
6
|
+
|
|
7
|
+
permissions:
|
|
8
|
+
contents: read
|
|
9
|
+
|
|
10
|
+
jobs:
|
|
11
|
+
checks:
|
|
12
|
+
name: Python ${{ matrix.python-version }}
|
|
13
|
+
runs-on: ubuntu-latest
|
|
14
|
+
strategy:
|
|
15
|
+
fail-fast: false
|
|
16
|
+
matrix:
|
|
17
|
+
python-version: ["3.11", "3.12", "3.13", "3.14"]
|
|
18
|
+
|
|
19
|
+
steps:
|
|
20
|
+
- uses: actions/checkout@v6
|
|
21
|
+
|
|
22
|
+
- uses: actions/setup-python@v6
|
|
23
|
+
with:
|
|
24
|
+
python-version: ${{ matrix.python-version }}
|
|
25
|
+
|
|
26
|
+
- uses: astral-sh/setup-uv@v7
|
|
27
|
+
|
|
28
|
+
- name: Install dependencies
|
|
29
|
+
run: uv sync --locked
|
|
30
|
+
|
|
31
|
+
- name: Lint
|
|
32
|
+
run: uv run ruff check .
|
|
33
|
+
|
|
34
|
+
- name: Check formatting
|
|
35
|
+
run: uv run ruff format --check .
|
|
36
|
+
|
|
37
|
+
- name: Run unit tests
|
|
38
|
+
run: uv run pytest
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
name: Weekly integration checks
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
schedule:
|
|
5
|
+
- cron: "17 3 * * 1"
|
|
6
|
+
workflow_dispatch:
|
|
7
|
+
|
|
8
|
+
permissions:
|
|
9
|
+
contents: read
|
|
10
|
+
|
|
11
|
+
jobs:
|
|
12
|
+
integration:
|
|
13
|
+
runs-on: ubuntu-latest
|
|
14
|
+
timeout-minutes: 30
|
|
15
|
+
|
|
16
|
+
steps:
|
|
17
|
+
- uses: actions/checkout@v6
|
|
18
|
+
|
|
19
|
+
- uses: actions/setup-python@v6
|
|
20
|
+
with:
|
|
21
|
+
python-version: "3.13"
|
|
22
|
+
|
|
23
|
+
- uses: astral-sh/setup-uv@v7
|
|
24
|
+
|
|
25
|
+
- name: Install dependencies
|
|
26
|
+
run: uv sync --locked
|
|
27
|
+
|
|
28
|
+
- name: Run live integration and archive symbol drift checks
|
|
29
|
+
run: uv run pytest --run-integration
|
|
@@ -8,13 +8,33 @@ permissions:
|
|
|
8
8
|
contents: read
|
|
9
9
|
|
|
10
10
|
jobs:
|
|
11
|
+
unit-tests:
|
|
12
|
+
runs-on: ubuntu-latest
|
|
13
|
+
|
|
14
|
+
steps:
|
|
15
|
+
- uses: actions/checkout@v6
|
|
16
|
+
|
|
17
|
+
- uses: actions/setup-python@v6
|
|
18
|
+
with:
|
|
19
|
+
python-version: "3.x"
|
|
20
|
+
|
|
21
|
+
- uses: astral-sh/setup-uv@v7
|
|
22
|
+
|
|
23
|
+
- name: Install dependencies
|
|
24
|
+
run: uv sync --locked
|
|
25
|
+
|
|
26
|
+
- name: Run unit tests
|
|
27
|
+
run: uv run pytest
|
|
28
|
+
|
|
11
29
|
release-build:
|
|
12
30
|
runs-on: ubuntu-latest
|
|
31
|
+
needs:
|
|
32
|
+
- unit-tests
|
|
13
33
|
|
|
14
34
|
steps:
|
|
15
|
-
- uses: actions/checkout@
|
|
35
|
+
- uses: actions/checkout@v6
|
|
16
36
|
|
|
17
|
-
- uses: actions/setup-python@
|
|
37
|
+
- uses: actions/setup-python@v6
|
|
18
38
|
with:
|
|
19
39
|
python-version: "3.x"
|
|
20
40
|
|
|
@@ -24,7 +44,7 @@ jobs:
|
|
|
24
44
|
python -m build
|
|
25
45
|
|
|
26
46
|
- name: Upload distributions
|
|
27
|
-
uses: actions/upload-artifact@
|
|
47
|
+
uses: actions/upload-artifact@v7
|
|
28
48
|
with:
|
|
29
49
|
name: release-dists
|
|
30
50
|
path: dist/
|
|
@@ -43,7 +63,7 @@ jobs:
|
|
|
43
63
|
|
|
44
64
|
steps:
|
|
45
65
|
- name: Retrieve release distributions
|
|
46
|
-
uses: actions/download-artifact@
|
|
66
|
+
uses: actions/download-artifact@v8
|
|
47
67
|
with:
|
|
48
68
|
name: release-dists
|
|
49
69
|
path: dist/
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
# AGENTS.md
|
|
2
|
+
|
|
3
|
+
## Project boundaries
|
|
4
|
+
|
|
5
|
+
- Before making changes, read `.AGENTS.local.md` at the repository root if it exists.
|
|
6
|
+
- This package manages historical archive discovery, downloading, local storage, and SHA256
|
|
7
|
+
verification. Checked-in source, tests, and configuration establish current behavior when
|
|
8
|
+
documentation conflicts with them.
|
|
9
|
+
- Use English for documentation, comments, docstrings, logs, CLI text, and developer-facing text.
|
|
10
|
+
Keep Python code compatible with Python 3.11.
|
|
11
|
+
- Keep `temp/` materials outside the public package surface and project documentation.
|
|
12
|
+
- Update the relevant documentation when behavior, public interfaces, or contributor workflows
|
|
13
|
+
change. Use one Conventional Commit per ticket.
|
|
14
|
+
|
|
15
|
+
## Task guidance
|
|
16
|
+
|
|
17
|
+
- Before changing Python code or tests, read [coding standards](docs/agents/coding-standards.md).
|
|
18
|
+
- Before structural or cross-layer changes, read [architecture](docs/architecture.md).
|
|
19
|
+
- Before adding a command, workflow, enum member, or command group, read
|
|
20
|
+
[extending the project](docs/extending.md).
|
|
21
|
+
- Before changing tests, fixtures, or test layout, read [test organization](docs/reference/testing.md).
|
|
22
|
+
- For module and CLI behavior, start at the [reference index](docs/reference/README.md).
|
|
23
|
+
- When creating or editing Markdown, use the installed
|
|
24
|
+
[writing-for-agents skill](.agents/skills/writing-for-agents/SKILL.md).
|
|
25
|
+
- When defining or changing domain terminology or recording architectural decisions, use the
|
|
26
|
+
installed [domain-modeling skill](.agents/skills/domain-modeling/SKILL.md).
|
|
27
|
+
Before domain work, read the [domain guide](docs/agents/domain.md) and its glossary/ADR pointers.
|
|
28
|
+
- When reading or updating issues and specs, follow the [issue tracker guide](docs/agents/issue-tracker.md).
|
|
29
|
+
- When triaging issues, use the [triage label mapping](docs/agents/triage-labels.md).
|
|
30
|
+
|
|
31
|
+
For agent skill setup in a fresh checkout, see [agent development](README.md#agent-development).
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
@AGENTS.md
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
# Historical archives
|
|
2
|
+
|
|
3
|
+
The domain covers discovering Binance historical archives, downloading them to local storage,
|
|
4
|
+
and verifying their integrity against SHA256 checksums.
|
|
5
|
+
|
|
6
|
+
## Language
|
|
7
|
+
|
|
8
|
+
**TradeType**:
|
|
9
|
+
The Binance market classification to which an archive belongs.
|
|
10
|
+
_Avoid_: Exchange type.
|
|
11
|
+
|
|
12
|
+
**DataFrequency**:
|
|
13
|
+
The archive partition frequency: the calendar period grouped into an archive file.
|
|
14
|
+
_Avoid_: Kline interval.
|
|
15
|
+
|
|
16
|
+
**DataType**:
|
|
17
|
+
The category of historical dataset contained in an archive.
|
|
18
|
+
_Avoid_: File format.
|
|
19
|
+
|
|
20
|
+
**ContractType**:
|
|
21
|
+
The futures contract settlement style, distinguishing perpetual contracts from contracts with
|
|
22
|
+
scheduled delivery.
|
|
23
|
+
_Avoid_: TradeType.
|
|
24
|
+
|
|
25
|
+
**interval**:
|
|
26
|
+
The time span represented by one K line, independent of the archive's DataFrequency.
|
|
27
|
+
_Avoid_: Archive frequency.
|
|
28
|
+
|
|
29
|
+
**Symbol**:
|
|
30
|
+
The market identifier for the instrument whose historical data is archived.
|
|
31
|
+
|
|
32
|
+
**Inference failure**:
|
|
33
|
+
A raw archive Symbol whose metadata cannot be inferred for its TradeType.
|
|
34
|
+
_Avoid_: Filtered out.
|
|
35
|
+
|
|
36
|
+
**Filtered out**:
|
|
37
|
+
An archive Symbol whose metadata was inferred successfully but does not satisfy the selected filters.
|
|
38
|
+
_Avoid_: Inference failure.
|
|
39
|
+
|
|
40
|
+
**Archive home**:
|
|
41
|
+
The root of the user's local historical archive storage.
|
|
42
|
+
|
|
43
|
+
**Downloaded**:
|
|
44
|
+
Successfully transferred to local storage. This does not establish Verified status.
|
|
45
|
+
_Avoid_: Verified.
|
|
46
|
+
|
|
47
|
+
**Up to date**:
|
|
48
|
+
Satisfies the rules for skipping a download. This does not establish Verified status.
|
|
49
|
+
_Avoid_: Verified.
|
|
50
|
+
|
|
51
|
+
**Verified**:
|
|
52
|
+
Has passed SHA256 comparison against the associated checksum.
|
|
53
|
+
_Avoid_: Downloaded, Up to date.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
|
-
Metadata-Version: 2.
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
2
|
Name: binance-datatool
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.2.0
|
|
4
4
|
Summary: A non-official Python toolkit for managing historical market data from data.binance.vision
|
|
5
5
|
Project-URL: Repository, https://github.com/lostleaf/binance-datatool
|
|
6
6
|
Author: lostleaf
|
|
@@ -22,15 +22,14 @@ Classifier: Topic :: Software Development :: Libraries :: Python Modules
|
|
|
22
22
|
Requires-Python: >=3.11
|
|
23
23
|
Requires-Dist: aiohttp>=3.13.5
|
|
24
24
|
Requires-Dist: loguru>=0.7.3
|
|
25
|
-
Requires-Dist: python-binance>=1.0.32
|
|
26
25
|
Requires-Dist: tenacity>=9.1.4
|
|
27
26
|
Requires-Dist: tqdm>=4.67.1
|
|
28
27
|
Requires-Dist: typer>=0.24.1
|
|
29
|
-
Requires-Dist: xmltodict>=1.0.4
|
|
30
28
|
Description-Content-Type: text/markdown
|
|
31
29
|
|
|
32
30
|
# Binance DataTool - Built for Agents, Data Delivered.
|
|
33
31
|
|
|
32
|
+
[](https://pypi.org/project/binance-datatool/)
|
|
34
33
|
[](https://www.python.org/)
|
|
35
34
|
[](LICENSE)
|
|
36
35
|
[](https://docs.astral.sh/ruff/)
|
|
@@ -40,11 +39,7 @@ A non-official Python toolkit for managing historical market data from
|
|
|
40
39
|
[data.binance.vision](https://data.binance.vision/) — designed
|
|
41
40
|
from the ground up for AI agents and quantitative workflows.
|
|
42
41
|
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
Most market-data tools are built for humans clicking through GUIs.
|
|
46
|
-
**binance-datatool** is different — every command is designed to be
|
|
47
|
-
called by AI agents just as easily as by a human:
|
|
42
|
+
**binance-datatool** is designed to make every command equally easy to call for AI agents and humans:
|
|
48
43
|
|
|
49
44
|
- **Composable CLI** — stdin/stdout pipelines let agents chain
|
|
50
45
|
commands without glue code
|
|
@@ -55,6 +50,13 @@ called by AI agents just as easily as by a human:
|
|
|
55
50
|
- **Zero interaction** — no prompts, no confirmations; dry-run mode
|
|
56
51
|
for safe previews
|
|
57
52
|
|
|
53
|
+
## Agent Skill
|
|
54
|
+
|
|
55
|
+
A packaged [Agent Skill](skills/binance-datatool/SKILL.md) is included in this repository.
|
|
56
|
+
|
|
57
|
+
It teaches agents how to use `binance-datatool` to discover, download, and
|
|
58
|
+
verify Binance historical market data — no manual command lookup required.
|
|
59
|
+
|
|
58
60
|
## Features
|
|
59
61
|
|
|
60
62
|
- **Multi-market coverage** — Spot, USD-M Futures, and COIN-M Futures
|
|
@@ -73,37 +75,39 @@ called by AI agents just as easily as by a human:
|
|
|
73
75
|
### Prerequisites
|
|
74
76
|
|
|
75
77
|
- [Python](https://www.python.org/) 3.11+
|
|
76
|
-
- [
|
|
78
|
+
- [pipx](https://pipx.pypa.io/) — installs the CLI in an isolated environment
|
|
77
79
|
- [aria2](https://aria2.github.io/) — required for the `download` command
|
|
78
80
|
|
|
79
81
|
### Install
|
|
80
82
|
|
|
81
83
|
```bash
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
84
|
+
# install the CLI
|
|
85
|
+
pipx install binance-datatool
|
|
86
|
+
# run `pipx ensurepath` if the command isn't found immediately
|
|
85
87
|
|
|
86
|
-
|
|
88
|
+
# system dependency for download command
|
|
89
|
+
# macOS
|
|
90
|
+
brew install aria2
|
|
91
|
+
# Ubuntu/Debian
|
|
92
|
+
sudo apt install aria2
|
|
87
93
|
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
uv sync
|
|
92
|
-
export BINANCE_DATATOOL_ARCHIVE_HOME=$HOME/crypto_data/binance_archive
|
|
94
|
+
export BINANCE_DATATOOL_ARCHIVE_HOME="$HOME/crypto_data/binance_archive"
|
|
95
|
+
|
|
96
|
+
binance-datatool --help
|
|
93
97
|
```
|
|
94
98
|
|
|
95
99
|
### Try it
|
|
96
100
|
|
|
97
101
|
```bash
|
|
98
102
|
# List USDT-quoted spot symbols, excluding stablecoins
|
|
99
|
-
|
|
103
|
+
binance-datatool list-symbols spot --quote USDT --exclude-stables
|
|
100
104
|
|
|
101
105
|
# Download daily 1m klines
|
|
102
|
-
|
|
103
|
-
|
|
|
106
|
+
binance-datatool list-symbols spot --quote USDT --exclude-stables \
|
|
107
|
+
| binance-datatool download spot --type klines --interval 1m
|
|
104
108
|
|
|
105
109
|
# Verify integrity
|
|
106
|
-
|
|
110
|
+
binance-datatool verify spot --type klines --interval 1m BTCUSDT
|
|
107
111
|
```
|
|
108
112
|
|
|
109
113
|
## Supported Data Types
|
|
@@ -115,9 +119,11 @@ uv run binance-datatool verify spot --type klines --interval 1m BTCUSDT
|
|
|
115
119
|
| Order book | `bookDepth`, `bookTicker` |
|
|
116
120
|
| Index data | `indexPriceKlines`, `markPriceKlines`, `premiumIndexKlines` |
|
|
117
121
|
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
122
|
+
The CLI accepts **daily** and **monthly** archive partitions. Availability depends on
|
|
123
|
+
the market and dataset; for example, futures funding rates use monthly archives.
|
|
124
|
+
Kline types require `--interval` (such as `1m`), independently of partition frequency.
|
|
125
|
+
Use `list-files` to discover what Binance publishes for a selection. See
|
|
126
|
+
[archive paths](docs/reference/archive/path-structure.md) and the [glossary](GLOSSARY.md).
|
|
121
127
|
|
|
122
128
|
## CLI Overview
|
|
123
129
|
|
|
@@ -129,18 +135,23 @@ directly on the root app:
|
|
|
129
135
|
| `list-symbols` | List available symbols on the remote archive |
|
|
130
136
|
| `list-files` | List archive files for given symbols |
|
|
131
137
|
| `download` | Download new or updated archive files |
|
|
132
|
-
| `verify` | Verify files against SHA256 checksums |
|
|
138
|
+
| `verify` | Verify local ZIP files against sibling SHA256 checksums; no network access |
|
|
133
139
|
|
|
134
140
|
> [!TIP]
|
|
135
|
-
>
|
|
136
|
-
>
|
|
141
|
+
> `list-files`, `download`, and `verify` accept symbols from **stdin** or as
|
|
142
|
+
> **positional args**, so they can consume `list-symbols` output:
|
|
137
143
|
>
|
|
138
144
|
> ```bash
|
|
139
145
|
> binance-datatool list-symbols um --quote USDT \
|
|
140
146
|
> | binance-datatool download um --type klines --interval 1m
|
|
141
147
|
> ```
|
|
142
148
|
|
|
143
|
-
Run `
|
|
149
|
+
Run `binance-datatool --help` for the full option reference.
|
|
150
|
+
|
|
151
|
+
Downloads count individual objects, including ZIP and checksum files. Successful transfer
|
|
152
|
+
or an up-to-date local copy does not establish SHA256 verification. Verification counts ZIPs
|
|
153
|
+
and distinguishes newly verified files from those skipped using cached verification. See
|
|
154
|
+
[command outcomes](docs/reference/cli/archive.md) for counts, cleanup, and exit semantics.
|
|
144
155
|
|
|
145
156
|
## Architecture
|
|
146
157
|
|
|
@@ -149,19 +160,13 @@ CLI Layer Thin Typer commands (argument parsing, output formatting)
|
|
|
149
160
|
│
|
|
150
161
|
Workflow Layer Business logic (diff computation, verification protocol)
|
|
151
162
|
│
|
|
152
|
-
Archive Layer
|
|
163
|
+
Archive Layer Remote listings and downloads, local directories, checksums, markers
|
|
153
164
|
│
|
|
154
165
|
Common Layer Shared types, enums, constants, and utilities
|
|
155
166
|
```
|
|
156
167
|
|
|
157
168
|
See [docs/architecture.md](docs/architecture.md) for a detailed design overview.
|
|
158
169
|
|
|
159
|
-
## Vision
|
|
160
|
-
|
|
161
|
-
binance-datatool is the data foundation for agent-driven quantitative
|
|
162
|
-
research. Next up: **Agent skills** — packaged capabilities that let AI
|
|
163
|
-
agents autonomously discover, fetch, and validate market data end-to-end.
|
|
164
|
-
|
|
165
170
|
## Documentation
|
|
166
171
|
|
|
167
172
|
| Document | Description |
|
|
@@ -170,3 +175,36 @@ agents autonomously discover, fetch, and validate market data end-to-end.
|
|
|
170
175
|
| [Module Reference](docs/reference/README.md) | Per-subpackage API reference |
|
|
171
176
|
| [Extending](docs/extending.md) | How to add commands, enums, workflows, and tests |
|
|
172
177
|
| [Testing Guide](docs/reference/testing.md) | Test organization and conventions |
|
|
178
|
+
| [Agent Skill](skills/binance-datatool/SKILL.md) | Packaged skill for agents |
|
|
179
|
+
|
|
180
|
+
## Agent development
|
|
181
|
+
|
|
182
|
+
Contributors using agents need the [Matt Pocock skills](https://github.com/mattpocock/skills)
|
|
183
|
+
collection recorded in [skills-lock.json](skills-lock.json), including `writing-for-agents`
|
|
184
|
+
for Markdown and `domain-modeling` for terminology and architectural decisions. These are
|
|
185
|
+
development tools; using the Python package or CLI does not require them. The packaged
|
|
186
|
+
[binance-datatool skill](skills/binance-datatool/SKILL.md) separately teaches CLI usage.
|
|
187
|
+
|
|
188
|
+
For a fresh checkout, install [uv](https://docs.astral.sh/uv/) for Python development and
|
|
189
|
+
Node.js/npm for the [skills installer](https://github.com/vercel-labs/skills), then run from
|
|
190
|
+
the repository root:
|
|
191
|
+
|
|
192
|
+
```bash
|
|
193
|
+
uv sync
|
|
194
|
+
npx skills experimental_install --agent codex claude-code --frozen-lockfile
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
Select the agent names you use. The installer restores the recorded collection into
|
|
198
|
+
`.agents/skills/` and links agent-specific directories as needed. Its `--dry-run` option
|
|
199
|
+
previews restoration without installing. Read [AGENTS.md](AGENTS.md) before working;
|
|
200
|
+
[CLAUDE.md](CLAUDE.md) imports that same guide.
|
|
201
|
+
|
|
202
|
+
The tracked skill record contains source repositories, skill paths, and content hashes.
|
|
203
|
+
It does not pin an upstream commit or guarantee reproducible restoration. Frozen installation
|
|
204
|
+
fails if fetched content would change the record. If upstream content has moved or changed,
|
|
205
|
+
leave updates to the maintainer: manually review the upstream changes, run the installer's
|
|
206
|
+
explicit update command when approved, and review the resulting skill-record diff. Skills
|
|
207
|
+
are not updated automatically by package setup or CI. Installer options are documented in
|
|
208
|
+
`npx skills --help`.
|
|
209
|
+
|
|
210
|
+
For Python and testing conventions, read [coding standards](docs/agents/coding-standards.md).
|
|
@@ -0,0 +1,181 @@
|
|
|
1
|
+
# Binance DataTool - Built for Agents, Data Delivered.
|
|
2
|
+
|
|
3
|
+
[](https://pypi.org/project/binance-datatool/)
|
|
4
|
+
[](https://www.python.org/)
|
|
5
|
+
[](LICENSE)
|
|
6
|
+
[](https://docs.astral.sh/ruff/)
|
|
7
|
+
[](https://docs.astral.sh/uv/)
|
|
8
|
+
|
|
9
|
+
A non-official Python toolkit for managing historical market data from
|
|
10
|
+
[data.binance.vision](https://data.binance.vision/) — designed
|
|
11
|
+
from the ground up for AI agents and quantitative workflows.
|
|
12
|
+
|
|
13
|
+
**binance-datatool** is designed to make every command equally easy to call for AI agents and humans:
|
|
14
|
+
|
|
15
|
+
- **Composable CLI** — stdin/stdout pipelines let agents chain
|
|
16
|
+
commands without glue code
|
|
17
|
+
- **Atomic operations** — each command does one thing well; agents
|
|
18
|
+
can inspect, decide, and act step by step
|
|
19
|
+
- **Deterministic output** — structured, parseable results that
|
|
20
|
+
agents can reason about
|
|
21
|
+
- **Zero interaction** — no prompts, no confirmations; dry-run mode
|
|
22
|
+
for safe previews
|
|
23
|
+
|
|
24
|
+
## Agent Skill
|
|
25
|
+
|
|
26
|
+
A packaged [Agent Skill](skills/binance-datatool/SKILL.md) is included in this repository.
|
|
27
|
+
|
|
28
|
+
It teaches agents how to use `binance-datatool` to discover, download, and
|
|
29
|
+
verify Binance historical market data — no manual command lookup required.
|
|
30
|
+
|
|
31
|
+
## Features
|
|
32
|
+
|
|
33
|
+
- **Multi-market coverage** — Spot, USD-M Futures, and COIN-M Futures
|
|
34
|
+
- **Smart symbol filtering** — filter by quote asset, exclude stablecoins
|
|
35
|
+
and leveraged tokens, select contract types
|
|
36
|
+
- **Resumable batch downloads** — diff-based sync via
|
|
37
|
+
[aria2](https://aria2.github.io/) that only fetches new or updated files
|
|
38
|
+
- **Data integrity verification** — SHA256 checksum validation with
|
|
39
|
+
timestamped marker caching
|
|
40
|
+
- **Composable pipelines** — Unix-friendly design; commands read from
|
|
41
|
+
stdin and write to stdout
|
|
42
|
+
- **Async I/O** — concurrent S3 listing and parallel checksum verification
|
|
43
|
+
|
|
44
|
+
## Quick Start
|
|
45
|
+
|
|
46
|
+
### Prerequisites
|
|
47
|
+
|
|
48
|
+
- [Python](https://www.python.org/) 3.11+
|
|
49
|
+
- [pipx](https://pipx.pypa.io/) — installs the CLI in an isolated environment
|
|
50
|
+
- [aria2](https://aria2.github.io/) — required for the `download` command
|
|
51
|
+
|
|
52
|
+
### Install
|
|
53
|
+
|
|
54
|
+
```bash
|
|
55
|
+
# install the CLI
|
|
56
|
+
pipx install binance-datatool
|
|
57
|
+
# run `pipx ensurepath` if the command isn't found immediately
|
|
58
|
+
|
|
59
|
+
# system dependency for download command
|
|
60
|
+
# macOS
|
|
61
|
+
brew install aria2
|
|
62
|
+
# Ubuntu/Debian
|
|
63
|
+
sudo apt install aria2
|
|
64
|
+
|
|
65
|
+
export BINANCE_DATATOOL_ARCHIVE_HOME="$HOME/crypto_data/binance_archive"
|
|
66
|
+
|
|
67
|
+
binance-datatool --help
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
### Try it
|
|
71
|
+
|
|
72
|
+
```bash
|
|
73
|
+
# List USDT-quoted spot symbols, excluding stablecoins
|
|
74
|
+
binance-datatool list-symbols spot --quote USDT --exclude-stables
|
|
75
|
+
|
|
76
|
+
# Download daily 1m klines
|
|
77
|
+
binance-datatool list-symbols spot --quote USDT --exclude-stables \
|
|
78
|
+
| binance-datatool download spot --type klines --interval 1m
|
|
79
|
+
|
|
80
|
+
# Verify integrity
|
|
81
|
+
binance-datatool verify spot --type klines --interval 1m BTCUSDT
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
## Supported Data Types
|
|
85
|
+
|
|
86
|
+
| Category | Types |
|
|
87
|
+
|----------|-------|
|
|
88
|
+
| Trade data | `klines`, `aggTrades`, `trades` |
|
|
89
|
+
| Derivatives | `fundingRate`, `liquidationSnapshot`, `metrics` |
|
|
90
|
+
| Order book | `bookDepth`, `bookTicker` |
|
|
91
|
+
| Index data | `indexPriceKlines`, `markPriceKlines`, `premiumIndexKlines` |
|
|
92
|
+
|
|
93
|
+
The CLI accepts **daily** and **monthly** archive partitions. Availability depends on
|
|
94
|
+
the market and dataset; for example, futures funding rates use monthly archives.
|
|
95
|
+
Kline types require `--interval` (such as `1m`), independently of partition frequency.
|
|
96
|
+
Use `list-files` to discover what Binance publishes for a selection. See
|
|
97
|
+
[archive paths](docs/reference/archive/path-structure.md) and the [glossary](GLOSSARY.md).
|
|
98
|
+
|
|
99
|
+
## CLI Overview
|
|
100
|
+
|
|
101
|
+
The entry point is `binance-datatool`. All data commands are available
|
|
102
|
+
directly on the root app:
|
|
103
|
+
|
|
104
|
+
| Command | Description |
|
|
105
|
+
|---------|-------------|
|
|
106
|
+
| `list-symbols` | List available symbols on the remote archive |
|
|
107
|
+
| `list-files` | List archive files for given symbols |
|
|
108
|
+
| `download` | Download new or updated archive files |
|
|
109
|
+
| `verify` | Verify local ZIP files against sibling SHA256 checksums; no network access |
|
|
110
|
+
|
|
111
|
+
> [!TIP]
|
|
112
|
+
> `list-files`, `download`, and `verify` accept symbols from **stdin** or as
|
|
113
|
+
> **positional args**, so they can consume `list-symbols` output:
|
|
114
|
+
>
|
|
115
|
+
> ```bash
|
|
116
|
+
> binance-datatool list-symbols um --quote USDT \
|
|
117
|
+
> | binance-datatool download um --type klines --interval 1m
|
|
118
|
+
> ```
|
|
119
|
+
|
|
120
|
+
Run `binance-datatool --help` for the full option reference.
|
|
121
|
+
|
|
122
|
+
Downloads count individual objects, including ZIP and checksum files. Successful transfer
|
|
123
|
+
or an up-to-date local copy does not establish SHA256 verification. Verification counts ZIPs
|
|
124
|
+
and distinguishes newly verified files from those skipped using cached verification. See
|
|
125
|
+
[command outcomes](docs/reference/cli/archive.md) for counts, cleanup, and exit semantics.
|
|
126
|
+
|
|
127
|
+
## Architecture
|
|
128
|
+
|
|
129
|
+
```
|
|
130
|
+
CLI Layer Thin Typer commands (argument parsing, output formatting)
|
|
131
|
+
│
|
|
132
|
+
Workflow Layer Business logic (diff computation, verification protocol)
|
|
133
|
+
│
|
|
134
|
+
Archive Layer Remote listings and downloads, local directories, checksums, markers
|
|
135
|
+
│
|
|
136
|
+
Common Layer Shared types, enums, constants, and utilities
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
See [docs/architecture.md](docs/architecture.md) for a detailed design overview.
|
|
140
|
+
|
|
141
|
+
## Documentation
|
|
142
|
+
|
|
143
|
+
| Document | Description |
|
|
144
|
+
|----------|-------------|
|
|
145
|
+
| [Architecture](docs/architecture.md) | Layered design, data flow, and key decisions |
|
|
146
|
+
| [Module Reference](docs/reference/README.md) | Per-subpackage API reference |
|
|
147
|
+
| [Extending](docs/extending.md) | How to add commands, enums, workflows, and tests |
|
|
148
|
+
| [Testing Guide](docs/reference/testing.md) | Test organization and conventions |
|
|
149
|
+
| [Agent Skill](skills/binance-datatool/SKILL.md) | Packaged skill for agents |
|
|
150
|
+
|
|
151
|
+
## Agent development
|
|
152
|
+
|
|
153
|
+
Contributors using agents need the [Matt Pocock skills](https://github.com/mattpocock/skills)
|
|
154
|
+
collection recorded in [skills-lock.json](skills-lock.json), including `writing-for-agents`
|
|
155
|
+
for Markdown and `domain-modeling` for terminology and architectural decisions. These are
|
|
156
|
+
development tools; using the Python package or CLI does not require them. The packaged
|
|
157
|
+
[binance-datatool skill](skills/binance-datatool/SKILL.md) separately teaches CLI usage.
|
|
158
|
+
|
|
159
|
+
For a fresh checkout, install [uv](https://docs.astral.sh/uv/) for Python development and
|
|
160
|
+
Node.js/npm for the [skills installer](https://github.com/vercel-labs/skills), then run from
|
|
161
|
+
the repository root:
|
|
162
|
+
|
|
163
|
+
```bash
|
|
164
|
+
uv sync
|
|
165
|
+
npx skills experimental_install --agent codex claude-code --frozen-lockfile
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
Select the agent names you use. The installer restores the recorded collection into
|
|
169
|
+
`.agents/skills/` and links agent-specific directories as needed. Its `--dry-run` option
|
|
170
|
+
previews restoration without installing. Read [AGENTS.md](AGENTS.md) before working;
|
|
171
|
+
[CLAUDE.md](CLAUDE.md) imports that same guide.
|
|
172
|
+
|
|
173
|
+
The tracked skill record contains source repositories, skill paths, and content hashes.
|
|
174
|
+
It does not pin an upstream commit or guarantee reproducible restoration. Frozen installation
|
|
175
|
+
fails if fetched content would change the record. If upstream content has moved or changed,
|
|
176
|
+
leave updates to the maintainer: manually review the upstream changes, run the installer's
|
|
177
|
+
explicit update command when approved, and review the resulting skill-record diff. Skills
|
|
178
|
+
are not updated automatically by package setup or CI. Installer options are documented in
|
|
179
|
+
`npx skills --help`.
|
|
180
|
+
|
|
181
|
+
For Python and testing conventions, read [coding standards](docs/agents/coding-standards.md).
|
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
Developer documentation for contributors, maintainers, and AI agents working on the
|
|
4
4
|
`binance-datatool` codebase.
|
|
5
5
|
|
|
6
|
-
For installation and usage instructions, see the repository root README.
|
|
6
|
+
For installation and usage instructions, see the [repository root README](../README.md).
|
|
7
7
|
|
|
8
8
|
## Core Docs
|
|
9
9
|
|
|
@@ -17,4 +17,6 @@ For installation and usage instructions, see the repository root README.
|
|
|
17
17
|
| Document | Description |
|
|
18
18
|
|----------|-------------|
|
|
19
19
|
| [Extending the Project](extending.md) | How to add commands, enums, workflows, and tests. |
|
|
20
|
+
| [Coding standards](agents/coding-standards.md) | Project Python, failure, inference, and test contracts. |
|
|
21
|
+
| [Domain guide](agents/domain.md) | Glossary and architectural-decision navigation. |
|
|
20
22
|
| [Testing](reference/testing.md) | Test directory layout, conventions, and shared fixtures. |
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
# Retain useful API conveniences and remove the ambiguous symbol count alias
|
|
2
|
+
|
|
3
|
+
Retain the module-level `list_symbols()`, all three symbol filters' `__call__` methods, and
|
|
4
|
+
`SymbolArchiveDir.clear_markers()`: their documented call shapes offer useful convenience
|
|
5
|
+
while reusing the underlying operations. The internal non-use established by
|
|
6
|
+
[Issue #34](https://github.com/lostleaf/binance-datatool/issues/34) does not establish external non-use.
|
|
7
|
+
|
|
8
|
+
Remove `ListSymbolsResult.unmatched_symbols` directly in the next normal release to make
|
|
9
|
+
the inference-failure count unambiguous. The maintainer accepts the compatibility break to
|
|
10
|
+
keep one canonical name for this count. Caller changes belong in the
|
|
11
|
+
[migration guidance](../reference/workflow/README.md#listsymbolsresult).
|