binance-datatool 0.1.0__tar.gz → 0.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (101) hide show
  1. binance_datatool-0.2.0/.github/workflows/ci.yml +38 -0
  2. binance_datatool-0.2.0/.github/workflows/integration.yml +29 -0
  3. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/.github/workflows/publish.yml +24 -4
  4. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/.gitignore +0 -1
  5. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/.pre-commit-config.yaml +1 -1
  6. binance_datatool-0.2.0/AGENTS.md +31 -0
  7. binance_datatool-0.2.0/CLAUDE.md +1 -0
  8. binance_datatool-0.2.0/GLOSSARY.md +53 -0
  9. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/PKG-INFO +75 -37
  10. binance_datatool-0.2.0/README.md +181 -0
  11. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/README.md +3 -1
  12. binance_datatool-0.2.0/docs/adr/0001-public-api-conveniences.md +11 -0
  13. binance_datatool-0.2.0/docs/agents/coding-standards.md +58 -0
  14. binance_datatool-0.2.0/docs/agents/domain.md +13 -0
  15. binance_datatool-0.2.0/docs/agents/issue-tracker.md +50 -0
  16. binance_datatool-0.2.0/docs/agents/triage-labels.md +15 -0
  17. binance_datatool-0.2.0/docs/architecture.md +37 -0
  18. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/extending.md +18 -29
  19. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/README.md +5 -4
  20. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/archive/README.md +4 -5
  21. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/archive/checksum.md +4 -2
  22. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/archive/client.md +10 -4
  23. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/archive/downloader.md +31 -3
  24. binance_datatool-0.2.0/docs/reference/archive/path-structure.md +210 -0
  25. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/archive/s3-protocol.md +8 -4
  26. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/archive/symbol_dir.md +35 -5
  27. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/cli/README.md +1 -1
  28. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/cli/archive.md +261 -54
  29. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/common/constants.md +1 -1
  30. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/common/enums.md +3 -3
  31. {binance_datatool-0.1.0/docs/reference/archive → binance_datatool-0.2.0/docs/reference/common}/filter.md +7 -7
  32. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/common/logging.md +3 -1
  33. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/common/path.md +40 -1
  34. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/common/progress.md +16 -4
  35. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/common/symbols.md +5 -2
  36. binance_datatool-0.2.0/docs/reference/performance.md +141 -0
  37. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/testing.md +29 -2
  38. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/workflow/README.md +123 -43
  39. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/pyproject.toml +1 -2
  40. binance_datatool-0.2.0/skills/binance-datatool/SKILL.md +424 -0
  41. binance_datatool-0.2.0/skills-lock.json +167 -0
  42. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/__init__.py +1 -1
  43. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/archive/__init__.py +2 -12
  44. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/archive/checksum.py +10 -5
  45. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/archive/client.py +86 -87
  46. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/archive/downloader.py +75 -8
  47. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/archive/symbol_dir.py +124 -19
  48. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/cli/archive.py +234 -108
  49. binance_datatool-0.2.0/src/binance_datatool/cli/output.py +32 -0
  50. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/common/__init__.py +12 -0
  51. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/common/constants.py +3 -3
  52. {binance_datatool-0.1.0/src/binance_datatool/archive → binance_datatool-0.2.0/src/binance_datatool/common}/filter.py +4 -3
  53. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/common/logging.py +7 -1
  54. binance_datatool-0.2.0/src/binance_datatool/common/path.py +91 -0
  55. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/common/progress.py +9 -121
  56. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/common/symbols.py +4 -4
  57. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/workflow/_shared.py +1 -11
  58. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/workflow/download.py +50 -8
  59. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/workflow/list_files.py +1 -1
  60. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/workflow/list_symbols.py +6 -4
  61. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/workflow/results.py +3 -6
  62. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/workflow/verify.py +93 -99
  63. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/conftest.py +45 -0
  64. binance_datatool-0.2.0/tests/test_archive_client.py +618 -0
  65. binance_datatool-0.2.0/tests/test_archive_workflow.py +1554 -0
  66. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/test_checksum.py +24 -1
  67. binance_datatool-0.2.0/tests/test_cli.py +2706 -0
  68. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/test_downloader.py +136 -1
  69. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/test_filter.py +84 -3
  70. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/test_path.py +1 -3
  71. binance_datatool-0.2.0/tests/test_progress.py +93 -0
  72. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/test_symbol_dir.py +40 -0
  73. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/test_symbols.py +186 -3
  74. binance_datatool-0.2.0/tests/test_verify_interrupt.py +396 -0
  75. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/uv.lock +3 -14
  76. binance_datatool-0.1.0/AGENTS.md +0 -85
  77. binance_datatool-0.1.0/CLAUDE.md +0 -1
  78. binance_datatool-0.1.0/README.md +0 -141
  79. binance_datatool-0.1.0/docs/architecture.md +0 -83
  80. binance_datatool-0.1.0/src/binance_datatool/common/path.py +0 -40
  81. binance_datatool-0.1.0/tests/test_archive_client.py +0 -404
  82. binance_datatool-0.1.0/tests/test_archive_workflow.py +0 -796
  83. binance_datatool-0.1.0/tests/test_cli.py +0 -1024
  84. binance_datatool-0.1.0/tests/test_progress.py +0 -28
  85. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/LICENSE +0 -0
  86. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/docs/reference/common/types.md +0 -0
  87. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/examples/download_spot_klines_usdt_1m.sh +0 -0
  88. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/examples/download_um_funding_usdt_perpetual.sh +0 -0
  89. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/examples/download_um_klines_usdt_1m.sh +0 -0
  90. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/cli/__init__.py +0 -0
  91. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/common/enums.py +0 -0
  92. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/common/types.py +0 -0
  93. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/py.typed +0 -0
  94. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/src/binance_datatool/workflow/__init__.py +0 -0
  95. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/data/BTCUSDT-1m-2020-01-01.zip +0 -0
  96. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/data/BTCUSDT-1m-2020-01-01.zip.CHECKSUM +0 -0
  97. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/data/BTCUSDT-1m-2020-01-02.zip +0 -0
  98. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/data/BTCUSDT-1m-2020-01-02.zip.CHECKSUM +0 -0
  99. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/data/BTCUSDT-1m-2020-01-03.zip +0 -0
  100. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/data/BTCUSDT-1m-2020-01-03.zip.CHECKSUM +0 -0
  101. {binance_datatool-0.1.0 → binance_datatool-0.2.0}/tests/test_enums.py +0 -0
@@ -0,0 +1,38 @@
1
+ name: CI
2
+
3
+ on:
4
+ push:
5
+ pull_request:
6
+
7
+ permissions:
8
+ contents: read
9
+
10
+ jobs:
11
+ checks:
12
+ name: Python ${{ matrix.python-version }}
13
+ runs-on: ubuntu-latest
14
+ strategy:
15
+ fail-fast: false
16
+ matrix:
17
+ python-version: ["3.11", "3.12", "3.13", "3.14"]
18
+
19
+ steps:
20
+ - uses: actions/checkout@v6
21
+
22
+ - uses: actions/setup-python@v6
23
+ with:
24
+ python-version: ${{ matrix.python-version }}
25
+
26
+ - uses: astral-sh/setup-uv@v7
27
+
28
+ - name: Install dependencies
29
+ run: uv sync --locked
30
+
31
+ - name: Lint
32
+ run: uv run ruff check .
33
+
34
+ - name: Check formatting
35
+ run: uv run ruff format --check .
36
+
37
+ - name: Run unit tests
38
+ run: uv run pytest
@@ -0,0 +1,29 @@
1
+ name: Weekly integration checks
2
+
3
+ on:
4
+ schedule:
5
+ - cron: "17 3 * * 1"
6
+ workflow_dispatch:
7
+
8
+ permissions:
9
+ contents: read
10
+
11
+ jobs:
12
+ integration:
13
+ runs-on: ubuntu-latest
14
+ timeout-minutes: 30
15
+
16
+ steps:
17
+ - uses: actions/checkout@v6
18
+
19
+ - uses: actions/setup-python@v6
20
+ with:
21
+ python-version: "3.13"
22
+
23
+ - uses: astral-sh/setup-uv@v7
24
+
25
+ - name: Install dependencies
26
+ run: uv sync --locked
27
+
28
+ - name: Run live integration and archive symbol drift checks
29
+ run: uv run pytest --run-integration
@@ -8,13 +8,33 @@ permissions:
8
8
  contents: read
9
9
 
10
10
  jobs:
11
+ unit-tests:
12
+ runs-on: ubuntu-latest
13
+
14
+ steps:
15
+ - uses: actions/checkout@v6
16
+
17
+ - uses: actions/setup-python@v6
18
+ with:
19
+ python-version: "3.x"
20
+
21
+ - uses: astral-sh/setup-uv@v7
22
+
23
+ - name: Install dependencies
24
+ run: uv sync --locked
25
+
26
+ - name: Run unit tests
27
+ run: uv run pytest
28
+
11
29
  release-build:
12
30
  runs-on: ubuntu-latest
31
+ needs:
32
+ - unit-tests
13
33
 
14
34
  steps:
15
- - uses: actions/checkout@v5
35
+ - uses: actions/checkout@v6
16
36
 
17
- - uses: actions/setup-python@v5
37
+ - uses: actions/setup-python@v6
18
38
  with:
19
39
  python-version: "3.x"
20
40
 
@@ -24,7 +44,7 @@ jobs:
24
44
  python -m build
25
45
 
26
46
  - name: Upload distributions
27
- uses: actions/upload-artifact@v4
47
+ uses: actions/upload-artifact@v7
28
48
  with:
29
49
  name: release-dists
30
50
  path: dist/
@@ -43,7 +63,7 @@ jobs:
43
63
 
44
64
  steps:
45
65
  - name: Retrieve release distributions
46
- uses: actions/download-artifact@v5
66
+ uses: actions/download-artifact@v8
47
67
  with:
48
68
  name: release-dists
49
69
  path: dist/
@@ -224,4 +224,3 @@ temp/
224
224
  .AGENTS.local.md
225
225
  .agents/
226
226
  .claude/
227
- skills-lock.json
@@ -1,6 +1,6 @@
1
1
  repos:
2
2
  - repo: https://github.com/astral-sh/ruff-pre-commit
3
- rev: v0.11.13
3
+ rev: v0.15.10
4
4
  hooks:
5
5
  - id: ruff
6
6
  args: [--fix]
@@ -0,0 +1,31 @@
1
+ # AGENTS.md
2
+
3
+ ## Project boundaries
4
+
5
+ - Before making changes, read `.AGENTS.local.md` at the repository root if it exists.
6
+ - This package manages historical archive discovery, downloading, local storage, and SHA256
7
+ verification. Checked-in source, tests, and configuration establish current behavior when
8
+ documentation conflicts with them.
9
+ - Use English for documentation, comments, docstrings, logs, CLI text, and developer-facing text.
10
+ Keep Python code compatible with Python 3.11.
11
+ - Keep `temp/` materials outside the public package surface and project documentation.
12
+ - Update the relevant documentation when behavior, public interfaces, or contributor workflows
13
+ change. Use one Conventional Commit per ticket.
14
+
15
+ ## Task guidance
16
+
17
+ - Before changing Python code or tests, read [coding standards](docs/agents/coding-standards.md).
18
+ - Before structural or cross-layer changes, read [architecture](docs/architecture.md).
19
+ - Before adding a command, workflow, enum member, or command group, read
20
+ [extending the project](docs/extending.md).
21
+ - Before changing tests, fixtures, or test layout, read [test organization](docs/reference/testing.md).
22
+ - For module and CLI behavior, start at the [reference index](docs/reference/README.md).
23
+ - When creating or editing Markdown, use the installed
24
+ [writing-for-agents skill](.agents/skills/writing-for-agents/SKILL.md).
25
+ - When defining or changing domain terminology or recording architectural decisions, use the
26
+ installed [domain-modeling skill](.agents/skills/domain-modeling/SKILL.md).
27
+ Before domain work, read the [domain guide](docs/agents/domain.md) and its glossary/ADR pointers.
28
+ - When reading or updating issues and specs, follow the [issue tracker guide](docs/agents/issue-tracker.md).
29
+ - When triaging issues, use the [triage label mapping](docs/agents/triage-labels.md).
30
+
31
+ For agent skill setup in a fresh checkout, see [agent development](README.md#agent-development).
@@ -0,0 +1 @@
1
+ @AGENTS.md
@@ -0,0 +1,53 @@
1
+ # Historical archives
2
+
3
+ The domain covers discovering Binance historical archives, downloading them to local storage,
4
+ and verifying their integrity against SHA256 checksums.
5
+
6
+ ## Language
7
+
8
+ **TradeType**:
9
+ The Binance market classification to which an archive belongs.
10
+ _Avoid_: Exchange type.
11
+
12
+ **DataFrequency**:
13
+ The archive partition frequency: the calendar period grouped into an archive file.
14
+ _Avoid_: Kline interval.
15
+
16
+ **DataType**:
17
+ The category of historical dataset contained in an archive.
18
+ _Avoid_: File format.
19
+
20
+ **ContractType**:
21
+ The futures contract settlement style, distinguishing perpetual contracts from contracts with
22
+ scheduled delivery.
23
+ _Avoid_: TradeType.
24
+
25
+ **interval**:
26
+ The time span represented by one K line, independent of the archive's DataFrequency.
27
+ _Avoid_: Archive frequency.
28
+
29
+ **Symbol**:
30
+ The market identifier for the instrument whose historical data is archived.
31
+
32
+ **Inference failure**:
33
+ A raw archive Symbol whose metadata cannot be inferred for its TradeType.
34
+ _Avoid_: Filtered out.
35
+
36
+ **Filtered out**:
37
+ An archive Symbol whose metadata was inferred successfully but does not satisfy the selected filters.
38
+ _Avoid_: Inference failure.
39
+
40
+ **Archive home**:
41
+ The root of the user's local historical archive storage.
42
+
43
+ **Downloaded**:
44
+ Successfully transferred to local storage. This does not establish Verified status.
45
+ _Avoid_: Verified.
46
+
47
+ **Up to date**:
48
+ Satisfies the rules for skipping a download. This does not establish Verified status.
49
+ _Avoid_: Verified.
50
+
51
+ **Verified**:
52
+ Has passed SHA256 comparison against the associated checksum.
53
+ _Avoid_: Downloaded, Up to date.
@@ -1,6 +1,6 @@
1
- Metadata-Version: 2.4
1
+ Metadata-Version: 2.5
2
2
  Name: binance-datatool
3
- Version: 0.1.0
3
+ Version: 0.2.0
4
4
  Summary: A non-official Python toolkit for managing historical market data from data.binance.vision
5
5
  Project-URL: Repository, https://github.com/lostleaf/binance-datatool
6
6
  Author: lostleaf
@@ -22,15 +22,14 @@ Classifier: Topic :: Software Development :: Libraries :: Python Modules
22
22
  Requires-Python: >=3.11
23
23
  Requires-Dist: aiohttp>=3.13.5
24
24
  Requires-Dist: loguru>=0.7.3
25
- Requires-Dist: python-binance>=1.0.32
26
25
  Requires-Dist: tenacity>=9.1.4
27
26
  Requires-Dist: tqdm>=4.67.1
28
27
  Requires-Dist: typer>=0.24.1
29
- Requires-Dist: xmltodict>=1.0.4
30
28
  Description-Content-Type: text/markdown
31
29
 
32
30
  # Binance DataTool - Built for Agents, Data Delivered.
33
31
 
32
+ [![PyPI](https://img.shields.io/pypi/v/binance-datatool?logo=pypi&logoColor=white)](https://pypi.org/project/binance-datatool/)
34
33
  [![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue?logo=python&logoColor=white)](https://www.python.org/)
35
34
  [![License: BSD-3-Clause](https://img.shields.io/badge/license-BSD--3--Clause-green)](LICENSE)
36
35
  [![Ruff](https://img.shields.io/badge/code%20style-ruff-d4aa00?logo=ruff&logoColor=white)](https://docs.astral.sh/ruff/)
@@ -40,11 +39,7 @@ A non-official Python toolkit for managing historical market data from
40
39
  [data.binance.vision](https://data.binance.vision/) — designed
41
40
  from the ground up for AI agents and quantitative workflows.
42
41
 
43
- ## Why Agent-First?
44
-
45
- Most market-data tools are built for humans clicking through GUIs.
46
- **binance-datatool** is different — every command is designed to be
47
- called by AI agents just as easily as by a human:
42
+ **binance-datatool** is designed to make every command equally easy to call for AI agents and humans:
48
43
 
49
44
  - **Composable CLI** — stdin/stdout pipelines let agents chain
50
45
  commands without glue code
@@ -55,6 +50,13 @@ called by AI agents just as easily as by a human:
55
50
  - **Zero interaction** — no prompts, no confirmations; dry-run mode
56
51
  for safe previews
57
52
 
53
+ ## Agent Skill
54
+
55
+ A packaged [Agent Skill](skills/binance-datatool/SKILL.md) is included in this repository.
56
+
57
+ It teaches agents how to use `binance-datatool` to discover, download, and
58
+ verify Binance historical market data — no manual command lookup required.
59
+
58
60
  ## Features
59
61
 
60
62
  - **Multi-market coverage** — Spot, USD-M Futures, and COIN-M Futures
@@ -73,37 +75,39 @@ called by AI agents just as easily as by a human:
73
75
  ### Prerequisites
74
76
 
75
77
  - [Python](https://www.python.org/) 3.11+
76
- - [uv](https://docs.astral.sh/uv/) — fast Python package manager
78
+ - [pipx](https://pipx.pypa.io/) — installs the CLI in an isolated environment
77
79
  - [aria2](https://aria2.github.io/) — required for the `download` command
78
80
 
79
81
  ### Install
80
82
 
81
83
  ```bash
82
- pip install binance-datatool
83
- export BINANCE_DATATOOL_ARCHIVE_HOME=$HOME/crypto_data/binance_archive
84
- ```
84
+ # install the CLI
85
+ pipx install binance-datatool
86
+ # run `pipx ensurepath` if the command isn't found immediately
85
87
 
86
- For development:
88
+ # system dependency for download command
89
+ # macOS
90
+ brew install aria2
91
+ # Ubuntu/Debian
92
+ sudo apt install aria2
87
93
 
88
- ```bash
89
- git clone https://github.com/lostleaf/binance-datatool.git
90
- cd binance-datatool
91
- uv sync
92
- export BINANCE_DATATOOL_ARCHIVE_HOME=$HOME/crypto_data/binance_archive
94
+ export BINANCE_DATATOOL_ARCHIVE_HOME="$HOME/crypto_data/binance_archive"
95
+
96
+ binance-datatool --help
93
97
  ```
94
98
 
95
99
  ### Try it
96
100
 
97
101
  ```bash
98
102
  # List USDT-quoted spot symbols, excluding stablecoins
99
- uv run binance-datatool list-symbols spot --quote USDT --exclude-stables
103
+ binance-datatool list-symbols spot --quote USDT --exclude-stables
100
104
 
101
105
  # Download daily 1m klines
102
- uv run binance-datatool list-symbols spot --quote USDT --exclude-stables \
103
- | uv run binance-datatool download spot --type klines --interval 1m
106
+ binance-datatool list-symbols spot --quote USDT --exclude-stables \
107
+ | binance-datatool download spot --type klines --interval 1m
104
108
 
105
109
  # Verify integrity
106
- uv run binance-datatool verify spot --type klines --interval 1m BTCUSDT
110
+ binance-datatool verify spot --type klines --interval 1m BTCUSDT
107
111
  ```
108
112
 
109
113
  ## Supported Data Types
@@ -115,9 +119,11 @@ uv run binance-datatool verify spot --type klines --interval 1m BTCUSDT
115
119
  | Order book | `bookDepth`, `bookTicker` |
116
120
  | Index data | `indexPriceKlines`, `markPriceKlines`, `premiumIndexKlines` |
117
121
 
118
- Each type is available in **daily** or **monthly** partitions. Kline types
119
- support all standard intervals (`1m` `3m` `5m` `15m` `30m` `1h` `2h` `4h`
120
- `6h` `8h` `12h` `1d` `3d` `1w` `1mo`).
122
+ The CLI accepts **daily** and **monthly** archive partitions. Availability depends on
123
+ the market and dataset; for example, futures funding rates use monthly archives.
124
+ Kline types require `--interval` (such as `1m`), independently of partition frequency.
125
+ Use `list-files` to discover what Binance publishes for a selection. See
126
+ [archive paths](docs/reference/archive/path-structure.md) and the [glossary](GLOSSARY.md).
121
127
 
122
128
  ## CLI Overview
123
129
 
@@ -129,18 +135,23 @@ directly on the root app:
129
135
  | `list-symbols` | List available symbols on the remote archive |
130
136
  | `list-files` | List archive files for given symbols |
131
137
  | `download` | Download new or updated archive files |
132
- | `verify` | Verify files against SHA256 checksums |
138
+ | `verify` | Verify local ZIP files against sibling SHA256 checksums; no network access |
133
139
 
134
140
  > [!TIP]
135
- > Every command accepts symbols from **stdin** or as **positional args**,
136
- > making it trivial to compose pipelines:
141
+ > `list-files`, `download`, and `verify` accept symbols from **stdin** or as
142
+ > **positional args**, so they can consume `list-symbols` output:
137
143
  >
138
144
  > ```bash
139
145
  > binance-datatool list-symbols um --quote USDT \
140
146
  > | binance-datatool download um --type klines --interval 1m
141
147
  > ```
142
148
 
143
- Run `uv run binance-datatool --help` for the full option reference.
149
+ Run `binance-datatool --help` for the full option reference.
150
+
151
+ Downloads count individual objects, including ZIP and checksum files. Successful transfer
152
+ or an up-to-date local copy does not establish SHA256 verification. Verification counts ZIPs
153
+ and distinguishes newly verified files from those skipped using cached verification. See
154
+ [command outcomes](docs/reference/cli/archive.md) for counts, cleanup, and exit semantics.
144
155
 
145
156
  ## Architecture
146
157
 
@@ -149,19 +160,13 @@ CLI Layer Thin Typer commands (argument parsing, output formatting)
149
160
  │
150
161
  Workflow Layer Business logic (diff computation, verification protocol)
151
162
  │
152
- Archive Layer S3 communication (HTTP listing, aria2 downloading, checksums)
163
+ Archive Layer Remote listings and downloads, local directories, checksums, markers
153
164
  │
154
165
  Common Layer Shared types, enums, constants, and utilities
155
166
  ```
156
167
 
157
168
  See [docs/architecture.md](docs/architecture.md) for a detailed design overview.
158
169
 
159
- ## Vision
160
-
161
- binance-datatool is the data foundation for agent-driven quantitative
162
- research. Next up: **Agent skills** — packaged capabilities that let AI
163
- agents autonomously discover, fetch, and validate market data end-to-end.
164
-
165
170
  ## Documentation
166
171
 
167
172
  | Document | Description |
@@ -170,3 +175,36 @@ agents autonomously discover, fetch, and validate market data end-to-end.
170
175
  | [Module Reference](docs/reference/README.md) | Per-subpackage API reference |
171
176
  | [Extending](docs/extending.md) | How to add commands, enums, workflows, and tests |
172
177
  | [Testing Guide](docs/reference/testing.md) | Test organization and conventions |
178
+ | [Agent Skill](skills/binance-datatool/SKILL.md) | Packaged skill for agents |
179
+
180
+ ## Agent development
181
+
182
+ Contributors using agents need the [Matt Pocock skills](https://github.com/mattpocock/skills)
183
+ collection recorded in [skills-lock.json](skills-lock.json), including `writing-for-agents`
184
+ for Markdown and `domain-modeling` for terminology and architectural decisions. These are
185
+ development tools; using the Python package or CLI does not require them. The packaged
186
+ [binance-datatool skill](skills/binance-datatool/SKILL.md) separately teaches CLI usage.
187
+
188
+ For a fresh checkout, install [uv](https://docs.astral.sh/uv/) for Python development and
189
+ Node.js/npm for the [skills installer](https://github.com/vercel-labs/skills), then run from
190
+ the repository root:
191
+
192
+ ```bash
193
+ uv sync
194
+ npx skills experimental_install --agent codex claude-code --frozen-lockfile
195
+ ```
196
+
197
+ Select the agent names you use. The installer restores the recorded collection into
198
+ `.agents/skills/` and links agent-specific directories as needed. Its `--dry-run` option
199
+ previews restoration without installing. Read [AGENTS.md](AGENTS.md) before working;
200
+ [CLAUDE.md](CLAUDE.md) imports that same guide.
201
+
202
+ The tracked skill record contains source repositories, skill paths, and content hashes.
203
+ It does not pin an upstream commit or guarantee reproducible restoration. Frozen installation
204
+ fails if fetched content would change the record. If upstream content has moved or changed,
205
+ leave updates to the maintainer: manually review the upstream changes, run the installer's
206
+ explicit update command when approved, and review the resulting skill-record diff. Skills
207
+ are not updated automatically by package setup or CI. Installer options are documented in
208
+ `npx skills --help`.
209
+
210
+ For Python and testing conventions, read [coding standards](docs/agents/coding-standards.md).
@@ -0,0 +1,181 @@
1
+ # Binance DataTool - Built for Agents, Data Delivered.
2
+
3
+ [![PyPI](https://img.shields.io/pypi/v/binance-datatool?logo=pypi&logoColor=white)](https://pypi.org/project/binance-datatool/)
4
+ [![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue?logo=python&logoColor=white)](https://www.python.org/)
5
+ [![License: BSD-3-Clause](https://img.shields.io/badge/license-BSD--3--Clause-green)](LICENSE)
6
+ [![Ruff](https://img.shields.io/badge/code%20style-ruff-d4aa00?logo=ruff&logoColor=white)](https://docs.astral.sh/ruff/)
7
+ [![uv](https://img.shields.io/badge/package%20manager-uv-de5fe9?logo=uv&logoColor=white)](https://docs.astral.sh/uv/)
8
+
9
+ A non-official Python toolkit for managing historical market data from
10
+ [data.binance.vision](https://data.binance.vision/) — designed
11
+ from the ground up for AI agents and quantitative workflows.
12
+
13
+ **binance-datatool** is designed to make every command equally easy to call for AI agents and humans:
14
+
15
+ - **Composable CLI** — stdin/stdout pipelines let agents chain
16
+ commands without glue code
17
+ - **Atomic operations** — each command does one thing well; agents
18
+ can inspect, decide, and act step by step
19
+ - **Deterministic output** — structured, parseable results that
20
+ agents can reason about
21
+ - **Zero interaction** — no prompts, no confirmations; dry-run mode
22
+ for safe previews
23
+
24
+ ## Agent Skill
25
+
26
+ A packaged [Agent Skill](skills/binance-datatool/SKILL.md) is included in this repository.
27
+
28
+ It teaches agents how to use `binance-datatool` to discover, download, and
29
+ verify Binance historical market data — no manual command lookup required.
30
+
31
+ ## Features
32
+
33
+ - **Multi-market coverage** — Spot, USD-M Futures, and COIN-M Futures
34
+ - **Smart symbol filtering** — filter by quote asset, exclude stablecoins
35
+ and leveraged tokens, select contract types
36
+ - **Resumable batch downloads** — diff-based sync via
37
+ [aria2](https://aria2.github.io/) that only fetches new or updated files
38
+ - **Data integrity verification** — SHA256 checksum validation with
39
+ timestamped marker caching
40
+ - **Composable pipelines** — Unix-friendly design; commands read from
41
+ stdin and write to stdout
42
+ - **Async I/O** — concurrent S3 listing and parallel checksum verification
43
+
44
+ ## Quick Start
45
+
46
+ ### Prerequisites
47
+
48
+ - [Python](https://www.python.org/) 3.11+
49
+ - [pipx](https://pipx.pypa.io/) — installs the CLI in an isolated environment
50
+ - [aria2](https://aria2.github.io/) — required for the `download` command
51
+
52
+ ### Install
53
+
54
+ ```bash
55
+ # install the CLI
56
+ pipx install binance-datatool
57
+ # run `pipx ensurepath` if the command isn't found immediately
58
+
59
+ # system dependency for download command
60
+ # macOS
61
+ brew install aria2
62
+ # Ubuntu/Debian
63
+ sudo apt install aria2
64
+
65
+ export BINANCE_DATATOOL_ARCHIVE_HOME="$HOME/crypto_data/binance_archive"
66
+
67
+ binance-datatool --help
68
+ ```
69
+
70
+ ### Try it
71
+
72
+ ```bash
73
+ # List USDT-quoted spot symbols, excluding stablecoins
74
+ binance-datatool list-symbols spot --quote USDT --exclude-stables
75
+
76
+ # Download daily 1m klines
77
+ binance-datatool list-symbols spot --quote USDT --exclude-stables \
78
+ | binance-datatool download spot --type klines --interval 1m
79
+
80
+ # Verify integrity
81
+ binance-datatool verify spot --type klines --interval 1m BTCUSDT
82
+ ```
83
+
84
+ ## Supported Data Types
85
+
86
+ | Category | Types |
87
+ |----------|-------|
88
+ | Trade data | `klines`, `aggTrades`, `trades` |
89
+ | Derivatives | `fundingRate`, `liquidationSnapshot`, `metrics` |
90
+ | Order book | `bookDepth`, `bookTicker` |
91
+ | Index data | `indexPriceKlines`, `markPriceKlines`, `premiumIndexKlines` |
92
+
93
+ The CLI accepts **daily** and **monthly** archive partitions. Availability depends on
94
+ the market and dataset; for example, futures funding rates use monthly archives.
95
+ Kline types require `--interval` (such as `1m`), independently of partition frequency.
96
+ Use `list-files` to discover what Binance publishes for a selection. See
97
+ [archive paths](docs/reference/archive/path-structure.md) and the [glossary](GLOSSARY.md).
98
+
99
+ ## CLI Overview
100
+
101
+ The entry point is `binance-datatool`. All data commands are available
102
+ directly on the root app:
103
+
104
+ | Command | Description |
105
+ |---------|-------------|
106
+ | `list-symbols` | List available symbols on the remote archive |
107
+ | `list-files` | List archive files for given symbols |
108
+ | `download` | Download new or updated archive files |
109
+ | `verify` | Verify local ZIP files against sibling SHA256 checksums; no network access |
110
+
111
+ > [!TIP]
112
+ > `list-files`, `download`, and `verify` accept symbols from **stdin** or as
113
+ > **positional args**, so they can consume `list-symbols` output:
114
+ >
115
+ > ```bash
116
+ > binance-datatool list-symbols um --quote USDT \
117
+ > | binance-datatool download um --type klines --interval 1m
118
+ > ```
119
+
120
+ Run `binance-datatool --help` for the full option reference.
121
+
122
+ Downloads count individual objects, including ZIP and checksum files. Successful transfer
123
+ or an up-to-date local copy does not establish SHA256 verification. Verification counts ZIPs
124
+ and distinguishes newly verified files from those skipped using cached verification. See
125
+ [command outcomes](docs/reference/cli/archive.md) for counts, cleanup, and exit semantics.
126
+
127
+ ## Architecture
128
+
129
+ ```
130
+ CLI Layer Thin Typer commands (argument parsing, output formatting)
131
+ │
132
+ Workflow Layer Business logic (diff computation, verification protocol)
133
+ │
134
+ Archive Layer Remote listings and downloads, local directories, checksums, markers
135
+ │
136
+ Common Layer Shared types, enums, constants, and utilities
137
+ ```
138
+
139
+ See [docs/architecture.md](docs/architecture.md) for a detailed design overview.
140
+
141
+ ## Documentation
142
+
143
+ | Document | Description |
144
+ |----------|-------------|
145
+ | [Architecture](docs/architecture.md) | Layered design, data flow, and key decisions |
146
+ | [Module Reference](docs/reference/README.md) | Per-subpackage API reference |
147
+ | [Extending](docs/extending.md) | How to add commands, enums, workflows, and tests |
148
+ | [Testing Guide](docs/reference/testing.md) | Test organization and conventions |
149
+ | [Agent Skill](skills/binance-datatool/SKILL.md) | Packaged skill for agents |
150
+
151
+ ## Agent development
152
+
153
+ Contributors using agents need the [Matt Pocock skills](https://github.com/mattpocock/skills)
154
+ collection recorded in [skills-lock.json](skills-lock.json), including `writing-for-agents`
155
+ for Markdown and `domain-modeling` for terminology and architectural decisions. These are
156
+ development tools; using the Python package or CLI does not require them. The packaged
157
+ [binance-datatool skill](skills/binance-datatool/SKILL.md) separately teaches CLI usage.
158
+
159
+ For a fresh checkout, install [uv](https://docs.astral.sh/uv/) for Python development and
160
+ Node.js/npm for the [skills installer](https://github.com/vercel-labs/skills), then run from
161
+ the repository root:
162
+
163
+ ```bash
164
+ uv sync
165
+ npx skills experimental_install --agent codex claude-code --frozen-lockfile
166
+ ```
167
+
168
+ Select the agent names you use. The installer restores the recorded collection into
169
+ `.agents/skills/` and links agent-specific directories as needed. Its `--dry-run` option
170
+ previews restoration without installing. Read [AGENTS.md](AGENTS.md) before working;
171
+ [CLAUDE.md](CLAUDE.md) imports that same guide.
172
+
173
+ The tracked skill record contains source repositories, skill paths, and content hashes.
174
+ It does not pin an upstream commit or guarantee reproducible restoration. Frozen installation
175
+ fails if fetched content would change the record. If upstream content has moved or changed,
176
+ leave updates to the maintainer: manually review the upstream changes, run the installer's
177
+ explicit update command when approved, and review the resulting skill-record diff. Skills
178
+ are not updated automatically by package setup or CI. Installer options are documented in
179
+ `npx skills --help`.
180
+
181
+ For Python and testing conventions, read [coding standards](docs/agents/coding-standards.md).
@@ -3,7 +3,7 @@
3
3
  Developer documentation for contributors, maintainers, and AI agents working on the
4
4
  `binance-datatool` codebase.
5
5
 
6
- For installation and usage instructions, see the repository root README.
6
+ For installation and usage instructions, see the [repository root README](../README.md).
7
7
 
8
8
  ## Core Docs
9
9
 
@@ -17,4 +17,6 @@ For installation and usage instructions, see the repository root README.
17
17
  | Document | Description |
18
18
  |----------|-------------|
19
19
  | [Extending the Project](extending.md) | How to add commands, enums, workflows, and tests. |
20
+ | [Coding standards](agents/coding-standards.md) | Project Python, failure, inference, and test contracts. |
21
+ | [Domain guide](agents/domain.md) | Glossary and architectural-decision navigation. |
20
22
  | [Testing](reference/testing.md) | Test directory layout, conventions, and shared fixtures. |
@@ -0,0 +1,11 @@
1
+ # Retain useful API conveniences and remove the ambiguous symbol count alias
2
+
3
+ Retain the module-level `list_symbols()`, all three symbol filters' `__call__` methods, and
4
+ `SymbolArchiveDir.clear_markers()`: their documented call shapes offer useful convenience
5
+ while reusing the underlying operations. The internal non-use established by
6
+ [Issue #34](https://github.com/lostleaf/binance-datatool/issues/34) does not establish external non-use.
7
+
8
+ Remove `ListSymbolsResult.unmatched_symbols` directly in the next normal release to make
9
+ the inference-failure count unambiguous. The maintainer accepts the compatibility break to
10
+ keep one canonical name for this count. Caller changes belong in the
11
+ [migration guidance](../reference/workflow/README.md#listsymbolsresult).