pydantic-tensorstore 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- pydantic_tensorstore-0.1.0/.github/DRIFT_TEMPLATE.md +14 -0
- pydantic_tensorstore-0.1.0/.github/ISSUE_TEMPLATE.md +15 -0
- pydantic_tensorstore-0.1.0/.github/TEST_FAIL_TEMPLATE.md +12 -0
- pydantic_tensorstore-0.1.0/.github/dependabot.yml +14 -0
- pydantic_tensorstore-0.1.0/.github/workflows/ci.yml +183 -0
- pydantic_tensorstore-0.1.0/.gitignore +115 -0
- pydantic_tensorstore-0.1.0/.pre-commit-config.yaml +42 -0
- pydantic_tensorstore-0.1.0/LICENSE +28 -0
- pydantic_tensorstore-0.1.0/PKG-INFO +215 -0
- pydantic_tensorstore-0.1.0/README.md +183 -0
- pydantic_tensorstore-0.1.0/examples/basic_usage.py +126 -0
- pydantic_tensorstore-0.1.0/pyproject.toml +184 -0
- pydantic_tensorstore-0.1.0/scripts/update_ts_schema.py +82 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/__init__.py +336 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_core/__init__.py +1 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_core/base.py +176 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_core/chunk_layout.py +209 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_core/codec.py +18 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_core/context.py +219 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_core/schema.py +56 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_core/spec.py +178 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_core/transform.py +277 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_core/zarr3_codecs.py +174 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_drivers/__init__.py +89 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_drivers/array.py +94 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_drivers/auto.py +25 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_drivers/cast.py +20 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_drivers/downsample.py +21 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_drivers/image.py +73 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_drivers/json.py +23 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_drivers/n5.py +199 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_drivers/neuroglancer_precomputed.py +168 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_drivers/stack.py +26 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_drivers/zarr.py +302 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_drivers/zarr3.py +259 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_kvstore/__init__.py +108 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_kvstore/base.py +59 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_kvstore/file.py +24 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_kvstore/gcs.py +23 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_kvstore/http.py +23 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_kvstore/kvstack.py +33 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_kvstore/memory.py +21 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_kvstore/neuroglancer_uint64_sharded.py +49 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_kvstore/ocdbt.py +75 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_kvstore/s3.py +36 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_kvstore/tsgrpc.py +19 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_kvstore/zarr3_sharding_indexed.py +25 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_kvstore/zip.py +13 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_types.py +223 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_validators.py +55 -0
- pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/py.typed +0 -0
- pydantic_tensorstore-0.1.0/tests/conftest.py +35 -0
- pydantic_tensorstore-0.1.0/tests/test_dtypes.py +50 -0
- pydantic_tensorstore-0.1.0/tests/test_dump.py +89 -0
- pydantic_tensorstore-0.1.0/tests/test_examples.py +13 -0
- pydantic_tensorstore-0.1.0/tests/test_schema_conformance.py +445 -0
- pydantic_tensorstore-0.1.0/tests/test_since.py +203 -0
- pydantic_tensorstore-0.1.0/tests/test_spec_validation.py +161 -0
- pydantic_tensorstore-0.1.0/tests/test_tensorstore.py +799 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/VERSION +1 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/docs_chunk_layout_schema.yml +185 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/docs_codec_schema.yml +20 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/docs_context_schema.yml +80 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/docs_index_domain_schema.yml +108 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/docs_index_transform_schema.yml +198 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/docs_schema_schema.yml +126 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/docs_tensorstore_schema.yml +272 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_array_schema.yml +23 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_auto_schema.yml +77 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_cast_schema.yml +55 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_downsample_schema.yml +110 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_image_avif_schema.yml +57 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_image_bmp_schema.yml +45 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_image_jpeg_schema.yml +50 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_image_png_schema.yml +50 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_image_tiff_schema.yml +50 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_image_webp_schema.yml +57 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_json_schema.yml +73 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_kvs_backed_chunk_driver_schema.yml +138 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_n5_schema.yml +308 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_neuroglancer_precomputed_schema.yml +259 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_stack_schema.yml +44 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_zarr3_schema.yml +647 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_zarr_schema.yml +357 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_kvstore_file_schema.yml +173 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_kvstore_gcs_schema.yml +152 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_kvstore_http_schema.yml +155 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_kvstore_kvstack_schema.yml +55 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_kvstore_memory_schema.yml +55 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_kvstore_neuroglancer_uint64_sharded_schema.yml +108 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_kvstore_ocdbt_schema.yml +232 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_kvstore_s3_schema.yml +299 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_kvstore_schema.yml +60 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_kvstore_tsgrpc_schema.yml +29 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_kvstore_zarr3_sharding_indexed_schema.yml +49 -0
- pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_kvstore_zip_schema.yml +67 -0
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "[drift-bot] models out of sync with latest tensorstore"
|
|
3
|
+
labels: [drift]
|
|
4
|
+
---
|
|
5
|
+
The weekly drift check failed: the models no longer match the newest tensorstore
|
|
6
|
+
release (its JSON schema files and/or runtime behaviour).
|
|
7
|
+
|
|
8
|
+
See the failed run for details: https://github.com/{{ env.GITHUB_REPOSITORY }}/actions/runs/{{ env.RUN_ID }}
|
|
9
|
+
|
|
10
|
+
To update:
|
|
11
|
+
|
|
12
|
+
1. `uv run scripts/update_ts_schema.py --latest`
|
|
13
|
+
2. Bump `TENSORSTORE_VERSION` in `src/pydantic_tensorstore/__init__.py`
|
|
14
|
+
3. `uv run pytest tests/test_schema_conformance.py` and fix the reported gaps
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
* pydantic-tensorstore version:
|
|
2
|
+
* Python version:
|
|
3
|
+
* Operating System:
|
|
4
|
+
|
|
5
|
+
### Description
|
|
6
|
+
|
|
7
|
+
Describe what you were trying to get done.
|
|
8
|
+
Tell us what happened, what went wrong, and what you expected to happen.
|
|
9
|
+
|
|
10
|
+
### What I Did
|
|
11
|
+
|
|
12
|
+
```
|
|
13
|
+
Paste the command(s) you ran and the output.
|
|
14
|
+
If there was a crash, please include the traceback here.
|
|
15
|
+
```
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "{{ env.TITLE }}"
|
|
3
|
+
labels: [bug]
|
|
4
|
+
---
|
|
5
|
+
The {{ workflow }} workflow failed on {{ date | date("YYYY-MM-DD HH:mm") }} UTC
|
|
6
|
+
|
|
7
|
+
The most recent failing test was on {{ env.PLATFORM }} py{{ env.PYTHON }}
|
|
8
|
+
with commit: {{ sha }}
|
|
9
|
+
|
|
10
|
+
Full run: https://github.com/{{ repo }}/actions/runs/{{ env.RUN_ID }}
|
|
11
|
+
|
|
12
|
+
(This post will be updated if another test fails, as long as this issue remains open.)
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
# https://docs.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
|
|
2
|
+
|
|
3
|
+
version: 2
|
|
4
|
+
updates:
|
|
5
|
+
- package-ecosystem: "github-actions"
|
|
6
|
+
directory: "/"
|
|
7
|
+
schedule:
|
|
8
|
+
interval: "quarterly"
|
|
9
|
+
commit-message:
|
|
10
|
+
prefix: "ci(dependabot):"
|
|
11
|
+
groups:
|
|
12
|
+
actions:
|
|
13
|
+
patterns:
|
|
14
|
+
- "*"
|
|
@@ -0,0 +1,183 @@
|
|
|
1
|
+
name: CI
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
branches: [main]
|
|
6
|
+
tags: [v*]
|
|
7
|
+
pull_request:
|
|
8
|
+
workflow_dispatch:
|
|
9
|
+
schedule:
|
|
10
|
+
# run every week (for --pre release tests)
|
|
11
|
+
- cron: "0 0 * * 0"
|
|
12
|
+
|
|
13
|
+
concurrency:
|
|
14
|
+
group: ${{ github.workflow }}-${{ github.ref }}
|
|
15
|
+
cancel-in-progress: true
|
|
16
|
+
|
|
17
|
+
jobs:
|
|
18
|
+
check-manifest:
|
|
19
|
+
runs-on: ubuntu-latest
|
|
20
|
+
steps:
|
|
21
|
+
- uses: actions/checkout@v7
|
|
22
|
+
- run: pipx run check-manifest
|
|
23
|
+
|
|
24
|
+
lint:
|
|
25
|
+
runs-on: ubuntu-latest
|
|
26
|
+
steps:
|
|
27
|
+
- uses: actions/checkout@v7
|
|
28
|
+
- uses: astral-sh/setup-uv@v7
|
|
29
|
+
with:
|
|
30
|
+
python-version: "3.13"
|
|
31
|
+
enable-cache: true
|
|
32
|
+
- run: uv run prek run --all-files
|
|
33
|
+
|
|
34
|
+
test:
|
|
35
|
+
name: ${{ matrix.platform }} (${{ matrix.python-version }}) [${{ matrix.resolution }}]
|
|
36
|
+
runs-on: ${{ matrix.platform }}
|
|
37
|
+
env:
|
|
38
|
+
UV_PRERELEASE: ${{ github.event_name == 'schedule' && 'allow' || 'if-necessary-or-explicit' }}
|
|
39
|
+
UV_RESOLUTION: ${{ matrix.resolution }}
|
|
40
|
+
PYTEST_ADDOPTS: ${{ matrix.resolution == 'lowest-direct' && '-W ignore' || '' }}
|
|
41
|
+
strategy:
|
|
42
|
+
fail-fast: false
|
|
43
|
+
matrix:
|
|
44
|
+
python-version: ["3.11", "3.12", "3.13", "3.14"]
|
|
45
|
+
platform: [ubuntu-latest, macos-latest, windows-latest]
|
|
46
|
+
resolution: ["highest", "lowest-direct"]
|
|
47
|
+
|
|
48
|
+
steps:
|
|
49
|
+
- uses: actions/checkout@v7
|
|
50
|
+
|
|
51
|
+
- name: 🐍 Set up Python ${{ matrix.python-version }}
|
|
52
|
+
uses: astral-sh/setup-uv@v7
|
|
53
|
+
with:
|
|
54
|
+
python-version: ${{ matrix.python-version }}
|
|
55
|
+
enable-cache: true
|
|
56
|
+
|
|
57
|
+
- name: Install Dependencies
|
|
58
|
+
run: uv sync --no-dev --group test
|
|
59
|
+
|
|
60
|
+
- name: 🧪 Run Tests
|
|
61
|
+
run: uv run --no-sync coverage run -p -m pytest -v
|
|
62
|
+
|
|
63
|
+
# If something goes wrong with --pre tests, we can open an issue in the repo
|
|
64
|
+
- name: 📝 Report --pre Failures
|
|
65
|
+
if: failure() && github.event_name == 'schedule'
|
|
66
|
+
uses: JasonEtco/create-an-issue@v2
|
|
67
|
+
env:
|
|
68
|
+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
69
|
+
PLATFORM: ${{ matrix.platform }}
|
|
70
|
+
PYTHON: ${{ matrix.python-version }}
|
|
71
|
+
RUN_ID: ${{ github.run_id }}
|
|
72
|
+
TITLE: "[test-bot] pip install --pre is failing"
|
|
73
|
+
with:
|
|
74
|
+
filename: .github/TEST_FAIL_TEMPLATE.md
|
|
75
|
+
update_existing: true
|
|
76
|
+
|
|
77
|
+
- name: Upload coverage
|
|
78
|
+
uses: actions/upload-artifact@v7
|
|
79
|
+
with:
|
|
80
|
+
name: covreport-${{ matrix.platform }}-py${{ matrix.python-version }}-${{ matrix.resolution }}
|
|
81
|
+
path: ./.coverage*
|
|
82
|
+
include-hidden-files: true
|
|
83
|
+
|
|
84
|
+
upload_coverage:
|
|
85
|
+
if: always()
|
|
86
|
+
needs: [test]
|
|
87
|
+
uses: pyapp-kit/workflows/.github/workflows/upload-coverage.yml@v2
|
|
88
|
+
secrets:
|
|
89
|
+
codecov_token: ${{ secrets.CODECOV_TOKEN }}
|
|
90
|
+
|
|
91
|
+
# Explicit tensorstore version ladder: oldest supported, a midpoint, the pinned
|
|
92
|
+
# (schema-verified) release, and whatever is newest on PyPI.
|
|
93
|
+
tensorstore-compat:
|
|
94
|
+
name: tensorstore ${{ matrix.tensorstore }}
|
|
95
|
+
runs-on: ubuntu-latest
|
|
96
|
+
strategy:
|
|
97
|
+
fail-fast: false
|
|
98
|
+
matrix:
|
|
99
|
+
tensorstore: ["min", "0.1.77", "pinned", "latest"]
|
|
100
|
+
steps:
|
|
101
|
+
- uses: actions/checkout@v7
|
|
102
|
+
- uses: astral-sh/setup-uv@v7
|
|
103
|
+
with:
|
|
104
|
+
python-version: "3.12"
|
|
105
|
+
enable-cache: true
|
|
106
|
+
- run: uv sync --no-dev --group test
|
|
107
|
+
- name: Install tensorstore ${{ matrix.tensorstore }}
|
|
108
|
+
run: |
|
|
109
|
+
case "${{ matrix.tensorstore }}" in
|
|
110
|
+
min) v=$(uv run --no-sync python -c "import pydantic_tensorstore as p; print(p.MIN_TENSORSTORE_VERSION)") ;;
|
|
111
|
+
pinned) v=$(uv run --no-sync python -c "import pydantic_tensorstore as p; print(p.TENSORSTORE_VERSION)") ;;
|
|
112
|
+
latest) uv pip install --upgrade tensorstore; exit 0 ;;
|
|
113
|
+
*) v="${{ matrix.tensorstore }}" ;;
|
|
114
|
+
esac
|
|
115
|
+
uv pip install "tensorstore==$v"
|
|
116
|
+
- run: uv run --no-sync python -c "from importlib.metadata import version; print('tensorstore', version('tensorstore'))"
|
|
117
|
+
- run: uv run --no-sync pytest -v
|
|
118
|
+
|
|
119
|
+
# Weekly: does the newest tensorstore release still match our models?
|
|
120
|
+
drift:
|
|
121
|
+
name: drift check (latest tensorstore)
|
|
122
|
+
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
|
|
123
|
+
runs-on: ubuntu-latest
|
|
124
|
+
steps:
|
|
125
|
+
- uses: actions/checkout@v7
|
|
126
|
+
- uses: astral-sh/setup-uv@v7
|
|
127
|
+
with:
|
|
128
|
+
python-version: "3.13"
|
|
129
|
+
enable-cache: true
|
|
130
|
+
- name: Fetch latest tensorstore schema
|
|
131
|
+
env:
|
|
132
|
+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
133
|
+
run: uv run --no-project scripts/update_ts_schema.py --latest --dest "$RUNNER_TEMP/ts_schema"
|
|
134
|
+
- name: Install latest tensorstore
|
|
135
|
+
run: uv sync --no-dev --group test --upgrade-package tensorstore
|
|
136
|
+
- name: Compare models against latest schema and runtime
|
|
137
|
+
env:
|
|
138
|
+
PTS_SCHEMA_DIR: ${{ runner.temp }}/ts_schema
|
|
139
|
+
run: |
|
|
140
|
+
uv run --no-sync python -c "from importlib.metadata import version; print('tensorstore', version('tensorstore'))"
|
|
141
|
+
uv run --no-sync pytest -v tests/test_schema_conformance.py tests/test_dtypes.py tests/test_since.py tests/test_tensorstore.py --deselect tests/test_schema_conformance.py::test_vendored_version_matches_package
|
|
142
|
+
- name: Report drift
|
|
143
|
+
if: failure()
|
|
144
|
+
uses: JasonEtco/create-an-issue@v2
|
|
145
|
+
env:
|
|
146
|
+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
147
|
+
RUN_ID: ${{ github.run_id }}
|
|
148
|
+
with:
|
|
149
|
+
filename: .github/DRIFT_TEMPLATE.md
|
|
150
|
+
update_existing: true
|
|
151
|
+
|
|
152
|
+
build-and-inspect-package:
|
|
153
|
+
name: Build & inspect package.
|
|
154
|
+
needs: [test, tensorstore-compat, lint]
|
|
155
|
+
runs-on: ubuntu-latest
|
|
156
|
+
steps:
|
|
157
|
+
- uses: actions/checkout@v7
|
|
158
|
+
with:
|
|
159
|
+
fetch-depth: 0
|
|
160
|
+
# no moving major tag as of v3; pin the full version
|
|
161
|
+
- uses: hynek/build-and-inspect-python-package@v3.0.1
|
|
162
|
+
|
|
163
|
+
upload-to-pypi:
|
|
164
|
+
name: Upload package to PyPI
|
|
165
|
+
needs: build-and-inspect-package
|
|
166
|
+
if: success() && startsWith(github.ref, 'refs/tags/') && github.event_name != 'schedule'
|
|
167
|
+
runs-on: ubuntu-latest
|
|
168
|
+
permissions:
|
|
169
|
+
id-token: write
|
|
170
|
+
contents: write
|
|
171
|
+
|
|
172
|
+
steps:
|
|
173
|
+
- name: Download built artifact to dist/
|
|
174
|
+
uses: actions/download-artifact@v8
|
|
175
|
+
with:
|
|
176
|
+
name: Packages
|
|
177
|
+
path: dist
|
|
178
|
+
- name: 🚢 Publish to PyPI
|
|
179
|
+
uses: pypa/gh-action-pypi-publish@release/v1
|
|
180
|
+
- uses: softprops/action-gh-release@v3
|
|
181
|
+
with:
|
|
182
|
+
generate_release_notes: true
|
|
183
|
+
files: "./dist/*"
|
|
@@ -0,0 +1,115 @@
|
|
|
1
|
+
# Byte-compiled / optimized / DLL files
|
|
2
|
+
__pycache__/
|
|
3
|
+
*.py[cod]
|
|
4
|
+
*$py.class
|
|
5
|
+
|
|
6
|
+
# C extensions
|
|
7
|
+
*.so
|
|
8
|
+
|
|
9
|
+
# Distribution / packaging
|
|
10
|
+
.Python
|
|
11
|
+
env/
|
|
12
|
+
build/
|
|
13
|
+
develop-eggs/
|
|
14
|
+
dist/
|
|
15
|
+
downloads/
|
|
16
|
+
eggs/
|
|
17
|
+
.eggs/
|
|
18
|
+
lib/
|
|
19
|
+
lib64/
|
|
20
|
+
parts/
|
|
21
|
+
sdist/
|
|
22
|
+
var/
|
|
23
|
+
wheels/
|
|
24
|
+
*.egg-info/
|
|
25
|
+
.installed.cfg
|
|
26
|
+
*.egg
|
|
27
|
+
|
|
28
|
+
.DS_Store
|
|
29
|
+
|
|
30
|
+
# PyInstaller
|
|
31
|
+
# Usually these files are written by a python script from a template
|
|
32
|
+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
|
|
33
|
+
*.manifest
|
|
34
|
+
*.spec
|
|
35
|
+
|
|
36
|
+
# Installer logs
|
|
37
|
+
pip-log.txt
|
|
38
|
+
pip-delete-this-directory.txt
|
|
39
|
+
|
|
40
|
+
# Unit test / coverage reports
|
|
41
|
+
htmlcov/
|
|
42
|
+
.tox/
|
|
43
|
+
.coverage
|
|
44
|
+
.coverage.*
|
|
45
|
+
.cache
|
|
46
|
+
nosetests.xml
|
|
47
|
+
coverage.xml
|
|
48
|
+
*.cover
|
|
49
|
+
.hypothesis/
|
|
50
|
+
.pytest_cache/
|
|
51
|
+
|
|
52
|
+
# Translations
|
|
53
|
+
*.mo
|
|
54
|
+
*.pot
|
|
55
|
+
|
|
56
|
+
# Django stuff:
|
|
57
|
+
*.log
|
|
58
|
+
local_settings.py
|
|
59
|
+
|
|
60
|
+
# Flask stuff:
|
|
61
|
+
instance/
|
|
62
|
+
.webassets-cache
|
|
63
|
+
|
|
64
|
+
# Scrapy stuff:
|
|
65
|
+
.scrapy
|
|
66
|
+
|
|
67
|
+
# Sphinx documentation
|
|
68
|
+
docs/_build/
|
|
69
|
+
|
|
70
|
+
# PyBuilder
|
|
71
|
+
target/
|
|
72
|
+
|
|
73
|
+
# Jupyter Notebook
|
|
74
|
+
.ipynb_checkpoints
|
|
75
|
+
|
|
76
|
+
# pyenv
|
|
77
|
+
.python-version
|
|
78
|
+
|
|
79
|
+
# celery beat schedule file
|
|
80
|
+
celerybeat-schedule
|
|
81
|
+
|
|
82
|
+
# SageMath parsed files
|
|
83
|
+
*.sage.py
|
|
84
|
+
|
|
85
|
+
# dotenv
|
|
86
|
+
.env
|
|
87
|
+
|
|
88
|
+
# virtualenv
|
|
89
|
+
.venv
|
|
90
|
+
venv/
|
|
91
|
+
ENV/
|
|
92
|
+
|
|
93
|
+
# Spyder project settings
|
|
94
|
+
.spyderproject
|
|
95
|
+
.spyproject
|
|
96
|
+
|
|
97
|
+
# Rope project settings
|
|
98
|
+
.ropeproject
|
|
99
|
+
|
|
100
|
+
# mkdocs documentation
|
|
101
|
+
/site
|
|
102
|
+
|
|
103
|
+
# mypy
|
|
104
|
+
.mypy_cache/
|
|
105
|
+
|
|
106
|
+
# ruff
|
|
107
|
+
.ruff_cache/
|
|
108
|
+
|
|
109
|
+
# IDE settings
|
|
110
|
+
.vscode/
|
|
111
|
+
.idea/
|
|
112
|
+
uv.lock
|
|
113
|
+
|
|
114
|
+
# local agent settings
|
|
115
|
+
.claude/
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
# enable pre-commit.ci at https://pre-commit.ci/
|
|
2
|
+
# it adds:
|
|
3
|
+
# 1. auto fixing pull requests
|
|
4
|
+
# 2. auto updating the pre-commit configuration
|
|
5
|
+
ci:
|
|
6
|
+
autoupdate_schedule: monthly
|
|
7
|
+
autofix_commit_msg: "style(pre-commit.ci): auto fixes [...]"
|
|
8
|
+
autoupdate_commit_msg: "ci(pre-commit.ci): autoupdate"
|
|
9
|
+
skip:
|
|
10
|
+
- mypy
|
|
11
|
+
|
|
12
|
+
repos:
|
|
13
|
+
- repo: https://github.com/abravalheri/validate-pyproject
|
|
14
|
+
rev: "0.26"
|
|
15
|
+
hooks:
|
|
16
|
+
- id: validate-pyproject
|
|
17
|
+
|
|
18
|
+
- repo: https://github.com/rhysd/actionlint
|
|
19
|
+
rev: v1.7.12
|
|
20
|
+
hooks:
|
|
21
|
+
- id: actionlint
|
|
22
|
+
files: "^\\.github/workflows/.*\\.ya?ml$"
|
|
23
|
+
|
|
24
|
+
- repo: https://github.com/adhtruong/mirrors-typos
|
|
25
|
+
rev: v1.50.1
|
|
26
|
+
hooks:
|
|
27
|
+
- id: typos
|
|
28
|
+
args: [--force-exclude] # omitting --write-changes
|
|
29
|
+
|
|
30
|
+
- repo: https://github.com/astral-sh/ruff-pre-commit
|
|
31
|
+
rev: v0.16.6
|
|
32
|
+
hooks:
|
|
33
|
+
- id: ruff-check
|
|
34
|
+
- id: ruff-format
|
|
35
|
+
|
|
36
|
+
- repo: local
|
|
37
|
+
hooks:
|
|
38
|
+
- id: mypy
|
|
39
|
+
name: mypy
|
|
40
|
+
entry: uv run --active mypy
|
|
41
|
+
language: system
|
|
42
|
+
pass_filenames: false
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
BSD 3-Clause License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2023, Talley Lambert
|
|
4
|
+
|
|
5
|
+
Redistribution and use in source and binary forms, with or without
|
|
6
|
+
modification, are permitted provided that the following conditions are met:
|
|
7
|
+
|
|
8
|
+
1. Redistributions of source code must retain the above copyright notice, this
|
|
9
|
+
list of conditions and the following disclaimer.
|
|
10
|
+
|
|
11
|
+
2. Redistributions in binary form must reproduce the above copyright notice,
|
|
12
|
+
this list of conditions and the following disclaimer in the documentation
|
|
13
|
+
and/or other materials provided with the distribution.
|
|
14
|
+
|
|
15
|
+
3. Neither the name of the copyright holder nor the names of its
|
|
16
|
+
contributors may be used to endorse or promote products derived from
|
|
17
|
+
this software without specific prior written permission.
|
|
18
|
+
|
|
19
|
+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
|
|
20
|
+
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
21
|
+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
|
|
22
|
+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
|
|
23
|
+
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
24
|
+
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
|
|
25
|
+
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
|
|
26
|
+
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
|
|
27
|
+
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
|
28
|
+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
@@ -0,0 +1,215 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: pydantic-tensorstore
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Pydantic models for TensorStore specifications
|
|
5
|
+
Project-URL: homepage, https://github.com/tlambert03/pydantic-tensorstore
|
|
6
|
+
Project-URL: repository, https://github.com/tlambert03/pydantic-tensorstore
|
|
7
|
+
Project-URL: changelog, https://github.com/tlambert03/pydantic-tensorstore/releases
|
|
8
|
+
Author-email: Talley Lambert <talley.lambert@gmail.com>
|
|
9
|
+
License: BSD-3-Clause
|
|
10
|
+
License-File: LICENSE
|
|
11
|
+
Keywords: pydantic,scientific-computing,tensorstore,validation
|
|
12
|
+
Classifier: Development Status :: 3 - Alpha
|
|
13
|
+
Classifier: License :: OSI Approved :: BSD License
|
|
14
|
+
Classifier: Programming Language :: Python :: 3
|
|
15
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
19
|
+
Classifier: Topic :: Scientific/Engineering
|
|
20
|
+
Classifier: Typing :: Typed
|
|
21
|
+
Requires-Python: >=3.11
|
|
22
|
+
Requires-Dist: numpy>=1.23.2
|
|
23
|
+
Requires-Dist: numpy>=2.1; python_version >= '3.13'
|
|
24
|
+
Requires-Dist: numpy>=2.3.2; python_version >= '3.14'
|
|
25
|
+
Requires-Dist: pydantic-core>=2.35.0; python_version >= '3.14'
|
|
26
|
+
Requires-Dist: pydantic>=2.11.0
|
|
27
|
+
Provides-Extra: tensorstore
|
|
28
|
+
Requires-Dist: tensorstore>=0.1.68; extra == 'tensorstore'
|
|
29
|
+
Requires-Dist: tensorstore>=0.1.70; (python_version >= '3.13') and extra == 'tensorstore'
|
|
30
|
+
Requires-Dist: tensorstore>=0.1.79; (python_version >= '3.14') and extra == 'tensorstore'
|
|
31
|
+
Description-Content-Type: text/markdown
|
|
32
|
+
|
|
33
|
+
# pydantic-tensorstore
|
|
34
|
+
|
|
35
|
+
[](https://github.com/tlambert03/pydantic-tensorstore/raw/main/LICENSE)
|
|
36
|
+
[](https://pypi.org/project/pydantic-tensorstore)
|
|
37
|
+
[](https://python.org)
|
|
38
|
+
[](https://github.com/tlambert03/pydantic-tensorstore/actions/workflows/ci.yml)
|
|
39
|
+
[](https://codecov.io/gh/tlambert03/pydantic-tensorstore)
|
|
40
|
+
|
|
41
|
+
*Type-safe, validated TensorStore specifications with Pydantic*
|
|
42
|
+
|
|
43
|
+
## Motivation
|
|
44
|
+
|
|
45
|
+
[TensorStore](https://github.com/google/tensorstore) is an exceptional C++ and
|
|
46
|
+
Python library for reading and writing large multi-dimensional arrays. It
|
|
47
|
+
supports numerous storage formats (Zarr, N5, Neuroglancer Precomputed) and
|
|
48
|
+
backends (local files, cloud storage, memory), making it incredibly powerful for
|
|
49
|
+
scientific computing and data analysis.
|
|
50
|
+
|
|
51
|
+
However, TensorStore has some pain points that this library attempts to address:
|
|
52
|
+
|
|
53
|
+
- **Poor type hinting**: TensorStore specifications are typically created as
|
|
54
|
+
dictionaries with minimal type information, making it difficult to know what
|
|
55
|
+
fields are available or required
|
|
56
|
+
- **Complex documentation**: Creating proper TensorStore JSON spec objects often
|
|
57
|
+
requires constantly referencing web documentation to understand the various
|
|
58
|
+
options and their relationships
|
|
59
|
+
- **No IDE support**: Without proper types, IDEs can't provide autocomplete,
|
|
60
|
+
validation, or refactoring support
|
|
61
|
+
|
|
62
|
+
**pydantic-tensorstore** solves these issues by providing:
|
|
63
|
+
|
|
64
|
+
- [x] **Full type safety** with Pydantic v2 models
|
|
65
|
+
- [x] **Excellent IDE support** with autocomplete and validation
|
|
66
|
+
- [x] **Clear, actionable error messages** when specifications are invalid
|
|
67
|
+
- [x] **Runtime validation** with detailed error reporting
|
|
68
|
+
- [x] **Seamless conversion** to native TensorStore specs
|
|
69
|
+
- [x] **Documentation** embedded in the type system (field descriptions state
|
|
70
|
+
TensorStore's defaults)
|
|
71
|
+
|
|
72
|
+
## Quick Example
|
|
73
|
+
|
|
74
|
+
Instead of wrestling with raw dictionaries:
|
|
75
|
+
|
|
76
|
+
```python
|
|
77
|
+
# Raw TensorStore (no types, no validation, no IDE support)
|
|
78
|
+
import tensorstore as ts
|
|
79
|
+
|
|
80
|
+
spec = {
|
|
81
|
+
"driver": "zarr",
|
|
82
|
+
"kvstore": {"driver": "file", "path": "/data/"},
|
|
83
|
+
"metadata": {
|
|
84
|
+
"chunks": [64, 64],
|
|
85
|
+
"compressor": {"id": "blosc", "cname": "lz4", "clevel": 5},
|
|
86
|
+
"dtype": "<f4", # Is this right? 🤔
|
|
87
|
+
},
|
|
88
|
+
}
|
|
89
|
+
# Hope it works! 🤞
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
Use type-safe, validated specifications with IDE autocompletion:
|
|
93
|
+
|
|
94
|
+
```python
|
|
95
|
+
# pydantic-tensorstore (full types, validation, IDE support)
|
|
96
|
+
import pydantic_tensorstore as pts
|
|
97
|
+
|
|
98
|
+
spec = pts.Zarr2Spec(
|
|
99
|
+
kvstore=pts.MemoryKvStore(),
|
|
100
|
+
metadata=pts.Zarr2Metadata(
|
|
101
|
+
chunks=[64, 64],
|
|
102
|
+
compressor=pts.Zarr2CompressorBlosc(cname="lz4", clevel=5),
|
|
103
|
+
dtype="<f4",
|
|
104
|
+
),
|
|
105
|
+
)
|
|
106
|
+
|
|
107
|
+
# Convert to native TensorStore when needed
|
|
108
|
+
ts_spec = spec.to_tensorstore() # requires tensorstore to be installed
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
To cast any dict (or JSON string, or `tensorstore.Spec`) to a validated spec:
|
|
112
|
+
|
|
113
|
+
```python
|
|
114
|
+
from pydantic_tensorstore import validate_spec
|
|
115
|
+
|
|
116
|
+
validated_spec = validate_spec(raw_dict)
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
## Installation
|
|
120
|
+
|
|
121
|
+
```bash
|
|
122
|
+
pip install pydantic-tensorstore
|
|
123
|
+
|
|
124
|
+
# with a compatible version of tensorstore installed for .to_tensorstore() support
|
|
125
|
+
pip install 'pydantic-tensorstore[tensorstore]'
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
## TensorStore compatibility
|
|
129
|
+
|
|
130
|
+
TensorStore's JSON spec is defined by the
|
|
131
|
+
[schema files in its repository](https://github.com/google/tensorstore/search?q=filename%3Aschema.yml).
|
|
132
|
+
The models here are checked field-by-field against those files for a single
|
|
133
|
+
pinned release, exposed as `pydantic_tensorstore.TENSORSTORE_VERSION`.
|
|
134
|
+
|
|
135
|
+
- **Verified against**: tensorstore `0.1.85` (`TENSORSTORE_VERSION`)
|
|
136
|
+
- **Tested with**: tensorstore `>= 0.1.68` (`MIN_TENSORSTORE_VERSION`). CI runs
|
|
137
|
+
the full suite against the oldest, a midpoint, the pinned, and the newest
|
|
138
|
+
release. Nearly everything the models can express already works on 0.1.68;
|
|
139
|
+
the dozen fields and data types TensorStore added later carry a `since`
|
|
140
|
+
marker.
|
|
141
|
+
- **Version check**: `spec.required_tensorstore_version()` reports the oldest
|
|
142
|
+
release a spec needs, and `to_tensorstore()` raises
|
|
143
|
+
`UnsupportedTensorStoreVersionError` naming the offending fields when the
|
|
144
|
+
installed tensorstore is older (pass `check_version=False` to skip).
|
|
145
|
+
- **Drift**: a weekly CI job re-checks the models against the newest
|
|
146
|
+
tensorstore release and opens an issue when something changes.
|
|
147
|
+
|
|
148
|
+
```python
|
|
149
|
+
>>> spec = pts.Zarr3Spec(kvstore="memory://", open_as_void=True)
|
|
150
|
+
>>> spec.required_tensorstore_version()
|
|
151
|
+
'0.1.85'
|
|
152
|
+
>>> spec.to_tensorstore() # with tensorstore 0.1.68 installed
|
|
153
|
+
UnsupportedTensorStoreVersionError: installed tensorstore 0.1.68 does not support:
|
|
154
|
+
open_as_void: requires tensorstore >= 0.1.85
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
Two design rules keep the models forward-compatible:
|
|
158
|
+
|
|
159
|
+
- **Optional fields default to `None`**, meaning "let TensorStore decide".
|
|
160
|
+
TensorStore's own default is documented in the field description. Dumping a
|
|
161
|
+
spec emits only the fields you set (an explicit `None` is kept as `null`,
|
|
162
|
+
since TensorStore gives `null` meaning for e.g. `compressor`).
|
|
163
|
+
- **KvStore URLs pass through.** `file://`, `memory://`, `s3://`, `gs://` and
|
|
164
|
+
`http(s)://` URLs are parsed into models; any other URL string (including
|
|
165
|
+
pipelines such as `"memory://a.zip|zip:"`) is kept verbatim and handed to
|
|
166
|
+
TensorStore unchanged.
|
|
167
|
+
|
|
168
|
+
## Known differences from TensorStore
|
|
169
|
+
|
|
170
|
+
A few deliberate or documented divergences, so they don't surprise you:
|
|
171
|
+
|
|
172
|
+
- **Partial specs are rejected.** TensorStore accepts `{"driver": "zarr3"}` with no
|
|
173
|
+
`kvstore`, to be filled in later via `ts.Spec.update()` or an `open()` argument.
|
|
174
|
+
The models require `kvstore` (and `cast`'s `dtype`), trading that pattern for a
|
|
175
|
+
clear "you forgot the kvstore" error. Specs produced by a real store always
|
|
176
|
+
include it.
|
|
177
|
+
- **Non-finite numbers become strings in JSON.** `model_dump_json()` writes
|
|
178
|
+
`"NaN"`, `"Infinity"` and `"-Infinity"`, which TensorStore reads back as floats.
|
|
179
|
+
Python-mode `model_dump()` keeps the float. Note that re-validating that JSON
|
|
180
|
+
gives you the string back, since `fill_value` is untyped.
|
|
181
|
+
- **`schema` is spelled `schema_` on the model,** because `schema` collides with a
|
|
182
|
+
Pydantic attribute. Both work at runtime, but mypy only accepts `schema_=`;
|
|
183
|
+
`validate_spec()` takes plain `"schema"` in a dict either way.
|
|
184
|
+
- **Arrays are not silently truncated.** `ArraySpec` refuses a `dtype` that would
|
|
185
|
+
lose data, rather than quietly rounding.
|
|
186
|
+
|
|
187
|
+
## Coverage
|
|
188
|
+
|
|
189
|
+
Every driver, kvstore, and context resource documented in tensorstore
|
|
190
|
+
`TENSORSTORE_VERSION` has a model.
|
|
191
|
+
|
|
192
|
+
| Kind | Models |
|
|
193
|
+
|---|---|
|
|
194
|
+
| Chunked drivers | `Zarr2Spec` (`zarr`/`zarr2`), `Zarr3Spec`, `N5Spec`, `NeuroglancerPrecomputedSpec` |
|
|
195
|
+
| Image drivers | `AvifSpec`, `BmpSpec`, `JpegSpec`, `PngSpec`, `TiffSpec`, `WebpSpec` |
|
|
196
|
+
| Other drivers | `ArraySpec`, `AutoSpec`, `CastSpec`, `DownsampleSpec`, `JsonSpec`, `StackSpec` |
|
|
197
|
+
| KvStores | `FileKvStore`, `MemoryKvStore`, `S3KvStore`, `GCSKvStore`, `HTTPKvStore`, `TsGrpcKvStore` |
|
|
198
|
+
| KvStore adapters | `OcdbtKvStore`, `ZipKvStore`, `KvStackKvStore`, `NeuroglancerUint64ShardedKvStore`, `Zarr3ShardingIndexedKvStore` |
|
|
199
|
+
| Codecs | `Zarr2Codec`, `Zarr3Codec` (+ every zarr3 codec), `N5Codec`, `NeuroglancerPrecomputedCodec` |
|
|
200
|
+
| Core | `Schema`, `ChunkLayout`, `IndexDomain`, `IndexTransform`, `Context` (+ every context resource), `Unit` |
|
|
201
|
+
|
|
202
|
+
`TensorStoreSpec` and `KvStore` are discriminated unions over all of the above;
|
|
203
|
+
`validate_spec()` and `validate_kvstore()` parse into them.
|
|
204
|
+
|
|
205
|
+
## Development
|
|
206
|
+
|
|
207
|
+
Tests compare the models against the vendored schema files in `tests/ts_schema/`
|
|
208
|
+
and, when tensorstore is installed, round-trip every spec through
|
|
209
|
+
`tensorstore.Spec`. To move to a newer tensorstore release:
|
|
210
|
+
|
|
211
|
+
```bash
|
|
212
|
+
uv run scripts/update_ts_schema.py --latest # refresh tests/ts_schema/
|
|
213
|
+
# bump TENSORSTORE_VERSION in src/pydantic_tensorstore/__init__.py
|
|
214
|
+
uv run pytest tests/test_schema_conformance.py # shows exactly what changed
|
|
215
|
+
```
|