pydantic-tensorstore 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (96) hide show
  1. pydantic_tensorstore-0.1.0/.github/DRIFT_TEMPLATE.md +14 -0
  2. pydantic_tensorstore-0.1.0/.github/ISSUE_TEMPLATE.md +15 -0
  3. pydantic_tensorstore-0.1.0/.github/TEST_FAIL_TEMPLATE.md +12 -0
  4. pydantic_tensorstore-0.1.0/.github/dependabot.yml +14 -0
  5. pydantic_tensorstore-0.1.0/.github/workflows/ci.yml +183 -0
  6. pydantic_tensorstore-0.1.0/.gitignore +115 -0
  7. pydantic_tensorstore-0.1.0/.pre-commit-config.yaml +42 -0
  8. pydantic_tensorstore-0.1.0/LICENSE +28 -0
  9. pydantic_tensorstore-0.1.0/PKG-INFO +215 -0
  10. pydantic_tensorstore-0.1.0/README.md +183 -0
  11. pydantic_tensorstore-0.1.0/examples/basic_usage.py +126 -0
  12. pydantic_tensorstore-0.1.0/pyproject.toml +184 -0
  13. pydantic_tensorstore-0.1.0/scripts/update_ts_schema.py +82 -0
  14. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/__init__.py +336 -0
  15. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_core/__init__.py +1 -0
  16. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_core/base.py +176 -0
  17. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_core/chunk_layout.py +209 -0
  18. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_core/codec.py +18 -0
  19. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_core/context.py +219 -0
  20. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_core/schema.py +56 -0
  21. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_core/spec.py +178 -0
  22. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_core/transform.py +277 -0
  23. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_core/zarr3_codecs.py +174 -0
  24. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_drivers/__init__.py +89 -0
  25. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_drivers/array.py +94 -0
  26. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_drivers/auto.py +25 -0
  27. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_drivers/cast.py +20 -0
  28. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_drivers/downsample.py +21 -0
  29. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_drivers/image.py +73 -0
  30. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_drivers/json.py +23 -0
  31. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_drivers/n5.py +199 -0
  32. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_drivers/neuroglancer_precomputed.py +168 -0
  33. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_drivers/stack.py +26 -0
  34. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_drivers/zarr.py +302 -0
  35. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_drivers/zarr3.py +259 -0
  36. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_kvstore/__init__.py +108 -0
  37. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_kvstore/base.py +59 -0
  38. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_kvstore/file.py +24 -0
  39. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_kvstore/gcs.py +23 -0
  40. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_kvstore/http.py +23 -0
  41. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_kvstore/kvstack.py +33 -0
  42. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_kvstore/memory.py +21 -0
  43. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_kvstore/neuroglancer_uint64_sharded.py +49 -0
  44. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_kvstore/ocdbt.py +75 -0
  45. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_kvstore/s3.py +36 -0
  46. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_kvstore/tsgrpc.py +19 -0
  47. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_kvstore/zarr3_sharding_indexed.py +25 -0
  48. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_kvstore/zip.py +13 -0
  49. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_types.py +223 -0
  50. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/_validators.py +55 -0
  51. pydantic_tensorstore-0.1.0/src/pydantic_tensorstore/py.typed +0 -0
  52. pydantic_tensorstore-0.1.0/tests/conftest.py +35 -0
  53. pydantic_tensorstore-0.1.0/tests/test_dtypes.py +50 -0
  54. pydantic_tensorstore-0.1.0/tests/test_dump.py +89 -0
  55. pydantic_tensorstore-0.1.0/tests/test_examples.py +13 -0
  56. pydantic_tensorstore-0.1.0/tests/test_schema_conformance.py +445 -0
  57. pydantic_tensorstore-0.1.0/tests/test_since.py +203 -0
  58. pydantic_tensorstore-0.1.0/tests/test_spec_validation.py +161 -0
  59. pydantic_tensorstore-0.1.0/tests/test_tensorstore.py +799 -0
  60. pydantic_tensorstore-0.1.0/tests/ts_schema/VERSION +1 -0
  61. pydantic_tensorstore-0.1.0/tests/ts_schema/docs_chunk_layout_schema.yml +185 -0
  62. pydantic_tensorstore-0.1.0/tests/ts_schema/docs_codec_schema.yml +20 -0
  63. pydantic_tensorstore-0.1.0/tests/ts_schema/docs_context_schema.yml +80 -0
  64. pydantic_tensorstore-0.1.0/tests/ts_schema/docs_index_domain_schema.yml +108 -0
  65. pydantic_tensorstore-0.1.0/tests/ts_schema/docs_index_transform_schema.yml +198 -0
  66. pydantic_tensorstore-0.1.0/tests/ts_schema/docs_schema_schema.yml +126 -0
  67. pydantic_tensorstore-0.1.0/tests/ts_schema/docs_tensorstore_schema.yml +272 -0
  68. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_array_schema.yml +23 -0
  69. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_auto_schema.yml +77 -0
  70. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_cast_schema.yml +55 -0
  71. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_downsample_schema.yml +110 -0
  72. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_image_avif_schema.yml +57 -0
  73. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_image_bmp_schema.yml +45 -0
  74. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_image_jpeg_schema.yml +50 -0
  75. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_image_png_schema.yml +50 -0
  76. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_image_tiff_schema.yml +50 -0
  77. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_image_webp_schema.yml +57 -0
  78. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_json_schema.yml +73 -0
  79. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_kvs_backed_chunk_driver_schema.yml +138 -0
  80. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_n5_schema.yml +308 -0
  81. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_neuroglancer_precomputed_schema.yml +259 -0
  82. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_stack_schema.yml +44 -0
  83. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_zarr3_schema.yml +647 -0
  84. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_driver_zarr_schema.yml +357 -0
  85. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_kvstore_file_schema.yml +173 -0
  86. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_kvstore_gcs_schema.yml +152 -0
  87. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_kvstore_http_schema.yml +155 -0
  88. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_kvstore_kvstack_schema.yml +55 -0
  89. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_kvstore_memory_schema.yml +55 -0
  90. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_kvstore_neuroglancer_uint64_sharded_schema.yml +108 -0
  91. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_kvstore_ocdbt_schema.yml +232 -0
  92. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_kvstore_s3_schema.yml +299 -0
  93. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_kvstore_schema.yml +60 -0
  94. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_kvstore_tsgrpc_schema.yml +29 -0
  95. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_kvstore_zarr3_sharding_indexed_schema.yml +49 -0
  96. pydantic_tensorstore-0.1.0/tests/ts_schema/tensorstore_kvstore_zip_schema.yml +67 -0
@@ -0,0 +1,14 @@
1
+ ---
2
+ title: "[drift-bot] models out of sync with latest tensorstore"
3
+ labels: [drift]
4
+ ---
5
+ The weekly drift check failed: the models no longer match the newest tensorstore
6
+ release (its JSON schema files and/or runtime behaviour).
7
+
8
+ See the failed run for details: https://github.com/{{ env.GITHUB_REPOSITORY }}/actions/runs/{{ env.RUN_ID }}
9
+
10
+ To update:
11
+
12
+ 1. `uv run scripts/update_ts_schema.py --latest`
13
+ 2. Bump `TENSORSTORE_VERSION` in `src/pydantic_tensorstore/__init__.py`
14
+ 3. `uv run pytest tests/test_schema_conformance.py` and fix the reported gaps
@@ -0,0 +1,15 @@
1
+ * pydantic-tensorstore version:
2
+ * Python version:
3
+ * Operating System:
4
+
5
+ ### Description
6
+
7
+ Describe what you were trying to get done.
8
+ Tell us what happened, what went wrong, and what you expected to happen.
9
+
10
+ ### What I Did
11
+
12
+ ```
13
+ Paste the command(s) you ran and the output.
14
+ If there was a crash, please include the traceback here.
15
+ ```
@@ -0,0 +1,12 @@
1
+ ---
2
+ title: "{{ env.TITLE }}"
3
+ labels: [bug]
4
+ ---
5
+ The {{ workflow }} workflow failed on {{ date | date("YYYY-MM-DD HH:mm") }} UTC
6
+
7
+ The most recent failing test was on {{ env.PLATFORM }} py{{ env.PYTHON }}
8
+ with commit: {{ sha }}
9
+
10
+ Full run: https://github.com/{{ repo }}/actions/runs/{{ env.RUN_ID }}
11
+
12
+ (This post will be updated if another test fails, as long as this issue remains open.)
@@ -0,0 +1,14 @@
1
+ # https://docs.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
2
+
3
+ version: 2
4
+ updates:
5
+ - package-ecosystem: "github-actions"
6
+ directory: "/"
7
+ schedule:
8
+ interval: "quarterly"
9
+ commit-message:
10
+ prefix: "ci(dependabot):"
11
+ groups:
12
+ actions:
13
+ patterns:
14
+ - "*"
@@ -0,0 +1,183 @@
1
+ name: CI
2
+
3
+ on:
4
+ push:
5
+ branches: [main]
6
+ tags: [v*]
7
+ pull_request:
8
+ workflow_dispatch:
9
+ schedule:
10
+ # run every week (for --pre release tests)
11
+ - cron: "0 0 * * 0"
12
+
13
+ concurrency:
14
+ group: ${{ github.workflow }}-${{ github.ref }}
15
+ cancel-in-progress: true
16
+
17
+ jobs:
18
+ check-manifest:
19
+ runs-on: ubuntu-latest
20
+ steps:
21
+ - uses: actions/checkout@v7
22
+ - run: pipx run check-manifest
23
+
24
+ lint:
25
+ runs-on: ubuntu-latest
26
+ steps:
27
+ - uses: actions/checkout@v7
28
+ - uses: astral-sh/setup-uv@v7
29
+ with:
30
+ python-version: "3.13"
31
+ enable-cache: true
32
+ - run: uv run prek run --all-files
33
+
34
+ test:
35
+ name: ${{ matrix.platform }} (${{ matrix.python-version }}) [${{ matrix.resolution }}]
36
+ runs-on: ${{ matrix.platform }}
37
+ env:
38
+ UV_PRERELEASE: ${{ github.event_name == 'schedule' && 'allow' || 'if-necessary-or-explicit' }}
39
+ UV_RESOLUTION: ${{ matrix.resolution }}
40
+ PYTEST_ADDOPTS: ${{ matrix.resolution == 'lowest-direct' && '-W ignore' || '' }}
41
+ strategy:
42
+ fail-fast: false
43
+ matrix:
44
+ python-version: ["3.11", "3.12", "3.13", "3.14"]
45
+ platform: [ubuntu-latest, macos-latest, windows-latest]
46
+ resolution: ["highest", "lowest-direct"]
47
+
48
+ steps:
49
+ - uses: actions/checkout@v7
50
+
51
+ - name: 🐍 Set up Python ${{ matrix.python-version }}
52
+ uses: astral-sh/setup-uv@v7
53
+ with:
54
+ python-version: ${{ matrix.python-version }}
55
+ enable-cache: true
56
+
57
+ - name: Install Dependencies
58
+ run: uv sync --no-dev --group test
59
+
60
+ - name: 🧪 Run Tests
61
+ run: uv run --no-sync coverage run -p -m pytest -v
62
+
63
+ # If something goes wrong with --pre tests, we can open an issue in the repo
64
+ - name: 📝 Report --pre Failures
65
+ if: failure() && github.event_name == 'schedule'
66
+ uses: JasonEtco/create-an-issue@v2
67
+ env:
68
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
69
+ PLATFORM: ${{ matrix.platform }}
70
+ PYTHON: ${{ matrix.python-version }}
71
+ RUN_ID: ${{ github.run_id }}
72
+ TITLE: "[test-bot] pip install --pre is failing"
73
+ with:
74
+ filename: .github/TEST_FAIL_TEMPLATE.md
75
+ update_existing: true
76
+
77
+ - name: Upload coverage
78
+ uses: actions/upload-artifact@v7
79
+ with:
80
+ name: covreport-${{ matrix.platform }}-py${{ matrix.python-version }}-${{ matrix.resolution }}
81
+ path: ./.coverage*
82
+ include-hidden-files: true
83
+
84
+ upload_coverage:
85
+ if: always()
86
+ needs: [test]
87
+ uses: pyapp-kit/workflows/.github/workflows/upload-coverage.yml@v2
88
+ secrets:
89
+ codecov_token: ${{ secrets.CODECOV_TOKEN }}
90
+
91
+ # Explicit tensorstore version ladder: oldest supported, a midpoint, the pinned
92
+ # (schema-verified) release, and whatever is newest on PyPI.
93
+ tensorstore-compat:
94
+ name: tensorstore ${{ matrix.tensorstore }}
95
+ runs-on: ubuntu-latest
96
+ strategy:
97
+ fail-fast: false
98
+ matrix:
99
+ tensorstore: ["min", "0.1.77", "pinned", "latest"]
100
+ steps:
101
+ - uses: actions/checkout@v7
102
+ - uses: astral-sh/setup-uv@v7
103
+ with:
104
+ python-version: "3.12"
105
+ enable-cache: true
106
+ - run: uv sync --no-dev --group test
107
+ - name: Install tensorstore ${{ matrix.tensorstore }}
108
+ run: |
109
+ case "${{ matrix.tensorstore }}" in
110
+ min) v=$(uv run --no-sync python -c "import pydantic_tensorstore as p; print(p.MIN_TENSORSTORE_VERSION)") ;;
111
+ pinned) v=$(uv run --no-sync python -c "import pydantic_tensorstore as p; print(p.TENSORSTORE_VERSION)") ;;
112
+ latest) uv pip install --upgrade tensorstore; exit 0 ;;
113
+ *) v="${{ matrix.tensorstore }}" ;;
114
+ esac
115
+ uv pip install "tensorstore==$v"
116
+ - run: uv run --no-sync python -c "from importlib.metadata import version; print('tensorstore', version('tensorstore'))"
117
+ - run: uv run --no-sync pytest -v
118
+
119
+ # Weekly: does the newest tensorstore release still match our models?
120
+ drift:
121
+ name: drift check (latest tensorstore)
122
+ if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
123
+ runs-on: ubuntu-latest
124
+ steps:
125
+ - uses: actions/checkout@v7
126
+ - uses: astral-sh/setup-uv@v7
127
+ with:
128
+ python-version: "3.13"
129
+ enable-cache: true
130
+ - name: Fetch latest tensorstore schema
131
+ env:
132
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
133
+ run: uv run --no-project scripts/update_ts_schema.py --latest --dest "$RUNNER_TEMP/ts_schema"
134
+ - name: Install latest tensorstore
135
+ run: uv sync --no-dev --group test --upgrade-package tensorstore
136
+ - name: Compare models against latest schema and runtime
137
+ env:
138
+ PTS_SCHEMA_DIR: ${{ runner.temp }}/ts_schema
139
+ run: |
140
+ uv run --no-sync python -c "from importlib.metadata import version; print('tensorstore', version('tensorstore'))"
141
+ uv run --no-sync pytest -v tests/test_schema_conformance.py tests/test_dtypes.py tests/test_since.py tests/test_tensorstore.py --deselect tests/test_schema_conformance.py::test_vendored_version_matches_package
142
+ - name: Report drift
143
+ if: failure()
144
+ uses: JasonEtco/create-an-issue@v2
145
+ env:
146
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
147
+ RUN_ID: ${{ github.run_id }}
148
+ with:
149
+ filename: .github/DRIFT_TEMPLATE.md
150
+ update_existing: true
151
+
152
+ build-and-inspect-package:
153
+ name: Build & inspect package.
154
+ needs: [test, tensorstore-compat, lint]
155
+ runs-on: ubuntu-latest
156
+ steps:
157
+ - uses: actions/checkout@v7
158
+ with:
159
+ fetch-depth: 0
160
+ # no moving major tag as of v3; pin the full version
161
+ - uses: hynek/build-and-inspect-python-package@v3.0.1
162
+
163
+ upload-to-pypi:
164
+ name: Upload package to PyPI
165
+ needs: build-and-inspect-package
166
+ if: success() && startsWith(github.ref, 'refs/tags/') && github.event_name != 'schedule'
167
+ runs-on: ubuntu-latest
168
+ permissions:
169
+ id-token: write
170
+ contents: write
171
+
172
+ steps:
173
+ - name: Download built artifact to dist/
174
+ uses: actions/download-artifact@v8
175
+ with:
176
+ name: Packages
177
+ path: dist
178
+ - name: 🚢 Publish to PyPI
179
+ uses: pypa/gh-action-pypi-publish@release/v1
180
+ - uses: softprops/action-gh-release@v3
181
+ with:
182
+ generate_release_notes: true
183
+ files: "./dist/*"
@@ -0,0 +1,115 @@
1
+ # Byte-compiled / optimized / DLL files
2
+ __pycache__/
3
+ *.py[cod]
4
+ *$py.class
5
+
6
+ # C extensions
7
+ *.so
8
+
9
+ # Distribution / packaging
10
+ .Python
11
+ env/
12
+ build/
13
+ develop-eggs/
14
+ dist/
15
+ downloads/
16
+ eggs/
17
+ .eggs/
18
+ lib/
19
+ lib64/
20
+ parts/
21
+ sdist/
22
+ var/
23
+ wheels/
24
+ *.egg-info/
25
+ .installed.cfg
26
+ *.egg
27
+
28
+ .DS_Store
29
+
30
+ # PyInstaller
31
+ # Usually these files are written by a python script from a template
32
+ # before PyInstaller builds the exe, so as to inject date/other infos into it.
33
+ *.manifest
34
+ *.spec
35
+
36
+ # Installer logs
37
+ pip-log.txt
38
+ pip-delete-this-directory.txt
39
+
40
+ # Unit test / coverage reports
41
+ htmlcov/
42
+ .tox/
43
+ .coverage
44
+ .coverage.*
45
+ .cache
46
+ nosetests.xml
47
+ coverage.xml
48
+ *.cover
49
+ .hypothesis/
50
+ .pytest_cache/
51
+
52
+ # Translations
53
+ *.mo
54
+ *.pot
55
+
56
+ # Django stuff:
57
+ *.log
58
+ local_settings.py
59
+
60
+ # Flask stuff:
61
+ instance/
62
+ .webassets-cache
63
+
64
+ # Scrapy stuff:
65
+ .scrapy
66
+
67
+ # Sphinx documentation
68
+ docs/_build/
69
+
70
+ # PyBuilder
71
+ target/
72
+
73
+ # Jupyter Notebook
74
+ .ipynb_checkpoints
75
+
76
+ # pyenv
77
+ .python-version
78
+
79
+ # celery beat schedule file
80
+ celerybeat-schedule
81
+
82
+ # SageMath parsed files
83
+ *.sage.py
84
+
85
+ # dotenv
86
+ .env
87
+
88
+ # virtualenv
89
+ .venv
90
+ venv/
91
+ ENV/
92
+
93
+ # Spyder project settings
94
+ .spyderproject
95
+ .spyproject
96
+
97
+ # Rope project settings
98
+ .ropeproject
99
+
100
+ # mkdocs documentation
101
+ /site
102
+
103
+ # mypy
104
+ .mypy_cache/
105
+
106
+ # ruff
107
+ .ruff_cache/
108
+
109
+ # IDE settings
110
+ .vscode/
111
+ .idea/
112
+ uv.lock
113
+
114
+ # local agent settings
115
+ .claude/
@@ -0,0 +1,42 @@
1
+ # enable pre-commit.ci at https://pre-commit.ci/
2
+ # it adds:
3
+ # 1. auto fixing pull requests
4
+ # 2. auto updating the pre-commit configuration
5
+ ci:
6
+ autoupdate_schedule: monthly
7
+ autofix_commit_msg: "style(pre-commit.ci): auto fixes [...]"
8
+ autoupdate_commit_msg: "ci(pre-commit.ci): autoupdate"
9
+ skip:
10
+ - mypy
11
+
12
+ repos:
13
+ - repo: https://github.com/abravalheri/validate-pyproject
14
+ rev: "0.26"
15
+ hooks:
16
+ - id: validate-pyproject
17
+
18
+ - repo: https://github.com/rhysd/actionlint
19
+ rev: v1.7.12
20
+ hooks:
21
+ - id: actionlint
22
+ files: "^\\.github/workflows/.*\\.ya?ml$"
23
+
24
+ - repo: https://github.com/adhtruong/mirrors-typos
25
+ rev: v1.50.1
26
+ hooks:
27
+ - id: typos
28
+ args: [--force-exclude] # omitting --write-changes
29
+
30
+ - repo: https://github.com/astral-sh/ruff-pre-commit
31
+ rev: v0.16.6
32
+ hooks:
33
+ - id: ruff-check
34
+ - id: ruff-format
35
+
36
+ - repo: local
37
+ hooks:
38
+ - id: mypy
39
+ name: mypy
40
+ entry: uv run --active mypy
41
+ language: system
42
+ pass_filenames: false
@@ -0,0 +1,28 @@
1
+ BSD 3-Clause License
2
+
3
+ Copyright (c) 2023, Talley Lambert
4
+
5
+ Redistribution and use in source and binary forms, with or without
6
+ modification, are permitted provided that the following conditions are met:
7
+
8
+ 1. Redistributions of source code must retain the above copyright notice, this
9
+ list of conditions and the following disclaimer.
10
+
11
+ 2. Redistributions in binary form must reproduce the above copyright notice,
12
+ this list of conditions and the following disclaimer in the documentation
13
+ and/or other materials provided with the distribution.
14
+
15
+ 3. Neither the name of the copyright holder nor the names of its
16
+ contributors may be used to endorse or promote products derived from
17
+ this software without specific prior written permission.
18
+
19
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
20
+ AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
21
+ IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
22
+ DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
23
+ FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
24
+ DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
25
+ SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
26
+ CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
27
+ OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
28
+ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
@@ -0,0 +1,215 @@
1
+ Metadata-Version: 2.5
2
+ Name: pydantic-tensorstore
3
+ Version: 0.1.0
4
+ Summary: Pydantic models for TensorStore specifications
5
+ Project-URL: homepage, https://github.com/tlambert03/pydantic-tensorstore
6
+ Project-URL: repository, https://github.com/tlambert03/pydantic-tensorstore
7
+ Project-URL: changelog, https://github.com/tlambert03/pydantic-tensorstore/releases
8
+ Author-email: Talley Lambert <talley.lambert@gmail.com>
9
+ License: BSD-3-Clause
10
+ License-File: LICENSE
11
+ Keywords: pydantic,scientific-computing,tensorstore,validation
12
+ Classifier: Development Status :: 3 - Alpha
13
+ Classifier: License :: OSI Approved :: BSD License
14
+ Classifier: Programming Language :: Python :: 3
15
+ Classifier: Programming Language :: Python :: 3.11
16
+ Classifier: Programming Language :: Python :: 3.12
17
+ Classifier: Programming Language :: Python :: 3.13
18
+ Classifier: Programming Language :: Python :: 3.14
19
+ Classifier: Topic :: Scientific/Engineering
20
+ Classifier: Typing :: Typed
21
+ Requires-Python: >=3.11
22
+ Requires-Dist: numpy>=1.23.2
23
+ Requires-Dist: numpy>=2.1; python_version >= '3.13'
24
+ Requires-Dist: numpy>=2.3.2; python_version >= '3.14'
25
+ Requires-Dist: pydantic-core>=2.35.0; python_version >= '3.14'
26
+ Requires-Dist: pydantic>=2.11.0
27
+ Provides-Extra: tensorstore
28
+ Requires-Dist: tensorstore>=0.1.68; extra == 'tensorstore'
29
+ Requires-Dist: tensorstore>=0.1.70; (python_version >= '3.13') and extra == 'tensorstore'
30
+ Requires-Dist: tensorstore>=0.1.79; (python_version >= '3.14') and extra == 'tensorstore'
31
+ Description-Content-Type: text/markdown
32
+
33
+ # pydantic-tensorstore
34
+
35
+ [![License](https://img.shields.io/pypi/l/pydantic-tensorstore.svg?color=green)](https://github.com/tlambert03/pydantic-tensorstore/raw/main/LICENSE)
36
+ [![PyPI](https://img.shields.io/pypi/v/pydantic-tensorstore.svg?color=green)](https://pypi.org/project/pydantic-tensorstore)
37
+ [![Python Version](https://img.shields.io/pypi/pyversions/pydantic-tensorstore.svg?color=green)](https://python.org)
38
+ [![CI](https://github.com/tlambert03/pydantic-tensorstore/actions/workflows/ci.yml/badge.svg)](https://github.com/tlambert03/pydantic-tensorstore/actions/workflows/ci.yml)
39
+ [![codecov](https://codecov.io/gh/tlambert03/pydantic-tensorstore/branch/main/graph/badge.svg)](https://codecov.io/gh/tlambert03/pydantic-tensorstore)
40
+
41
+ *Type-safe, validated TensorStore specifications with Pydantic*
42
+
43
+ ## Motivation
44
+
45
+ [TensorStore](https://github.com/google/tensorstore) is an exceptional C++ and
46
+ Python library for reading and writing large multi-dimensional arrays. It
47
+ supports numerous storage formats (Zarr, N5, Neuroglancer Precomputed) and
48
+ backends (local files, cloud storage, memory), making it incredibly powerful for
49
+ scientific computing and data analysis.
50
+
51
+ However, TensorStore has some pain points that this library attempts to address:
52
+
53
+ - **Poor type hinting**: TensorStore specifications are typically created as
54
+ dictionaries with minimal type information, making it difficult to know what
55
+ fields are available or required
56
+ - **Complex documentation**: Creating proper TensorStore JSON spec objects often
57
+ requires constantly referencing web documentation to understand the various
58
+ options and their relationships
59
+ - **No IDE support**: Without proper types, IDEs can't provide autocomplete,
60
+ validation, or refactoring support
61
+
62
+ **pydantic-tensorstore** solves these issues by providing:
63
+
64
+ - [x] **Full type safety** with Pydantic v2 models
65
+ - [x] **Excellent IDE support** with autocomplete and validation
66
+ - [x] **Clear, actionable error messages** when specifications are invalid
67
+ - [x] **Runtime validation** with detailed error reporting
68
+ - [x] **Seamless conversion** to native TensorStore specs
69
+ - [x] **Documentation** embedded in the type system (field descriptions state
70
+ TensorStore's defaults)
71
+
72
+ ## Quick Example
73
+
74
+ Instead of wrestling with raw dictionaries:
75
+
76
+ ```python
77
+ # Raw TensorStore (no types, no validation, no IDE support)
78
+ import tensorstore as ts
79
+
80
+ spec = {
81
+ "driver": "zarr",
82
+ "kvstore": {"driver": "file", "path": "/data/"},
83
+ "metadata": {
84
+ "chunks": [64, 64],
85
+ "compressor": {"id": "blosc", "cname": "lz4", "clevel": 5},
86
+ "dtype": "<f4", # Is this right? 🤔
87
+ },
88
+ }
89
+ # Hope it works! 🤞
90
+ ```
91
+
92
+ Use type-safe, validated specifications with IDE autocompletion:
93
+
94
+ ```python
95
+ # pydantic-tensorstore (full types, validation, IDE support)
96
+ import pydantic_tensorstore as pts
97
+
98
+ spec = pts.Zarr2Spec(
99
+ kvstore=pts.MemoryKvStore(),
100
+ metadata=pts.Zarr2Metadata(
101
+ chunks=[64, 64],
102
+ compressor=pts.Zarr2CompressorBlosc(cname="lz4", clevel=5),
103
+ dtype="<f4",
104
+ ),
105
+ )
106
+
107
+ # Convert to native TensorStore when needed
108
+ ts_spec = spec.to_tensorstore() # requires tensorstore to be installed
109
+ ```
110
+
111
+ To cast any dict (or JSON string, or `tensorstore.Spec`) to a validated spec:
112
+
113
+ ```python
114
+ from pydantic_tensorstore import validate_spec
115
+
116
+ validated_spec = validate_spec(raw_dict)
117
+ ```
118
+
119
+ ## Installation
120
+
121
+ ```bash
122
+ pip install pydantic-tensorstore
123
+
124
+ # with a compatible version of tensorstore installed for .to_tensorstore() support
125
+ pip install 'pydantic-tensorstore[tensorstore]'
126
+ ```
127
+
128
+ ## TensorStore compatibility
129
+
130
+ TensorStore's JSON spec is defined by the
131
+ [schema files in its repository](https://github.com/google/tensorstore/search?q=filename%3Aschema.yml).
132
+ The models here are checked field-by-field against those files for a single
133
+ pinned release, exposed as `pydantic_tensorstore.TENSORSTORE_VERSION`.
134
+
135
+ - **Verified against**: tensorstore `0.1.85` (`TENSORSTORE_VERSION`)
136
+ - **Tested with**: tensorstore `>= 0.1.68` (`MIN_TENSORSTORE_VERSION`). CI runs
137
+ the full suite against the oldest, a midpoint, the pinned, and the newest
138
+ release. Nearly everything the models can express already works on 0.1.68;
139
+ the dozen fields and data types TensorStore added later carry a `since`
140
+ marker.
141
+ - **Version check**: `spec.required_tensorstore_version()` reports the oldest
142
+ release a spec needs, and `to_tensorstore()` raises
143
+ `UnsupportedTensorStoreVersionError` naming the offending fields when the
144
+ installed tensorstore is older (pass `check_version=False` to skip).
145
+ - **Drift**: a weekly CI job re-checks the models against the newest
146
+ tensorstore release and opens an issue when something changes.
147
+
148
+ ```python
149
+ >>> spec = pts.Zarr3Spec(kvstore="memory://", open_as_void=True)
150
+ >>> spec.required_tensorstore_version()
151
+ '0.1.85'
152
+ >>> spec.to_tensorstore() # with tensorstore 0.1.68 installed
153
+ UnsupportedTensorStoreVersionError: installed tensorstore 0.1.68 does not support:
154
+ open_as_void: requires tensorstore >= 0.1.85
155
+ ```
156
+
157
+ Two design rules keep the models forward-compatible:
158
+
159
+ - **Optional fields default to `None`**, meaning "let TensorStore decide".
160
+ TensorStore's own default is documented in the field description. Dumping a
161
+ spec emits only the fields you set (an explicit `None` is kept as `null`,
162
+ since TensorStore gives `null` meaning for e.g. `compressor`).
163
+ - **KvStore URLs pass through.** `file://`, `memory://`, `s3://`, `gs://` and
164
+ `http(s)://` URLs are parsed into models; any other URL string (including
165
+ pipelines such as `"memory://a.zip|zip:"`) is kept verbatim and handed to
166
+ TensorStore unchanged.
167
+
168
+ ## Known differences from TensorStore
169
+
170
+ A few deliberate or documented divergences, so they don't surprise you:
171
+
172
+ - **Partial specs are rejected.** TensorStore accepts `{"driver": "zarr3"}` with no
173
+ `kvstore`, to be filled in later via `ts.Spec.update()` or an `open()` argument.
174
+ The models require `kvstore` (and `cast`'s `dtype`), trading that pattern for a
175
+ clear "you forgot the kvstore" error. Specs produced by a real store always
176
+ include it.
177
+ - **Non-finite numbers become strings in JSON.** `model_dump_json()` writes
178
+ `"NaN"`, `"Infinity"` and `"-Infinity"`, which TensorStore reads back as floats.
179
+ Python-mode `model_dump()` keeps the float. Note that re-validating that JSON
180
+ gives you the string back, since `fill_value` is untyped.
181
+ - **`schema` is spelled `schema_` on the model,** because `schema` collides with a
182
+ Pydantic attribute. Both work at runtime, but mypy only accepts `schema_=`;
183
+ `validate_spec()` takes plain `"schema"` in a dict either way.
184
+ - **Arrays are not silently truncated.** `ArraySpec` refuses a `dtype` that would
185
+ lose data, rather than quietly rounding.
186
+
187
+ ## Coverage
188
+
189
+ Every driver, kvstore, and context resource documented in tensorstore
190
+ `TENSORSTORE_VERSION` has a model.
191
+
192
+ | Kind | Models |
193
+ |---|---|
194
+ | Chunked drivers | `Zarr2Spec` (`zarr`/`zarr2`), `Zarr3Spec`, `N5Spec`, `NeuroglancerPrecomputedSpec` |
195
+ | Image drivers | `AvifSpec`, `BmpSpec`, `JpegSpec`, `PngSpec`, `TiffSpec`, `WebpSpec` |
196
+ | Other drivers | `ArraySpec`, `AutoSpec`, `CastSpec`, `DownsampleSpec`, `JsonSpec`, `StackSpec` |
197
+ | KvStores | `FileKvStore`, `MemoryKvStore`, `S3KvStore`, `GCSKvStore`, `HTTPKvStore`, `TsGrpcKvStore` |
198
+ | KvStore adapters | `OcdbtKvStore`, `ZipKvStore`, `KvStackKvStore`, `NeuroglancerUint64ShardedKvStore`, `Zarr3ShardingIndexedKvStore` |
199
+ | Codecs | `Zarr2Codec`, `Zarr3Codec` (+ every zarr3 codec), `N5Codec`, `NeuroglancerPrecomputedCodec` |
200
+ | Core | `Schema`, `ChunkLayout`, `IndexDomain`, `IndexTransform`, `Context` (+ every context resource), `Unit` |
201
+
202
+ `TensorStoreSpec` and `KvStore` are discriminated unions over all of the above;
203
+ `validate_spec()` and `validate_kvstore()` parse into them.
204
+
205
+ ## Development
206
+
207
+ Tests compare the models against the vendored schema files in `tests/ts_schema/`
208
+ and, when tensorstore is installed, round-trip every spec through
209
+ `tensorstore.Spec`. To move to a newer tensorstore release:
210
+
211
+ ```bash
212
+ uv run scripts/update_ts_schema.py --latest # refresh tests/ts_schema/
213
+ # bump TENSORSTORE_VERSION in src/pydantic_tensorstore/__init__.py
214
+ uv run pytest tests/test_schema_conformance.py # shows exactly what changed
215
+ ```