atlas-python 0.9.0__tar.gz → 0.10.0__tar.gz

atlas_python-0.10.0/.github/workflows/atlas-python-docs.yaml

@@ -0,0 +1,88 @@
+name: atlas-python-docs
+on:
+    workflow_dispatch:
+        inputs:
+            version:
+                description: "Docs version to publish (e.g. 1.2.3). Leave blank to publish as 'dev'."
+                required: false
+                type: string
+            alias:
+                description: "Alias to update (e.g. latest)."
+                required: false
+                default: dev
+                type: string
+    release:
+        types: [published]
+    push:
+        tags:
+            - "v*" # triggers on tags like v1.0.0, v2.3, etc.
+permissions:
+    contents: write
+jobs:
+    deploy:
+        runs-on: ubuntu-latest
+        steps:
+            - uses: actions/checkout@v4
+            - name: Configure Git Credentials
+              run: |
+                  git config user.name github-actions[bot]
+                  git config user.email 41898282+github-actions[bot]@users.noreply.github.com
+
+            - name: Extract version
+              id: vars
+              env:
+                  INPUT_VERSION: ${{ github.event.inputs.version }}
+                  INPUT_ALIAS: ${{ github.event.inputs.alias }}
+              run: |
+                  if [[ "$GITHUB_REF" == refs/tags/v* ]]; then
+                      # Tag push or release: refs/tags/v1.2.3 -> version=1.2.3, alias=latest
+                      VERSION="${GITHUB_REF#refs/tags/v}"
+                      ALIAS=latest
+                  else
+                      # Manual run: use provided inputs, falling back to 'dev'
+                      VERSION="${INPUT_VERSION:-dev}"
+                      ALIAS="${INPUT_ALIAS:-dev}"
+                  fi
+                  echo "version=$VERSION" >> $GITHUB_OUTPUT
+                  echo "alias=$ALIAS" >> $GITHUB_OUTPUT
+
+            - name: Show version
+              run: |
+                  echo "Deploying docs version '${{ steps.vars.outputs.version }}' (alias '${{ steps.vars.outputs.alias }}')"
+
+            - uses: actions/setup-python@v6
+              with:
+                  python-version: "3.12"
+            - run: echo "cache_id=$(date --utc '+%V')" >> $GITHUB_ENV
+            - uses: actions/cache@v4
+              with:
+                  key: mkdocs-material-${{ env.cache_id }}
+                  path: ~/.cache
+                  restore-keys: |
+                      mkdocs-material-
+            - run: pip install mkdocs-material mkdocstrings[python] mike
+              # Build & install the package so the compiled `atlas._atlas` extension is
+              # importable — mkdocstrings preloads it to resolve the re-exported API.
+              # Editable (-e) so maturin places the compiled `_atlas.*.so` into the
+              # python-source tree (atlas-python/python/atlas/), which is where griffe's
+              # `paths: ["python"]` search path looks for it. A plain install puts it in
+              # site-packages, outside that search path, and the alias fails to resolve.
+              # ubuntu-latest ships a Rust toolchain, so maturin can build from source.
+            - run: pip install -e ./atlas-python
+            - env:
+                  VERSION: ${{ steps.vars.outputs.version }}
+                  ALIAS: ${{ steps.vars.outputs.alias }}
+              run: |
+                  git fetch origin gh-pages --depth=1 || true
+                  # mike rejects an alias identical to the version, so only pass it when it differs
+                  if [[ "$ALIAS" == "$VERSION" ]]; then
+                      mike deploy --push --update-aliases -F atlas-python/mkdocs.yml "$VERSION"
+                  else
+                      mike deploy --push --update-aliases -F atlas-python/mkdocs.yml "$VERSION" "$ALIAS"
+                  fi
+                  # The site root must always redirect to `latest`, so only (re)write the
+                  # root index.html when this run deploys the `latest` alias (releases).
+                  # Manual `dev` runs are published at /dev/ but never touch the root.
+                  if [[ "$ALIAS" == "latest" ]]; then
+                      mike set-default --push -F atlas-python/mkdocs.yml latest
+                  fi

{atlas_python-0.9.0 → atlas_python-0.10.0}/.gitignore

@@ -28,4 +28,8 @@ raw_smoke_tests/
 pyatlas/python/pyatlas/_pyatlas.pyd
 pyatlas/python/pyatlas/_pyatlas.abi3.so
 playground/
-data/
+data/
+atlas-python/python/atlas/_atlas.pyd
+atlas-python/python/atlas/_atlas.abi3.so
+atlas-python/python/atlas/_atlas*.so
+atlas-python/python/atlas/_atlas.pdb

{atlas_python-0.9.0 → atlas_python-0.10.0}/CONTRIBUTING.md

@@ -83,7 +83,7 @@ Key design points:
 - **Sync Python API, backed by an internal multi-threaded tokio runtime.** Each blocking call uses `py.allow_threads(|| RT.block_on(...))` so other Python threads can run.
 - **Type dispatch via macros.** Atlas's read/write methods are generic over dtype (`T: ArrayElement`); the bindings dispatch at runtime via a `numeric_dispatch!` macro in [pyatlas/src/dataset.rs](pyatlas/src/dataset.rs).
 - **`numpy`-zero-copy on the numeric path.** Python `np.ndarray` ↔ `ndarray::ArrayView` via the `numpy` crate.
-- **xarray integration** lives in [pyatlas/python/pyatlas/xarray.py](pyatlas/python/pyatlas/xarray.py). The `Atlas.add_xr_dataset` Rust pymethod and the `ds.atlas.write` accessor both delegate to the `_write_xarray_new_dataset` helper. dask-backed variables are streamed one chunk at a time via `arr.blocks[idx].compute()`.
+- **xarray integration** lives in [pyatlas/python/pyatlas/xarray.py](pyatlas/python/pyatlas/xarray.py). The `Atlas.add_xarray_dataset` Rust pymethod and the `ds.atlas.write` accessor both delegate to the `_write_xarray_new_dataset` helper. dask-backed variables are streamed one chunk at a time via `arr.blocks[idx].compute()`.
 
 ---
 

{atlas_python-0.9.0 → atlas_python-0.10.0}/Cargo.lock

@@ -28,9 +28,9 @@ checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c"
 
 [[package]]
 name = "array-format"
-version = "0.9.0"
+version = "0.10.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d91c6d4d26f667272f8930d41bc95118716e12858baf115940a626510e7b5873"
+checksum = "a637a3b5f23ba45eb53f09e643b363163114f0e94d0f6f05f330134cb86d381b"
 dependencies = [
  "bytes",
  "futures",
@@ -71,7 +71,7 @@ dependencies = [
 
 [[package]]
 name = "atlas-python"
-version = "0.9.0"
+version = "0.10.0"
 dependencies = [
  "atlas-rust",
  "ndarray",
@@ -86,7 +86,7 @@ dependencies = [
 
 [[package]]
 name = "atlas-rust"
-version = "0.9.0"
+version = "0.10.0"
 dependencies = [
  "array-format",
  "chrono",

{atlas_python-0.9.0 → atlas_python-0.10.0}/Cargo.toml

@@ -5,11 +5,11 @@ members = ["atlas-python"]
 # Registry name on crates.io (`atlas` is taken). The library is still
 # imported as `atlas` — see `[lib]` below.
 name = "atlas-rust"
-version = "0.9.0"
+version = "0.10.0"
 edition = "2024"
 description = "Directory-based store for thousands of N-dimensional datasets local or remote using object storage."
 license = "Apache-2.0"
-repository = "https://github.com/robinskil/atlas"
+repository = "https://github.com/maris-development/atlas"
 readme = "README.md"
 keywords = ["array", "ndarray", "scientific", "storage", "object-store"]
 categories = ["science", "data-structures", "filesystem"]
@@ -22,7 +22,7 @@ all-features = true
 rustdoc-args = ["--cfg", "docsrs"]
 
 [dependencies]
-array-format = "0.9.0"
+array-format = "0.10.0"
 object_store = "0.13"
 serde = { version = "1", features = ["derive"] }
 serde_json = "1"

{atlas_python-0.9.0 → atlas_python-0.10.0}/LICENSE

@@ -186,7 +186,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright [yyyy] [name of copyright owner]
+   Copyright 2026 Robin Kooyman
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

atlas_python-0.10.0/PKG-INFO

@@ -0,0 +1,291 @@
+Metadata-Version: 2.4
+Name: atlas-python
+Version: 0.10.0
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Rust
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering
+Requires-Dist: numpy>=1.23
+Requires-Dist: xarray>=2023.1
+Requires-Dist: dask>=2023.1
+Requires-Dist: zarr>=3 ; extra == 'bench'
+Requires-Dist: numcodecs ; extra == 'bench'
+Requires-Dist: netcdf4 ; extra == 'bench'
+Requires-Dist: obstore>=0.9 ; extra == 'cloud'
+Requires-Dist: pytest ; extra == 'test'
+Requires-Dist: netcdf4 ; extra == 'test'
+Provides-Extra: bench
+Provides-Extra: cloud
+Provides-Extra: test
+License-File: LICENSE
+Summary: Python bindings for the ATLAS array store
+Keywords: array,ndarray,xarray,scientific,storage,object-store
+Author-email: Robin Kooyman <robin.kooyman.work@gmail.com>
+License-Expression: Apache-2.0
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
+Project-URL: Homepage, https://github.com/maris-development/atlas
+Project-URL: Issues, https://github.com/maris-development/atlas/issues
+Project-URL: Repository, https://github.com/maris-development/atlas
+
+# atlas-python
+
+Python bindings for **ATLAS** (Aggregated Tensor Large Array Store) — a directory-based store
+for many similarly-shaped N-dimensional arrays, backed by local files or any object store
+(S3 / GCS / Azure / HTTP). Built on a Rust core with a synchronous, NumPy-native API and
+first-class [xarray](https://docs.xarray.dev) integration.
+
+```bash
+pip install atlas-python
+```
+
+```python
+import atlas
+```
+
+| Extra | Install | Adds |
+| --- | --- | --- |
+| cloud | `pip install "atlas-python[cloud]"` | S3 / GCS / Azure / HTTP backends via [obstore](https://github.com/developmentseed/obstore) |
+
+`numpy`, `xarray`, and `dask` are installed automatically.
+
+## Quick start
+
+```python
+import numpy as np
+import atlas
+
+# The `with` block flushes (== close) on exit. Nothing is persisted before that.
+with atlas.Atlas.create("/tmp/my_store", codec="zstd") as store:   # "zstd" | "lz4" | "none"
+    ds = store.create_dataset("jan_2024")
+    ds.define_array(
+        "temperature",
+        dtype="float32",
+        dims=["lat", "lon"],
+        shape=[8, 16],
+        chunk_shape=[4, 8],
+        fill_value=float("nan"),   # unwritten cells read back as NaN; NaN cells count as nulls in stats
+    )
+    ds.write_array("temperature", start=[0, 0], data=np.full((8, 16), 20.0, dtype=np.float32))
+    ds.set_attribute("month", 1)
+    ds.set_attribute("station", "KNMI")
+
+# Reopen and read
+store = atlas.Atlas.open("/tmp/my_store")
+ds = store.open_dataset("jan_2024")
+arr = ds.read_array("temperature")                    # full read -> np.ndarray
+chunk = ds.read_array("temperature", [0, 0], [4, 8])  # partial read
+stats = ds.array_stats("temperature")                 # {"row_count", "null_count", "min", "max"}
+month = ds.get_attribute("month")                     # 1
+```
+
+## Durability model
+
+This is the one concept to internalise: **writes are buffered in memory and only hit disk on
+`flush()`.**
+
+The store's metadata is loaded once on `open`/`create`. Every subsequent mutation — creating
+datasets, defining arrays, `write_array`, `set_attribute` — updates in-memory state only.
+**Nothing reaches disk until `store.flush()` (equivalently `store.close()`, or the `with store:`
+block exiting).** Dropping an `Atlas` without flushing abandons every pending write.
+
+The payoff: N consecutive writes amortise to a single flush — one delta file per touched array
+name and one metadata rewrite, no matter how many datasets you touched.
+
+```python
+store = atlas.Atlas.create("/tmp/my_store")
+# ... many create_dataset / write_array calls ...
+store.flush()   # the single durability boundary
+```
+
+## xarray integration
+
+Importing `atlas` registers an accessor at `xr.Dataset.atlas`, so the integration is always
+available. The store must exist first; you then append xarray datasets to it.
+
+```python
+import numpy as np, xarray as xr, atlas
+
+ds = xr.Dataset(
+    data_vars={
+        "temperature": (["lat", "lon"], np.arange(8 * 16, dtype=np.float32).reshape(8, 16),
+                        {"units": "C", "long_name": "surface temperature"}),
+    },
+    coords={"lat": np.arange(8, dtype=np.float32), "lon": np.arange(16, dtype=np.float32)},
+    attrs={"month": 1, "station": "KNMI"},
+)
+
+with atlas.Atlas.create("/tmp/my_store") as store:
+    store.add_xarray_dataset(ds, "jan_2024")     # store-side method
+    ds.atlas.write(store, "jan_2025")        # xarray accessor (same effect)
+
+# Read back as an xr.Dataset
+store = atlas.Atlas.open("/tmp/my_store")
+ds_back = store.open_as_xarray_dataset("jan_2024")
+xr.testing.assert_identical(ds, ds_back)
+```
+
+### Bulk ingestion
+
+`add_xarray_dataset` never flushes by itself — N consecutive calls accumulate in memory and a single
+`flush()` (or the `with` exit) persists everything.
+
+```python
+import glob, os, atlas, xarray as xr
+
+with atlas.Atlas.create("/tmp/store") as store:
+    for nc_path in sorted(glob.glob("*.nc")):
+        name = os.path.splitext(os.path.basename(nc_path))[0]
+        store.add_xarray_dataset(xr.open_dataset(nc_path), name)
+# One delta file per array name across the whole batch (not one per file).
+```
+
+### Streaming dask-backed writes
+
+If a variable's `.data` is a `dask.array.Array` (e.g. from `xr.open_dataset(path, chunks=...)`
+or `ds.chunk({...})`), `add_xarray_dataset` / `ds.atlas.write` stream **one dask block at a time**
+into the store rather than materialising the whole array. The dask chunk shape becomes the
+on-disk `chunk_shape`, so the layout maps 1:1. Peak memory ≈ one chunk per variable.
+
+```python
+ds = xr.open_dataset("big.nc", chunks={"time": 100, "lat": -1, "lon": -1})
+with atlas.Atlas.create("/tmp/store") as store:
+    store.add_xarray_dataset(ds, "big")     # streams chunk-by-chunk
+```
+
+Pass `chunks={var: [...]}` to `add_xarray_dataset` / `ds.atlas.write` to override the on-disk chunk
+shape independently of dask's chunking.
+
+### Lazy dask-backed reads
+
+`store.open_as_xarray_dataset(name)` returns each variable dask-backed whenever it was stored with non-trivial
+chunking (`chunk_shape != shape`); the dask `chunks` tuple mirrors the on-disk chunk grid and each
+on-disk chunk is one dask task. Full-shape arrays (and 0-D scalars) come back eager as numpy. Call
+`.compute()` to materialise, or slice / `map_blocks` to operate lazily.
+
+```python
+ds_back = atlas.Atlas.open("/tmp/store").open_as_xarray_dataset("big")
+ds_back["temperature"].data              # -> dask.array.Array
+ds_back["temperature"][0:100].compute()  # reads exactly one chunk
+```
+
+Reads run under dask's **threaded** scheduler only — the `DatasetView` captured in the graph isn't
+picklable, so call `.compute()` before handing off to distributed/multiprocessing schedulers.
+
+### How xarray maps onto the store
+
+| Item | How it's stored |
+| --- | --- |
+| Each coord / data variable | A separate array, with `dims` mapped 1:1. |
+| Dataset attrs | Dataset attributes, plain keys. |
+| Per-variable attrs | Flattened as `{var}.{attr}` at the dataset attr level. |
+| Per-variable `_FillValue` | Consumed by `define_array` as a typed fill value (source `Dataset.attrs` is not mutated). See [Missing data](#missing-data). |
+| Coord vs data_var distinction | JSON list in the internal `_pyatlas_coords` attr. |
+| Non-scalar attr values (list, ndarray) | JSON-encoded string with a `json:` prefix marker. |
+
+Each `add_xarray_dataset` / `ds.atlas.write` creates a *new* dataset — there is no append-into-existing
+mode.
+
+### Missing data
+
+When a dataset is opened with `mask_and_scale=True` (xarray's default), masked cells become `NaN`
+(floats) / `NaT` (datetimes) and `_FillValue` is moved into `var.encoding`. So those cells are
+recorded as **null** (counted in `null_count`, excluded from min/max stats), arrays default to a
+sentinel fill on write: `NaN` for floats, `NaT` for `datetime64[ns]`, and `""` for strings (integers
+have none). Missing string cells (`None`/`NaN`) are substituted with the string fill and a warning is
+emitted, since a string can't be stored as null directly.
+
+Override per write with `fill_value` — a bare scalar (numeric arrays) or a `{var: scalar}` dict
+(`None` disables the default for that var); an explicit CF `_FillValue` attribute still takes
+precedence over the default:
+
+```python
+store.add_xarray_dataset(ds, "jan_2024", fill_value={"temperature": -9999.0})
+```
+
+See the [xarray guide](https://github.com/maris-development/atlas/blob/main/atlas-python/docs/guides/xarray.md#fill-values-and-missing-data)
+for the full resolution order.
+
+## Supported dtypes
+
+| numpy dtype | atlas dtype |
+| --- | --- |
+| `int8/16/32/64`, `uint8/16/32/64`, `float32/64` | matching numeric |
+| `datetime64[ns]` | `timestamp_nanoseconds` (aliases: `timestamp_ns`, `datetime64[ns]`) |
+| `object` (`str`/`bytes`), `\|S<n>`, `\|U<n>` | `string` (variable-length; reads return Python `str`) |
+
+- 0-D scalar arrays (`shape=[]`) are supported for every dtype above.
+- `bool` is available as an *attribute* type but not as an array dtype.
+- `binary`, `list[...]`, `fixed_size_list[...,N]` are reserved for a later release.
+
+## Cloud / object storage
+
+With the `cloud` extra, `Atlas.open` / `Atlas.create` accept an
+[obstore](https://github.com/developmentseed/obstore)-constructed S3 / GCS / Azure / HTTP store
+handle instead of a local path. The path-based local-filesystem API works without it. See the
+[cloud storage guide](https://github.com/maris-development/atlas/blob/main/atlas-python/docs/guides/cloud-storage.md).
+
+## API reference
+
+### `atlas.Atlas`
+
+| Method | Description |
+| --- | --- |
+| `Atlas.create(path, codec="zstd")` | Create a new store at `path`. |
+| `Atlas.open(path)` | Open an existing store. |
+| `create_dataset(name) -> DatasetView` | New dataset (in-memory until flush). |
+| `open_dataset(name) -> DatasetView` | Existing dataset. |
+| `delete_dataset(name)` | Remove a dataset (persisted on next `flush`). |
+| `list_datasets() -> list[str]` | All dataset names. |
+| `list_arrays() -> list[str]` | Distinct array names across datasets. |
+| `dataset_exists(name) -> bool` | Existence check. |
+| `add_xarray_dataset(ds, name, chunks=None, fill_value=None)` | Append an `xarray.Dataset` (does **not** flush). `fill_value` overrides the per-array fill (scalar or `{var: scalar}`); see [Missing data](#missing-data). |
+| `open_as_xarray_dataset(name) -> xr.Dataset` | Read a dataset back (chunked vars come back dask-backed). |
+| `open_as_many_xarray_dataset(names, concat_dim="dataset") -> xr.Dataset` | Open many datasets stacked along `concat_dim` (eager numpy). |
+| `flush()` | The single durability boundary — persist everything. |
+| `close()` | Alias for `flush()`; also the `with`-block exit. |
+| `compact()` | Reclaim tombstoned space across cached array files. |
+| `__enter__` / `__exit__` | Context-manager support (`__exit__` calls `close()`). |
+
+### `atlas.DatasetView`
+
+| Method | Description |
+| --- | --- |
+| `name` (property) | Dataset name. |
+| `list_arrays() -> list[str]` | Array names in this dataset. |
+| `define_array(name, dtype, dims, shape, chunk_shape=None, fill_value=None)` | Declare a new array. `fill_value` is a Python scalar matching the dtype; unwritten cells read back as it, and *written* cells equal to it count as nulls in `array_stats`. Dtype is enforced (`TypeError` on mismatch, `OverflowError` for out-of-range ints). |
+| `write_array(name, start, data)` | Write a numpy ndarray (matching the stored dtype). |
+| `read_array(name, start=None, shape=None) -> np.ndarray \| None` | Read full or partial; `None` if the array isn't in this dataset. |
+| `delete_array(name)` | Tombstone the array within this dataset. |
+| `array_meta(name) -> dict \| None` | `{"dtype", "shape", "chunk_shape", "dimension_names"}`. |
+| `array_stats(name) -> dict \| None` | `{"row_count", "null_count", "min", "max"}` — populated after `flush()`. |
+| `set_attribute(key, value, dtype=None)` | Type inferred from the Python value; pass `dtype` to override (e.g. `"int8"`, `"float32"`, `"timestamp_nanoseconds"`). On disk: bool, int64, float64, string, timestamp_nanoseconds. |
+| `get_attribute(key)` / `attributes()` | Single attribute or dict of all. |
+
+`DatasetView` does **not** expose its own `flush` / `compact` — both go through the parent `Atlas`.
+
+## Examples
+
+Runnable, self-contained scripts (each writes to a temp directory):
+
+- [01_basics.py](https://github.com/maris-development/atlas/blob/main/atlas-python/examples/01_basics.py) — create a store, define arrays, set attributes, reopen, read back.
+- [02_xarray.py](https://github.com/maris-development/atlas/blob/main/atlas-python/examples/02_xarray.py) — round-trip an `xr.Dataset` via both `store.add_xarray_dataset(...)` and the `ds.atlas.write(...)` accessor.
+- [03_dask_streaming.py](https://github.com/maris-development/atlas/blob/main/atlas-python/examples/03_dask_streaming.py) — stream a dask-chunked `xr.Dataset` in one chunk at a time.
+- [09_missing_data.py](https://github.com/maris-development/atlas/blob/main/atlas-python/examples/09_missing_data.py) — masked/missing cells with default `NaN` / `NaT` / `""` fills, `null_count`, and `fill_value=` overrides.
+
+## Performance
+
+ATLAS is tuned for collections of many similarly-shaped datasets. On a "1000 datasets" benchmark
+against netCDF4 and Zarr v3, the bulk read paths (`Atlas.open_as_many_xarray_dataset` /
+`Atlas.read_array_across_stacked`) beat Zarr by ~2.8× on large chunked slice reads, and on small
+per-dataset workloads ATLAS leads on both reads and writes. See the
+[benchmarks](https://github.com/maris-development/atlas/tree/main/atlas-python/benchmarks) for the full
+methodology, numbers, and an API picker for the fastest read path per workload.
+
+## Links
+
+- Source & issues: <https://github.com/maris-development/atlas>
+- License: Apache-2.0
+

{atlas_python-0.9.0 → atlas_python-0.10.0}/README.md

@@ -1,7 +1,20 @@
 # ATLAS — Aggregated Tensor Large Array Store
 
+[![CI](https://github.com/maris-development/atlas/actions/workflows/ci.yaml/badge.svg)](https://github.com/maris-development/atlas/actions/workflows/ci.yaml)
+[![crates.io](https://img.shields.io/crates/v/atlas-rust.svg?logo=rust)](https://crates.io/crates/atlas-rust)
+[![docs.rs](https://img.shields.io/docsrs/atlas-rust?logo=docsdotrs&label=docs.rs)](https://docs.rs/atlas-rust)
+[![PyPI](https://img.shields.io/pypi/v/atlas-python.svg?logo=pypi&logoColor=white)](https://pypi.org/project/atlas-python/)
+[![Python docs](https://img.shields.io/badge/docs-atlas--python-blue?logo=materialformkdocs&logoColor=white)](https://maris-development.github.io/atlas/)
+[![License](https://img.shields.io/crates/l/atlas-rust.svg)](LICENSE)
+
 A directory-based store for thousands of named datasets, each holding N-dimensional typed arrays. Built on top of the [`array-format`](https://github.com/robinskil/array-format) (`.af`) binary format, with configurable compression (Zstd, LZ4, or none), chunked I/O, and an [`object_store`](https://crates.io/crates/object_store) backend that works on local disk, S3, GCS, Azure Blob, and in-memory.
 
+**Think of it as a "zip" for N-dimensional datasets.** Where a `.zip` bundles many files into one archive, ATLAS gathers many NetCDF / Zarr-style datasets — anything that's a set of named N-dimensional arrays — into a single high-performance collection. But it's more than a bundle: instead of storing each dataset whole, ATLAS lays the data out **variable-first**, so every dataset's `temperature` lives in one file. That makes it cheap to scan or slice a single variable across thousands of datasets at once, while still reading back any individual dataset as a normal NetCDF/xarray-like object.
+
+> **Looking for Python?** The Python bindings live in [`atlas-python/`](atlas-python/) — `pip install atlas-python`, then `import atlas`. They add a NumPy-native API and first-class [xarray](https://docs.xarray.dev) integration. See the [atlas-python README](atlas-python/README.md) for usage.
+>
+> The rest of this document covers **how the format works and the Rust crate**.
+
 ---
 
 ## What it does
@@ -251,7 +264,7 @@ Choose LZ4 when decompression throughput matters more than storage size (e.g. la
 
 ### Write path
 
-Writes are buffered in-memory across the whole atlas. Calling `Atlas::flush()` compresses and writes every modified block across every cached array file, then rewrites `atlas.json` atomically (a single `PUT`). The write path scales with the number of modified chunks, not the number of datasets, and N consecutive `add_xr_dataset` / `create_dataset` calls amortise to a single flush.
+Writes are buffered in-memory across the whole atlas. Calling `Atlas::flush()` compresses and writes every modified block across every cached array file, then rewrites `atlas.json` atomically (a single `PUT`). The write path scales with the number of modified chunks, not the number of datasets, and N consecutive `add_xarray_dataset` / `create_dataset` calls amortise to a single flush.
 
 ### Compaction
 
@@ -295,7 +308,7 @@ the `(0:25, 0:25, 0:12)` slice down to chunk-level reads.
 | **atlas-bulk** | 1 store, `read_array_across_stacked` with slice push-down | **2.12** | 59 | 6387 |
 | **atlas + `--use-dask`** | 1 store, dask-threaded per-dataset `view.read_arrays(...)` | **3.21** | 60 | 6387 |
 | zarr | 1000 separate stores, `xr.open_mfdataset(parallel=True).isel(...)` | 5.99 | 38 | 6392 |
-| atlas (default) | 1 store, serial per-dataset `to_xarray(...).isel(...).load()` | 10.23 | 51 | 6387 |
+| atlas (default) | 1 store, serial per-dataset `open_as_xarray_dataset(...).isel(...).load()` | 10.23 | 51 | 6387 |
 | netcdf | 1000 `.nc` files, `xr.open_mfdataset(parallel=True).isel(...)` | 13.91 | 122 | 5596 |
 
 #### `--case profile --datasets 1000` (~67 MB raw — per-dataset overhead dominates)
@@ -326,9 +339,9 @@ For tiny-shape datasets the comparison is dominated by per-dataset overhead, whi
 
 ### What the numbers say
 
-- **Reads on gridded** (decompression-dominated, with realistic chunking + slice push-down): `atlas-bulk` reads in **2.12s vs zarr's 6.00s — 2.8× faster**. `atlas + --use-dask` (using the new `view.read_arrays(...)` fast path) hits **3.21s — 1.9× faster than zarr**. The win comes from atlas's structural advantage (one open of one file per variable, in-memory metadata) combined with APIs that bypass `to_xarray`'s xr.Dataset + per-chunk dask graph overhead. zarr and netcdf both push the slice down through `open_mfdataset(...).isel(...)` via dask graph optimization — they're not penalised by the chunking; atlas just wins by amortising metadata and skipping per-dataset Python overhead.
+- **Reads on gridded** (decompression-dominated, with realistic chunking + slice push-down): `atlas-bulk` reads in **2.12s vs zarr's 6.00s — 2.8× faster**. `atlas + --use-dask` (using the new `view.read_arrays(...)` fast path) hits **3.21s — 1.9× faster than zarr**. The win comes from atlas's structural advantage (one open of one file per variable, in-memory metadata) combined with APIs that bypass `open_as_xarray_dataset`'s xr.Dataset + per-chunk dask graph overhead. zarr and netcdf both push the slice down through `open_mfdataset(...).isel(...)` via dask graph optimization — they're not penalised by the chunking; atlas just wins by amortising metadata and skipping per-dataset Python overhead.
 - **Reads on profile** (overhead-dominated): atlas wins by an order of magnitude. `atlas-bulk` is **~50× faster than zarr** (0.08s vs 4.07s) because the per-dataset I/O is small enough that everything is overhead — and atlas's structural design is built for exactly that case. Default serial `atlas` is still ~12× faster than zarr.
-- **Default `atlas.to_xarray` iteration is currently SLOW on chunked storage** (10.23s vs zarr's 6.00s). It goes through `to_xarray(name)` which returns dask-backed arrays per chunk (8 chunks/var × 3 vars × 1000 datasets = 24,000 dask delayed tasks just to build the graph). Dask graph overhead exceeds the parallelism win. **The fix is to use `view.read_arrays(vars, start, shape)` for per-dataset slice reads** — that's what `atlas + --use-dask` does internally and why it hits 3.21s. For cross-dataset reads of the *same* slice from many datasets, prefer `Atlas.to_xarray_many` / `read_array_across_stacked` (the `atlas-bulk` row).
+- **Default `atlas.open_as_xarray_dataset` iteration is currently SLOW on chunked storage** (10.23s vs zarr's 6.00s). It goes through `open_as_xarray_dataset(name)` which returns dask-backed arrays per chunk (8 chunks/var × 3 vars × 1000 datasets = 24,000 dask delayed tasks just to build the graph). Dask graph overhead exceeds the parallelism win. **The fix is to use `view.read_arrays(vars, start, shape)` for per-dataset slice reads** — that's what `atlas + --use-dask` does internally and why it hits 3.21s. For cross-dataset reads of the *same* slice from many datasets, prefer `Atlas.open_as_many_xarray_dataset` / `read_array_across_stacked` (the `atlas-bulk` row).
 - **Workload sensitivity**: `--use-dask` helps when per-dataset decompression is the bottleneck and *hurts* when per-dataset overhead is the bottleneck (profile: default `atlas` 0.32s → dask 2.03s). Picking the right API for the workload matters more than picking dask vs serial.
 - **Writes on gridded**: zarr is fastest (~31s vs atlas's ~50s) — each `to_zarr(...)` just dumps bytes into a per-dataset subtree, while atlas does more bookkeeping per call (schema registration, per-dataset attrs into the shared `atlas.msgpack.zst`, block addressing in the shared array file). Atlas still beats netcdf decisively.
 - **Writes on profile**: atlas wins ~12× (0.91s vs zarr's 11.43s, netcdf's 2.98s). At small per-dataset sizes the metadata write dominates, and atlas's amortised single-flush model wins.

{atlas_python-0.9.0 → atlas_python-0.10.0}/atlas-python/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "atlas-python"
-version = "0.9.0"
+version = "0.10.0"
 edition = "2021"
 readme = "README.md"