atlas-python 0.9.1__tar.gz → 0.10.0__tar.gz

atlas_python-0.10.0/.github/workflows/atlas-python-docs.yaml

@@ -0,0 +1,88 @@
+name: atlas-python-docs
+on:
+    workflow_dispatch:
+        inputs:
+            version:
+                description: "Docs version to publish (e.g. 1.2.3). Leave blank to publish as 'dev'."
+                required: false
+                type: string
+            alias:
+                description: "Alias to update (e.g. latest)."
+                required: false
+                default: dev
+                type: string
+    release:
+        types: [published]
+    push:
+        tags:
+            - "v*" # triggers on tags like v1.0.0, v2.3, etc.
+permissions:
+    contents: write
+jobs:
+    deploy:
+        runs-on: ubuntu-latest
+        steps:
+            - uses: actions/checkout@v4
+            - name: Configure Git Credentials
+              run: |
+                  git config user.name github-actions[bot]
+                  git config user.email 41898282+github-actions[bot]@users.noreply.github.com
+
+            - name: Extract version
+              id: vars
+              env:
+                  INPUT_VERSION: ${{ github.event.inputs.version }}
+                  INPUT_ALIAS: ${{ github.event.inputs.alias }}
+              run: |
+                  if [[ "$GITHUB_REF" == refs/tags/v* ]]; then
+                      # Tag push or release: refs/tags/v1.2.3 -> version=1.2.3, alias=latest
+                      VERSION="${GITHUB_REF#refs/tags/v}"
+                      ALIAS=latest
+                  else
+                      # Manual run: use provided inputs, falling back to 'dev'
+                      VERSION="${INPUT_VERSION:-dev}"
+                      ALIAS="${INPUT_ALIAS:-dev}"
+                  fi
+                  echo "version=$VERSION" >> $GITHUB_OUTPUT
+                  echo "alias=$ALIAS" >> $GITHUB_OUTPUT
+
+            - name: Show version
+              run: |
+                  echo "Deploying docs version '${{ steps.vars.outputs.version }}' (alias '${{ steps.vars.outputs.alias }}')"
+
+            - uses: actions/setup-python@v6
+              with:
+                  python-version: "3.12"
+            - run: echo "cache_id=$(date --utc '+%V')" >> $GITHUB_ENV
+            - uses: actions/cache@v4
+              with:
+                  key: mkdocs-material-${{ env.cache_id }}
+                  path: ~/.cache
+                  restore-keys: |
+                      mkdocs-material-
+            - run: pip install mkdocs-material mkdocstrings[python] mike
+              # Build & install the package so the compiled `atlas._atlas` extension is
+              # importable — mkdocstrings preloads it to resolve the re-exported API.
+              # Editable (-e) so maturin places the compiled `_atlas.*.so` into the
+              # python-source tree (atlas-python/python/atlas/), which is where griffe's
+              # `paths: ["python"]` search path looks for it. A plain install puts it in
+              # site-packages, outside that search path, and the alias fails to resolve.
+              # ubuntu-latest ships a Rust toolchain, so maturin can build from source.
+            - run: pip install -e ./atlas-python
+            - env:
+                  VERSION: ${{ steps.vars.outputs.version }}
+                  ALIAS: ${{ steps.vars.outputs.alias }}
+              run: |
+                  git fetch origin gh-pages --depth=1 || true
+                  # mike rejects an alias identical to the version, so only pass it when it differs
+                  if [[ "$ALIAS" == "$VERSION" ]]; then
+                      mike deploy --push --update-aliases -F atlas-python/mkdocs.yml "$VERSION"
+                  else
+                      mike deploy --push --update-aliases -F atlas-python/mkdocs.yml "$VERSION" "$ALIAS"
+                  fi
+                  # The site root must always redirect to `latest`, so only (re)write the
+                  # root index.html when this run deploys the `latest` alias (releases).
+                  # Manual `dev` runs are published at /dev/ but never touch the root.
+                  if [[ "$ALIAS" == "latest" ]]; then
+                      mike set-default --push -F atlas-python/mkdocs.yml latest
+                  fi

{atlas_python-0.9.1 → atlas_python-0.10.0}/.gitignore

@@ -28,4 +28,8 @@ raw_smoke_tests/
 pyatlas/python/pyatlas/_pyatlas.pyd
 pyatlas/python/pyatlas/_pyatlas.abi3.so
 playground/
-data/
+data/
+atlas-python/python/atlas/_atlas.pyd
+atlas-python/python/atlas/_atlas.abi3.so
+atlas-python/python/atlas/_atlas*.so
+atlas-python/python/atlas/_atlas.pdb

{atlas_python-0.9.1 → atlas_python-0.10.0}/CONTRIBUTING.md

@@ -83,7 +83,7 @@ Key design points:
 - **Sync Python API, backed by an internal multi-threaded tokio runtime.** Each blocking call uses `py.allow_threads(|| RT.block_on(...))` so other Python threads can run.
 - **Type dispatch via macros.** Atlas's read/write methods are generic over dtype (`T: ArrayElement`); the bindings dispatch at runtime via a `numeric_dispatch!` macro in [pyatlas/src/dataset.rs](pyatlas/src/dataset.rs).
 - **`numpy`-zero-copy on the numeric path.** Python `np.ndarray` ↔ `ndarray::ArrayView` via the `numpy` crate.
-- **xarray integration** lives in [pyatlas/python/pyatlas/xarray.py](pyatlas/python/pyatlas/xarray.py). The `Atlas.add_xr_dataset` Rust pymethod and the `ds.atlas.write` accessor both delegate to the `_write_xarray_new_dataset` helper. dask-backed variables are streamed one chunk at a time via `arr.blocks[idx].compute()`.
+- **xarray integration** lives in [pyatlas/python/pyatlas/xarray.py](pyatlas/python/pyatlas/xarray.py). The `Atlas.add_xarray_dataset` Rust pymethod and the `ds.atlas.write` accessor both delegate to the `_write_xarray_new_dataset` helper. dask-backed variables are streamed one chunk at a time via `arr.blocks[idx].compute()`.
 
 ---
 

{atlas_python-0.9.1 → atlas_python-0.10.0}/Cargo.lock

@@ -28,9 +28,9 @@ checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c"
 
 [[package]]
 name = "array-format"
-version = "0.9.0"
+version = "0.10.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d91c6d4d26f667272f8930d41bc95118716e12858baf115940a626510e7b5873"
+checksum = "a637a3b5f23ba45eb53f09e643b363163114f0e94d0f6f05f330134cb86d381b"
 dependencies = [
  "bytes",
  "futures",
@@ -71,7 +71,7 @@ dependencies = [
 
 [[package]]
 name = "atlas-python"
-version = "0.9.1"
+version = "0.10.0"
 dependencies = [
  "atlas-rust",
  "ndarray",
@@ -86,7 +86,7 @@ dependencies = [
 
 [[package]]
 name = "atlas-rust"
-version = "0.9.1"
+version = "0.10.0"
 dependencies = [
  "array-format",
  "chrono",

{atlas_python-0.9.1 → atlas_python-0.10.0}/Cargo.toml

@@ -5,7 +5,7 @@ members = ["atlas-python"]
 # Registry name on crates.io (`atlas` is taken). The library is still
 # imported as `atlas` — see `[lib]` below.
 name = "atlas-rust"
-version = "0.9.1"
+version = "0.10.0"
 edition = "2024"
 description = "Directory-based store for thousands of N-dimensional datasets local or remote using object storage."
 license = "Apache-2.0"
@@ -22,7 +22,7 @@ all-features = true
 rustdoc-args = ["--cfg", "docsrs"]
 
 [dependencies]
-array-format = "0.9.0"
+array-format = "0.10.0"
 object_store = "0.13"
 serde = { version = "1", features = ["derive"] }
 serde_json = "1"

{atlas_python-0.9.1 → atlas_python-0.10.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: atlas-python
-Version: 0.9.1
+Version: 0.10.0
 Classifier: Development Status :: 3 - Alpha
 Classifier: Intended Audience :: Science/Research
 Classifier: Programming Language :: Python :: 3
@@ -118,18 +118,18 @@ ds = xr.Dataset(
 )
 
 with atlas.Atlas.create("/tmp/my_store") as store:
-    store.add_xr_dataset(ds, "jan_2024")     # store-side method
+    store.add_xarray_dataset(ds, "jan_2024")     # store-side method
     ds.atlas.write(store, "jan_2025")        # xarray accessor (same effect)
 
 # Read back as an xr.Dataset
 store = atlas.Atlas.open("/tmp/my_store")
-ds_back = store.to_xarray("jan_2024")
+ds_back = store.open_as_xarray_dataset("jan_2024")
 xr.testing.assert_identical(ds, ds_back)
 ```
 
 ### Bulk ingestion
 
-`add_xr_dataset` never flushes by itself — N consecutive calls accumulate in memory and a single
+`add_xarray_dataset` never flushes by itself — N consecutive calls accumulate in memory and a single
 `flush()` (or the `with` exit) persists everything.
 
 ```python
@@ -138,35 +138,35 @@ import glob, os, atlas, xarray as xr
 with atlas.Atlas.create("/tmp/store") as store:
     for nc_path in sorted(glob.glob("*.nc")):
         name = os.path.splitext(os.path.basename(nc_path))[0]
-        store.add_xr_dataset(xr.open_dataset(nc_path), name)
+        store.add_xarray_dataset(xr.open_dataset(nc_path), name)
 # One delta file per array name across the whole batch (not one per file).
 ```
 
 ### Streaming dask-backed writes
 
 If a variable's `.data` is a `dask.array.Array` (e.g. from `xr.open_dataset(path, chunks=...)`
-or `ds.chunk({...})`), `add_xr_dataset` / `ds.atlas.write` stream **one dask block at a time**
+or `ds.chunk({...})`), `add_xarray_dataset` / `ds.atlas.write` stream **one dask block at a time**
 into the store rather than materialising the whole array. The dask chunk shape becomes the
 on-disk `chunk_shape`, so the layout maps 1:1. Peak memory ≈ one chunk per variable.
 
 ```python
 ds = xr.open_dataset("big.nc", chunks={"time": 100, "lat": -1, "lon": -1})
 with atlas.Atlas.create("/tmp/store") as store:
-    store.add_xr_dataset(ds, "big")     # streams chunk-by-chunk
+    store.add_xarray_dataset(ds, "big")     # streams chunk-by-chunk
 ```
 
-Pass `chunks={var: [...]}` to `add_xr_dataset` / `ds.atlas.write` to override the on-disk chunk
+Pass `chunks={var: [...]}` to `add_xarray_dataset` / `ds.atlas.write` to override the on-disk chunk
 shape independently of dask's chunking.
 
 ### Lazy dask-backed reads
 
-`store.to_xarray(name)` returns each variable dask-backed whenever it was stored with non-trivial
+`store.open_as_xarray_dataset(name)` returns each variable dask-backed whenever it was stored with non-trivial
 chunking (`chunk_shape != shape`); the dask `chunks` tuple mirrors the on-disk chunk grid and each
 on-disk chunk is one dask task. Full-shape arrays (and 0-D scalars) come back eager as numpy. Call
 `.compute()` to materialise, or slice / `map_blocks` to operate lazily.
 
 ```python
-ds_back = atlas.Atlas.open("/tmp/store").to_xarray("big")
+ds_back = atlas.Atlas.open("/tmp/store").open_as_xarray_dataset("big")
 ds_back["temperature"].data              # -> dask.array.Array
 ds_back["temperature"][0:100].compute()  # reads exactly one chunk
 ```
@@ -181,13 +181,33 @@ picklable, so call `.compute()` before handing off to distributed/multiprocessin
 | Each coord / data variable | A separate array, with `dims` mapped 1:1. |
 | Dataset attrs | Dataset attributes, plain keys. |
 | Per-variable attrs | Flattened as `{var}.{attr}` at the dataset attr level. |
-| Per-variable `_FillValue` | Consumed by `define_array` as a typed fill value (source `Dataset.attrs` is not mutated). |
+| Per-variable `_FillValue` | Consumed by `define_array` as a typed fill value (source `Dataset.attrs` is not mutated). See [Missing data](#missing-data). |
 | Coord vs data_var distinction | JSON list in the internal `_pyatlas_coords` attr. |
 | Non-scalar attr values (list, ndarray) | JSON-encoded string with a `json:` prefix marker. |
 
-Each `add_xr_dataset` / `ds.atlas.write` creates a *new* dataset — there is no append-into-existing
+Each `add_xarray_dataset` / `ds.atlas.write` creates a *new* dataset — there is no append-into-existing
 mode.
 
+### Missing data
+
+When a dataset is opened with `mask_and_scale=True` (xarray's default), masked cells become `NaN`
+(floats) / `NaT` (datetimes) and `_FillValue` is moved into `var.encoding`. So those cells are
+recorded as **null** (counted in `null_count`, excluded from min/max stats), arrays default to a
+sentinel fill on write: `NaN` for floats, `NaT` for `datetime64[ns]`, and `""` for strings (integers
+have none). Missing string cells (`None`/`NaN`) are substituted with the string fill and a warning is
+emitted, since a string can't be stored as null directly.
+
+Override per write with `fill_value` — a bare scalar (numeric arrays) or a `{var: scalar}` dict
+(`None` disables the default for that var); an explicit CF `_FillValue` attribute still takes
+precedence over the default:
+
+```python
+store.add_xarray_dataset(ds, "jan_2024", fill_value={"temperature": -9999.0})
+```
+
+See the [xarray guide](https://github.com/maris-development/atlas/blob/main/atlas-python/docs/guides/xarray.md#fill-values-and-missing-data)
+for the full resolution order.
+
 ## Supported dtypes
 
 | numpy dtype | atlas dtype |
@@ -221,8 +241,9 @@ handle instead of a local path. The path-based local-filesystem API works withou
 | `list_datasets() -> list[str]` | All dataset names. |
 | `list_arrays() -> list[str]` | Distinct array names across datasets. |
 | `dataset_exists(name) -> bool` | Existence check. |
-| `add_xr_dataset(ds, name, chunks=None)` | Append an `xarray.Dataset` (does **not** flush). |
-| `to_xarray(name) -> xr.Dataset` | Read a dataset back (chunked vars come back dask-backed). |
+| `add_xarray_dataset(ds, name, chunks=None, fill_value=None)` | Append an `xarray.Dataset` (does **not** flush). `fill_value` overrides the per-array fill (scalar or `{var: scalar}`); see [Missing data](#missing-data). |
+| `open_as_xarray_dataset(name) -> xr.Dataset` | Read a dataset back (chunked vars come back dask-backed). |
+| `open_as_many_xarray_dataset(names, concat_dim="dataset") -> xr.Dataset` | Open many datasets stacked along `concat_dim` (eager numpy). |
 | `flush()` | The single durability boundary — persist everything. |
 | `close()` | Alias for `flush()`; also the `with`-block exit. |
 | `compact()` | Reclaim tombstoned space across cached array files. |
@@ -250,13 +271,14 @@ handle instead of a local path. The path-based local-filesystem API works withou
 Runnable, self-contained scripts (each writes to a temp directory):
 
 - [01_basics.py](https://github.com/maris-development/atlas/blob/main/atlas-python/examples/01_basics.py) — create a store, define arrays, set attributes, reopen, read back.
-- [02_xarray.py](https://github.com/maris-development/atlas/blob/main/atlas-python/examples/02_xarray.py) — round-trip an `xr.Dataset` via both `store.add_xr_dataset(...)` and the `ds.atlas.write(...)` accessor.
+- [02_xarray.py](https://github.com/maris-development/atlas/blob/main/atlas-python/examples/02_xarray.py) — round-trip an `xr.Dataset` via both `store.add_xarray_dataset(...)` and the `ds.atlas.write(...)` accessor.
 - [03_dask_streaming.py](https://github.com/maris-development/atlas/blob/main/atlas-python/examples/03_dask_streaming.py) — stream a dask-chunked `xr.Dataset` in one chunk at a time.
+- [09_missing_data.py](https://github.com/maris-development/atlas/blob/main/atlas-python/examples/09_missing_data.py) — masked/missing cells with default `NaN` / `NaT` / `""` fills, `null_count`, and `fill_value=` overrides.
 
 ## Performance
 
 ATLAS is tuned for collections of many similarly-shaped datasets. On a "1000 datasets" benchmark
-against netCDF4 and Zarr v3, the bulk read paths (`Atlas.to_xarray_many` /
+against netCDF4 and Zarr v3, the bulk read paths (`Atlas.open_as_many_xarray_dataset` /
 `Atlas.read_array_across_stacked`) beat Zarr by ~2.8× on large chunked slice reads, and on small
 per-dataset workloads ATLAS leads on both reads and writes. See the
 [benchmarks](https://github.com/maris-development/atlas/tree/main/atlas-python/benchmarks) for the full

{atlas_python-0.9.1 → atlas_python-0.10.0}/README.md

@@ -1,5 +1,12 @@
 # ATLAS — Aggregated Tensor Large Array Store
 
+[![CI](https://github.com/maris-development/atlas/actions/workflows/ci.yaml/badge.svg)](https://github.com/maris-development/atlas/actions/workflows/ci.yaml)
+[![crates.io](https://img.shields.io/crates/v/atlas-rust.svg?logo=rust)](https://crates.io/crates/atlas-rust)
+[![docs.rs](https://img.shields.io/docsrs/atlas-rust?logo=docsdotrs&label=docs.rs)](https://docs.rs/atlas-rust)
+[![PyPI](https://img.shields.io/pypi/v/atlas-python.svg?logo=pypi&logoColor=white)](https://pypi.org/project/atlas-python/)
+[![Python docs](https://img.shields.io/badge/docs-atlas--python-blue?logo=materialformkdocs&logoColor=white)](https://maris-development.github.io/atlas/)
+[![License](https://img.shields.io/crates/l/atlas-rust.svg)](LICENSE)
+
 A directory-based store for thousands of named datasets, each holding N-dimensional typed arrays. Built on top of the [`array-format`](https://github.com/robinskil/array-format) (`.af`) binary format, with configurable compression (Zstd, LZ4, or none), chunked I/O, and an [`object_store`](https://crates.io/crates/object_store) backend that works on local disk, S3, GCS, Azure Blob, and in-memory.
 
 **Think of it as a "zip" for N-dimensional datasets.** Where a `.zip` bundles many files into one archive, ATLAS gathers many NetCDF / Zarr-style datasets — anything that's a set of named N-dimensional arrays — into a single high-performance collection. But it's more than a bundle: instead of storing each dataset whole, ATLAS lays the data out **variable-first**, so every dataset's `temperature` lives in one file. That makes it cheap to scan or slice a single variable across thousands of datasets at once, while still reading back any individual dataset as a normal NetCDF/xarray-like object.
@@ -257,7 +264,7 @@ Choose LZ4 when decompression throughput matters more than storage size (e.g. la
 
 ### Write path
 
-Writes are buffered in-memory across the whole atlas. Calling `Atlas::flush()` compresses and writes every modified block across every cached array file, then rewrites `atlas.json` atomically (a single `PUT`). The write path scales with the number of modified chunks, not the number of datasets, and N consecutive `add_xr_dataset` / `create_dataset` calls amortise to a single flush.
+Writes are buffered in-memory across the whole atlas. Calling `Atlas::flush()` compresses and writes every modified block across every cached array file, then rewrites `atlas.json` atomically (a single `PUT`). The write path scales with the number of modified chunks, not the number of datasets, and N consecutive `add_xarray_dataset` / `create_dataset` calls amortise to a single flush.
 
 ### Compaction
 
@@ -301,7 +308,7 @@ the `(0:25, 0:25, 0:12)` slice down to chunk-level reads.
 | **atlas-bulk** | 1 store, `read_array_across_stacked` with slice push-down | **2.12** | 59 | 6387 |
 | **atlas + `--use-dask`** | 1 store, dask-threaded per-dataset `view.read_arrays(...)` | **3.21** | 60 | 6387 |
 | zarr | 1000 separate stores, `xr.open_mfdataset(parallel=True).isel(...)` | 5.99 | 38 | 6392 |
-| atlas (default) | 1 store, serial per-dataset `to_xarray(...).isel(...).load()` | 10.23 | 51 | 6387 |
+| atlas (default) | 1 store, serial per-dataset `open_as_xarray_dataset(...).isel(...).load()` | 10.23 | 51 | 6387 |
 | netcdf | 1000 `.nc` files, `xr.open_mfdataset(parallel=True).isel(...)` | 13.91 | 122 | 5596 |
 
 #### `--case profile --datasets 1000` (~67 MB raw — per-dataset overhead dominates)
@@ -332,9 +339,9 @@ For tiny-shape datasets the comparison is dominated by per-dataset overhead, whi
 
 ### What the numbers say
 
-- **Reads on gridded** (decompression-dominated, with realistic chunking + slice push-down): `atlas-bulk` reads in **2.12s vs zarr's 6.00s — 2.8× faster**. `atlas + --use-dask` (using the new `view.read_arrays(...)` fast path) hits **3.21s — 1.9× faster than zarr**. The win comes from atlas's structural advantage (one open of one file per variable, in-memory metadata) combined with APIs that bypass `to_xarray`'s xr.Dataset + per-chunk dask graph overhead. zarr and netcdf both push the slice down through `open_mfdataset(...).isel(...)` via dask graph optimization — they're not penalised by the chunking; atlas just wins by amortising metadata and skipping per-dataset Python overhead.
+- **Reads on gridded** (decompression-dominated, with realistic chunking + slice push-down): `atlas-bulk` reads in **2.12s vs zarr's 6.00s — 2.8× faster**. `atlas + --use-dask` (using the new `view.read_arrays(...)` fast path) hits **3.21s — 1.9× faster than zarr**. The win comes from atlas's structural advantage (one open of one file per variable, in-memory metadata) combined with APIs that bypass `open_as_xarray_dataset`'s xr.Dataset + per-chunk dask graph overhead. zarr and netcdf both push the slice down through `open_mfdataset(...).isel(...)` via dask graph optimization — they're not penalised by the chunking; atlas just wins by amortising metadata and skipping per-dataset Python overhead.
 - **Reads on profile** (overhead-dominated): atlas wins by an order of magnitude. `atlas-bulk` is **~50× faster than zarr** (0.08s vs 4.07s) because the per-dataset I/O is small enough that everything is overhead — and atlas's structural design is built for exactly that case. Default serial `atlas` is still ~12× faster than zarr.
-- **Default `atlas.to_xarray` iteration is currently SLOW on chunked storage** (10.23s vs zarr's 6.00s). It goes through `to_xarray(name)` which returns dask-backed arrays per chunk (8 chunks/var × 3 vars × 1000 datasets = 24,000 dask delayed tasks just to build the graph). Dask graph overhead exceeds the parallelism win. **The fix is to use `view.read_arrays(vars, start, shape)` for per-dataset slice reads** — that's what `atlas + --use-dask` does internally and why it hits 3.21s. For cross-dataset reads of the *same* slice from many datasets, prefer `Atlas.to_xarray_many` / `read_array_across_stacked` (the `atlas-bulk` row).
+- **Default `atlas.open_as_xarray_dataset` iteration is currently SLOW on chunked storage** (10.23s vs zarr's 6.00s). It goes through `open_as_xarray_dataset(name)` which returns dask-backed arrays per chunk (8 chunks/var × 3 vars × 1000 datasets = 24,000 dask delayed tasks just to build the graph). Dask graph overhead exceeds the parallelism win. **The fix is to use `view.read_arrays(vars, start, shape)` for per-dataset slice reads** — that's what `atlas + --use-dask` does internally and why it hits 3.21s. For cross-dataset reads of the *same* slice from many datasets, prefer `Atlas.open_as_many_xarray_dataset` / `read_array_across_stacked` (the `atlas-bulk` row).
 - **Workload sensitivity**: `--use-dask` helps when per-dataset decompression is the bottleneck and *hurts* when per-dataset overhead is the bottleneck (profile: default `atlas` 0.32s → dask 2.03s). Picking the right API for the workload matters more than picking dask vs serial.
 - **Writes on gridded**: zarr is fastest (~31s vs atlas's ~50s) — each `to_zarr(...)` just dumps bytes into a per-dataset subtree, while atlas does more bookkeeping per call (schema registration, per-dataset attrs into the shared `atlas.msgpack.zst`, block addressing in the shared array file). Atlas still beats netcdf decisively.
 - **Writes on profile**: atlas wins ~12× (0.91s vs zarr's 11.43s, netcdf's 2.98s). At small per-dataset sizes the metadata write dominates, and atlas's amortised single-flush model wins.

{atlas_python-0.9.1 → atlas_python-0.10.0}/atlas-python/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "atlas-python"
-version = "0.9.1"
+version = "0.10.0"
 edition = "2021"
 readme = "README.md"
 

{atlas_python-0.9.1 → atlas_python-0.10.0}/atlas-python/README.md

@@ -86,18 +86,18 @@ ds = xr.Dataset(
 )
 
 with atlas.Atlas.create("/tmp/my_store") as store:
-    store.add_xr_dataset(ds, "jan_2024")     # store-side method
+    store.add_xarray_dataset(ds, "jan_2024")     # store-side method
     ds.atlas.write(store, "jan_2025")        # xarray accessor (same effect)
 
 # Read back as an xr.Dataset
 store = atlas.Atlas.open("/tmp/my_store")
-ds_back = store.to_xarray("jan_2024")
+ds_back = store.open_as_xarray_dataset("jan_2024")
 xr.testing.assert_identical(ds, ds_back)
 ```
 
 ### Bulk ingestion
 
-`add_xr_dataset` never flushes by itself — N consecutive calls accumulate in memory and a single
+`add_xarray_dataset` never flushes by itself — N consecutive calls accumulate in memory and a single
 `flush()` (or the `with` exit) persists everything.
 
 ```python
@@ -106,35 +106,35 @@ import glob, os, atlas, xarray as xr
 with atlas.Atlas.create("/tmp/store") as store:
     for nc_path in sorted(glob.glob("*.nc")):
         name = os.path.splitext(os.path.basename(nc_path))[0]
-        store.add_xr_dataset(xr.open_dataset(nc_path), name)
+        store.add_xarray_dataset(xr.open_dataset(nc_path), name)
 # One delta file per array name across the whole batch (not one per file).
 ```
 
 ### Streaming dask-backed writes
 
 If a variable's `.data` is a `dask.array.Array` (e.g. from `xr.open_dataset(path, chunks=...)`
-or `ds.chunk({...})`), `add_xr_dataset` / `ds.atlas.write` stream **one dask block at a time**
+or `ds.chunk({...})`), `add_xarray_dataset` / `ds.atlas.write` stream **one dask block at a time**
 into the store rather than materialising the whole array. The dask chunk shape becomes the
 on-disk `chunk_shape`, so the layout maps 1:1. Peak memory ≈ one chunk per variable.
 
 ```python
 ds = xr.open_dataset("big.nc", chunks={"time": 100, "lat": -1, "lon": -1})
 with atlas.Atlas.create("/tmp/store") as store:
-    store.add_xr_dataset(ds, "big")     # streams chunk-by-chunk
+    store.add_xarray_dataset(ds, "big")     # streams chunk-by-chunk
 ```
 
-Pass `chunks={var: [...]}` to `add_xr_dataset` / `ds.atlas.write` to override the on-disk chunk
+Pass `chunks={var: [...]}` to `add_xarray_dataset` / `ds.atlas.write` to override the on-disk chunk
 shape independently of dask's chunking.
 
 ### Lazy dask-backed reads
 
-`store.to_xarray(name)` returns each variable dask-backed whenever it was stored with non-trivial
+`store.open_as_xarray_dataset(name)` returns each variable dask-backed whenever it was stored with non-trivial
 chunking (`chunk_shape != shape`); the dask `chunks` tuple mirrors the on-disk chunk grid and each
 on-disk chunk is one dask task. Full-shape arrays (and 0-D scalars) come back eager as numpy. Call
 `.compute()` to materialise, or slice / `map_blocks` to operate lazily.
 
 ```python
-ds_back = atlas.Atlas.open("/tmp/store").to_xarray("big")
+ds_back = atlas.Atlas.open("/tmp/store").open_as_xarray_dataset("big")
 ds_back["temperature"].data              # -> dask.array.Array
 ds_back["temperature"][0:100].compute()  # reads exactly one chunk
 ```
@@ -149,13 +149,33 @@ picklable, so call `.compute()` before handing off to distributed/multiprocessin
 | Each coord / data variable | A separate array, with `dims` mapped 1:1. |
 | Dataset attrs | Dataset attributes, plain keys. |
 | Per-variable attrs | Flattened as `{var}.{attr}` at the dataset attr level. |
-| Per-variable `_FillValue` | Consumed by `define_array` as a typed fill value (source `Dataset.attrs` is not mutated). |
+| Per-variable `_FillValue` | Consumed by `define_array` as a typed fill value (source `Dataset.attrs` is not mutated). See [Missing data](#missing-data). |
 | Coord vs data_var distinction | JSON list in the internal `_pyatlas_coords` attr. |
 | Non-scalar attr values (list, ndarray) | JSON-encoded string with a `json:` prefix marker. |
 
-Each `add_xr_dataset` / `ds.atlas.write` creates a *new* dataset — there is no append-into-existing
+Each `add_xarray_dataset` / `ds.atlas.write` creates a *new* dataset — there is no append-into-existing
 mode.
 
+### Missing data
+
+When a dataset is opened with `mask_and_scale=True` (xarray's default), masked cells become `NaN`
+(floats) / `NaT` (datetimes) and `_FillValue` is moved into `var.encoding`. So those cells are
+recorded as **null** (counted in `null_count`, excluded from min/max stats), arrays default to a
+sentinel fill on write: `NaN` for floats, `NaT` for `datetime64[ns]`, and `""` for strings (integers
+have none). Missing string cells (`None`/`NaN`) are substituted with the string fill and a warning is
+emitted, since a string can't be stored as null directly.
+
+Override per write with `fill_value` — a bare scalar (numeric arrays) or a `{var: scalar}` dict
+(`None` disables the default for that var); an explicit CF `_FillValue` attribute still takes
+precedence over the default:
+
+```python
+store.add_xarray_dataset(ds, "jan_2024", fill_value={"temperature": -9999.0})
+```
+
+See the [xarray guide](https://github.com/maris-development/atlas/blob/main/atlas-python/docs/guides/xarray.md#fill-values-and-missing-data)
+for the full resolution order.
+
 ## Supported dtypes
 
 | numpy dtype | atlas dtype |
@@ -189,8 +209,9 @@ handle instead of a local path. The path-based local-filesystem API works withou
 | `list_datasets() -> list[str]` | All dataset names. |
 | `list_arrays() -> list[str]` | Distinct array names across datasets. |
 | `dataset_exists(name) -> bool` | Existence check. |
-| `add_xr_dataset(ds, name, chunks=None)` | Append an `xarray.Dataset` (does **not** flush). |
-| `to_xarray(name) -> xr.Dataset` | Read a dataset back (chunked vars come back dask-backed). |
+| `add_xarray_dataset(ds, name, chunks=None, fill_value=None)` | Append an `xarray.Dataset` (does **not** flush). `fill_value` overrides the per-array fill (scalar or `{var: scalar}`); see [Missing data](#missing-data). |
+| `open_as_xarray_dataset(name) -> xr.Dataset` | Read a dataset back (chunked vars come back dask-backed). |
+| `open_as_many_xarray_dataset(names, concat_dim="dataset") -> xr.Dataset` | Open many datasets stacked along `concat_dim` (eager numpy). |
 | `flush()` | The single durability boundary — persist everything. |
 | `close()` | Alias for `flush()`; also the `with`-block exit. |
 | `compact()` | Reclaim tombstoned space across cached array files. |
@@ -218,13 +239,14 @@ handle instead of a local path. The path-based local-filesystem API works withou
 Runnable, self-contained scripts (each writes to a temp directory):
 
 - [01_basics.py](https://github.com/maris-development/atlas/blob/main/atlas-python/examples/01_basics.py) — create a store, define arrays, set attributes, reopen, read back.
-- [02_xarray.py](https://github.com/maris-development/atlas/blob/main/atlas-python/examples/02_xarray.py) — round-trip an `xr.Dataset` via both `store.add_xr_dataset(...)` and the `ds.atlas.write(...)` accessor.
+- [02_xarray.py](https://github.com/maris-development/atlas/blob/main/atlas-python/examples/02_xarray.py) — round-trip an `xr.Dataset` via both `store.add_xarray_dataset(...)` and the `ds.atlas.write(...)` accessor.
 - [03_dask_streaming.py](https://github.com/maris-development/atlas/blob/main/atlas-python/examples/03_dask_streaming.py) — stream a dask-chunked `xr.Dataset` in one chunk at a time.
+- [09_missing_data.py](https://github.com/maris-development/atlas/blob/main/atlas-python/examples/09_missing_data.py) — masked/missing cells with default `NaN` / `NaT` / `""` fills, `null_count`, and `fill_value=` overrides.
 
 ## Performance
 
 ATLAS is tuned for collections of many similarly-shaped datasets. On a "1000 datasets" benchmark
-against netCDF4 and Zarr v3, the bulk read paths (`Atlas.to_xarray_many` /
+against netCDF4 and Zarr v3, the bulk read paths (`Atlas.open_as_many_xarray_dataset` /
 `Atlas.read_array_across_stacked`) beat Zarr by ~2.8× on large chunked slice reads, and on small
 per-dataset workloads ATLAS leads on both reads and writes. See the
 [benchmarks](https://github.com/maris-development/atlas/tree/main/atlas-python/benchmarks) for the full

{atlas_python-0.9.1 → atlas_python-0.10.0}/atlas-python/benchmarks/README.md

@@ -49,7 +49,7 @@ replace them, so one run can compare multiple variants head-to-head.
 
 | Backend | Always              | Added by `--atlas-bulk` | Added by `--netcdf-groups` | Added by `--zarr-groups` |
 |---|---|---|---|---|
-| atlas         | 1 store, N datasets (iterate `to_xarray` or `view.read_arrays`) | (use `read_array_across_stacked`) | — | — |
+| atlas         | 1 store, N datasets (iterate `open_as_xarray_dataset` or `view.read_arrays`) | (use `read_array_across_stacked`) | — | — |
 | atlas-bulk    | — | (this row) | — | — |
 | netcdf        | N separate `.nc` files    | — | 1 `.nc` file w/ N netCDF4 groups | — |
 | netcdf-groups | —                         | — | (this row)                       | — |
@@ -65,7 +65,7 @@ read pattern (not the layout) — see "Read pattern" below.
 Read the slice from every dataset. Each backend uses its canonical
 "many datasets" idiom:
 
-- **atlas (default, no `--use-dask`)** — iterate `to_xarray(name).isel(slice).load()`.
+- **atlas (default, no `--use-dask`)** — iterate `open_as_xarray_dataset(name).isel(slice).load()`.
   Returns a full xr.Dataset per dataset and slices in xarray. On chunked
   storage this pays per-chunk dask graph build overhead; slow on `gridded`.
 - **atlas (with `--use-dask`)** — fast path: each dask worker calls
@@ -100,14 +100,14 @@ When set, `--use-dask` does three things:
    per-dataset slice load is wrapped in `dask.delayed` and dispatched via
    `dask.compute(*, scheduler="threads")`. For atlas specifically, this
    path uses `view.read_arrays(vars, start, shape)` (the fast Rust dict
-   path) rather than `to_xarray(...).isel(...).load()`, avoiding the
+   path) rather than `open_as_xarray_dataset(...).isel(...).load()`, avoiding the
    xr.Dataset + dask graph overhead that dominates default `atlas`.
 2. **Reads (mfdataset paths)** — default netcdf, default zarr: already use
    dask internally; the flag only constrains the thread pool to
    `--dask-workers` if specified.
 3. **Writes** — `generate_dataset()` returns dask-backed `xr.DataArray`s
    (2 chunks along `time`). Each backend's xarray write triggers a dask
-   compute that streams chunks. Atlas's `add_xr_dataset` already does this
+   compute that streams chunks. Atlas's `add_xarray_dataset` already does this
    chunk-by-chunk; netCDF/zarr `to_*` handle it transparently.
 
 Use `--dask-workers N` to set the thread count; defaults to dask's default

{atlas_python-0.9.1 → atlas_python-0.10.0}/atlas-python/benchmarks/bench_collection.py

@@ -1,7 +1,7 @@
 """Compare atlas / netCDF / zarr on a 1000-dataset collection workload.
 
 Each backend uses its canonical "many datasets" layout and read pattern:
-    atlas  — 1 store, 1000 datasets;       iterate `to_xarray(name)`
+    atlas  — 1 store, 1000 datasets;       iterate `open_as_xarray_dataset(name)`
     netCDF — 1000 .nc files;               `xr.open_mfdataset(files, ...)`
     zarr   — 1 store, 1000 groups;         iterate `xr.open_zarr(..., group=name)`
 
@@ -76,7 +76,7 @@ def bench_atlas(
             meta_compression="zstd",
         ) as store:
             for i in range(n_datasets):
-                store.add_xr_dataset(
+                store.add_xarray_dataset(
                     generate_dataset(i, case, use_dask=use_dask),
                     f"ds_{i:04d}",
                     chunks=chunks_per_var,
@@ -97,7 +97,7 @@ def bench_atlas(
         from dask.delayed import delayed
 
         # Fast path: per-dataset slice reads via `view.read_arrays(...)`
-        # bypass to_xarray's xr.Dataset + dask-graph build overhead. The
+        # bypass open_as_xarray_dataset's xr.Dataset + dask-graph build overhead. The
         # dask scheduler still parallelises across datasets; each task does
         # just one Rust call per variable with the slice already applied.
         start = [s.start for s in indexers.values()]
@@ -114,7 +114,7 @@ def bench_atlas(
     else:
         with time_block() as rt:
             for i in range(n_datasets):
-                ds = store.to_xarray(f"ds_{i:04d}")
+                ds = store.open_as_xarray_dataset(f"ds_{i:04d}")
                 _ = ds[var_names].isel(indexers).load()
                 progress_tick(i, n_datasets, "read")
     log_phase("atlas", "read", f"{rt.elapsed:.3f}s")
@@ -122,7 +122,7 @@ def bench_atlas(
     return BackendResult(name="atlas", write_s=wt.elapsed, read_s=rt.elapsed, size_bytes=size)
 
 
-# ── atlas (bulk, single-call to_xarray_many) ────────────────────────────
+# ── atlas (bulk, single-call open_as_many_xarray_dataset) ────────────────────────────
 
 
 def bench_atlas_bulk(
@@ -134,7 +134,7 @@ def bench_atlas_bulk(
     dask_workers: int | None = None,
 ) -> BackendResult:
     """Same write phase as `bench_atlas`; read phase is one
-    `Atlas.to_xarray_many(...)` call — the atlas-native equivalent of
+    `Atlas.open_as_many_xarray_dataset(...)` call — the atlas-native equivalent of
     `xr.open_mfdataset`. Stacked Dataset is sliced and `.load()`-ed in one
     dask compute."""
     import dask.config as dask_config
@@ -158,7 +158,7 @@ def bench_atlas_bulk(
             meta_compression="zstd",
         ) as store:
             for i, name in enumerate(names):
-                store.add_xr_dataset(
+                store.add_xarray_dataset(
                     generate_dataset(i, case, use_dask=use_dask),
                     name,
                     chunks=chunks_per_var,
@@ -179,7 +179,7 @@ def bench_atlas_bulk(
     # Push the slice down through the low-level API so atlas only decompresses
     # the chunks overlapping the slice (matches what zarr/netcdf's
     # `open_mfdataset(...).isel(...).load()` does via dask graph optimization).
-    # `to_xarray_many(...).isel(...)` would also work but slices in numpy
+    # `open_as_many_xarray_dataset(...).isel(...)` would also work but slices in numpy
     # *after* decompressing the full per-dataset chunks, wasting work when
     # `chunk_shape != shape`.
     start = [s.start for s in indexers.values()]
@@ -544,7 +544,7 @@ def parse_args() -> argparse.Namespace:
         "--atlas-bulk",
         action="store_true",
         help=(
-            "Add an `atlas-bulk` row that reads via Atlas.to_xarray_many "
+            "Add an `atlas-bulk` row that reads via Atlas.open_as_many_xarray_dataset "
             "(atlas-native equivalent of xr.open_mfdataset) instead of "
             "iterating per-dataset. Additive — the default `atlas` row still runs."
         ),

{atlas_python-0.9.1 → atlas_python-0.10.0}/atlas-python/docs/benchmarks.md

@@ -23,7 +23,7 @@ actually reach for:
   numpy (opt in with `--atlas-bulk`).
 
 The default (serial) atlas read path — iterate
-`view.to_xarray(name).isel(...).load()` — isn't in the chart; it scales
+`view.open_as_xarray_dataset(name).isel(...).load()` — isn't in the chart; it scales
 linearly with N and is fine for small fleets, but at the scales below
 (100+ datasets) you should already be reaching for one of the parallel
 paths. `--netcdf-groups` / `--zarr-groups` add layouts that use one file
@@ -96,7 +96,7 @@ threaded `dask.delayed` fan-out has more overhead than tokio's
 - **`atlas + --use-dask`** picks up most of atlas-bulk's win without
   giving up the dask graph: 1.6–4× faster than `zarr+dask` across the
   sweep, and the natural choice when downstream `xr` code needs to stay
-  lazy (a `to_xarray(...).isel(...)` slice chain composes with the
+  lazy (a `open_as_xarray_dataset(...).isel(...)` slice chain composes with the
   upstream graph, where `atlas-bulk` returns eager numpy). Reach for
   serial `atlas` only when the fleet is small enough that the dask
   fan-out overhead doesn't pay back.
@@ -146,14 +146,14 @@ Update the `RESULTS` dict in that script and re-run after a fresh sweep.
 ## API picker for reads (in rough order of speed)
 
 - **Cross-dataset slice of the same vars across many datasets** →
-  [`Atlas.to_xarray_many`](reference/atlas.md) /
+  [`Atlas.open_as_many_xarray_dataset`](reference/atlas.md) /
   [`Atlas.read_array_across_stacked`](reference/atlas.md) (the
   `atlas-bulk` path; one Rust call per variable).
 - **Per-dataset slice reads inside a dask worker** →
   [`view.read_arrays(vars, start, shape)`](reference/dataset-view.md)
   (returns `dict[str, np.ndarray]`; skips xr.Dataset + per-chunk dask
   graph). This is what `bench_atlas` with `--use-dask` does internally.
-- **Natural xarray code** → `to_xarray(name).isel(...).load()`. Most
+- **Natural xarray code** → `open_as_xarray_dataset(name).isel(...).load()`. Most
   ergonomic but pays per-chunk dask graph build overhead on chunked
   storage.