atlas-python 0.9.1__tar.gz → 0.12.0__tar.gz

atlas_python-0.12.0/.github/workflows/atlas-python-docs.yaml

@@ -0,0 +1,88 @@
+name: atlas-python-docs
+on:
+    workflow_dispatch:
+        inputs:
+            version:
+                description: "Docs version to publish (e.g. 1.2.3). Leave blank to publish as 'dev'."
+                required: false
+                type: string
+            alias:
+                description: "Alias to update (e.g. latest)."
+                required: false
+                default: dev
+                type: string
+    release:
+        types: [published]
+    push:
+        tags:
+            - "v*" # triggers on tags like v1.0.0, v2.3, etc.
+permissions:
+    contents: write
+jobs:
+    deploy:
+        runs-on: ubuntu-latest
+        steps:
+            - uses: actions/checkout@v4
+            - name: Configure Git Credentials
+              run: |
+                  git config user.name github-actions[bot]
+                  git config user.email 41898282+github-actions[bot]@users.noreply.github.com
+
+            - name: Extract version
+              id: vars
+              env:
+                  INPUT_VERSION: ${{ github.event.inputs.version }}
+                  INPUT_ALIAS: ${{ github.event.inputs.alias }}
+              run: |
+                  if [[ "$GITHUB_REF" == refs/tags/v* ]]; then
+                      # Tag push or release: refs/tags/v1.2.3 -> version=1.2.3, alias=latest
+                      VERSION="${GITHUB_REF#refs/tags/v}"
+                      ALIAS=latest
+                  else
+                      # Manual run: use provided inputs, falling back to 'dev'
+                      VERSION="${INPUT_VERSION:-dev}"
+                      ALIAS="${INPUT_ALIAS:-dev}"
+                  fi
+                  echo "version=$VERSION" >> $GITHUB_OUTPUT
+                  echo "alias=$ALIAS" >> $GITHUB_OUTPUT
+
+            - name: Show version
+              run: |
+                  echo "Deploying docs version '${{ steps.vars.outputs.version }}' (alias '${{ steps.vars.outputs.alias }}')"
+
+            - uses: actions/setup-python@v6
+              with:
+                  python-version: "3.12"
+            - run: echo "cache_id=$(date --utc '+%V')" >> $GITHUB_ENV
+            - uses: actions/cache@v4
+              with:
+                  key: mkdocs-material-${{ env.cache_id }}
+                  path: ~/.cache
+                  restore-keys: |
+                      mkdocs-material-
+            - run: pip install mkdocs-material mkdocstrings[python] mike
+              # Build & install the package so the compiled `atlas._atlas` extension is
+              # importable — mkdocstrings preloads it to resolve the re-exported API.
+              # Editable (-e) so maturin places the compiled `_atlas.*.so` into the
+              # python-source tree (atlas-python/python/atlas/), which is where griffe's
+              # `paths: ["python"]` search path looks for it. A plain install puts it in
+              # site-packages, outside that search path, and the alias fails to resolve.
+              # ubuntu-latest ships a Rust toolchain, so maturin can build from source.
+            - run: pip install -e ./atlas-python
+            - env:
+                  VERSION: ${{ steps.vars.outputs.version }}
+                  ALIAS: ${{ steps.vars.outputs.alias }}
+              run: |
+                  git fetch origin gh-pages --depth=1 || true
+                  # mike rejects an alias identical to the version, so only pass it when it differs
+                  if [[ "$ALIAS" == "$VERSION" ]]; then
+                      mike deploy --push --update-aliases -F atlas-python/mkdocs.yml "$VERSION"
+                  else
+                      mike deploy --push --update-aliases -F atlas-python/mkdocs.yml "$VERSION" "$ALIAS"
+                  fi
+                  # The site root must always redirect to `latest`, so only (re)write the
+                  # root index.html when this run deploys the `latest` alias (releases).
+                  # Manual `dev` runs are published at /dev/ but never touch the root.
+                  if [[ "$ALIAS" == "latest" ]]; then
+                      mike set-default --push -F atlas-python/mkdocs.yml latest
+                  fi

{atlas_python-0.9.1 → atlas_python-0.12.0}/.gitignore

@@ -28,4 +28,8 @@ raw_smoke_tests/
 pyatlas/python/pyatlas/_pyatlas.pyd
 pyatlas/python/pyatlas/_pyatlas.abi3.so
 playground/
-data/
+data/
+atlas-python/python/atlas/_atlas.pyd
+atlas-python/python/atlas/_atlas.abi3.so
+atlas-python/python/atlas/_atlas*.so
+atlas-python/python/atlas/_atlas.pdb

{atlas_python-0.9.1 → atlas_python-0.12.0}/CONTRIBUTING.md

@@ -83,7 +83,7 @@ Key design points:
 - **Sync Python API, backed by an internal multi-threaded tokio runtime.** Each blocking call uses `py.allow_threads(|| RT.block_on(...))` so other Python threads can run.
 - **Type dispatch via macros.** Atlas's read/write methods are generic over dtype (`T: ArrayElement`); the bindings dispatch at runtime via a `numeric_dispatch!` macro in [pyatlas/src/dataset.rs](pyatlas/src/dataset.rs).
 - **`numpy`-zero-copy on the numeric path.** Python `np.ndarray` ↔ `ndarray::ArrayView` via the `numpy` crate.
-- **xarray integration** lives in [pyatlas/python/pyatlas/xarray.py](pyatlas/python/pyatlas/xarray.py). The `Atlas.add_xr_dataset` Rust pymethod and the `ds.atlas.write` accessor both delegate to the `_write_xarray_new_dataset` helper. dask-backed variables are streamed one chunk at a time via `arr.blocks[idx].compute()`.
+- **xarray integration** lives in [pyatlas/python/pyatlas/xarray.py](pyatlas/python/pyatlas/xarray.py). The `Atlas.add_xarray_dataset` Rust pymethod and the `ds.atlas.write` accessor both delegate to the `_write_xarray_new_dataset` helper. dask-backed variables are streamed one chunk at a time via `arr.blocks[idx].compute()`.
 
 ---
 

{atlas_python-0.9.1 → atlas_python-0.12.0}/Cargo.lock

@@ -28,9 +28,9 @@ checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c"
 
 [[package]]
 name = "array-format"
-version = "0.9.0"
+version = "0.11.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d91c6d4d26f667272f8930d41bc95118716e12858baf115940a626510e7b5873"
+checksum = "f894cd59132aa6a00c360fef55469100c39da19c556bbf44a46bd647f7252fae"
 dependencies = [
  "bytes",
  "futures",
@@ -71,7 +71,7 @@ dependencies = [
 
 [[package]]
 name = "atlas-python"
-version = "0.9.1"
+version = "0.12.0"
 dependencies = [
  "atlas-rust",
  "ndarray",
@@ -86,7 +86,7 @@ dependencies = [
 
 [[package]]
 name = "atlas-rust"
-version = "0.9.1"
+version = "0.12.0"
 dependencies = [
  "array-format",
  "chrono",

{atlas_python-0.9.1 → atlas_python-0.12.0}/Cargo.toml

@@ -5,7 +5,7 @@ members = ["atlas-python"]
 # Registry name on crates.io (`atlas` is taken). The library is still
 # imported as `atlas` — see `[lib]` below.
 name = "atlas-rust"
-version = "0.9.1"
+version = "0.12.0"
 edition = "2024"
 description = "Directory-based store for thousands of N-dimensional datasets local or remote using object storage."
 license = "Apache-2.0"
@@ -22,7 +22,7 @@ all-features = true
 rustdoc-args = ["--cfg", "docsrs"]
 
 [dependencies]
-array-format = "0.9.0"
+array-format = "0.11.0"
 object_store = "0.13"
 serde = { version = "1", features = ["derive"] }
 serde_json = "1"

{atlas_python-0.9.1 → atlas_python-0.12.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: atlas-python
-Version: 0.9.1
+Version: 0.12.0
 Classifier: Development Status :: 3 - Alpha
 Classifier: Intended Audience :: Science/Research
 Classifier: Programming Language :: Python :: 3
@@ -118,18 +118,18 @@ ds = xr.Dataset(
 )
 
 with atlas.Atlas.create("/tmp/my_store") as store:
-    store.add_xr_dataset(ds, "jan_2024")     # store-side method
+    store.add_xarray_dataset(ds, "jan_2024")     # store-side method
     ds.atlas.write(store, "jan_2025")        # xarray accessor (same effect)
 
 # Read back as an xr.Dataset
 store = atlas.Atlas.open("/tmp/my_store")
-ds_back = store.to_xarray("jan_2024")
+ds_back = store.open_as_xarray_dataset("jan_2024")
 xr.testing.assert_identical(ds, ds_back)
 ```
 
 ### Bulk ingestion
 
-`add_xr_dataset` never flushes by itself — N consecutive calls accumulate in memory and a single
+`add_xarray_dataset` never flushes by itself — N consecutive calls accumulate in memory and a single
 `flush()` (or the `with` exit) persists everything.
 
 ```python
@@ -138,35 +138,35 @@ import glob, os, atlas, xarray as xr
 with atlas.Atlas.create("/tmp/store") as store:
     for nc_path in sorted(glob.glob("*.nc")):
         name = os.path.splitext(os.path.basename(nc_path))[0]
-        store.add_xr_dataset(xr.open_dataset(nc_path), name)
+        store.add_xarray_dataset(xr.open_dataset(nc_path), name)
 # One delta file per array name across the whole batch (not one per file).
 ```
 
 ### Streaming dask-backed writes
 
 If a variable's `.data` is a `dask.array.Array` (e.g. from `xr.open_dataset(path, chunks=...)`
-or `ds.chunk({...})`), `add_xr_dataset` / `ds.atlas.write` stream **one dask block at a time**
+or `ds.chunk({...})`), `add_xarray_dataset` / `ds.atlas.write` stream **one dask block at a time**
 into the store rather than materialising the whole array. The dask chunk shape becomes the
 on-disk `chunk_shape`, so the layout maps 1:1. Peak memory ≈ one chunk per variable.
 
 ```python
 ds = xr.open_dataset("big.nc", chunks={"time": 100, "lat": -1, "lon": -1})
 with atlas.Atlas.create("/tmp/store") as store:
-    store.add_xr_dataset(ds, "big")     # streams chunk-by-chunk
+    store.add_xarray_dataset(ds, "big")     # streams chunk-by-chunk
 ```
 
-Pass `chunks={var: [...]}` to `add_xr_dataset` / `ds.atlas.write` to override the on-disk chunk
+Pass `chunks={var: [...]}` to `add_xarray_dataset` / `ds.atlas.write` to override the on-disk chunk
 shape independently of dask's chunking.
 
 ### Lazy dask-backed reads
 
-`store.to_xarray(name)` returns each variable dask-backed whenever it was stored with non-trivial
+`store.open_as_xarray_dataset(name)` returns each variable dask-backed whenever it was stored with non-trivial
 chunking (`chunk_shape != shape`); the dask `chunks` tuple mirrors the on-disk chunk grid and each
 on-disk chunk is one dask task. Full-shape arrays (and 0-D scalars) come back eager as numpy. Call
 `.compute()` to materialise, or slice / `map_blocks` to operate lazily.
 
 ```python
-ds_back = atlas.Atlas.open("/tmp/store").to_xarray("big")
+ds_back = atlas.Atlas.open("/tmp/store").open_as_xarray_dataset("big")
 ds_back["temperature"].data              # -> dask.array.Array
 ds_back["temperature"][0:100].compute()  # reads exactly one chunk
 ```
@@ -179,15 +179,35 @@ picklable, so call `.compute()` before handing off to distributed/multiprocessin
 | Item | How it's stored |
 | --- | --- |
 | Each coord / data variable | A separate array, with `dims` mapped 1:1. |
-| Dataset attrs | Dataset attributes, plain keys. |
-| Per-variable attrs | Flattened as `{var}.{attr}` at the dataset attr level. |
-| Per-variable `_FillValue` | Consumed by `define_array` as a typed fill value (source `Dataset.attrs` is not mutated). |
+| Dataset attrs | Dataset-level (global) attributes, plain keys. |
+| Per-variable attrs | Real per-variable attributes on each variable's array. |
+| Per-variable `_FillValue` | Consumed by `define_array` as a typed fill value (source `Dataset.attrs` is not mutated). See [Missing data](#missing-data). |
 | Coord vs data_var distinction | JSON list in the internal `_pyatlas_coords` attr. |
 | Non-scalar attr values (list, ndarray) | JSON-encoded string with a `json:` prefix marker. |
 
-Each `add_xr_dataset` / `ds.atlas.write` creates a *new* dataset — there is no append-into-existing
+Each `add_xarray_dataset` / `ds.atlas.write` creates a *new* dataset — there is no append-into-existing
 mode.
 
+### Missing data
+
+When a dataset is opened with `mask_and_scale=True` (xarray's default), masked cells become `NaN`
+(floats) / `NaT` (datetimes) and `_FillValue` is moved into `var.encoding`. So those cells are
+recorded as **null** (counted in `null_count`, excluded from min/max stats), arrays default to a
+sentinel fill on write: `NaN` for floats, `NaT` for `datetime64[ns]`, and `""` for strings (integers
+have none). Missing string cells (`None`/`NaN`) are substituted with the string fill and a warning is
+emitted, since a string can't be stored as null directly.
+
+Override per write with `fill_value` — a bare scalar (numeric arrays) or a `{var: scalar}` dict
+(`None` disables the default for that var); an explicit CF `_FillValue` attribute still takes
+precedence over the default:
+
+```python
+store.add_xarray_dataset(ds, "jan_2024", fill_value={"temperature": -9999.0})
+```
+
+See the [xarray guide](https://github.com/maris-development/atlas/blob/main/atlas-python/docs/guides/xarray.md#fill-values-and-missing-data)
+for the full resolution order.
+
 ## Supported dtypes
 
 | numpy dtype | atlas dtype |
@@ -221,8 +241,9 @@ handle instead of a local path. The path-based local-filesystem API works withou
 | `list_datasets() -> list[str]` | All dataset names. |
 | `list_arrays() -> list[str]` | Distinct array names across datasets. |
 | `dataset_exists(name) -> bool` | Existence check. |
-| `add_xr_dataset(ds, name, chunks=None)` | Append an `xarray.Dataset` (does **not** flush). |
-| `to_xarray(name) -> xr.Dataset` | Read a dataset back (chunked vars come back dask-backed). |
+| `add_xarray_dataset(ds, name, chunks=None, fill_value=None)` | Append an `xarray.Dataset` (does **not** flush). `fill_value` overrides the per-array fill (scalar or `{var: scalar}`); see [Missing data](#missing-data). |
+| `open_as_xarray_dataset(name) -> xr.Dataset` | Read a dataset back (chunked vars come back dask-backed). |
+| `open_as_many_xarray_dataset(names, concat_dim="dataset") -> xr.Dataset` | Open many datasets stacked along `concat_dim` (eager numpy). |
 | `flush()` | The single durability boundary — persist everything. |
 | `close()` | Alias for `flush()`; also the `with`-block exit. |
 | `compact()` | Reclaim tombstoned space across cached array files. |
@@ -240,8 +261,10 @@ handle instead of a local path. The path-based local-filesystem API works withou
 | `delete_array(name)` | Tombstone the array within this dataset. |
 | `array_meta(name) -> dict \| None` | `{"dtype", "shape", "chunk_shape", "dimension_names"}`. |
 | `array_stats(name) -> dict \| None` | `{"row_count", "null_count", "min", "max"}` — populated after `flush()`. |
-| `set_attribute(key, value, dtype=None)` | Type inferred from the Python value; pass `dtype` to override (e.g. `"int8"`, `"float32"`, `"timestamp_nanoseconds"`). On disk: bool, int64, float64, string, timestamp_nanoseconds. |
-| `get_attribute(key)` / `attributes()` | Single attribute or dict of all. |
+| `set_attribute(key, value, dtype=None)` | Set a dataset-level (global) attribute. Type inferred from the Python value; pass `dtype` to override (e.g. `"int8"`, `"float32"`, `"timestamp_nanoseconds"`). Values are stored in the `_global` `.af` file. |
+| `get_attribute(key)` / `attributes()` | Single global attribute or dict of all. |
+| `set_array_attribute(array, key, value, dtype=None)` | Set a per-variable attribute on `array` (e.g. `units`); `KeyError` if the array isn't defined. |
+| `get_array_attribute(array, key)` / `array_attributes(array)` | Single per-variable attribute or dict of all, for `array`. |
 
 `DatasetView` does **not** expose its own `flush` / `compact` — both go through the parent `Atlas`.
 
@@ -250,13 +273,14 @@ handle instead of a local path. The path-based local-filesystem API works withou
 Runnable, self-contained scripts (each writes to a temp directory):
 
 - [01_basics.py](https://github.com/maris-development/atlas/blob/main/atlas-python/examples/01_basics.py) — create a store, define arrays, set attributes, reopen, read back.
-- [02_xarray.py](https://github.com/maris-development/atlas/blob/main/atlas-python/examples/02_xarray.py) — round-trip an `xr.Dataset` via both `store.add_xr_dataset(...)` and the `ds.atlas.write(...)` accessor.
+- [02_xarray.py](https://github.com/maris-development/atlas/blob/main/atlas-python/examples/02_xarray.py) — round-trip an `xr.Dataset` via both `store.add_xarray_dataset(...)` and the `ds.atlas.write(...)` accessor.
 - [03_dask_streaming.py](https://github.com/maris-development/atlas/blob/main/atlas-python/examples/03_dask_streaming.py) — stream a dask-chunked `xr.Dataset` in one chunk at a time.
+- [09_missing_data.py](https://github.com/maris-development/atlas/blob/main/atlas-python/examples/09_missing_data.py) — masked/missing cells with default `NaN` / `NaT` / `""` fills, `null_count`, and `fill_value=` overrides.
 
 ## Performance
 
 ATLAS is tuned for collections of many similarly-shaped datasets. On a "1000 datasets" benchmark
-against netCDF4 and Zarr v3, the bulk read paths (`Atlas.to_xarray_many` /
+against netCDF4 and Zarr v3, the bulk read paths (`Atlas.open_as_many_xarray_dataset` /
 `Atlas.read_array_across_stacked`) beat Zarr by ~2.8× on large chunked slice reads, and on small
 per-dataset workloads ATLAS leads on both reads and writes. See the
 [benchmarks](https://github.com/maris-development/atlas/tree/main/atlas-python/benchmarks) for the full

{atlas_python-0.9.1 → atlas_python-0.12.0}/README.md

@@ -1,5 +1,12 @@
 # ATLAS — Aggregated Tensor Large Array Store
 
+[![CI](https://github.com/maris-development/atlas/actions/workflows/ci.yaml/badge.svg)](https://github.com/maris-development/atlas/actions/workflows/ci.yaml)
+[![crates.io](https://img.shields.io/crates/v/atlas-rust.svg?logo=rust)](https://crates.io/crates/atlas-rust)
+[![docs.rs](https://img.shields.io/docsrs/atlas-rust?logo=docsdotrs&label=docs.rs)](https://docs.rs/atlas-rust)
+[![PyPI](https://img.shields.io/pypi/v/atlas-python.svg?logo=pypi&logoColor=white)](https://pypi.org/project/atlas-python/)
+[![Python docs](https://img.shields.io/badge/docs-atlas--python-blue?logo=materialformkdocs&logoColor=white)](https://maris-development.github.io/atlas/)
+[![License](https://img.shields.io/crates/l/atlas-rust.svg)](LICENSE)
+
 A directory-based store for thousands of named datasets, each holding N-dimensional typed arrays. Built on top of the [`array-format`](https://github.com/robinskil/array-format) (`.af`) binary format, with configurable compression (Zstd, LZ4, or none), chunked I/O, and an [`object_store`](https://crates.io/crates/object_store) backend that works on local disk, S3, GCS, Azure Blob, and in-memory.
 
 **Think of it as a "zip" for N-dimensional datasets.** Where a `.zip` bundles many files into one archive, ATLAS gathers many NetCDF / Zarr-style datasets — anything that's a set of named N-dimensional arrays — into a single high-performance collection. But it's more than a bundle: instead of storing each dataset whole, ATLAS lays the data out **variable-first**, so every dataset's `temperature` lives in one file. That makes it cheap to scan or slice a single variable across thousands of datasets at once, while still reading back any individual dataset as a normal NetCDF/xarray-like object.
@@ -14,26 +21,30 @@ A directory-based store for thousands of named datasets, each holding N-dimensio
 
 `atlas` is designed for workloads where you have a large collection of similarly-shaped datasets — such as one dataset per time step, sensor station, or simulation run — and you want to query a single variable (e.g. `temperature`) across all of them efficiently.
 
-Each dataset is a named group of N-dimensional arrays with typed per-dataset attributes. Datasets that share an array name (e.g. every `jan_2024`, `feb_2024`, … all have `temperature`) are stored together in the same physical file, keyed by dataset name inside the file.
+Each dataset is a named group of N-dimensional arrays with typed attributes — both dataset-level (global) attributes and per-variable attributes. Datasets that share an array name (e.g. every `jan_2024`, `feb_2024`, … all have `temperature`) are stored together in the same physical file, keyed by dataset name inside the file.
 
 ```text
 my_store/
-├── atlas.json               ← dataset registry and per-dataset attributes (JSON)
+├── atlas.json               ← interned per-dataset schema + attribute-key namespace (JSON)
+├── _global/
+│   └── data.af         ← dataset-level (global) attribute values, one entry per dataset
 ├── temperature/
-│   └── data.af         ← one ArrayFile holding temperature for every dataset
+│   └── data.af         ← one ArrayFile: temperature + its per-variable attributes, per dataset
 ├── pressure/
 │   └── data.af
 └── time/
     └── data.af
 ```
 
+`atlas.json` holds only the **schema** of the collection. Attribute **values** live in the `.af` files: per-variable attributes on the variable's own file, and dataset-level attributes in the reserved `_global` file.
+
 ---
 
 ## Durability model
 
-`atlas.json` is read **once** when the store is opened or created. Every subsequent mutation — `create_dataset`, `define_array`, `set_attribute`, `delete_array`, `delete_dataset` — only touches the in-memory `StoreMeta`; writes to `ArrayFile`s buffer inside the per-array in-memory layer. **Nothing reaches disk until `Atlas::flush()` (or `Atlas::close()`).** Dropping an `Atlas` without flushing abandons every pending in-memory write.
+`atlas.json` is read **once** when the store is opened or created. Every subsequent mutation — `create_dataset`, `define_array`, `set_attribute`, `set_array_attribute`, `delete_array`, `delete_dataset` — only touches in-memory state; array writes buffer inside the per-array in-memory layer and attribute writes buffer in a pending-attribute map. **Nothing reaches disk until `Atlas::flush()` (or `Atlas::close()`).** Dropping an `Atlas` without flushing abandons every pending in-memory write.
 
-A single `Atlas::flush()` walks every cached `ArrayFile` (writing deltas + stats) and then serialises the in-memory `StoreMeta` to `atlas.json`. This gives one durability boundary for the whole store: N datasets ⇒ one delta file per touched array name (not one per dataset) and one `atlas.json` rewrite (not N).
+A single `Atlas::flush()` drains the buffered attributes into their `.af` files, walks every cached `ArrayFile` (writing deltas + stats), and then serialises the in-memory schema to `atlas.json`. This gives one durability boundary for the whole store: N datasets ⇒ one delta file per touched array name (not one per dataset) and one `atlas.json` rewrite (not N).
 
 `DatasetView` is a borrowed handle into the atlas's shared meta — it has no `flush()` of its own.
 
@@ -43,14 +54,25 @@ A single `Atlas::flush()` walks every cached `ArrayFile` (writing deltas + stats
 
 ### `atlas.json`
 
-The registry is a plain JSON file written on `Atlas::flush()` / `Atlas::close()`. It stores:
+The schema is a plain JSON file written on `Atlas::flush()` / `Atlas::close()`. It stores:
+
+- **Store version** — for format upgrades. The current version is `2`; stores written by an older atlas are rejected on open (re-export to upgrade).
+- **Interned schema pool** — each *distinct* per-dataset schema is stored once and referenced by index. A dataset's schema is its array schemas (per array: dtype, shape, chunk shape, named dimensions, codec) plus its **attribute-key namespace** (the names of its global and per-variable attribute keys). A collection of thousands of homogeneous datasets stores its schema a single time.
+- **Dataset registry** — a map of dataset name → schema-pool index.
+- **Merged schema** — a collection-wide summary listing every unique array (with its dtype widened across all datasets, its dimensions, and its per-variable attribute types) and every global attribute type. Descriptive only — reads always use each dataset's own schema — but handy for tools that want one view of "what's in here." Also exposed programmatically via `Atlas::merged_schema()`.
+
+When the same array name (or attribute key) appears in more than one dataset with different types, the merged type is **widened**. Widening is allowed only within numeric types (e.g. `int16` ∪ `int32` → `int32`; `int32` ∪ `float32` → `float64`) or between `string` and `timestamp` (→ `string`).
+
+Any other combination — e.g. an `int32` array in one dataset and a `string` array under the same name in another — can't merge. The dataset is **still stored** under its own type (each dataset keeps the type it declared, and its data reads back normally); the merged schema simply keeps the **first-seen** type. Whether that's reported as a warning or an error is set by `StoreConfig::on_type_mismatch`:
+
+| `TypeMismatchPolicy` | Behaviour |
+| --- | --- |
+| `Warn` (default) | Logs a `tracing` warning and carries on — an ingest of many heterogeneous files won't abort because one disagrees. |
+| `Error` | `define_array` / `set_attribute` / `set_array_attribute` return `Error::TypeMismatch`. |
 
-- **Store version** — for future format upgrades.
-- **Dataset names** — the complete list of datasets in the store.
-- **Per-dataset attributes** — typed key-value pairs serialized as plain JSON values: bool, 64-bit integer, 64-bit float, UTF-8 string, or an RFC 3339 nanosecond-precision timestamp string (e.g. `"2023-11-15T07:33:20.123456789Z"`). Atlas's `Attr` enum has five variants (`Bool`/`Int64`/`Float64`/`String`/`TimestampNanoseconds`); there are no narrow integer/float types on disk.
-- **Array schemas** — per array: dtype, shape, chunk shape, named dimensions, and the codec used when the array was first written.
+The policy is a **per-session** choice, not an on-disk property: pass it to `Atlas::create`, or to `Atlas::open_with_config` / `Atlas::open_path_with_config` when opening an existing collection (plain `open` / `open_path` use the default). Note that these open variants honour only `on_type_mismatch` — the codec and metadata format are always detected from disk.
 
-Because `atlas.json` is human-readable and self-describing, you can inspect or audit the store contents with any JSON tool without needing the library.
+Attribute **values** are **not** in `atlas.json` — only their key names are (as part of the schema). Values live in the `.af` files and are read via the `array-format` attribute API. Because `atlas.json` is human-readable and self-describing, you can still inspect or audit the collection's schema with any JSON tool without needing the library.
 
 ### `<array_name>/data.af`
 
@@ -60,6 +82,7 @@ Each array variable gets its own subdirectory with a single `data.af` binary fil
 - **Chunked layout** — arrays are split into chunks of a user-specified shape, so partial reads and writes touch only the relevant blocks.
 - **Configurable compression** — each block is compressed with the codec set when the store was created (default: Zstd; also LZ4 and uncompressed). The codec is persisted in `atlas.json` and restored automatically on `open` — no need to pass it again. Block target size is 8 MiB.
 - **Per-array fill value** — `define_array` accepts an optional `FillValue` (one of `Bool`/`Int`/`UInt`/`Float`/`String`). Unwritten cells read back as the fill value, and any *written* cell equal to it is counted as a null in the persisted stats (see below). Fill values are stored in the per-array footer; `atlas.json` is not extended.
+- **Attribute values** — per-variable attributes (e.g. `units`) are stored in the footer of the variable's own `data.af`, keyed by dataset. Dataset-level (global) attributes live the same way in the reserved `_global/data.af`. Atlas's `Attr` type mirrors `array-format`'s `AttributeValue` (bool, sized ints/uints, `f32`/`f64`, string, binary, and typed lists) plus a nanosecond timestamp variant stored as an RFC 3339 string.
 - **Persisted statistics** — on `Atlas::flush()`, min, max, null count, and row count are computed per array per dataset and stored alongside the data. Cells equal to the array's fill value are tallied in `null_count` and excluded from `min`/`max` (NaN fills match NaN cells by bit pattern). Statistics survive store reopening.
 - **In-memory caches** — a 256 MiB decoded block cache and a 64 MiB raw I/O cache sit in front of the object store for repeated reads.
 
@@ -257,7 +280,7 @@ Choose LZ4 when decompression throughput matters more than storage size (e.g. la
 
 ### Write path
 
-Writes are buffered in-memory across the whole atlas. Calling `Atlas::flush()` compresses and writes every modified block across every cached array file, then rewrites `atlas.json` atomically (a single `PUT`). The write path scales with the number of modified chunks, not the number of datasets, and N consecutive `add_xr_dataset` / `create_dataset` calls amortise to a single flush.
+Writes are buffered in-memory across the whole atlas. Calling `Atlas::flush()` compresses and writes every modified block across every cached array file, then rewrites `atlas.json` atomically (a single `PUT`). The write path scales with the number of modified chunks, not the number of datasets, and N consecutive `add_xarray_dataset` / `create_dataset` calls amortise to a single flush.
 
 ### Compaction
 
@@ -301,7 +324,7 @@ the `(0:25, 0:25, 0:12)` slice down to chunk-level reads.
 | **atlas-bulk** | 1 store, `read_array_across_stacked` with slice push-down | **2.12** | 59 | 6387 |
 | **atlas + `--use-dask`** | 1 store, dask-threaded per-dataset `view.read_arrays(...)` | **3.21** | 60 | 6387 |
 | zarr | 1000 separate stores, `xr.open_mfdataset(parallel=True).isel(...)` | 5.99 | 38 | 6392 |
-| atlas (default) | 1 store, serial per-dataset `to_xarray(...).isel(...).load()` | 10.23 | 51 | 6387 |
+| atlas (default) | 1 store, serial per-dataset `open_as_xarray_dataset(...).isel(...).load()` | 10.23 | 51 | 6387 |
 | netcdf | 1000 `.nc` files, `xr.open_mfdataset(parallel=True).isel(...)` | 13.91 | 122 | 5596 |
 
 #### `--case profile --datasets 1000` (~67 MB raw — per-dataset overhead dominates)
@@ -332,9 +355,9 @@ For tiny-shape datasets the comparison is dominated by per-dataset overhead, whi
 
 ### What the numbers say
 
-- **Reads on gridded** (decompression-dominated, with realistic chunking + slice push-down): `atlas-bulk` reads in **2.12s vs zarr's 6.00s — 2.8× faster**. `atlas + --use-dask` (using the new `view.read_arrays(...)` fast path) hits **3.21s — 1.9× faster than zarr**. The win comes from atlas's structural advantage (one open of one file per variable, in-memory metadata) combined with APIs that bypass `to_xarray`'s xr.Dataset + per-chunk dask graph overhead. zarr and netcdf both push the slice down through `open_mfdataset(...).isel(...)` via dask graph optimization — they're not penalised by the chunking; atlas just wins by amortising metadata and skipping per-dataset Python overhead.
+- **Reads on gridded** (decompression-dominated, with realistic chunking + slice push-down): `atlas-bulk` reads in **2.12s vs zarr's 6.00s — 2.8× faster**. `atlas + --use-dask` (using the new `view.read_arrays(...)` fast path) hits **3.21s — 1.9× faster than zarr**. The win comes from atlas's structural advantage (one open of one file per variable, in-memory metadata) combined with APIs that bypass `open_as_xarray_dataset`'s xr.Dataset + per-chunk dask graph overhead. zarr and netcdf both push the slice down through `open_mfdataset(...).isel(...)` via dask graph optimization — they're not penalised by the chunking; atlas just wins by amortising metadata and skipping per-dataset Python overhead.
 - **Reads on profile** (overhead-dominated): atlas wins by an order of magnitude. `atlas-bulk` is **~50× faster than zarr** (0.08s vs 4.07s) because the per-dataset I/O is small enough that everything is overhead — and atlas's structural design is built for exactly that case. Default serial `atlas` is still ~12× faster than zarr.
-- **Default `atlas.to_xarray` iteration is currently SLOW on chunked storage** (10.23s vs zarr's 6.00s). It goes through `to_xarray(name)` which returns dask-backed arrays per chunk (8 chunks/var × 3 vars × 1000 datasets = 24,000 dask delayed tasks just to build the graph). Dask graph overhead exceeds the parallelism win. **The fix is to use `view.read_arrays(vars, start, shape)` for per-dataset slice reads** — that's what `atlas + --use-dask` does internally and why it hits 3.21s. For cross-dataset reads of the *same* slice from many datasets, prefer `Atlas.to_xarray_many` / `read_array_across_stacked` (the `atlas-bulk` row).
+- **Default `atlas.open_as_xarray_dataset` iteration is currently SLOW on chunked storage** (10.23s vs zarr's 6.00s). It goes through `open_as_xarray_dataset(name)` which returns dask-backed arrays per chunk (8 chunks/var × 3 vars × 1000 datasets = 24,000 dask delayed tasks just to build the graph). Dask graph overhead exceeds the parallelism win. **The fix is to use `view.read_arrays(vars, start, shape)` for per-dataset slice reads** — that's what `atlas + --use-dask` does internally and why it hits 3.21s. For cross-dataset reads of the *same* slice from many datasets, prefer `Atlas.open_as_many_xarray_dataset` / `read_array_across_stacked` (the `atlas-bulk` row).
 - **Workload sensitivity**: `--use-dask` helps when per-dataset decompression is the bottleneck and *hurts* when per-dataset overhead is the bottleneck (profile: default `atlas` 0.32s → dask 2.03s). Picking the right API for the workload matters more than picking dask vs serial.
 - **Writes on gridded**: zarr is fastest (~31s vs atlas's ~50s) — each `to_zarr(...)` just dumps bytes into a per-dataset subtree, while atlas does more bookkeeping per call (schema registration, per-dataset attrs into the shared `atlas.msgpack.zst`, block addressing in the shared array file). Atlas still beats netcdf decisively.
 - **Writes on profile**: atlas wins ~12× (0.91s vs zarr's 11.43s, netcdf's 2.98s). At small per-dataset sizes the metadata write dominates, and atlas's amortised single-flush model wins.

{atlas_python-0.9.1 → atlas_python-0.12.0}/atlas-python/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "atlas-python"
-version = "0.9.1"
+version = "0.12.0"
 edition = "2021"
 readme = "README.md"
 

{atlas_python-0.9.1 → atlas_python-0.12.0}/atlas-python/README.md

@@ -86,18 +86,18 @@ ds = xr.Dataset(
 )
 
 with atlas.Atlas.create("/tmp/my_store") as store:
-    store.add_xr_dataset(ds, "jan_2024")     # store-side method
+    store.add_xarray_dataset(ds, "jan_2024")     # store-side method
     ds.atlas.write(store, "jan_2025")        # xarray accessor (same effect)
 
 # Read back as an xr.Dataset
 store = atlas.Atlas.open("/tmp/my_store")
-ds_back = store.to_xarray("jan_2024")
+ds_back = store.open_as_xarray_dataset("jan_2024")
 xr.testing.assert_identical(ds, ds_back)
 ```
 
 ### Bulk ingestion
 
-`add_xr_dataset` never flushes by itself — N consecutive calls accumulate in memory and a single
+`add_xarray_dataset` never flushes by itself — N consecutive calls accumulate in memory and a single
 `flush()` (or the `with` exit) persists everything.
 
 ```python
@@ -106,35 +106,35 @@ import glob, os, atlas, xarray as xr
 with atlas.Atlas.create("/tmp/store") as store:
     for nc_path in sorted(glob.glob("*.nc")):
         name = os.path.splitext(os.path.basename(nc_path))[0]
-        store.add_xr_dataset(xr.open_dataset(nc_path), name)
+        store.add_xarray_dataset(xr.open_dataset(nc_path), name)
 # One delta file per array name across the whole batch (not one per file).
 ```
 
 ### Streaming dask-backed writes
 
 If a variable's `.data` is a `dask.array.Array` (e.g. from `xr.open_dataset(path, chunks=...)`
-or `ds.chunk({...})`), `add_xr_dataset` / `ds.atlas.write` stream **one dask block at a time**
+or `ds.chunk({...})`), `add_xarray_dataset` / `ds.atlas.write` stream **one dask block at a time**
 into the store rather than materialising the whole array. The dask chunk shape becomes the
 on-disk `chunk_shape`, so the layout maps 1:1. Peak memory ≈ one chunk per variable.
 
 ```python
 ds = xr.open_dataset("big.nc", chunks={"time": 100, "lat": -1, "lon": -1})
 with atlas.Atlas.create("/tmp/store") as store:
-    store.add_xr_dataset(ds, "big")     # streams chunk-by-chunk
+    store.add_xarray_dataset(ds, "big")     # streams chunk-by-chunk
 ```
 
-Pass `chunks={var: [...]}` to `add_xr_dataset` / `ds.atlas.write` to override the on-disk chunk
+Pass `chunks={var: [...]}` to `add_xarray_dataset` / `ds.atlas.write` to override the on-disk chunk
 shape independently of dask's chunking.
 
 ### Lazy dask-backed reads
 
-`store.to_xarray(name)` returns each variable dask-backed whenever it was stored with non-trivial
+`store.open_as_xarray_dataset(name)` returns each variable dask-backed whenever it was stored with non-trivial
 chunking (`chunk_shape != shape`); the dask `chunks` tuple mirrors the on-disk chunk grid and each
 on-disk chunk is one dask task. Full-shape arrays (and 0-D scalars) come back eager as numpy. Call
 `.compute()` to materialise, or slice / `map_blocks` to operate lazily.
 
 ```python
-ds_back = atlas.Atlas.open("/tmp/store").to_xarray("big")
+ds_back = atlas.Atlas.open("/tmp/store").open_as_xarray_dataset("big")
 ds_back["temperature"].data              # -> dask.array.Array
 ds_back["temperature"][0:100].compute()  # reads exactly one chunk
 ```
@@ -147,15 +147,35 @@ picklable, so call `.compute()` before handing off to distributed/multiprocessin
 | Item | How it's stored |
 | --- | --- |
 | Each coord / data variable | A separate array, with `dims` mapped 1:1. |
-| Dataset attrs | Dataset attributes, plain keys. |
-| Per-variable attrs | Flattened as `{var}.{attr}` at the dataset attr level. |
-| Per-variable `_FillValue` | Consumed by `define_array` as a typed fill value (source `Dataset.attrs` is not mutated). |
+| Dataset attrs | Dataset-level (global) attributes, plain keys. |
+| Per-variable attrs | Real per-variable attributes on each variable's array. |
+| Per-variable `_FillValue` | Consumed by `define_array` as a typed fill value (source `Dataset.attrs` is not mutated). See [Missing data](#missing-data). |
 | Coord vs data_var distinction | JSON list in the internal `_pyatlas_coords` attr. |
 | Non-scalar attr values (list, ndarray) | JSON-encoded string with a `json:` prefix marker. |
 
-Each `add_xr_dataset` / `ds.atlas.write` creates a *new* dataset — there is no append-into-existing
+Each `add_xarray_dataset` / `ds.atlas.write` creates a *new* dataset — there is no append-into-existing
 mode.
 
+### Missing data
+
+When a dataset is opened with `mask_and_scale=True` (xarray's default), masked cells become `NaN`
+(floats) / `NaT` (datetimes) and `_FillValue` is moved into `var.encoding`. So those cells are
+recorded as **null** (counted in `null_count`, excluded from min/max stats), arrays default to a
+sentinel fill on write: `NaN` for floats, `NaT` for `datetime64[ns]`, and `""` for strings (integers
+have none). Missing string cells (`None`/`NaN`) are substituted with the string fill and a warning is
+emitted, since a string can't be stored as null directly.
+
+Override per write with `fill_value` — a bare scalar (numeric arrays) or a `{var: scalar}` dict
+(`None` disables the default for that var); an explicit CF `_FillValue` attribute still takes
+precedence over the default:
+
+```python
+store.add_xarray_dataset(ds, "jan_2024", fill_value={"temperature": -9999.0})
+```
+
+See the [xarray guide](https://github.com/maris-development/atlas/blob/main/atlas-python/docs/guides/xarray.md#fill-values-and-missing-data)
+for the full resolution order.
+
 ## Supported dtypes
 
 | numpy dtype | atlas dtype |
@@ -189,8 +209,9 @@ handle instead of a local path. The path-based local-filesystem API works withou
 | `list_datasets() -> list[str]` | All dataset names. |
 | `list_arrays() -> list[str]` | Distinct array names across datasets. |
 | `dataset_exists(name) -> bool` | Existence check. |
-| `add_xr_dataset(ds, name, chunks=None)` | Append an `xarray.Dataset` (does **not** flush). |
-| `to_xarray(name) -> xr.Dataset` | Read a dataset back (chunked vars come back dask-backed). |
+| `add_xarray_dataset(ds, name, chunks=None, fill_value=None)` | Append an `xarray.Dataset` (does **not** flush). `fill_value` overrides the per-array fill (scalar or `{var: scalar}`); see [Missing data](#missing-data). |
+| `open_as_xarray_dataset(name) -> xr.Dataset` | Read a dataset back (chunked vars come back dask-backed). |
+| `open_as_many_xarray_dataset(names, concat_dim="dataset") -> xr.Dataset` | Open many datasets stacked along `concat_dim` (eager numpy). |
 | `flush()` | The single durability boundary — persist everything. |
 | `close()` | Alias for `flush()`; also the `with`-block exit. |
 | `compact()` | Reclaim tombstoned space across cached array files. |
@@ -208,8 +229,10 @@ handle instead of a local path. The path-based local-filesystem API works withou
 | `delete_array(name)` | Tombstone the array within this dataset. |
 | `array_meta(name) -> dict \| None` | `{"dtype", "shape", "chunk_shape", "dimension_names"}`. |
 | `array_stats(name) -> dict \| None` | `{"row_count", "null_count", "min", "max"}` — populated after `flush()`. |
-| `set_attribute(key, value, dtype=None)` | Type inferred from the Python value; pass `dtype` to override (e.g. `"int8"`, `"float32"`, `"timestamp_nanoseconds"`). On disk: bool, int64, float64, string, timestamp_nanoseconds. |
-| `get_attribute(key)` / `attributes()` | Single attribute or dict of all. |
+| `set_attribute(key, value, dtype=None)` | Set a dataset-level (global) attribute. Type inferred from the Python value; pass `dtype` to override (e.g. `"int8"`, `"float32"`, `"timestamp_nanoseconds"`). Values are stored in the `_global` `.af` file. |
+| `get_attribute(key)` / `attributes()` | Single global attribute or dict of all. |
+| `set_array_attribute(array, key, value, dtype=None)` | Set a per-variable attribute on `array` (e.g. `units`); `KeyError` if the array isn't defined. |
+| `get_array_attribute(array, key)` / `array_attributes(array)` | Single per-variable attribute or dict of all, for `array`. |
 
 `DatasetView` does **not** expose its own `flush` / `compact` — both go through the parent `Atlas`.
 
@@ -218,13 +241,14 @@ handle instead of a local path. The path-based local-filesystem API works withou
 Runnable, self-contained scripts (each writes to a temp directory):
 
 - [01_basics.py](https://github.com/maris-development/atlas/blob/main/atlas-python/examples/01_basics.py) — create a store, define arrays, set attributes, reopen, read back.
-- [02_xarray.py](https://github.com/maris-development/atlas/blob/main/atlas-python/examples/02_xarray.py) — round-trip an `xr.Dataset` via both `store.add_xr_dataset(...)` and the `ds.atlas.write(...)` accessor.
+- [02_xarray.py](https://github.com/maris-development/atlas/blob/main/atlas-python/examples/02_xarray.py) — round-trip an `xr.Dataset` via both `store.add_xarray_dataset(...)` and the `ds.atlas.write(...)` accessor.
 - [03_dask_streaming.py](https://github.com/maris-development/atlas/blob/main/atlas-python/examples/03_dask_streaming.py) — stream a dask-chunked `xr.Dataset` in one chunk at a time.
+- [09_missing_data.py](https://github.com/maris-development/atlas/blob/main/atlas-python/examples/09_missing_data.py) — masked/missing cells with default `NaN` / `NaT` / `""` fills, `null_count`, and `fill_value=` overrides.
 
 ## Performance
 
 ATLAS is tuned for collections of many similarly-shaped datasets. On a "1000 datasets" benchmark
-against netCDF4 and Zarr v3, the bulk read paths (`Atlas.to_xarray_many` /
+against netCDF4 and Zarr v3, the bulk read paths (`Atlas.open_as_many_xarray_dataset` /
 `Atlas.read_array_across_stacked`) beat Zarr by ~2.8× on large chunked slice reads, and on small
 per-dataset workloads ATLAS leads on both reads and writes. See the
 [benchmarks](https://github.com/maris-development/atlas/tree/main/atlas-python/benchmarks) for the full