atlas-python 0.10.0__tar.gz → 0.13.0__tar.gz

{atlas_python-0.10.0 → atlas_python-0.13.0}/.gitignore

@@ -33,3 +33,5 @@ atlas-python/python/atlas/_atlas.pyd
 atlas-python/python/atlas/_atlas.abi3.so
 atlas-python/python/atlas/_atlas*.so
 atlas-python/python/atlas/_atlas.pdb
+test-collection-netcdf-seadatanet/
+batch_to_atlas.py

{atlas_python-0.10.0 → atlas_python-0.13.0}/Cargo.lock

@@ -28,9 +28,9 @@ checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c"
 
 [[package]]
 name = "array-format"
-version = "0.10.0"
+version = "0.12.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "a637a3b5f23ba45eb53f09e643b363163114f0e94d0f6f05f330134cb86d381b"
+checksum = "f7379c3303a9b9f0693ccd040d85da31e302a601efeafae8a6cb69341e42de41"
 dependencies = [
  "bytes",
  "futures",
@@ -71,7 +71,7 @@ dependencies = [
 
 [[package]]
 name = "atlas-python"
-version = "0.10.0"
+version = "0.13.0"
 dependencies = [
  "atlas-rust",
  "ndarray",
@@ -86,7 +86,7 @@ dependencies = [
 
 [[package]]
 name = "atlas-rust"
-version = "0.10.0"
+version = "0.13.0"
 dependencies = [
  "array-format",
  "chrono",

{atlas_python-0.10.0 → atlas_python-0.13.0}/Cargo.toml

@@ -5,7 +5,7 @@ members = ["atlas-python"]
 # Registry name on crates.io (`atlas` is taken). The library is still
 # imported as `atlas` — see `[lib]` below.
 name = "atlas-rust"
-version = "0.10.0"
+version = "0.13.0"
 edition = "2024"
 description = "Directory-based store for thousands of N-dimensional datasets local or remote using object storage."
 license = "Apache-2.0"
@@ -22,7 +22,7 @@ all-features = true
 rustdoc-args = ["--cfg", "docsrs"]
 
 [dependencies]
-array-format = "0.10.0"
+array-format = "0.12.0"
 object_store = "0.13"
 serde = { version = "1", features = ["derive"] }
 serde_json = "1"

{atlas_python-0.10.0 → atlas_python-0.13.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: atlas-python
-Version: 0.10.0
+Version: 0.13.0
 Classifier: Development Status :: 3 - Alpha
 Classifier: Intended Audience :: Science/Research
 Classifier: Programming Language :: Python :: 3
@@ -179,8 +179,8 @@ picklable, so call `.compute()` before handing off to distributed/multiprocessin
 | Item | How it's stored |
 | --- | --- |
 | Each coord / data variable | A separate array, with `dims` mapped 1:1. |
-| Dataset attrs | Dataset attributes, plain keys. |
-| Per-variable attrs | Flattened as `{var}.{attr}` at the dataset attr level. |
+| Dataset attrs | Dataset-level (global) attributes, plain keys. |
+| Per-variable attrs | Real per-variable attributes on each variable's array. |
 | Per-variable `_FillValue` | Consumed by `define_array` as a typed fill value (source `Dataset.attrs` is not mutated). See [Missing data](#missing-data). |
 | Coord vs data_var distinction | JSON list in the internal `_pyatlas_coords` attr. |
 | Non-scalar attr values (list, ndarray) | JSON-encoded string with a `json:` prefix marker. |
@@ -220,6 +220,40 @@ for the full resolution order.
 - `bool` is available as an *attribute* type but not as an array dtype.
 - `binary`, `list[...]`, `fixed_size_list[...,N]` are reserved for a later release.
 
+## Cross-dataset pruning
+
+`ds.array_stats(name)` answers "what's the range of `temperature` in *this* dataset?" one at a
+time. To ask "which of my 10 000 datasets could contain a value above 25?" as a single
+vectorised scan, use the **pruning index** — a flattened, columnar view of the per-dataset
+statistics, one row per dataset:
+
+```python
+import numpy as np
+
+store = atlas.Atlas.open("/tmp/my_store")
+
+# Footer only — every column's collection-wide min/max, no column data fetched.
+summ = store.column_summaries()
+if summ["temperature"]["max"] < 25.0:
+    ...  # nothing can match — skip the column entirely
+
+# Fetch just the columns you need; each is a dict of numpy arrays over the row space.
+idx = store.pruning_index(arrays=["temperature"])
+col = idx["columns"]["temperature"]
+
+ok   = col["present"] & col["stats_valid"] & idx["live"]   # exclude gaps + deleted rows
+hits = np.where(ok & (col["max"] > 25.0))[0]
+candidates = [idx["datasets"][i] for i in hits]            # row -> dataset name
+```
+
+The return value is self-describing: `idx["datasets"][i]` names row `i`, `idx["live"]` is the
+delete mask (always `&` it in), and datasets that don't declare a column are explicit gaps
+(`present` is `False`, with `row_count` 0). Only the columns you ask for are fetched from
+storage, so the cost is independent of how many other columns the collection has. `min`/`max`
+keep their source type (strings come back as `bytes`, timestamps as int64 nanoseconds). See the
+[stats & scans guide](https://github.com/maris-development/atlas/blob/main/atlas-python/docs/guides/stats.md)
+for the full API, including `global_attrs=` / `array_attrs=` columns.
+
 ## Cloud / object storage
 
 With the `cloud` extra, `Atlas.open` / `Atlas.create` accept an
@@ -244,9 +278,13 @@ handle instead of a local path. The path-based local-filesystem API works withou
 | `add_xarray_dataset(ds, name, chunks=None, fill_value=None)` | Append an `xarray.Dataset` (does **not** flush). `fill_value` overrides the per-array fill (scalar or `{var: scalar}`); see [Missing data](#missing-data). |
 | `open_as_xarray_dataset(name) -> xr.Dataset` | Read a dataset back (chunked vars come back dask-backed). |
 | `open_as_many_xarray_dataset(names, concat_dim="dataset") -> xr.Dataset` | Open many datasets stacked along `concat_dim` (eager numpy). |
+| `pruning_index(arrays=None, global_attrs=None, array_attrs=None) -> dict` | Flattened statistics for **only** the requested columns; see [Cross-dataset pruning](#cross-dataset-pruning). |
+| `column_summaries() -> dict` | Every column's collection-wide min/max, read from the index footer alone (no column data fetched). |
+| `dataset_row(name) -> int \| None` | The dataset's fixed row ordinal in the pruning index. |
+| `row_slots() -> int` | Total row slots (live + tombstoned) — the pruning index's height. |
 | `flush()` | The single durability boundary — persist everything. |
 | `close()` | Alias for `flush()`; also the `with`-block exit. |
-| `compact()` | Reclaim tombstoned space across cached array files. |
+| `compact()` | Reclaim tombstoned space across cached array files, and renumber row ordinals. |
 | `__enter__` / `__exit__` | Context-manager support (`__exit__` calls `close()`). |
 
 ### `atlas.DatasetView`
@@ -261,8 +299,10 @@ handle instead of a local path. The path-based local-filesystem API works withou
 | `delete_array(name)` | Tombstone the array within this dataset. |
 | `array_meta(name) -> dict \| None` | `{"dtype", "shape", "chunk_shape", "dimension_names"}`. |
 | `array_stats(name) -> dict \| None` | `{"row_count", "null_count", "min", "max"}` — populated after `flush()`. |
-| `set_attribute(key, value, dtype=None)` | Type inferred from the Python value; pass `dtype` to override (e.g. `"int8"`, `"float32"`, `"timestamp_nanoseconds"`). On disk: bool, int64, float64, string, timestamp_nanoseconds. |
-| `get_attribute(key)` / `attributes()` | Single attribute or dict of all. |
+| `set_attribute(key, value, dtype=None)` | Set a dataset-level (global) attribute. Type inferred from the Python value; pass `dtype` to override (e.g. `"int8"`, `"float32"`, `"timestamp_nanoseconds"`). Values are stored in the `_global` `.af` file. |
+| `get_attribute(key)` / `attributes()` | Single global attribute or dict of all. |
+| `set_array_attribute(array, key, value, dtype=None)` | Set a per-variable attribute on `array` (e.g. `units`); `KeyError` if the array isn't defined. |
+| `get_array_attribute(array, key)` / `array_attributes(array)` | Single per-variable attribute or dict of all, for `array`. |
 
 `DatasetView` does **not** expose its own `flush` / `compact` — both go through the parent `Atlas`.
 

{atlas_python-0.10.0 → atlas_python-0.13.0}/README.md

@@ -21,26 +21,31 @@ A directory-based store for thousands of named datasets, each holding N-dimensio
 
 `atlas` is designed for workloads where you have a large collection of similarly-shaped datasets — such as one dataset per time step, sensor station, or simulation run — and you want to query a single variable (e.g. `temperature`) across all of them efficiently.
 
-Each dataset is a named group of N-dimensional arrays with typed per-dataset attributes. Datasets that share an array name (e.g. every `jan_2024`, `feb_2024`, … all have `temperature`) are stored together in the same physical file, keyed by dataset name inside the file.
+Each dataset is a named group of N-dimensional arrays with typed attributes — both dataset-level (global) attributes and per-variable attributes. Datasets that share an array name (e.g. every `jan_2024`, `feb_2024`, … all have `temperature`) are stored together in the same physical file, keyed by dataset name inside the file.
 
 ```text
 my_store/
-├── atlas.json               ← dataset registry and per-dataset attributes (JSON)
+├── atlas.json               ← interned per-dataset schema + attribute-key namespace (JSON)
+├── pruning.idx              ← columnar min/max/count index across all datasets (see below)
+├── _global/
+│   └── data.af         ← dataset-level (global) attribute values, one entry per dataset
 ├── temperature/
-│   └── data.af         ← one ArrayFile holding temperature for every dataset
+│   └── data.af         ← one ArrayFile: temperature + its per-variable attributes, per dataset
 ├── pressure/
 │   └── data.af
 └── time/
     └── data.af
 ```
 
+`atlas.json` holds only the **schema** of the collection. Attribute **values** live in the `.af` files: per-variable attributes on the variable's own file, and dataset-level attributes in the reserved `_global` file. `pruning.idx` is a derived, columnar statistics index — everything you need to prune a query to a handful of candidate datasets without opening a single `.af` file.
+
 ---
 
 ## Durability model
 
-`atlas.json` is read **once** when the store is opened or created. Every subsequent mutation — `create_dataset`, `define_array`, `set_attribute`, `delete_array`, `delete_dataset` — only touches the in-memory `StoreMeta`; writes to `ArrayFile`s buffer inside the per-array in-memory layer. **Nothing reaches disk until `Atlas::flush()` (or `Atlas::close()`).** Dropping an `Atlas` without flushing abandons every pending in-memory write.
+`atlas.json` is read **once** when the store is opened or created. Every subsequent mutation — `create_dataset`, `define_array`, `set_attribute`, `set_array_attribute`, `delete_array`, `delete_dataset` — only touches in-memory state; array writes buffer inside the per-array in-memory layer and attribute writes buffer in a pending-attribute map. **Nothing reaches disk until `Atlas::flush()` (or `Atlas::close()`).** Dropping an `Atlas` without flushing abandons every pending in-memory write.
 
-A single `Atlas::flush()` walks every cached `ArrayFile` (writing deltas + stats) and then serialises the in-memory `StoreMeta` to `atlas.json`. This gives one durability boundary for the whole store: N datasets ⇒ one delta file per touched array name (not one per dataset) and one `atlas.json` rewrite (not N).
+A single `Atlas::flush()` drains the buffered attributes into their `.af` files, walks every cached `ArrayFile` (writing deltas + stats), fills and writes the `pruning.idx` statistics index, and then serialises the in-memory schema to `atlas.json`. This gives one durability boundary for the whole store: N datasets ⇒ one delta file per touched array name (not one per dataset) and one `atlas.json` rewrite (not N).
 
 `DatasetView` is a borrowed handle into the atlas's shared meta — it has no `flush()` of its own.
 
@@ -50,14 +55,25 @@ A single `Atlas::flush()` walks every cached `ArrayFile` (writing deltas + stats
 
 ### `atlas.json`
 
-The registry is a plain JSON file written on `Atlas::flush()` / `Atlas::close()`. It stores:
+The schema is a plain JSON file written on `Atlas::flush()` / `Atlas::close()`. It stores:
+
+- **Store version** — for format upgrades. The current version is `3`; stores written by an older atlas are rejected on open with a clear `Error::UnsupportedVersion` (re-export to upgrade). Version 3 added the per-dataset **row ordinals + tombstone mask** that back the pruning index.
+- **Interned schema pool** — each *distinct* per-dataset schema is stored once and referenced by index. A dataset's schema is its array schemas (per array: dtype, shape, chunk shape, named dimensions, codec) plus its **attribute-key namespace** (the names of its global and per-variable attribute keys). A collection of thousands of homogeneous datasets stores its schema a single time.
+- **Dataset registry** — a map of dataset name → schema-pool index.
+- **Merged schema** — a collection-wide summary listing every unique array (with its dtype widened across all datasets, its dimensions, and its per-variable attribute types) and every global attribute type. Descriptive only — reads always use each dataset's own schema — but handy for tools that want one view of "what's in here." Also exposed programmatically via `Atlas::merged_schema()`.
 
-- **Store version** — for future format upgrades.
-- **Dataset names** — the complete list of datasets in the store.
-- **Per-dataset attributes** — typed key-value pairs serialized as plain JSON values: bool, 64-bit integer, 64-bit float, UTF-8 string, or an RFC 3339 nanosecond-precision timestamp string (e.g. `"2023-11-15T07:33:20.123456789Z"`). Atlas's `Attr` enum has five variants (`Bool`/`Int64`/`Float64`/`String`/`TimestampNanoseconds`); there are no narrow integer/float types on disk.
-- **Array schemas** — per array: dtype, shape, chunk shape, named dimensions, and the codec used when the array was first written.
+When the same array name (or attribute key) appears in more than one dataset with different types, the merged type is **widened**. Widening is allowed only within numeric types (e.g. `int16` ∪ `int32` → `int32`; `int32` ∪ `float32` → `float64`) or between `string` and `timestamp` (→ `string`).
 
-Because `atlas.json` is human-readable and self-describing, you can inspect or audit the store contents with any JSON tool without needing the library.
+Any other combination — e.g. an `int32` array in one dataset and a `string` array under the same name in another — can't merge. The dataset is **still stored** under its own type (each dataset keeps the type it declared, and its data reads back normally); the merged schema simply keeps the **first-seen** type. Whether that's reported as a warning or an error is set by `StoreConfig::on_type_mismatch`:
+
+| `TypeMismatchPolicy` | Behaviour |
+| --- | --- |
+| `Warn` (default) | Logs a `tracing` warning and carries on — an ingest of many heterogeneous files won't abort because one disagrees. |
+| `Error` | `define_array` / `set_attribute` / `set_array_attribute` return `Error::TypeMismatch`. |
+
+The policy is a **per-session** choice, not an on-disk property: pass it to `Atlas::create`, or to `Atlas::open_with_config` / `Atlas::open_path_with_config` when opening an existing collection (plain `open` / `open_path` use the default). Note that these open variants honour only `on_type_mismatch` — the codec and metadata format are always detected from disk.
+
+Attribute **values** are **not** in `atlas.json` — only their key names are (as part of the schema). Values live in the `.af` files and are read via the `array-format` attribute API. Because `atlas.json` is human-readable and self-describing, you can still inspect or audit the collection's schema with any JSON tool without needing the library.
 
 ### `<array_name>/data.af`
 
@@ -67,9 +83,56 @@ Each array variable gets its own subdirectory with a single `data.af` binary fil
 - **Chunked layout** — arrays are split into chunks of a user-specified shape, so partial reads and writes touch only the relevant blocks.
 - **Configurable compression** — each block is compressed with the codec set when the store was created (default: Zstd; also LZ4 and uncompressed). The codec is persisted in `atlas.json` and restored automatically on `open` — no need to pass it again. Block target size is 8 MiB.
 - **Per-array fill value** — `define_array` accepts an optional `FillValue` (one of `Bool`/`Int`/`UInt`/`Float`/`String`). Unwritten cells read back as the fill value, and any *written* cell equal to it is counted as a null in the persisted stats (see below). Fill values are stored in the per-array footer; `atlas.json` is not extended.
+- **Attribute values** — per-variable attributes (e.g. `units`) are stored in the footer of the variable's own `data.af`, keyed by dataset. Dataset-level (global) attributes live the same way in the reserved `_global/data.af`. Atlas's `Attr` type mirrors `array-format`'s `AttributeValue` (bool, sized ints/uints, `f32`/`f64`, string, binary, and typed lists) plus a nanosecond timestamp variant stored as an RFC 3339 string.
 - **Persisted statistics** — on `Atlas::flush()`, min, max, null count, and row count are computed per array per dataset and stored alongside the data. Cells equal to the array's fill value are tallied in `null_count` and excluded from `min`/`max` (NaN fills match NaN cells by bit pattern). Statistics survive store reopening.
 - **In-memory caches** — a 256 MiB decoded block cache and a 64 MiB raw I/O cache sit in front of the object store for repeated reads.
 
+### `pruning.idx`
+
+The per-array `.af` stats above answer "what's the min/max of `temperature` in *this* dataset?" one dataset at a time. `pruning.idx` answers the **cross-dataset** question — "which of my 10 000 datasets could possibly contain a value above 25?" — as a single vectorised scan, without opening any `.af` file.
+
+It is a flattened, **columnar** index, written at the store root on every `flush()`:
+
+- **One logical row per dataset**, positional — row *i* is the dataset at ordinal *i*. Deleted datasets keep their slot (see [Deletes](#deletes-and-compaction) below) and are hidden by a liveness mask, so a row number never changes underneath a cached index.
+- **One column per array, per global attribute, and per (array, attribute) pair.** Each column holds, for every dataset: a `present` bit, a `stats_valid` bit, `min`, `max`, `row_count`, and `null_count`. Datasets that don't declare that array/attribute are **explicit gaps** (`present = false`), so columns stay aligned no matter how heterogeneous the collection is.
+- **Compacted and column-addressable.** Sparsity is the norm (~26 % density on a real 10 000-dataset collection), so on disk each column stores values only for its present rows, keyed by the bitmap. Columns are compressed individually (Zstd by default, via `StoreConfig::pruning_compression`) and located through a footer directory — reading 2 of 200 columns fetches ~2/200ths of the file via ranged reads, never the whole thing.
+- **Guarded against staleness.** The footer carries a `meta_epoch` that must match `atlas.json`; an index left behind by a crash between the two writes is rejected (`Error::CorruptIndex`) rather than silently mis-attributing rows.
+
+The read API is a two-step funnel — rule columns out from the footer alone, then fetch only the survivors:
+
+```rust
+use atlas::{ColumnKey, StatVal};
+
+// Footer only — every column's collection-wide min/max, no column data fetched.
+for (key, summary) in store.column_summaries().await? {
+    // e.g. skip a column whose global max can't satisfy the predicate.
+    let _ = summary.might_match(|_, hi| hi > &StatVal::Float(25.0));
+}
+
+// Fetch just the columns you need. The returned index is self-describing:
+// it carries the liveness mask and the row↔name mapping.
+let key = ColumnKey::array("temperature");
+let index = store.pruning_index(std::slice::from_ref(&key)).await?;
+if let Some(view) = index.view(&key) {
+    // Deleted and absent rows are already excluded.
+    for row in view.candidates(|_, hi| hi > &StatVal::Float(25.0)) {
+        println!("candidate: {}", index.dataset_name(row).unwrap());
+    }
+}
+```
+
+`min`/`max` keep the type the statistic was computed with (string min/max are lexicographic bytes; timestamps are integer nanoseconds); a column's collection-wide *declared* type is the merged dtype from `atlas.json`. **Attribute columns currently record presence only** — which datasets carry the key — not attribute values; filtering on an attribute's *value* through the index isn't wired up yet.
+
+---
+
+## Deletes and compaction
+
+`delete_dataset` and `delete_array` are **logical**. A deleted dataset is *tombstoned* — its slot stays in `atlas.json` (so every later dataset keeps its row ordinal) and a liveness bit is cleared; its entries in the shared `.af` files are marked deleted but the bytes remain. Reads, `list_datasets`, the merged schema, and the pruning index all skip tombstoned datasets, so a delete is invisible to consumers immediately, but nothing is reclaimed yet.
+
+`Atlas::compact()` is the garbage collector. It rewrites every cached `.af` file with the tombstoned regions dropped, **renumbers** the surviving datasets to close the ordinal holes, and rebuilds the pruning index against the new numbering. It is the only operation that changes a dataset's row ordinal — so a row number cached from before a compact is stale afterwards.
+
+Re-creating a deleted name reuses its slot (and pruning-index row), starting from a clean schema — the revived dataset never inherits the previous occupant's stats.
+
 ---
 
 ## Quick start
@@ -133,8 +196,9 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
 | **Array** | An N-dimensional typed array with named dimensions and an optional chunk shape. |
 | **Attribute** | A typed scalar attached to a dataset (metadata, not array data). |
 | **Array file** | One `data.af` file per variable name, shared across all datasets that define that variable. |
-| **Flush** | `Atlas::flush()` — the single durability boundary. Persists every cached array file and rewrites `atlas.json` from the in-memory `StoreMeta`. Must be called explicitly (or via `Atlas::close()` / Python's `with atlas:`). |
-| **Compact** | `Atlas::compact()` rewrites every cached `.af` file to reclaim space after deletes. |
+| **Pruning index** | `pruning.idx` — a columnar min/max/count index across all datasets. Query it with `Atlas::pruning_index(&[ColumnKey])` / `column_summaries()` to prune to candidate datasets without reading array data. |
+| **Flush** | `Atlas::flush()` — the single durability boundary. Persists every cached array file, the pruning index, and rewrites `atlas.json` from the in-memory `StoreMeta`. Must be called explicitly (or via `Atlas::close()` / Python's `with atlas:`). |
+| **Compact** | `Atlas::compact()` rewrites every cached `.af` file to reclaim space after deletes, and renumbers dataset row ordinals. |
 | **StoreConfig** | Configuration passed to `Atlas::create`. Currently holds the compression `Codec`. |
 | **Codec** | Compression codec for new array blocks: `Codec::Zstd` (default), `Codec::Lz4`, or `Codec::Uncompressed`. Persisted in `atlas.json`; `open` reads it automatically. |
 
@@ -199,6 +263,7 @@ Reading `temperature` for 1 000 datasets means opening exactly **one file**. Thi
 | Cross-dataset column scan | Slow (N file opens) | Slow (N directory opens) | Fast (1 file open) |
 | Partial reads | Yes | Yes | Yes |
 | Statistics (min/max/nulls) | No | No | Yes (persisted on flush) |
+| Cross-dataset stat pruning | No | No | Yes (`pruning.idx`, vectorised) |
 | Self-describing metadata | Yes | Yes | Yes (`atlas.json`) |
 | Language support | C/Python/Julia/… | Python/Java/… | Rust |
 | Mutable after write | Limited | Yes | Yes (chunked overwrites + compact) |
@@ -246,10 +311,12 @@ Two caches sit in front of the object store:
 
 The decoded cache means repeated reads of the same chunk cost only a hash-map lookup. Both caches are shared across all `DatasetView`s that open the same `Atlas`.
 
-### Persisted statistics
+### Persisted statistics and pruning
 
 Min, max, null count, and row count are computed and persisted on every `Atlas::flush()`. Downstream systems can read these statistics from the opened `DatasetView` without touching array data at all — useful for query planning, dashboards, or data-quality checks.
 
+For **cross-dataset** query planning, the [`pruning.idx`](#pruningidx) index turns "which datasets could match?" into a vectorised scan over compacted, column-addressed statistics. A footer-only `column_summaries()` rules whole columns out for free; `pruning_index(&[…])` then fetches just the requested columns via ranged reads. On a real 10 000-dataset collection the whole index is ~0.3 MiB compressed, and a single-column query reads it in a couple of milliseconds — so a query engine can prune a fleet of datasets to a handful of candidates before opening a single `.af` file.
+
 ### Compression
 
 The codec is chosen once at store creation via `StoreConfig` and written into `atlas.json`. `Atlas::open` reads it from there, so callers never need to repeat the codec choice. Each array also records its own codec in `atlas.json`, so a store can theoretically hold arrays written with different codecs if the schema is migrated.
@@ -264,7 +331,7 @@ Choose LZ4 when decompression throughput matters more than storage size (e.g. la
 
 ### Write path
 
-Writes are buffered in-memory across the whole atlas. Calling `Atlas::flush()` compresses and writes every modified block across every cached array file, then rewrites `atlas.json` atomically (a single `PUT`). The write path scales with the number of modified chunks, not the number of datasets, and N consecutive `add_xarray_dataset` / `create_dataset` calls amortise to a single flush.
+Writes are buffered in-memory across the whole atlas. Attribute writes buffer in a shared, **interned** pending map — identical values (a NetCDF collection is ~94 % duplicate attribute values) are stored once and shared, so the buffer stays small during a large ingest. Calling `Atlas::flush()` compresses and writes every modified block across every cached array file, writes the pruning index, then rewrites `atlas.json` atomically (a single `PUT`). The write path scales with the number of modified chunks, not the number of datasets, and N consecutive `add_xarray_dataset` / `create_dataset` calls amortise to a single flush.
 
 ### Compaction
 

{atlas_python-0.10.0 → atlas_python-0.13.0}/atlas-python/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "atlas-python"
-version = "0.10.0"
+version = "0.13.0"
 edition = "2021"
 readme = "README.md"
 

{atlas_python-0.10.0 → atlas_python-0.13.0}/atlas-python/README.md

@@ -147,8 +147,8 @@ picklable, so call `.compute()` before handing off to distributed/multiprocessin
 | Item | How it's stored |
 | --- | --- |
 | Each coord / data variable | A separate array, with `dims` mapped 1:1. |
-| Dataset attrs | Dataset attributes, plain keys. |
-| Per-variable attrs | Flattened as `{var}.{attr}` at the dataset attr level. |
+| Dataset attrs | Dataset-level (global) attributes, plain keys. |
+| Per-variable attrs | Real per-variable attributes on each variable's array. |
 | Per-variable `_FillValue` | Consumed by `define_array` as a typed fill value (source `Dataset.attrs` is not mutated). See [Missing data](#missing-data). |
 | Coord vs data_var distinction | JSON list in the internal `_pyatlas_coords` attr. |
 | Non-scalar attr values (list, ndarray) | JSON-encoded string with a `json:` prefix marker. |
@@ -188,6 +188,40 @@ for the full resolution order.
 - `bool` is available as an *attribute* type but not as an array dtype.
 - `binary`, `list[...]`, `fixed_size_list[...,N]` are reserved for a later release.
 
+## Cross-dataset pruning
+
+`ds.array_stats(name)` answers "what's the range of `temperature` in *this* dataset?" one at a
+time. To ask "which of my 10 000 datasets could contain a value above 25?" as a single
+vectorised scan, use the **pruning index** — a flattened, columnar view of the per-dataset
+statistics, one row per dataset:
+
+```python
+import numpy as np
+
+store = atlas.Atlas.open("/tmp/my_store")
+
+# Footer only — every column's collection-wide min/max, no column data fetched.
+summ = store.column_summaries()
+if summ["temperature"]["max"] < 25.0:
+    ...  # nothing can match — skip the column entirely
+
+# Fetch just the columns you need; each is a dict of numpy arrays over the row space.
+idx = store.pruning_index(arrays=["temperature"])
+col = idx["columns"]["temperature"]
+
+ok   = col["present"] & col["stats_valid"] & idx["live"]   # exclude gaps + deleted rows
+hits = np.where(ok & (col["max"] > 25.0))[0]
+candidates = [idx["datasets"][i] for i in hits]            # row -> dataset name
+```
+
+The return value is self-describing: `idx["datasets"][i]` names row `i`, `idx["live"]` is the
+delete mask (always `&` it in), and datasets that don't declare a column are explicit gaps
+(`present` is `False`, with `row_count` 0). Only the columns you ask for are fetched from
+storage, so the cost is independent of how many other columns the collection has. `min`/`max`
+keep their source type (strings come back as `bytes`, timestamps as int64 nanoseconds). See the
+[stats & scans guide](https://github.com/maris-development/atlas/blob/main/atlas-python/docs/guides/stats.md)
+for the full API, including `global_attrs=` / `array_attrs=` columns.
+
 ## Cloud / object storage
 
 With the `cloud` extra, `Atlas.open` / `Atlas.create` accept an
@@ -212,9 +246,13 @@ handle instead of a local path. The path-based local-filesystem API works withou
 | `add_xarray_dataset(ds, name, chunks=None, fill_value=None)` | Append an `xarray.Dataset` (does **not** flush). `fill_value` overrides the per-array fill (scalar or `{var: scalar}`); see [Missing data](#missing-data). |
 | `open_as_xarray_dataset(name) -> xr.Dataset` | Read a dataset back (chunked vars come back dask-backed). |
 | `open_as_many_xarray_dataset(names, concat_dim="dataset") -> xr.Dataset` | Open many datasets stacked along `concat_dim` (eager numpy). |
+| `pruning_index(arrays=None, global_attrs=None, array_attrs=None) -> dict` | Flattened statistics for **only** the requested columns; see [Cross-dataset pruning](#cross-dataset-pruning). |
+| `column_summaries() -> dict` | Every column's collection-wide min/max, read from the index footer alone (no column data fetched). |
+| `dataset_row(name) -> int \| None` | The dataset's fixed row ordinal in the pruning index. |
+| `row_slots() -> int` | Total row slots (live + tombstoned) — the pruning index's height. |
 | `flush()` | The single durability boundary — persist everything. |
 | `close()` | Alias for `flush()`; also the `with`-block exit. |
-| `compact()` | Reclaim tombstoned space across cached array files. |
+| `compact()` | Reclaim tombstoned space across cached array files, and renumber row ordinals. |
 | `__enter__` / `__exit__` | Context-manager support (`__exit__` calls `close()`). |
 
 ### `atlas.DatasetView`
@@ -229,8 +267,10 @@ handle instead of a local path. The path-based local-filesystem API works withou
 | `delete_array(name)` | Tombstone the array within this dataset. |
 | `array_meta(name) -> dict \| None` | `{"dtype", "shape", "chunk_shape", "dimension_names"}`. |
 | `array_stats(name) -> dict \| None` | `{"row_count", "null_count", "min", "max"}` — populated after `flush()`. |
-| `set_attribute(key, value, dtype=None)` | Type inferred from the Python value; pass `dtype` to override (e.g. `"int8"`, `"float32"`, `"timestamp_nanoseconds"`). On disk: bool, int64, float64, string, timestamp_nanoseconds. |
-| `get_attribute(key)` / `attributes()` | Single attribute or dict of all. |
+| `set_attribute(key, value, dtype=None)` | Set a dataset-level (global) attribute. Type inferred from the Python value; pass `dtype` to override (e.g. `"int8"`, `"float32"`, `"timestamp_nanoseconds"`). Values are stored in the `_global` `.af` file. |
+| `get_attribute(key)` / `attributes()` | Single global attribute or dict of all. |
+| `set_array_attribute(array, key, value, dtype=None)` | Set a per-variable attribute on `array` (e.g. `units`); `KeyError` if the array isn't defined. |
+| `get_array_attribute(array, key)` / `array_attributes(array)` | Single per-variable attribute or dict of all, for `array`. |
 
 `DatasetView` does **not** expose its own `flush` / `compact` — both go through the parent `Atlas`.
 

atlas_python-0.13.0/atlas-python/docs/guides/attributes.md

@@ -0,0 +1,115 @@
+# Attributes
+
+Each [`DatasetView`](../reference/dataset-view.md) carries typed attributes at
+two levels: **dataset-level** (global) attributes and **per-variable**
+attributes on individual arrays. Attribute *values* are stored in the `.af`
+files — global values in a reserved `_global` file, per-variable values on the
+variable's own file. Only the attribute *key names* are recorded in
+`atlas.json` (as part of the schema), so listing which attributes exist is
+cheap and doesn't load any array bytes.
+
+## Dataset-level (global) attributes
+
+```python
+ds = atlas.create_dataset("jan_2024")
+
+ds.set_attribute("month", 1)            # inferred int
+ds.set_attribute("station", "KNMI")     # inferred str
+ds.set_attribute("calibrated", True)    # inferred bool
+
+ds.get_attribute("month")               # -> 1
+ds.get_attribute("missing")             # -> None
+ds.attributes()                         # -> {"month": 1, "station": "KNMI", "calibrated": True}
+```
+
+## Per-variable attributes
+
+Attributes can also attach to a specific array (e.g. `units` on `temperature`):
+
+```python
+ds.define_array("temperature", dtype="float32", dims=["lat", "lon"], shape=[4, 8])
+ds.set_array_attribute("temperature", "units", "degC")
+ds.set_array_attribute("temperature", "valid_range", [-40.0, 60.0], dtype="f64")
+
+ds.get_array_attribute("temperature", "units")   # -> "degC"
+ds.array_attributes("temperature")               # -> {"units": "degC", "valid_range": [...]}
+```
+
+`set_array_attribute` raises `KeyError` if the array isn't defined in the
+dataset.
+
+## Durability
+
+Attribute writes are buffered in memory and only reach disk on
+[`atlas.flush()`](durability.md) (or leaving a `with atlas:` block), together
+with the array data. The reserved `_global/data.af` file is created lazily —
+only once a dataset actually sets a global attribute.
+
+## The on-disk type system
+
+Atlas's `Attr` type mirrors the underlying `array-format` `AttributeValue`:
+
+| On-disk type | Python type returned on read |
+|---|---|
+| `bool` | `bool` |
+| `int8` / `int16` / `int32` / `int64` | `int` |
+| `uint8` / `uint16` / `uint32` / `uint64` | `int` |
+| `float32` / `float64` | `float` |
+| `string` | `str` |
+| `binary` | `bytes` |
+| `timestamp_nanoseconds` | `int` (nanoseconds; stored as an RFC 3339 string) |
+| any of the above as a list | `list` |
+
+Type is inferred from the Python value by default — `int` → `int64`,
+`float` → `float64`, `str` → `string`, `bytes` → `binary`, `bool` → `bool`.
+
+## Overriding inferred types
+
+Pass `dtype=` to narrow or force a specific type (works on both
+`set_attribute` and `set_array_attribute`):
+
+```python
+ds.set_attribute("sensor_id", 7, dtype="int8")          # stored as int8, range-checked
+ds.set_attribute("ratio", 0.5, dtype="float32")         # stored as float32
+ds.set_attribute("observed_at",
+                 np.datetime64("2024-01-15T10:00", "ns").astype("int64").item(),
+                 dtype="timestamp_nanoseconds")
+```
+
+Unlike earlier versions, width hints now preserve the storage type: `dtype="int8"`
+stores an 8-bit integer, not a widened `int64`. A `timestamp_nanoseconds`
+attribute is stored as an RFC 3339 string and restored to a timestamp on read.
+
+## Per-variable xarray attributes
+
+When you write an `xr.Dataset` via `atlas.add_xarray_dataset(ds, name)`, each
+variable's `attrs` are stored as **real per-variable attributes** on that
+variable's array, and the dataset's own `attrs` become dataset-level
+attributes:
+
+```python
+ds = xr.Dataset(
+    data_vars={"temperature": xr.DataArray(arr, dims=["lat", "lon"],
+                                            attrs={"units": "C"})},
+    attrs={"station": "KNMI"},
+)
+atlas.add_xarray_dataset(ds, "jan_2024")
+
+view = atlas.open_dataset("jan_2024")
+view.attributes()                        # {"station": "KNMI"}
+view.array_attributes("temperature")     # {"units": "C"}
+```
+
+On read, `atlas.open_as_xarray_dataset("jan_2024")` puts each variable's attrs
+back on the right `DataArray` and the global attrs back on the Dataset.
+
+See [xarray integration](xarray.md) for the full storage convention.
+
+## JSON-encoded "complex" attributes
+
+xarray attribute values are sometimes nested dicts or other structures that
+don't map to atlas's on-disk types. The xarray bridge JSON-encodes those and
+prefixes the string with `json:`, decoding transparently on read. You generally
+don't need to think about this, but it's why some values come back as strings
+starting with `json:` if you read them through the raw attribute API. Simple
+lists of scalars are stored natively as typed-list attributes.

{atlas_python-0.10.0 → atlas_python-0.13.0}/atlas-python/docs/guides/datasets-and-arrays.md

@@ -4,18 +4,24 @@
 
 An [`Atlas`](../reference/atlas.md) is a directory-backed handle. It owns:
 
-1. An in-memory **`StoreMeta`** — every dataset, every array schema, every
-   attribute. Loaded once at `open`/`create`, mutated by every write,
-   persisted on `flush()`.
+1. An in-memory **`StoreMeta`** — the collection's schema: every dataset and
+   every array schema, plus the attribute-key namespace (which global and
+   per-variable attribute keys exist). Distinct dataset schemas are interned,
+   so thousands of identical schemas cost one copy. Loaded once at
+   `open`/`create`, mutated by every schema write, persisted to `atlas.json`
+   on `flush()`.
 2. A set of **array file caches** — one buffer per array name across the
-   whole store. Pending writes accumulate here until `flush()`.
+   whole store. Pending array writes accumulate here until `flush()`.
+3. A **pending-attribute buffer** — attribute *values* set via `set_attribute`
+   / `set_array_attribute`, drained into the `.af` files on `flush()`.
 
 A [`DatasetView`](../reference/dataset-view.md) is a typed handle into a
 single logical dataset. It exposes the per-dataset array schemas, the
-per-array statistics, and the per-dataset attribute dict. Mutations through
-the view (`define_array`, `write_array`, `set_attribute`, …) update the
-parent atlas's in-memory state; nothing reaches disk until you flush the
-*atlas*.
+per-array statistics, dataset-level (global) attributes, and per-variable
+attributes. Mutations through the view (`define_array`, `write_array`,
+`set_attribute`, `set_array_attribute`, …) update the parent atlas's in-memory
+state; attribute *values* live in the `.af` files, and nothing reaches disk
+until you flush the *atlas*.
 
 ```text
 Atlas ── StoreMeta (in-memory) ─┬─ DatasetView "jan_2024"
@@ -50,6 +56,27 @@ jan = atlas.open_dataset("jan_2024")
 atlas.delete_dataset("feb_2024")
 ```
 
+## Deleting datasets
+
+`delete_dataset` is **logical**. The dataset immediately disappears from
+`list_datasets()`, reads, the merged schema, and the pruning index — but under
+the hood it is *tombstoned*: its slot stays in `atlas.json` and its bytes stay
+in the shared array files. This keeps every other dataset's **row ordinal
+stable** (`dataset_row(name)` never shifts under you, which is what lets a
+cached [pruning index](stats.md) stay valid across a delete).
+
+```python
+atlas.delete_dataset("feb_2024")
+atlas.dataset_exists("feb_2024")   # False — invisible immediately
+atlas.row_slots()                  # still counts the dead slot until compact
+```
+
+Call [`atlas.compact()`](durability.md) to actually reclaim the space: it
+rewrites the array files without the tombstoned regions and **renumbers** the
+surviving datasets to close the ordinal holes — the only operation that changes
+a row ordinal. Re-creating a deleted name reuses its slot with a clean schema
+(the revived dataset never inherits the old one's data or stats).
+
 ## Declaring an array
 
 `define_array` records the schema (dtype, dims, shape, chunking, fill
@@ -142,3 +169,40 @@ jan.delete_array("temperature")    # tombstone within this dataset
 
 The array's bytes inside the shared physical file are tombstoned; reclaim
 the space with [`atlas.compact()`](durability.md).
+
+## Merged schema and type widening
+
+`atlas.json` also carries a collection-wide **merged schema** — every unique
+array (with its dtype, dimensions, and per-variable attribute types) and every
+global attribute type, folded across all datasets. Read it with
+`atlas.merged_schema()`:
+
+```python
+merged = store.merged_schema()
+merged["arrays"]["temperature"]["dtype"]        # widened dtype across datasets
+merged["arrays"]["temperature"]["attributes"]   # {attr_key: dtype}
+merged["global_attributes"]                     # {attr_key: dtype}
+```
+
+When the same array name or attribute key appears in multiple datasets with
+different types, the merged type is **widened** — but only within numeric types
+(`int16` ∪ `int32` → `int32`, `int32` ∪ `float32` → `float64`) or between
+`string` and `timestamp` (→ `string`).
+
+Anything else can't merge (e.g. an `int32` array in one dataset and a `string`
+array under the same name in another). The dataset is **still stored** under its
+own type and reads back normally; the merged schema just keeps the
+**first-seen** type. `on_type_mismatch` decides how that's reported:
+
+```python
+# "warn" (default): stored, logs a warning, merged keeps the first type
+store = atlas.Atlas.create("/tmp/store")
+store = atlas.Atlas.open("/tmp/store")
+
+# "error": the mismatching define_array / set_attribute raises ValueError
+store = atlas.Atlas.create("/tmp/store", on_type_mismatch="error")
+store = atlas.Atlas.open("/tmp/store", on_type_mismatch="error")
+```
+
+It's a per-session choice — it isn't stored in `atlas.json`, so pass it each
+time you open the collection.