atlas-python 0.13.1__tar.gz → 0.14.0__tar.gz

{atlas_python-0.13.1 → atlas_python-0.14.0}/Cargo.lock

@@ -71,7 +71,7 @@ dependencies = [
 
 [[package]]
 name = "atlas-python"
-version = "0.13.1"
+version = "0.14.0"
 dependencies = [
  "atlas-rust",
  "ndarray",
@@ -86,7 +86,7 @@ dependencies = [
 
 [[package]]
 name = "atlas-rust"
-version = "0.13.1"
+version = "0.14.0"
 dependencies = [
  "array-format",
  "chrono",

{atlas_python-0.13.1 → atlas_python-0.14.0}/Cargo.toml

@@ -5,7 +5,7 @@ members = ["atlas-python"]
 # Registry name on crates.io (`atlas` is taken). The library is still
 # imported as `atlas` — see `[lib]` below.
 name = "atlas-rust"
-version = "0.13.1"
+version = "0.14.0"
 edition = "2024"
 description = "Directory-based store for thousands of N-dimensional datasets local or remote using object storage."
 license = "Apache-2.0"

{atlas_python-0.13.1 → atlas_python-0.14.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: atlas-python
-Version: 0.13.1
+Version: 0.14.0
 Classifier: Development Status :: 3 - Alpha
 Classifier: Intended Audience :: Science/Research
 Classifier: Programming Language :: Python :: 3

{atlas_python-0.13.1 → atlas_python-0.14.0}/README.md

@@ -14,6 +14,8 @@ A directory-based store for thousands of named datasets, each holding N-dimensio
 > **Looking for Python?** The Python bindings live in [`atlas-python/`](atlas-python/) — `pip install atlas-python`, then `import atlas`. They add a NumPy-native API and first-class [xarray](https://docs.xarray.dev) integration. See the [atlas-python README](atlas-python/README.md) for usage.
 >
 > The rest of this document covers **how the format works and the Rust crate**.
+>
+> **Architecture deep-dive?** [`docs/`](docs/) has focused walkthroughs with diagrams — [architecture](docs/architecture.md), [storage layout](docs/storage-layout.md), [data model](docs/data-model.md), [metadata](docs/metadata.md), [write path & durability](docs/write-path.md), [the pruning index](docs/pruning-index.md), and [Python/xarray](docs/python-xarray.md).
 
 ---
 
@@ -26,7 +28,6 @@ Each dataset is a named group of N-dimensional arrays with typed attributes —
 ```text
 my_store/
 ├── atlas.json               ← interned per-dataset schema + attribute-key namespace (JSON)
-├── pruning.idx              ← columnar min/max/count index across all datasets (see below)
 ├── _global/
 │   └── data.af         ← dataset-level (global) attribute values, one entry per dataset
 ├── temperature/
@@ -37,7 +38,7 @@ my_store/
     └── data.af
 ```
 
-`atlas.json` holds only the **schema** of the collection. Attribute **values** live in the `.af` files: per-variable attributes on the variable's own file, and dataset-level attributes in the reserved `_global` file. `pruning.idx` is a derived, columnar statistics index — everything you need to prune a query to a handful of candidate datasets without opening a single `.af` file.
+`atlas.json` holds only the **schema** of the collection. Attribute **values** live in the `.af` files: per-variable attributes on the variable's own file, and dataset-level attributes in the reserved `_global` file. There is **no separate index file** — the cross-dataset pruning index is built on demand from the `.af` statistics (see below), so it can never go stale and adds nothing to a write.
 
 ---
 
@@ -45,7 +46,7 @@ my_store/
 
 `atlas.json` is read **once** when the store is opened or created. Every subsequent mutation — `create_dataset`, `define_array`, `set_attribute`, `set_array_attribute`, `delete_array`, `delete_dataset` — only touches in-memory state; array writes buffer inside the per-array in-memory layer and attribute writes buffer in a pending-attribute map. **Nothing reaches disk until `Atlas::flush()` (or `Atlas::close()`).** Dropping an `Atlas` without flushing abandons every pending in-memory write.
 
-A single `Atlas::flush()` drains the buffered attributes into their `.af` files, walks every cached `ArrayFile` (writing deltas + stats), fills and writes the `pruning.idx` statistics index, and then serialises the in-memory schema to `atlas.json`. This gives one durability boundary for the whole store: N datasets ⇒ one delta file per touched array name (not one per dataset) and one `atlas.json` rewrite (not N).
+A single `Atlas::flush()` drains the buffered attributes into their `.af` files, walks every cached `ArrayFile` (writing deltas + stats), and then serialises the in-memory schema to `atlas.json`. This gives one durability boundary for the whole store: N datasets ⇒ one delta file per touched array name (not one per dataset) and one `atlas.json` rewrite (not N). There is no index to write — the pruning index is derived from the `.af` stats on demand.
 
 `DatasetView` is a borrowed handle into the atlas's shared meta — it has no `flush()` of its own.
 
@@ -87,28 +88,20 @@ Each array variable gets its own subdirectory with a single `data.af` binary fil
 - **Persisted statistics** — on `Atlas::flush()`, min, max, null count, and row count are computed per array per dataset and stored alongside the data. Cells equal to the array's fill value are tallied in `null_count` and excluded from `min`/`max` (NaN fills match NaN cells by bit pattern). Statistics survive store reopening.
 - **In-memory caches** — a 256 MiB decoded block cache and a 64 MiB raw I/O cache sit in front of the object store for repeated reads.
 
-### `pruning.idx`
-
-The per-array `.af` stats above answer "what's the min/max of `temperature` in *this* dataset?" one dataset at a time. `pruning.idx` answers the **cross-dataset** question — "which of my 10 000 datasets could possibly contain a value above 25?" — as a single vectorised scan, without opening any `.af` file.
+### Pruning index (built on demand — no index file)
 
-It is a flattened, **columnar** index, written at the store root on every `flush()`:
+The per-array `.af` stats above answer "what's the min/max of `temperature` in *this* dataset?" one dataset at a time. The **pruning index** answers the **cross-dataset** question — "which of my 1 000 000 datasets could possibly contain a value above 25?" — as a single vectorised scan.
 
-- **One logical row per dataset**, positional — row *i* is the dataset at ordinal *i*. Deleted datasets keep their slot (see [Deletes](#deletes-and-compaction) below) and are hidden by a liveness mask, so a row number never changes underneath a cached index.
-- **One column per array, per global attribute, and per (array, attribute) pair.** Each column holds, for every dataset: a `present` bit, a `stats_valid` bit, `min`, `max`, `row_count`, and `null_count`. Datasets that don't declare that array/attribute are **explicit gaps** (`present = false`), so columns stay aligned no matter how heterogeneous the collection is.
-- **Compacted and column-addressable.** Sparsity is the norm (~26 % density on a real 10 000-dataset collection), so on disk each column stores values only for its present rows, keyed by the bitmap. Columns are compressed individually (Zstd by default, via `StoreConfig::pruning_compression`) and located through a footer directory — reading 2 of 200 columns fetches ~2/200ths of the file via ranged reads, never the whole thing.
-- **Guarded against staleness.** The footer carries a `meta_epoch` that must match `atlas.json`; an index left behind by a crash between the two writes is rejected (`Error::CorruptIndex`) rather than silently mis-attributing rows.
+There is **no `pruning.idx` file**. The `.af` stats *are* the source of truth, so the index is a denormalized pivot of data that already exists — Atlas rebuilds it **on demand** for the requested columns instead of persisting (and maintaining) a copy:
 
-The read API is a two-step funnel — rule columns out from the footer alone, then fetch only the survivors:
+- `pruning_index(cols)` reads each requested array's `StatsFile` once and scatters its per-dataset entries into a flat, length-N column by dataset ordinal. Cost is proportional to the *requested* columns, not the (possibly 50 000+) total — so wide collections stay cheap.
+- **One logical row per dataset**, positional — row *i* is the dataset at ordinal *i*. Deleted datasets keep their slot and are hidden by a liveness mask.
+- **Columns**: one per array, per global attribute, and per (array, attribute) pair. Each holds, per dataset, a `present` bit, a `stats_valid` bit, `min`, `max`, `row_count`, `null_count`. Attribute columns now carry each dataset's **value** as a point range `[v, v]`, so you can range-prune on attributes too.
+- Always consistent (no epoch, nothing to go stale), zero extra write memory, and nothing to write at `flush()`. Columns are built in parallel and the `name → ordinal` pivot map is cached across calls.
 
 ```rust
 use atlas::{ColumnKey, StatVal};
 
-// Footer only — every column's collection-wide min/max, no column data fetched.
-for (key, summary) in store.column_summaries().await? {
-    // e.g. skip a column whose global max can't satisfy the predicate.
-    let _ = summary.might_match(|_, hi| hi > &StatVal::Float(25.0));
-}
-
 // Fetch just the columns you need. The returned index is self-describing:
 // it carries the liveness mask and the row↔name mapping.
 let key = ColumnKey::array("temperature");
@@ -119,9 +112,12 @@ if let Some(view) = index.view(&key) {
         println!("candidate: {}", index.dataset_name(row).unwrap());
     }
 }
+
+// column_summaries() folds each column's collection-wide min/max/present_count
+// (touches every column — a decide-what-to-read helper, not a hot path).
 ```
 
-`min`/`max` keep the type the statistic was computed with (string min/max are lexicographic bytes; timestamps are integer nanoseconds); a column's collection-wide *declared* type is the merged dtype from `atlas.json`. **Attribute columns currently record presence only** — which datasets carry the key — not attribute values; filtering on an attribute's *value* through the index isn't wired up yet.
+`min`/`max` keep the type the statistic was computed with (string min/max are lexicographic bytes; timestamps are integer nanoseconds). **See [`docs/pruning-index.md`](docs/pruning-index.md) for the full design, diagrams, and benchmarks.**
 
 ---
 
@@ -196,7 +192,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
 | **Array** | An N-dimensional typed array with named dimensions and an optional chunk shape. |
 | **Attribute** | A typed scalar attached to a dataset (metadata, not array data). |
 | **Array file** | One `data.af` file per variable name, shared across all datasets that define that variable. |
-| **Pruning index** | `pruning.idx` — a columnar min/max/count index across all datasets. Query it with `Atlas::pruning_index(&[ColumnKey])` / `column_summaries()` to prune to candidate datasets without reading array data. |
+| **Pruning index** | A columnar min/max/count table across all datasets, **built on demand** from the `.af` stats (no index file). Query it with `Atlas::pruning_index(&[ColumnKey])` / `column_summaries()` to prune to candidate datasets without reading array data. |
 | **Flush** | `Atlas::flush()` — the single durability boundary. Persists every cached array file, the pruning index, and rewrites `atlas.json` from the in-memory `StoreMeta`. Must be called explicitly (or via `Atlas::close()` / Python's `with atlas:`). |
 | **Compact** | `Atlas::compact()` rewrites every cached `.af` file to reclaim space after deletes, and renumbers dataset row ordinals. |
 | **StoreConfig** | Configuration passed to `Atlas::create`. Currently holds the compression `Codec`. |
@@ -263,7 +259,7 @@ Reading `temperature` for 1 000 datasets means opening exactly **one file**. Thi
 | Cross-dataset column scan | Slow (N file opens) | Slow (N directory opens) | Fast (1 file open) |
 | Partial reads | Yes | Yes | Yes |
 | Statistics (min/max/nulls) | No | No | Yes (persisted on flush) |
-| Cross-dataset stat pruning | No | No | Yes (`pruning.idx`, vectorised) |
+| Cross-dataset stat pruning | No | No | Yes (on-demand columnar index, vectorised) |
 | Self-describing metadata | Yes | Yes | Yes (`atlas.json`) |
 | Language support | C/Python/Julia/… | Python/Java/… | Rust |
 | Mutable after write | Limited | Yes | Yes (chunked overwrites + compact) |
@@ -315,7 +311,7 @@ The decoded cache means repeated reads of the same chunk cost only a hash-map lo
 
 Min, max, null count, and row count are computed and persisted on every `Atlas::flush()`. Downstream systems can read these statistics from the opened `DatasetView` without touching array data at all — useful for query planning, dashboards, or data-quality checks.
 
-For **cross-dataset** query planning, the [`pruning.idx`](#pruningidx) index turns "which datasets could match?" into a vectorised scan over compacted, column-addressed statistics. A footer-only `column_summaries()` rules whole columns out for free; `pruning_index(&[…])` then fetches just the requested columns via ranged reads. On a real 10 000-dataset collection the whole index is ~0.3 MiB compressed, and a single-column query reads it in a couple of milliseconds — so a query engine can prune a fleet of datasets to a handful of candidates before opening a single `.af` file.
+For **cross-dataset** query planning, the [pruning index](#pruning-index-built-on-demand--no-index-file) turns "which datasets could match?" into a vectorised scan over the array statistics. It's built on demand: `pruning_index(&[…])` reads just the requested columns' `.af` stats and pivots them into flat per-dataset vectors (columns built in parallel, the ordinal map cached); `column_summaries()` folds every column's collection-wide range to rule whole columns out. So a query engine can prune a fleet of datasets to a handful of candidates before opening a single `.af` file — with nothing to persist or keep in sync. See [`docs/pruning-index.md`](docs/pruning-index.md).
 
 ### Compression
 

{atlas_python-0.13.1 → atlas_python-0.14.0}/atlas-python/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "atlas-python"
-version = "0.13.1"
+version = "0.14.0"
 edition = "2021"
 readme = "README.md"
 

atlas_python-0.14.0/docs/README.md

@@ -0,0 +1,67 @@
+# Atlas — Architecture Documentation
+
+Atlas (**A**ggregated **T**ensor **L**arge **A**rray **S**tore) is a
+directory-based store for **thousands to millions of named datasets**, each a
+collection of N-dimensional arrays with attributes, backed by object storage
+(local filesystem, S3, …).
+
+This folder explains how the pieces fit together. Read the docs in order for a
+full picture, or jump to a topic.
+
+| # | Doc | What it covers |
+|---|-----|----------------|
+| 1 | [architecture.md](architecture.md) | The crates, the layers, and how a call flows top to bottom |
+| 2 | [storage-layout.md](storage-layout.md) | What's on disk: `atlas.json`, `<array>/data.af`, `_global` |
+| 3 | [data-model.md](data-model.md) | Datasets, arrays, array-name sharing, ordinals, attributes, dtypes |
+| 4 | [metadata.md](metadata.md) | `atlas.json` / `StoreMeta`: schema interning, the type index, tombstones |
+| 5 | [write-path.md](write-path.md) | create → write → flush → compact, buffering, and the durability boundary |
+| 6 | [pruning-index.md](pruning-index.md) | The flat statistics table, **built on demand** from the array files |
+| 7 | [python-xarray.md](python-xarray.md) | The `atlas-python` bindings and the xarray integration |
+
+## The 10,000-foot view
+
+```
+        ┌───────────────────────────────────────────────────────────────┐
+        │  Consumers:  Rust API   •   Python (xarray)   •   CLI/notebooks │
+        └───────────────────────────────┬───────────────────────────────┘
+                                        │
+        ┌───────────────────────────────▼───────────────────────────────┐
+        │  atlas-rust  — the store                                        │
+        │                                                                 │
+        │   Atlas         one store: create/open, flush, compact,         │
+        │                 pruning_index (query stats across datasets)     │
+        │   DatasetView   one dataset: define/write/read arrays + attrs   │
+        │   StoreMeta     the schema + ordinals (atlas.json)              │
+        │   Pruning       a flat stats table, assembled on demand         │
+        └───────────────────────────────┬───────────────────────────────┘
+                                        │  typed columnar arrays + stats
+        ┌───────────────────────────────▼───────────────────────────────┐
+        │  array-format — one physical file per array name                │
+        │  chunked · compressed · delta layers · per-dataset statistics   │
+        └───────────────────────────────┬───────────────────────────────┘
+                                        │
+        ┌───────────────────────────────▼───────────────────────────────┐
+        │  object_store —  local FS  •  S3  •  GCS  •  Azure  •  memory    │
+        └───────────────────────────────────────────────────────────────┘
+```
+
+## The one idea that makes Atlas different
+
+**Datasets that share an array name are co-located in one physical file, keyed
+by dataset name.** A thousand sensors that each record `temperature` don't make
+a thousand files — they make **one** `temperature/data.af`, with a thousand
+entries inside. This is what lets Atlas scan a variable *across* datasets
+cheaply, and it's the backbone of the [pruning index](pruning-index.md).
+
+```
+   1,000 datasets ── each declares "temperature" and "salinity" ──►
+
+   store/
+   ├── temperature/data.af   ← 1,000 entries (one per dataset)
+   └── salinity/data.af      ← 1,000 entries (one per dataset)
+
+   NOT: store/ds0/…, store/ds1/…, …  (1,000 directories)
+```
+
+See [data-model.md](data-model.md) for why, and [pruning-index.md](pruning-index.md)
+for what it buys you.

atlas_python-0.14.0/docs/architecture.md

@@ -0,0 +1,104 @@
+# 1. Architecture
+
+Atlas is three layers stacked on object storage. Each layer has one job.
+
+```
+   ┌──────────────────────────────────────────────────────────────────────┐
+   │  atlas-python  (PyO3 bindings + xarray integration)                    │
+   │  • Atlas / DatasetView wrappers        • ds.atlas.write(store)          │
+   │  • numpy ⇄ atlas dtype mapping         • add_xarray_dataset(ds, name)   │
+   │  Releases the GIL, moves array data zero-copy into the core.           │
+   └───────────────────────────────────┬──────────────────────────────────┘
+                                       │
+   ┌───────────────────────────────────▼──────────────────────────────────┐
+   │  atlas-rust  (the core — this crate)                                   │
+   │                                                                        │
+   │   store/     Atlas: create/open, flush, compact, delete, queries       │
+   │   dataset/   DatasetView: define/write/read arrays, get/set attrs      │
+   │   meta/      StoreMeta: the schema, ordinals, type index (atlas.json)  │
+   │   pruning/   the flat statistics table, assembled on demand            │
+   │   schema/    DType, Attr, ArraySchema — the type system                │
+   │   array.rs   AtlasArray: the lazy per-array-file handle + cache        │
+   │   config.rs  Codec, MetaFormat, StoreConfig, TypeMismatchPolicy        │
+   │   error.rs   Error / Result                                            │
+   └───────────────────────────────────┬──────────────────────────────────┘
+                                       │
+   ┌───────────────────────────────────▼──────────────────────────────────┐
+   │  array-format  (sibling crate — the columnar file format)              │
+   │   ArrayFile: one physical file holding many datasets' entries for one   │
+   │   array name. Chunked, per-chunk compressed, append-friendly via delta  │
+   │   layers, with a per-dataset StatsFile and per-entry attributes.        │
+   └───────────────────────────────────┬──────────────────────────────────┘
+                                       │
+   ┌───────────────────────────────────▼──────────────────────────────────┐
+   │  object_store  (local FS, S3, GCS, Azure, in-memory)                   │
+   └──────────────────────────────────────────────────────────────────────┘
+```
+
+## Who owns what
+
+| Layer | Owns | Does **not** touch |
+|-------|------|--------------------|
+| `atlas-python` | ergonomics, numpy/xarray glue | storage, stats |
+| `atlas-rust` | the *collection* — which datasets exist, their schema, cross-dataset queries | how one array's bytes are laid out |
+| `array-format` | one physical array file — chunking, compression, stats, attributes | anything about *other* arrays or the collection |
+| `object_store` | bytes in and out of a backend | structure |
+
+The boundary that matters: **atlas-rust knows about the collection; array-format
+knows about a single file.** Atlas never reaches into an array's byte layout;
+array-format never knows two arrays belong to the same dataset.
+
+## The central types
+
+```
+   Atlas ──────────────► one store (a directory / prefix)
+     │  create_dataset(name) -> DatasetView
+     │  open_dataset(name)   -> DatasetView
+     │  flush() / compact()
+     │  pruning_index(cols)  -> PruningIndex     (cross-dataset stats)
+     │  read_array_across(…) -> stacked arrays   (one var, many datasets)
+     │
+     ├── StoreMeta  (in memory, persisted as atlas.json)
+     │     datasets: name → schema  (insertion order = ordinal, tombstoned)
+     │     schema_pool, type index, codec, meta_epoch
+     │
+     ├── ArrayCache  (name → AtlasArray, lazy)
+     │     one handle per physical array file, opened on first use
+     │
+     └── ordinal_map cache   (name → ordinal, for the pruning pivot)
+
+   DatasetView ────────► one dataset inside a store
+     │  define_array / write_array / read_array
+     │  set_attribute / get_attribute  (dataset-global)
+     │  set_array_attribute / …        (per-array)
+     └── all mutations buffer in memory; nothing hits disk until Atlas::flush()
+```
+
+## How a call flows
+
+**Write** (`view.write_array("temp", …)`):
+
+```
+   DatasetView.write_array
+     └─ ArrayCache.get_or_insert("temp")     → AtlasArray (lazy handle)
+         └─ AtlasArray.get().await           → opens/creates temp/data.af
+             └─ ArrayFile.write_array        → buffers into the pending delta
+                                               (spills 4 MiB blocks to a tempfile;
+                                                nothing durable yet)
+```
+
+**Durability boundary** (`atlas.flush()`): commits every touched `ArrayFile`'s
+pending delta to a sidecar layer, computes each array's `StatsFile`, and writes
+`atlas.json`. See [write-path.md](write-path.md).
+
+**Cross-dataset query** (`atlas.pruning_index([temp])`): reads `temp/data.af`'s
+`StatsFile`, pivots the per-dataset entries into a flat length-N column by
+ordinal. **No separate index is stored** — see [pruning-index.md](pruning-index.md).
+
+## Runtime & concurrency
+
+- Everything is `async` on Tokio. `Atlas` is `Send + Sync`.
+- Each physical array file is guarded by its own `tokio::sync::RwLock`, so
+  different arrays are independent — the basis for the parallel pruning reads.
+- Store metadata is a `parking_lot::Mutex` held only for short, non-`await`
+  critical sections.

atlas_python-0.14.0/docs/data-model.md

@@ -0,0 +1,108 @@
+# 3. Data Model
+
+## Datasets and arrays
+
+A **dataset** is a named collection of **arrays** (N-dimensional, typed) plus
+attributes. Think of one NetCDF file → one dataset; its variables → arrays.
+
+```
+   dataset "GL_PR_BO_JLKU"
+   ├── array  TIME       datetime64[ns]   dims: [time]
+   ├── array  LATITUDE   float32          dims: [time]
+   ├── array  TEMP       float64          dims: [time, depth]
+   ├── attr   platform_code = "BO"        (dataset-global)
+   └── TEMP.units = "degree_Celsius"      (per-array attribute)
+```
+
+## Array-name sharing (the core idea)
+
+Two datasets that declare an array with the **same name** are stored in the
+**same physical file**, keyed by dataset name. The store holds *N* physical
+arrays where *N* = number of distinct array names, regardless of dataset count.
+
+```
+   ds "jan"  ── TEMP, PSAL ──┐
+   ds "feb"  ── TEMP, PSAL ──┤──►  TEMP/data.af   { jan, feb, mar }
+   ds "mar"  ── TEMP, PSAL ──┘     PSAL/data.af   { jan, feb, mar }
+```
+
+This is what makes "read one variable across all datasets" and the pruning
+index cheap: the variable's data (and stats) are already gathered in one file.
+
+**Corollary — normalize names.** If `jan` calls it `TEMP` and `feb` calls it
+`TEMP_SBE`, they don't co-locate — you get a file per variant. For heterogeneous
+ingest (per-file/station-specific names), map names to a shared schema *before*
+writing. A store with a handful of shared array names scales to millions of
+datasets; a store with per-dataset-unique names does not.
+
+## Ordinals — a dataset's fixed row number
+
+Every dataset has an **ordinal**: its position in the insertion-ordered dataset
+list. The ordinal is a dataset's row in every cross-dataset view (the pruning
+index, `read_array_across`).
+
+```
+   ordinal   0        1        2        3
+   dataset   ds0      ds1      ds2      ds3
+```
+
+Ordinals are **stable**: deleting a dataset *tombstones* its slot (a liveness
+bit flips to false) rather than removing it, so no later dataset's ordinal
+shifts. The one operation that renumbers is [`compact`](write-path.md), which
+drops tombstones and closes the gaps.
+
+```
+   delete ds1:
+   ordinal   0        1(dead)  2        3
+   dataset   ds0      —        ds2      ds3      ← ds2, ds3 keep ordinals 2, 3
+
+   compact:
+   ordinal   0        1        2
+   dataset   ds0      ds2      ds3               ← renumbered, gaps closed
+```
+
+## Attributes
+
+Two scopes, both key→value:
+
+- **Dataset-global** (`set_attribute`) — e.g. `platform_code`, `cruise_id`.
+  Stored on the dataset's entry in `_global/data.af`.
+- **Per-array** (`set_array_attribute`) — e.g. `units`, `valid_range`.
+  Stored on the array's entry in `<array>/data.af`.
+
+Values are scalars or lists (`Attr`): bool, ints, floats, string, binary, and
+list variants. The pruning index can range-prune on **scalar** attribute values
+(a per-dataset value becomes a point range `[v, v]`); list-valued attributes are
+tracked as present without a range.
+
+## The type system
+
+Atlas array dtypes (`DType`):
+
+```
+   UInt8/16/32/64   Int8/16/32/64   Float32/64
+   String           TimestampNs (nanoseconds since epoch)
+   List { child }
+```
+
+The numpy/xarray mapping (in `atlas-python`) covers these plus
+`datetime64[ns]` → `TimestampNs` and `timedelta64` → `Int64` (+ a marker for
+round-trip). Object/bytes/unicode → `String`.
+
+### Type widening across datasets
+
+Datasets sharing an array name may disagree on exact dtype; Atlas computes a
+**merged type** for the collection:
+
+```
+   widen(Int32, Int64)      = Int64
+   widen(Int32, Float32)    = Float64   (int + float promotes to f64)
+   widen(UInt16, UInt32)    = UInt32
+   widen(String, TimestampNs) = String  (timestamps render as strings)
+   widen(List<Int8>, List<Int16>) = List<Int16>   (element-wise)
+```
+
+Incompatible combinations (e.g. a numeric vs. a string array under the same
+name) are **rejected at define time** under `TypeMismatchPolicy::Error`, or
+warned under `::Warn`. The merged schema is what `merged_schema()` and the
+pruning index's collection-wide summaries report.

atlas_python-0.14.0/docs/metadata.md

@@ -0,0 +1,90 @@
+# 4. Metadata (`atlas.json` / `StoreMeta`)
+
+`StoreMeta` is the in-memory model of the collection's *structure*; `atlas.json`
+is its on-disk form. It holds no bulk data and no attribute values — only
+schema, ordinals, and the type index.
+
+```
+   StoreMeta
+   ├── version            format version
+   ├── codec              default Codec for new arrays (Zstd / Lz4 / none)
+   ├── meta_epoch         monotonic counter, bumped on every save
+   │
+   ├── datasets:  IndexMap<name, Arc<DatasetSchema>>     ← insertion order = ordinal
+   │              (includes tombstones — never removed except by compact)
+   ├── live:      Vec<bool>                              ← liveness bit per ordinal
+   │
+   ├── schema_pool: HashMap<hash, Vec<Arc<DatasetSchema>>>   ← interning pool
+   └── index:     TypeIndex                              ← derived: name → merged type
+```
+
+## Schema interning
+
+Most collections are **homogeneous** — every dataset from one NetCDF folder has
+the same variables with the same dtypes. Storing that schema once per dataset
+would be wasteful, so identical `DatasetSchema`s are **interned**: deduplicated
+by content hash into `schema_pool`, and every dataset holds an `Arc` to the
+shared instance.
+
+```
+   1,000 identical datasets  ──►  schema_pool: { hash → [ one DatasetSchema ] }
+                                  datasets:    1,000 × Arc(same schema)
+```
+
+Editing a dataset's schema after interning uses `Arc::make_mut`, copying it out
+of the pool first (copy-on-write), so shared instances are never mutated.
+
+A `DatasetSchema` is:
+
+```
+   DatasetSchema
+   ├── arrays:       IndexMap<name, ArraySchema>          (dtype, shape, chunks, dims, codec, fill)
+   ├── global_attrs: IndexMap<key, DTypeS>                (attribute-key namespace)
+   └── array_attrs:  IndexMap<array, IndexMap<key, DTypeS>>
+```
+
+## Tombstones and ordinals
+
+Deletes are logical. `unrecord_dataset` clears the liveness bit; the entry stays
+in `datasets` so ordinals below and above it don't shift. Enumeration goes
+through the `live_*` methods so dead entries never leak:
+
+```
+   live_datasets()  → (ordinal, name, schema) for live rows only
+   row_slots()      → total slots incl. tombstones  = the pruning row count
+   live_count()     → live datasets only
+   live_mask()      → Vec<bool>, applied by pruning views to hide deleted rows
+```
+
+`compact` is the only thing that physically drops tombstones and renumbers (see
+[write-path.md](write-path.md)).
+
+## The type index
+
+`TypeIndex` is **derived** from `datasets` (never set directly). It maps each
+array name / attribute key to its collection-wide **merged type**, so that:
+
+- defining an array can be checked against what other datasets already declared
+  (widen or reject — see [data-model.md](data-model.md));
+- `merged_schema()` can report the whole collection's unified shape without
+  re-scanning every dataset.
+
+It is rebuilt whenever the dataset set changes.
+
+## `meta_epoch`
+
+A monotonic counter bumped on every `save` (flush/compact). Historically it also
+tagged a persisted pruning index so a stale index could be detected. **The
+pruning index is now built on demand** (there's nothing persisted to go stale),
+so `meta_epoch` is purely a save/version counter today.
+
+## On-disk format (`MetaFormat`)
+
+| Format | File | Use |
+|--------|------|-----|
+| `Json` (default) | `atlas.json` | human-readable, diff-able |
+| `MsgPack` | `atlas.msgpack` | compact binary |
+| `MsgPack` + compression | `atlas.msgpack.zst` / `.lz4` | large collections |
+
+On `open`, the format and codec are **auto-detected** from the files present —
+no config needed to reopen.