atlas-python 0.14.0__tar.gz → 0.16.0__tar.gz

atlas_python-0.16.0/CONTRIBUTING.md

@@ -0,0 +1,275 @@
+# Contributing to ATLAS
+
+What the project is, how the pieces fit, and how to get a development
+environment working.
+
+For end-user docs see [README.md](README.md) (Rust) and
+[atlas-python/README.md](atlas-python/README.md) (Python).
+
+---
+
+## What this is
+
+**ATLAS** (Aggregated Tensor Large Array Store) keeps thousands of named
+datasets in one immutable file. A dataset is a set of named N-dimensional
+arrays with attributes — the shape a NetCDF file or an `xarray.Dataset` has.
+
+The design goal is that *knowing what a collection holds* costs one request,
+however many datasets it holds.
+
+The repository is a Cargo workspace with two crates:
+
+| Crate | Purpose |
+|---|---|
+| `atlas-rust` (workspace root, `[lib] name = "atlas"`) | The format, the writer, the reader |
+| `atlas-python` ([atlas-python/](atlas-python/)) | PyO3 bindings, the five operations, the `atlas` command |
+
+---
+
+## Architecture
+
+### On-disk layout
+
+```text
+my_collection/
+├── data.atlas      ATLS │ segment │ segment │ … │ footer │ trailer
+└── deleted.mask    optional: ordinals of deleted datasets
+```
+
+One write-once file. Each dataset is a contiguous segment — itself a complete
+[`array-format`](https://github.com/robinskil/array-format) file — and the
+footer records every dataset's name, segment byte range, schema, and attributes.
+
+Byte-level detail is in [docs/format.md](docs/format.md). The rest of
+[docs/](docs/) covers the layers, the data model, and the two paths.
+
+**The format is Rust only.** `atlas-python` holds no format knowledge — grep it
+for `ATLS` and you get nothing. One implementation of the bytes, one place for a
+bug to live. Keep it that way.
+
+### Rust crate
+
+| Path | Role |
+|---|---|
+| [src/lib.rs](src/lib.rs) | Public re-exports, `validate_name`, thread-safety asserts |
+| [src/format/mod.rs](src/format/mod.rs) | Container framing: magic, header, trailer |
+| [src/format/footer.rs](src/format/footer.rs) | `CollectionFooter`, `DatasetEntry`, `AttrS`, the interner |
+| [src/format/mask.rs](src/format/mask.rs) | The deletion mask codec |
+| [src/format/segment_store.rs](src/format/segment_store.rs) | `ObjectStore` adapter over one byte range |
+| [src/writer/mod.rs](src/writer/mod.rs) | `AtlasWriter`, `DatasetWriter` |
+| [src/reader/mod.rs](src/reader/mod.rs) | `Atlas`, `DatasetView` |
+| [src/schema/](src/schema/) | `ArraySchema`, `DatasetSchema`, `Attr`, dtype serde |
+| [src/config.rs](src/config.rs) | `Codec`, `WriterConfig` |
+| [src/error.rs](src/error.rs) | `Error` / `Result` |
+
+The API is async (tokio). Reads take no locks — the data is immutable. Writes
+share one `tokio::sync::Mutex` over the output stream, held only for a dataset's
+append.
+
+### Python bindings
+
+Mixed Python/Rust maturin layout:
+
+```text
+atlas-python/
+├── Cargo.toml               cdylib named `_atlas`
+├── pyproject.toml           maturin build backend; declares the `atlas` script
+├── python/atlas/
+│   ├── __init__.py          the five operations. `__all__` is the surface
+│   ├── __init__.pyi         type stubs (PEP 561) — the authoritative contract
+│   ├── py.typed             marker
+│   ├── _ops.py              the operations themselves
+│   ├── _cli.py              argument parsing and output formatting
+│   ├── _source.py           path or URL → something the bindings accept
+│   └── xarray.py            the xarray → atlas mapping. Internal
+├── src/
+│   ├── lib.rs               #[pymodule] wiring
+│   ├── runtime.rs           shared OnceLock<tokio::Runtime>
+│   ├── error.rs             atlas::Error → PyErr
+│   ├── source.rs            AtlasSource (path | obstore handle), codec parsing
+│   ├── dtype.rs             dtype string ⇄ DType
+│   ├── attr.rs              Python value ⇄ Attr
+│   ├── writer.rs            PyAtlasWriter, PyDatasetWriter — internal
+│   └── reader.rs            PyAtlas, PyDatasetView — internal
+├── tests/                   pytest, plus make_fixture.py
+└── examples/                runnable scripts
+```
+
+Key points:
+
+- **Five operations, and that is the whole surface.** `create`, `remove`,
+  `list_datasets`, `describe`, `info`, as a library and as `atlas create|rm|ls|
+  show|info`. `__all__` in `__init__.py` is the contract; a test asserts it.
+- **`_atlas` is private.** It still exposes writer and reader classes because
+  the operations need them, but nothing outside `_ops.py` touches them, and
+  they are not exported.
+- **Python writes; Rust reads array data.** There is no `read_array` on the
+  Python side. See [docs/read-path.md](docs/read-path.md) for the reasoning.
+- **NetCDF is the only ingest route.** `create` scans a directory, sorts it, and
+  writes one dataset per file — sorting is what makes ordinals reproducible.
+- **Sync Python API over a tokio runtime.** Each blocking call uses
+  `py.detach(|| runtime().block_on(...))` so other Python threads keep running.
+- **numpy zero-copy on the numeric path**, via the `numpy` crate. Strings are
+  the exception — they are extracted element by element.
+- **Type dispatch via macros.** `define_array` / `write_array` are generic over
+  `T: ArrayElement`; the bindings dispatch at runtime through
+  `numeric_dispatch!` in [atlas-python/src/writer.rs](atlas-python/src/writer.rs).
+
+---
+
+## Prerequisites
+
+| Tool | Why |
+|---|---|
+| **Rust** stable, 1.85+ (edition 2024) | Build the workspace |
+| **Python ≥ 3.10** | The wheel targets `abi3-py310` |
+| **`maturin`** | Build the extension. `pip install maturin` |
+
+---
+
+## Rust: build and test
+
+```bash
+cargo build --workspace
+cargo test -p atlas-rust
+```
+
+`cargo test --workspace` does not work: `atlas-python` links
+`pyo3/extension-module`, so its test binary has no libpython to link against.
+Test the core crate; test the bindings through pytest.
+
+On macOS, `cargo build --workspace` also fails to link the cdylib for the same
+reason. Use `cargo check --workspace` for type checking and `maturin develop`
+for a real build.
+
+Examples:
+
+```bash
+cargo run --example lifecycle
+cargo run --example sensor_fleet
+cargo run --example weather_store
+```
+
+---
+
+## Python: build and test
+
+```bash
+python3 -m venv .venv
+source .venv/bin/activate
+pip install --upgrade pip maturin
+
+maturin develop --extras test --manifest-path atlas-python/Cargo.toml
+pytest atlas-python/tests -v
+```
+
+The install is editable, so changes under `python/atlas/*.py` take effect at
+once. **Rust changes need `maturin develop` again** — otherwise pytest runs
+against the previously built binary.
+
+---
+
+## Test fixtures
+
+Two committed fixtures pin behaviour that a round-trip test would miss.
+
+**`tests/fixtures/golden_v1/`** — a v1 container, read back by
+[tests/golden.rs](tests/golden.rs) with every value asserted. If a change breaks
+compatibility with an existing container, this catches it. Regenerate only when
+you intend to break the format, which means bumping `FORMAT_VERSION`:
+
+```bash
+cargo test --test golden -- --ignored regenerate
+```
+
+The writer's output is deliberately not compared byte for byte — zstd makes no
+promise of stable output across versions. Only the framing this crate produces
+itself is asserted exactly.
+
+**`tests/fixtures/from_python/`** — written by
+[atlas-python/tests/make_fixture.py](atlas-python/tests/make_fixture.py) and read
+back by [tests/cross_fixture.rs](tests/cross_fixture.rs). This is the only thing
+verifying that the bytes the Python xarray layer writes are the bytes it meant,
+now that pytest cannot read arrays. Regenerate after changing the write path:
+
+```bash
+python atlas-python/tests/make_fixture.py
+```
+
+---
+
+## Common workflows
+
+### Adding an array dtype
+
+1. Confirm `array-format` supports it as an element type.
+2. Add the dispatch arm in `numeric_dispatch!`
+   ([atlas-python/src/writer.rs](atlas-python/src/writer.rs)), or an explicit
+   branch alongside String / TimestampNs.
+3. Extend [atlas-python/src/dtype.rs](atlas-python/src/dtype.rs) to parse the
+   name.
+4. Add it to `_NUMPY_TO_ATLAS` in
+   [atlas-python/python/atlas/xarray.py](atlas-python/python/atlas/xarray.py) if
+   it has a numpy equivalent.
+5. Test in both suites, and extend the cross fixture if it is worth pinning.
+
+### Adding to the Python surface
+
+The surface is deliberately five operations. Before adding a sixth, check the
+job cannot be done by composing the ones that exist.
+
+If it genuinely needs to be new:
+
+1. Implement whatever it needs on `Atlas` / `DatasetView` in `src/`.
+2. Expose it through [atlas-python/src/reader.rs](atlas-python/src/reader.rs)
+   or [writer.rs](atlas-python/src/writer.rs), releasing the GIL with
+   `py.detach` for anything that blocks.
+3. Add the operation to
+   [atlas-python/python/atlas/_ops.py](atlas-python/python/atlas/_ops.py) and
+   export it from `__init__.py`.
+4. Add a subcommand in
+   [atlas-python/python/atlas/_cli.py](atlas-python/python/atlas/_cli.py), with
+   both a text renderer and `--json`.
+5. Add the stub to
+   [atlas-python/python/atlas/\_\_init\_\_.pyi](atlas-python/python/atlas/__init__.pyi)
+   — that file is the contract.
+6. `maturin develop`, then test it in both `test_ops.py` and `test_cli.py`.
+
+### Touching the on-disk format
+
+Any change to the framing, the footer struct, or the mask is **breaking** — the
+footer is compact MessagePack, so field order is part of the format.
+
+1. Bump `FORMAT_VERSION` in [src/format/mod.rs](src/format/mod.rs).
+2. Regenerate the golden fixture and update `tests/golden.rs` to the new
+   version.
+3. Update [docs/format.md](docs/format.md) — it is a specification, not a
+   summary.
+
+There is no migration path, by design. A collection that predates the change is
+rewritten, not upgraded.
+
+---
+
+## Code style
+
+- **Rust**: `cargo fmt` before committing, default rustfmt.
+  `cargo clippy --all-targets` should be clean.
+- **Python**: 4-space indentation, imports grouped stdlib / third-party / local.
+  No formatter is enforced; match what is there.
+- **Comments explain why, not what.** The code already shows what. A comment
+  earns its place by recording a constraint, a trade-off, or a trap — the kind
+  of thing the next reader would otherwise have to rediscover.
+- **Tests are named as claims.** `an_end_past_the_segment_is_clamped_not_leaked`
+  says what it protects; `test_read_3` does not.
+
+---
+
+## Pull requests
+
+1. Branch from `main`.
+2. `cargo test -p atlas-rust`, `cargo clippy --all-targets`, and
+   `pytest atlas-python/tests` must pass.
+3. If you change the format, the public API, or user-visible behaviour, update
+   the relevant docs in the same PR.
+4. Keep commits focused; squash fixups before review.

{atlas_python-0.14.0 → atlas_python-0.16.0}/Cargo.lock

@@ -71,7 +71,7 @@ dependencies = [
 
 [[package]]
 name = "atlas-python"
-version = "0.14.0"
+version = "0.16.0"
 dependencies = [
  "atlas-rust",
  "ndarray",
@@ -86,20 +86,19 @@ dependencies = [
 
 [[package]]
 name = "atlas-rust"
-version = "0.14.0"
+version = "0.16.0"
 dependencies = [
  "array-format",
+ "async-trait",
+ "bytes",
  "chrono",
  "futures",
  "indexmap",
- "lz4_flex",
  "ndarray",
- "num_cpus",
  "object_store",
  "parking_lot",
  "rmp-serde",
  "serde",
- "serde_json",
  "tempfile",
  "thiserror 2.0.18",
  "tokio",
@@ -568,12 +567,6 @@ version = "0.5.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea"
 
-[[package]]
-name = "hermit-abi"
-version = "0.5.2"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "fc0fef456e4baa96da950455cd02c081ca953b141298e41db3fc7e36b1da849c"
-
 [[package]]
 name = "http"
 version = "1.4.0"
@@ -1064,16 +1057,6 @@ dependencies = [
  "autocfg",
 ]
 
-[[package]]
-name = "num_cpus"
-version = "1.17.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "91df4bbde75afed763b708b7eee1e8e7651e02d97f6d5dd763e89367e957b23b"
-dependencies = [
- "hermit-abi",
- "libc",
-]
-
 [[package]]
 name = "numpy"
 version = "0.28.0"

{atlas_python-0.14.0 → atlas_python-0.16.0}/Cargo.toml

@@ -5,9 +5,9 @@ members = ["atlas-python"]
 # Registry name on crates.io (`atlas` is taken). The library is still
 # imported as `atlas` — see `[lib]` below.
 name = "atlas-rust"
-version = "0.14.0"
+version = "0.16.0"
 edition = "2024"
-description = "Directory-based store for thousands of N-dimensional datasets local or remote using object storage."
+description = "Single-file immutable store for thousands of N-dimensional datasets, local or on object storage."
 license = "Apache-2.0"
 repository = "https://github.com/maris-development/atlas"
 readme = "README.md"
@@ -22,23 +22,24 @@ all-features = true
 rustdoc-args = ["--cfg", "docsrs"]
 
 [dependencies]
-array-format = "0.12.0"
+# Exact pin: the container format embeds this crate's files verbatim, so a
+# change to its bytes is a change to the atlas format.
+array-format = "=0.12.0"
 object_store = "0.13"
+async-trait = "0.1"
+bytes = "1"
 serde = { version = "1", features = ["derive"] }
-serde_json = "1"
 rmp-serde = "1"
 zstd = "0.13"
-lz4_flex = "0.11"
 thiserror = "2"
 ndarray = "0.17"
 indexmap = { version = "2", features = ["serde"] }
 chrono = { version = "0.4", default-features = false, features = ["std"] }
 parking_lot = "0.12"
-tokio = { version = "1", features = ["fs", "sync", "rt"] }
+tempfile = "3"
+tokio = { version = "1", features = ["fs", "io-util", "sync", "rt"] }
 tracing = "0.1"
 futures = "0.3"
-num_cpus = "1"
 
 [dev-dependencies]
-tempfile = "3"
 tokio = { version = "1", features = ["rt", "rt-multi-thread", "macros"] }

atlas_python-0.16.0/PKG-INFO

@@ -0,0 +1,169 @@
+Metadata-Version: 2.4
+Name: atlas-python
+Version: 0.16.0
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Rust
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering
+Requires-Dist: numpy>=1.23
+Requires-Dist: xarray>=2023.1
+Requires-Dist: dask>=2023.1
+Requires-Dist: obstore>=0.9 ; extra == 'cloud'
+Requires-Dist: pytest ; extra == 'test'
+Requires-Dist: netcdf4 ; extra == 'test'
+Provides-Extra: cloud
+Provides-Extra: test
+License-File: LICENSE
+Summary: Python bindings for the ATLAS array store
+Keywords: array,ndarray,xarray,scientific,storage,object-store
+Author-email: Robin Kooyman <robin.kooyman.work@gmail.com>
+License-Expression: Apache-2.0
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
+Project-URL: Homepage, https://github.com/maris-development/atlas
+Project-URL: Issues, https://github.com/maris-development/atlas/issues
+Project-URL: Repository, https://github.com/maris-development/atlas
+
+# atlas-python
+
+Thousands of NetCDF datasets in one immutable file. It sits on local disk or on
+object storage: S3, GCS, Azure, or HTTP. A Rust core, five operations, and one
+command.
+
+```bash
+pip install atlas-python
+```
+
+```bash
+atlas create /data/nc /data/collection
+atlas ls     /data/collection
+atlas show   /data/collection 2024-01
+atlas info   /data/collection
+atlas rm     /data/collection 2024-02 2024-03
+```
+
+| Extra | Install | Adds |
+|---|---|---|
+| cloud | `pip install "atlas-python[cloud]"` | S3 / GCS / Azure / HTTP via [obstore](https://github.com/developmentseed/obstore) |
+
+`numpy`, `xarray`, and `dask` install automatically.
+
+## Five operations
+
+The same five as a library:
+
+```python
+import atlas
+
+atlas.create("/data/nc", "/data/collection")   # from a directory of NetCDF files
+atlas.list_datasets("/data/collection")        # ['2024-01', '2024-02', '2024-03']
+atlas.describe("/data/collection", "2024-01")  # types, shapes, attrs, statistics
+atlas.info("/data/collection")                 # counts, size, codec, statistics
+atlas.remove("/data/collection", ["2024-02"])  # updates the mask
+```
+
+Every one takes a local path, a URL, or an obstore handle:
+
+```bash
+atlas ls s3://my-bucket/collections/2024 --region eu-west-1
+```
+
+```python
+atlas.list_datasets("s3://my-bucket/collections/2024", region="eu-west-1")
+```
+
+## Two things to internalise
+
+**One write builds a collection.** There is no append, no in-place update, and
+no `flush`. The file has a valid trailer, or it is no collection. To change a
+dataset, rebuild the collection. `remove` is the one exception. It writes a
+small mask file, and never touches the container. It therefore reclaims no
+space, and moves no ordinal.
+
+**Python writes. Rust reads array data.** From Python you build a collection
+and read its *metadata*. That is the dataset names, the array types, the
+shapes, the chunk shapes, the fill values, the attributes, and the statistics
+of the write. There is no `read_array`. Array values come from the Rust API.
+
+That split makes the read side free. The footer an open already fetched answers
+every metadata call. A catalogue of a thousand datasets is therefore one
+request.
+
+## What `show` gives you
+
+```text
+$ atlas show /data/collection 2024-01
+dataset 2024-01 {
+dimensions:
+	lat = 4 ;
+	lon = 6 ;
+variables:
+	float32 temperature(lat, lon) ;
+		temperature:_FillValue = nan ;
+		temperature:units = "celsius" ;
+		// stats: count=24  min=1.0  max=24.0
+	string station(lat) ;
+		// stats: count=4  min="a"  max="d"
+
+// global attributes:
+		:month = 1 ;
+
+// ordinal 0, segment bytes 8..1691
+}
+```
+
+The shape follows `ncdump -h`. It adds the statistics of the write: the
+minimum, the maximum, and how many elements are missing. `--json` on any read
+command gives the same content as a structure.
+
+## Ingest
+
+`create` scans a directory for `.nc`, `.nc4`, `.cdf`, and `.netcdf`. It sorts
+them, and writes one dataset per file, named after the stem. Each coordinate
+and data variable becomes an array. Each variable attribute becomes a per-array
+attribute. `_FillValue` becomes the fill of the array.
+
+Each file opens with dask chunking. A file far larger than memory therefore
+streams block by block. `--chunk-size` sets the block budget, and defaults to
+128 MiB. It is about the memory ceiling per variable. Those blocks also become
+the stored chunk shape.
+
+Nothing at the destination is readable until every file lands. A failure
+part-way leaves no collection, and not a partial one. `on_error="skip"`, or
+`--skip-errors`, trades that for progress.
+
+## dtypes
+
+| numpy | atlas |
+|---|---|
+| int / uint widths, `float32`, `float64` | the same |
+| `datetime64[ns]` | `timestamp_nanoseconds` |
+| `timedelta64[*]` | `int64` nanoseconds, plus a unit marker |
+| `object` / `S` / `U` | `string` |
+
+`bool`, `binary`, and the list types work as an *attribute* value. No one of
+them works yet as an array element type.
+
+## Documentation
+
+The full docs sit at **<https://maris-development.github.io/atlas/>**. They
+hold the [command reference], and guides for [creating], [inspecting],
+[removing], [dtypes], [reading data], and [cloud storage].
+
+The format itself is documented in
+[`docs/`](https://github.com/maris-development/atlas/tree/main/docs).
+
+[command reference]: https://maris-development.github.io/atlas/cli/
+[creating]: https://maris-development.github.io/atlas/guides/creating/
+[inspecting]: https://maris-development.github.io/atlas/guides/inspecting/
+[removing]: https://maris-development.github.io/atlas/guides/removing/
+[dtypes]: https://maris-development.github.io/atlas/guides/dtypes/
+[reading data]: https://maris-development.github.io/atlas/guides/reading-data/
+[cloud storage]: https://maris-development.github.io/atlas/guides/cloud-storage/
+
+## License
+
+Apache-2.0. See [LICENSE](https://github.com/maris-development/atlas/blob/main/LICENSE).
+