xarray-binfile 0.2.0__tar.gz → 0.3.0__tar.gz

{xarray_binfile-0.2.0 → xarray_binfile-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: xarray-binfile
-Version: 0.2.0
+Version: 0.3.0
 Summary: Read and write raw binary files using the familiar interface from the Xarray library
 Project-URL: Source, https://github.com/fschuch/xarray-binfile
 Project-URL: Tracker, https://github.com/fschuch/xarray-binfile/issues

{xarray_binfile-0.2.0 → xarray_binfile-0.3.0}/docs/_toc.yml

@@ -13,6 +13,7 @@ parts:
       - file: usage/spec-getter-protocols.md
       - file: tutorials/read.ipynb
       - file: tutorials/write.ipynb
+      - file: tutorials/stacked-variables.ipynb
       - file: tutorials/parallel-and-larger-than-memory.ipynb
       - file: tutorials/benchmarking.ipynb
       - file: tutorials/examples.ipynb

xarray_binfile-0.3.0/docs/tutorials/stacked-variables.ipynb

@@ -0,0 +1,311 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "0",
+   "metadata": {},
+   "source": [
+    "# Stack And Split Variables Across Files\n",
+    "\n",
+    "Many solvers store a vector field as one file per component (`ux-0001.bin`, `uy-0001.bin`, `uz-0001.bin`) and a set of scalar fractions as numbered variables (`phi1-0001.bin`, `phi2-0001.bin`). In memory, you would rather work with one array `u` carrying a component dimension `i`, and one array `phi` carrying a fraction dimension `n`.\n",
+    "\n",
+    "xarray-binfile describes that encoding once, with a `VariableStack`, and uses it in both directions:\n",
+    "\n",
+    "- **split** on write: `u` with `i = [\"x\", \"y\", \"z\"]` becomes the variables `ux`, `uy` and `uz`, each written to its own files\n",
+    "- **stack** on read: `ux`, `uy` and `uz` found on disk are concatenated back into `u`\n",
+    "\n",
+    "Nothing about `u`, `x`, `y`, `z` or `phi` is hard-coded in the library: the dimension name, the name template and the allowed values all come from the declaration."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "1",
+   "metadata": {},
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "import pathlib\n",
+    "import tempfile\n",
+    "\n",
+    "import numpy as np\n",
+    "import xarray as xr\n",
+    "\n",
+    "import xarray_binfile  # noqa: F401  (registers the .binary_engine accessors)\n",
+    "from xarray_binfile.conventions import Layout, StepIndexedFiles, VariableStack\n",
+    "\n",
+    "directory_holder = tempfile.TemporaryDirectory()\n",
+    "directory = pathlib.Path(directory_holder.name)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2",
+   "metadata": {},
+   "source": [
+    "## Declare the stacks\n",
+    "\n",
+    "A `VariableStack` has three parts:\n",
+    "\n",
+    "- `dim`: the dimension that lives in the variable names rather than in the files\n",
+    "- `template`: how a variable name is built from the base `{name}` and the value of `dim`\n",
+    "- `values`: the values the dimension may take, in the order the stacked coordinate should follow\n",
+    "\n",
+    "The template is parsed with `FilenamePattern`, so integer values can carry zero-fill (`\"{name}{n:02d}\"`) and are converted back to `int` on read. `values` is required because templates without a separator would otherwise match almost any name: `pp` fits `\"{name}{i}\"` as `p` + `p`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "3",
+   "metadata": {},
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "velocity = VariableStack(\"i\", \"{name}{i}\", values=(\"x\", \"y\", \"z\"))\n",
+    "scalars = VariableStack(\"n\", \"{name}{n:d}\", values=range(1, 10), names=(\"phi\",))\n",
+    "velocity, scalars"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4",
+   "metadata": {},
+   "source": [
+    "The optional `names` restricts stacking to given base names. Without it, a vorticity component written as `w3` would be picked up by `scalars` as `w` with `n = [3]`, which is harmless but probably not what you want.\n",
+    "\n",
+    "## Attach them to a convention\n",
+    "\n",
+    "Every shipped convention accepts `stacks`. On write, arrays are split in the order the stacks are listed, before the per-file split along `time`; on read, `open` rebuilds the stacked arrays by default."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "5",
+   "metadata": {},
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "layout = Layout(\n",
+    "    {\n",
+    "        \"x\": np.linspace(0.0, 1.0, 8, dtype=np.float32),\n",
+    "        \"y\": np.linspace(0.0, 2.0, 6, dtype=np.float32),\n",
+    "        \"z\": np.linspace(0.0, 0.5, 4, dtype=np.float32),\n",
+    "    },\n",
+    "    dtype=np.float32,\n",
+    ")\n",
+    "convention = StepIndexedFiles(\n",
+    "    layout,\n",
+    "    pattern=\"{name}-{step:03d}.bin\",\n",
+    "    time_step=0.25,\n",
+    "    stacks=[velocity, scalars],\n",
+    ")\n",
+    "convention"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6",
+   "metadata": {},
+   "source": [
+    "## Build a dataset with stacked arrays\n",
+    "\n",
+    "`u` carries `i`, `phi` carries `n`, and `pp` carries neither. All of them share the mesh and the `time` dimension."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "7",
+   "metadata": {},
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "rng = np.random.default_rng(0)\n",
+    "time = np.arange(3) * 0.25\n",
+    "\n",
+    "\n",
+    "def field(**extra):\n",
+    "    coords = {**extra, **layout.coords, \"time\": time}\n",
+    "    shape = tuple(len(values) for values in coords.values())\n",
+    "    return xr.DataArray(rng.random(shape, dtype=np.float32), coords=coords)\n",
+    "\n",
+    "\n",
+    "dataset = xr.Dataset(\n",
+    "    {\n",
+    "        \"u\": field(i=[\"x\", \"y\", \"z\"]),\n",
+    "        \"phi\": field(n=[1, 2]),\n",
+    "        \"pp\": field(),\n",
+    "    }\n",
+    ")\n",
+    "dataset"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "8",
+   "metadata": {},
+   "source": [
+    "## Split on write\n",
+    "\n",
+    "Writing the dataset through the convention produces one file per component, per fraction and per step. `u` never appears on disk; `pp` is written as is."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "9",
+   "metadata": {},
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "dataset.binary_engine.to_file(convention.writer, directory)\n",
+    "sorted(path.name for path in directory.glob(\"*.bin\"))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "10",
+   "metadata": {},
+   "source": [
+    "The split happens before the file split, so the per-file slices are plain `(x, y, z)` arrays. You can inspect the plan without writing anything by iterating the writer:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "11",
+   "metadata": {},
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "[(spec.filename, spec.sub_array.dims) for spec in convention.writer(dataset[\"u\"])][:4]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "12",
+   "metadata": {},
+   "source": [
+    "## Stack on read\n",
+    "\n",
+    "`open` lists the files that follow the pattern, opens them lazily with `xarray.open_mfdataset`, and applies the stacks. The result has the same variables as the original dataset, with `i` and `n` as proper coordinates."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "13",
+   "metadata": {},
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "opened = convention.open(directory, chunks={\"time\": 1})\n",
+    "opened"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "14",
+   "metadata": {},
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "for name in dataset.data_vars:\n",
+    "    xr.testing.assert_allclose(\n",
+    "        opened[name].transpose(*dataset[name].dims).load(), dataset[name]\n",
+    "    )\n",
+    "print(\"round trip matches\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "15",
+   "metadata": {},
+   "source": [
+    "Stacking is lazy, so a Dask-backed dataset stays Dask-backed, and `i` and `n` can be used like any other coordinate:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "16",
+   "metadata": {},
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "speed = np.sqrt((opened[\"u\"] ** 2).sum(\"i\")).rename(\"speed\")\n",
+    "speed.isel(time=0, z=0).compute()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "17",
+   "metadata": {},
+   "source": [
+    "## Opt out, or stack by hand\n",
+    "\n",
+    "Pass `stack=False` to see the variables exactly as they are on disk. The same declarations are available as functions for datasets opened another way: `VariableStack.stack` for one dimension, `stack_variables` for a sequence, and `split_variables` for the inverse."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "18",
+   "metadata": {},
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "raw = convention.open(directory, stack=False)\n",
+    "sorted(raw.data_vars)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "19",
+   "metadata": {},
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "from xarray_binfile.conventions import split_variables, stack_variables\n",
+    "\n",
+    "restacked = stack_variables(raw, [velocity, scalars])\n",
+    "pieces = [piece.name for piece in split_variables(dataset[\"u\"], [velocity])]\n",
+    "sorted(restacked.data_vars), pieces"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "20",
+   "metadata": {},
+   "source": [
+    "## Guard rails\n",
+    "\n",
+    "A value that is not declared cannot be written, because the reader would have no way to stack it back:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "21",
+   "metadata": {},
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "bad = dataset[\"u\"].assign_coords(i=[\"x\", \"y\", \"w\"])\n",
+    "try:\n",
+    "    list(convention.writer(bad))\n",
+    "except ValueError as error:\n",
+    "    print(error)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "22",
+   "metadata": {},
+   "execution_count": null,
+   "outputs": [],
+   "source": [
+    "directory_holder.cleanup()"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

{xarray_binfile-0.2.0 → xarray_binfile-0.3.0}/docs/usage/spec-getter-protocols.md

@@ -33,6 +33,8 @@ Contract:
 - `coords`: mapping of dimension name to coordinate array, in on-disk order
 - `name`: variable name
 - `attrs`: optional attributes
+- `order`: memory layout of the file, `"C"` (default, last dimension varies fastest, as NumPy's `tofile` and C programs write) or `"F"` (first dimension varies fastest, as Fortran programs such as 2DECOMP&FFT and Xcompact3d write). Both describe the same bytes: a Fortran file with dims `(x, y, z)` is also a C file with dims `(z, y, x)`. Pick the one that keeps your dimension names in their natural order
+- `coord_attrs`: optional attributes per coordinate, attached on read (for example `{"x": {"units": "m"}}`)
 
 The backend validates the file size against `coords` and `dtype` before reading anything, so a wrong shape or byte order fails early instead of producing garbage.
 
@@ -50,6 +52,9 @@ Contract:
 - `filename`: output path. A relative path is resolved against the directory passed to `to_file`, which must exist, and may include sub-folders, which are created on demand, but it must stay inside that directory (`..` escaping it is rejected; symbolic links are not followed for this check). An absolute path is used as is, so one call can target several locations
 - `sub_array`: array slice to write into that file, already transposed to on-disk order
 - `dtype`: optional on-disk data type; when set, the sub-array is cast right before serialization (including byte order, for example `"<f4"`). When omitted, the in-memory dtype and native byte order are written as-is, so make sure they match what your read specs getter declares.
+- `order`: memory layout of the file, `"C"` (default) or `"F"`, with the same meaning as on `ReadSpecs`. `tofile` always serializes in C order, so Fortran order is obtained by flattening the array first.
+
+`to_file` also accepts a `progress` wrapper applied to the iterator of write specs, for example `progress=tqdm` or `progress=functools.partial(tqdm, desc="ux")`, to report progress without the library depending on a progress-bar package.
 
 ```{warning}
 Writes are eager and whole-file only. Each `sub_array` is fully loaded into memory (a Dask compute for lazy data) and its file is written in a single pass — the backend never appends to, patches, or resumes a file. Files are staged in a temporary file next to their destination and atomically moved into place with `os.replace` once complete, so an interrupted write never leaves a truncated file behind. That protects against partially-updated, corrupted files, but it means every `sub_array` you yield must fit in memory. Split large arrays into more, smaller files, and reach for xarray-supported formats such as NetCDF or Zarr when you need streaming or partial writes.
@@ -61,8 +66,18 @@ See the [API reference](../references/api-reference.rst) for the authoritative s
 
 The module `xarray_binfile.conventions` provides small, ready-to-use implementations of both protocols. Each convention exposes a `reader` (the read specs getter) and a `writer` (the write specs getter) built from the same two ingredients, so files written by one can always be read back by the other:
 
-- `FilenamePattern`: one `str.format`-style template such as `"{name}-{step:04d}.bin"`. The regular expression used for reading is derived from the template, so the two cannot drift apart. Matching is anchored to the whole filename, and a zero-fill width is a minimum width (step `12345` written with `04d` still reads back).
-- `Layout`: the dimension order, coordinate values and dtype of one file on disk. Readers attach these coordinates to every file; writers refuse arrays that do not match them.
+- `FilenamePattern`: one `str.format`-style template such as `"{name}-{step:04d}.bin"`. The regular expression used for reading is derived from the template, so the two cannot drift apart. Matching is anchored to the whole filename, and a zero-fill width is a minimum width (step `12345` written with `04d` still reads back). Templates without a separator between a name ending in a digit and the number are ambiguous under that rule (`phi1000` reads as `phi` + `1000`); pass `exact_width=True` to match exactly the declared width instead (`phi1` + `000`), which also refuses wider steps at write time. `glob()` turns the template into a pattern for `pathlib.Path.glob` (`"*-*.bin"`, or `"ux-*.bin"` with `glob(name="ux")`), and a `name` value may start with folder segments (`"geometry/epsi"`), which are kept in front of the formatted filename.
+- `Layout`: the dimension order, coordinate values, dtype and memory `order` of one file on disk, plus optional `coord_attrs`. Readers attach these coordinates to every file; writers refuse arrays that do not match them.
+
+Every shipped convention inherits `Convention`, the base class that turns a `reader`/`writer` pair into a full convention: `name_of_file(path)`, `accepts(path)`, `files(directory)`, `stack(dataset)` and `open(directory, ...)`, all derived from `reader` by default and overridden where something cheaper exists. Subclass it for your own conventions; a bare object with only `reader` and `writer` is wrapped through `Convention.adapt` when it joins a `FolderConventions` or `PatternConventions`, so it gets the same defaults.
+
+Every shipped convention also offers:
+
+- `files(directory)`: the files in a folder that follow its pattern, and `open(directory, ...)`: those files as one lazy dataset through `xarray.open_mfdataset`, with `variables=[...]` to open a subset and any other keyword (`chunks`, `parallel`) forwarded. Discovery is driven by the anchored pattern only, so unrelated files in the data folder (an XDMF index, notes, backups, hidden files, the temporary files of an interrupted write) are never opened
+- `names`: the variable names the convention accepts. When the pattern alone is too permissive, for example a bare `"{name}"` template for extension-less files that would also match `README` or `Makefile`, `names=("epsilon",)` restricts reading, discovery and writing to the listed variables, and lets a `PatternConventions` move on to the next member
+- `stacks`: dimensions encoded in the variable names, see [Stacked variables](#stacked-variables)
+- `name_of`: a hook returning the name to write an array under, in place of `data_array.name` (for example `lambda da: da.attrs["file_name"]`)
+- `time_dtype` (time-series conventions): the dtype of the `time` coordinate on read, for example `np.float32` to match single-precision data
 
 ### Time-series numbered by step
 
@@ -118,6 +133,37 @@ from xarray_binfile.conventions import StaticFiles
 static = StaticFiles(layout, pattern="{name}.bin")
 ```
 
+### Stacked variables
+
+Solvers often store a vector field as one file per component (`ux`, `uy`, `uz`) and a set of scalar fractions as numbered variables (`phi1`, `phi2`). A `VariableStack` declares that encoding once: the dimension it stands for, the template that builds a variable name from `{name}` and the value, and the values it may take. Conventions split arrays along those dimensions on write and stack the variables back on read:
+
+```python
+from xarray_binfile.conventions import VariableStack
+
+velocity = VariableStack("i", "{name}{i}", values=("x", "y", "z"))
+scalars = VariableStack("n", "{name}{n:d}", values=range(1, 10), names=("phi",))
+convention = StepIndexedFiles(layout, stacks=[velocity, scalars])
+
+# u with i = ["x", "y", "z"] is written as ux-0001.bin, uy-0001.bin, uz-0001.bin
+dataset.binary_engine.to_file(convention.writer, output_dir)
+
+# ux, uy, uz found on disk come back as u with the coordinate i
+opened = convention.open(output_dir)          # stacks by default
+raw = convention.open(output_dir, stack=False)  # ux, uy, uz as on disk
+```
+
+`values` is required because templates without a separator match almost any name (`pp` fits `"{name}{i}"` as `p` + `p`); it also fixes the order of the stacked coordinate, and writing a coordinate value that is not listed raises. On read each value is matched literally, so `vortx` is `vort` + `x`, and `names` (any collection, kept as a `frozenset`; a bare string is rejected) says which base names to reassemble; without it, a group needs `min_components` (two by default) components, so a lone `vorticity` is not read as `vorticit` + `y`. Stacking refuses to overwrite a variable that already carries the base name. Writing always splits an array carrying the dimension, whatever its name. The same declarations work on their own through `VariableStack.stack`, `stack_variables` and `split_variables`. See the [stacked variables tutorial](../tutorials/stacked-variables.ipynb).
+
+### Mixed conventions in one folder
+
+When files following different conventions share a folder, for example `ux-0001.bin` next to `epsi.bin`, `PatternConventions` tries its members in order on read and keeps the first whose pattern accepts the filename, so list the most specific pattern first. On write it offers the array to every member and requires exactly one to accept it, based on dimensions and coordinates:
+
+```python
+from xarray_binfile.conventions import PatternConventions
+
+conventions = PatternConventions([StepIndexedFiles(layout), StaticFiles(layout)])
+```
+
 ### Shapes that depend on the folder
 
 Projects often keep arrays of different shapes in different folders:
@@ -132,7 +178,7 @@ caseC/
 		epsi.bin
 ```
 
-`FolderConventions` dispatches to one convention per folder. Reading picks the convention whose folder matches the end of the file's parent path, preferring the most specific folder when several match (`snapshots/3d` over `3d`). Writing offers the array to every convention and requires exactly one to accept it, based on the array dimensions and coordinates, then writes into that folder. If no convention or more than one accepts the array, a `LayoutMismatchError` is raised: the backend never guesses.
+`FolderConventions` dispatches to one convention per folder. Reading picks the convention whose folder matches the end of the file's parent path, preferring the most specific folder when several match (`snapshots/3d` over `3d`). The dataset root is registered as `"."` and matches any file not claimed by a more specific folder. Writing offers the array to every convention and requires exactly one to accept it, based on the array dimensions and coordinates, then writes into that folder. If no convention or more than one accepts the array, a `LayoutMismatchError` is raised: the backend never guesses. An array whose name starts with a registered folder (`"static/epsi"`) is sent to that folder's convention directly, under the remaining name, which resolves the ambiguity between folders sharing one layout. `files(directory)` and `open(directory)` walk every registered folder; `PatternConventions` can be a member, for example as the root convention.
 
 ```python
 from xarray_binfile.conventions import FolderConventions
@@ -146,9 +192,7 @@ conventions = FolderConventions(
     }
 )
 
-dataset = xr.open_mfdataset(
-    sorted(case_dir.rglob("*.bin")), engine="binfile", read_specs_getter=conventions.reader
-)
+dataset = conventions.open(case_dir)  # or xr.open_mfdataset(conventions.files(case_dir), engine="binfile", read_specs_getter=conventions.reader)
 dataset.binary_engine.to_file(conventions.writer, output_dir)
 ```
 
@@ -189,7 +233,7 @@ Write specs getter:
 
 - Derive the read and write filename rules from one definition.
 - Anchor filename matching to the whole name so stale or backup files are rejected.
-- Keep dimension order explicit and consistent.
+- Keep dimension order explicit and consistent, and declare the memory `order` (`"C"` or `"F"`) that matches the program that wrote the files.
 - Make dtype and endianness explicit.
 - Fail loudly on layout mismatch instead of broadcasting or squeezing.
 - Test read/write round-trips (`xarray.testing.assert_identical`).
@@ -202,4 +246,5 @@ Raw binaries are machine-specific unless conventions are explicit. Endianness (l
 
 - [Built-in tutorial: reading](../tutorials/read.ipynb)
 - [Built-in tutorial: writing](../tutorials/write.ipynb)
+- [Built-in tutorial: stacked variables](../tutorials/stacked-variables.ipynb)
 - [API reference](../references/api-reference.rst)

{xarray_binfile-0.2.0 → xarray_binfile-0.3.0}/src/xarray_binfile/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.2.0'
-__version_tuple__ = version_tuple = (0, 2, 0)
+__version__ = version = '0.3.0'
+__version_tuple__ = version_tuple = (0, 3, 0)
 
 __commit_id__ = commit_id = None

{xarray_binfile-0.2.0 → xarray_binfile-0.3.0}/src/xarray_binfile/conventions/__init__.py

@@ -5,8 +5,9 @@ Start from the convention closest to your project's naming scheme, or copy one
 and adapt it when none fits.
 """
 
+from xarray_binfile.conventions.base import Convention
 from xarray_binfile.conventions.filename_pattern import FilenamePattern
-from xarray_binfile.conventions.folders import FolderConventions
+from xarray_binfile.conventions.folders import FolderConventions, PatternConventions
 from xarray_binfile.conventions.getters import (
     StaticFiles,
     StepIndexedFiles,
@@ -14,14 +15,24 @@ from xarray_binfile.conventions.getters import (
 )
 from xarray_binfile.conventions.layout import Layout, LayoutMismatchError
 from xarray_binfile.conventions.protocol import ConventionProtocol
+from xarray_binfile.conventions.stacking import (
+    VariableStack,
+    split_variables,
+    stack_variables,
+)
 
 __all__ = [
+    "Convention",
     "ConventionProtocol",
     "FilenamePattern",
     "FolderConventions",
     "Layout",
     "LayoutMismatchError",
+    "PatternConventions",
     "StaticFiles",
     "StepIndexedFiles",
     "TimeStampedFiles",
+    "VariableStack",
+    "split_variables",
+    "stack_variables",
 ]