brainstem-nwb 0.1.0__tar.gz

brainstem_nwb-0.1.0/.gitignore

@@ -0,0 +1,18 @@
+__pycache__/
+*.py[cod]
+.venv/
+venv/
+build/
+dist/
+*.egg-info/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+coverage.xml
+htmlcov/
+site/
+*.nwb
+!tests/fixtures/**/*.nwb
+.DS_Store
+/out/

brainstem_nwb-0.1.0/CHANGELOG.md

@@ -0,0 +1,99 @@
+# Changelog
+
+All notable changes to brainstem-nwb are documented here. The project follows semantic versioning. The mapping version, which tracks changes to what is written to NWB files, is listed separately.
+
+## 0.1.0 (2026-11-07)
+
+### The BrainSTEM data model is read from the server (mapping version 1.3.0)
+
+- brainstem-nwb no longer ships a copy of the BrainSTEM data model. It reads the record types, fields, links and type vocabularies from the server's reference documents (`/reference/data/`), caches them per server for an hour, and uses the cached copy when the server cannot be reached (`DATA_MODEL_CACHED`). `BRAINSTEM_NWB_DATA_MODEL` names a directory with the documents for offline machines; `brainstem-nwb data-model` fetches and shows the model.
+- The manifest's `data_model` names the server, the data model generation, the export format and a checksum of the model used. A generation this release was not written for is reported as `DATA_MODEL_UNSUPPORTED`.
+- Follows the production data model (`specimen-1`): data acquisitions and manipulations link procedures only through their interfaces, as the server's session export does; specimens record `laterality` (no `hemisphere` or `baseline_conditions`).
+- Planned and cancelled sessions get no file (`SESSION_NOT_PERFORMED`); planned and cancelled procedures and specimen procedures are left out of the file and kept in the manifest (`PROCEDURE_NOT_PERFORMED`). Records without a status are performed.
+- The code is MIT. The BrainSTEM data model and schemas are © BrainSTEM, all rights reserved; embedded schemas carry BrainSTEM's notice in the `schemas` table description.
+
+### NWB extensions (mapping version 1.2.0)
+
+- `--extension brainstem` writes every included BrainSTEM record, with every field and the links between them, as `ndx-brainstem` tables (pre-release extension, 0.2.x; install with `pip install "brainstem-nwb[extensions]"`).
+- With `--extension brainstem`, coordinates of procedures and interfaces in stereotaxic absolute and Allen CCF systems are also written as `ndx-anatomical-localization` tables, linked to their `ndx-brainstem` rows (`--extension localization` is the same option).
+- The `ndx-brainstem` schema table (extension 0.2.0) lists every BrainSTEM JSON schema the file's tables name, with its stable URL. `--embed-schemas` fetches the schemas from the server's public schema service and embeds them. Coordinates get their own `coordinates_schema` column.
+- Materials link directly to their catalog consumable (`catalog_entry`), also when the usage names a stock. Stocks keep only their typed details (batch, titer, volume, aliquots).
+- `--herd` writes ontology terms for species, strains, regions and device models as a HERD zip next to each file.
+- Licenses, data storages and inventories are no longer written to NWB files (NWBFile.protocol lists protocols only); they stay in the manifest.
+- An NWB Inspector check that crashes is reported as a warning, not an error.
+
+### Export format 2.1 (mapping version 1.1.0)
+
+- **Session exports.** Reads session exports (`export_info.scope`, `session_id`, `project_ids`). `export --session UUID` fetches one session's export through `BrainstemClient.export_session` when the API tools provide it. A session in several projects lists all of them in `experiment_description` and `keywords`.
+- **Catalog records the export now includes:**
+  - Brain regions and atlases: electrode-group and stimulus-site `location` from region acronyms, with full names and atlas in descriptions and procedure text.
+  - Suppliers: `DeviceModel` with manufacturer and model number for hardware devices and probe designs; suppliers in material descriptions.
+  - Publications: `related_publications` DOIs.
+  - User names: who performed each procedure, episode personnel, and opt-in `experimenter_source = "session_creator"`.
+  - Log entries: `Subject.weight` from the latest weighing before the session.
+  - Behavioral paradigms and categories, setup types, regulatory authorities.
+- **Restricted lab resources.** Resources the exporting user cannot read (`export_info.restricted`) are named in the text and reported as `RESTRICTED_LAB_RESOURCES`.
+- **Contract fixtures.** Regenerated from the server's export builder, as a project export and a session export.
+
+### First version
+
+First release: metadata-only export. Mapping version 1.0.0. Manifest and validation report schemas 1.0.0.
+
+#### Input
+
+- Reads BrainSTEM project exports (format 2.x), from a downloaded file (`.json` or `.json.gz`) or fetched with `BrainstemClient.export_project()`.
+
+#### Export units
+
+- One NWB file per session and subject, named `sub-<subject>/sub-<subject>_ses-<session>.nwb`.
+- Each record is attributed to the subject it belongs to, and every decision is documented.
+- Specimen (ex vivo) sessions are supported.
+
+#### What is written
+
+**Session and subject**
+
+- Session fields and project context.
+- `Subject`: species, optionally as an NCBI Taxonomy IRI; strain; sex; date of birth; age; genotype.
+
+**Procedures and manipulations**
+
+- Surgery, virus and pharmacology narratives, from procedures and their materials.
+- Stimulus notes, from manipulations.
+
+**Devices**
+
+- Devices from equipment and experimental interfaces.
+- Electrode groups.
+- Optogenetic stimulus sites.
+
+**Other**
+
+- The epochs table.
+- Ex vivo slices.
+- Behaviors in the notes.
+- A provenance block.
+
+#### Manifest and validation
+
+- **Conversion manifest:** a value state for every NWB destination, coded warnings, raw-data references, and the records used.
+- **Validation:**
+  - pynwb schema validation.
+  - NWB Inspector, with the DANDI configuration.
+  - BrainSTEM semantic checks.
+  - Reports in JSON and text.
+
+#### Tools
+
+- **CLI:** `export`, `manifest`, `validate`, `fetch`, `mapping`, `schema`.
+- **Python API:** `export_project`, `export_session`, `build_manifest`, `validate_output`, `load_export`, `fetch_export`.
+
+#### Reproducibility
+
+- Deterministic NWB identifiers.
+- `--reproducible` mode.
+- Idempotent re-runs: files that are already up to date are kept.
+
+#### Raw data (Phase 4)
+
+- Designed only: the converter protocol, registry, data-path file and NeuroConv metadata hand-off.

brainstem_nwb-0.1.0/LICENSE

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) 2026 Peter C. Petersen - BrainSTEM Team
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

brainstem_nwb-0.1.0/PKG-INFO

@@ -0,0 +1,158 @@
+Metadata-Version: 2.5
+Name: brainstem-nwb
+Version: 0.1.0
+Summary: Export BrainSTEM experimental metadata to Neurodata Without Borders (NWB) files
+Project-URL: Homepage, https://github.com/brainstem-org/brainstem-nwb
+Project-URL: Documentation, https://github.com/brainstem-org/brainstem-nwb/tree/main/docs
+Project-URL: Issues, https://github.com/brainstem-org/brainstem-nwb/issues
+Author: BrainSTEM Team
+License-Expression: MIT
+License-File: LICENSE
+Keywords: brainstem,dandi,electronic lab notebook,metadata,neuroscience,nwb
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
+Requires-Python: >=3.10
+Requires-Dist: brainstem-python-api-tools>=2.1
+Requires-Dist: click>=8.1
+Requires-Dist: nwbinspector>=0.7
+Requires-Dist: pydantic>=2.6
+Requires-Dist: pynwb<5,>=4.0
+Requires-Dist: requests>=2.28
+Requires-Dist: tomli>=2.0; python_version < '3.11'
+Provides-Extra: dev
+Requires-Dist: jsonschema>=4.18; extra == 'dev'
+Requires-Dist: mypy>=1.11; extra == 'dev'
+Requires-Dist: pytest-cov>=5; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Provides-Extra: docs
+Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
+Provides-Extra: extensions
+Requires-Dist: ndx-brainstem<0.3,>=0.2; extra == 'extensions'
+Provides-Extra: neuroconv
+Requires-Dist: neuroconv[ecephys-minimal]>=0.10; extra == 'neuroconv'
+Description-Content-Type: text/markdown
+
+# brainstem-nwb
+
+Export experimental metadata from [BrainSTEM](https://www.brainstem.org) to
+[Neurodata Without Borders (NWB)](https://www.nwb.org) files.
+
+`brainstem-nwb` reads a BrainSTEM project export and writes one NWB file per session and subject,
+following NWB and DANDI conventions. Each file holds what BrainSTEM knows about the experiment:
+- session and subject
+- surgeries, viruses and drugs
+- devices, electrode groups and optogenetic sites
+- acquisitions, behaviors, manipulations and epochs
+
+Alongside the files it writes a versioned **conversion manifest** and a **validation report**.
+
+This release exports **metadata only**. Measured data (recordings, video, imaging) is not converted
+yet; the files describe it and point to it. Raw-data conversion with
+[NeuroConv](https://github.com/catalystneuro/neuroconv), starting with flat binary extracellular
+recordings, is designed and planned (see [docs/design/phase4-raw-data.md](docs/design/phase4-raw-data.md)).
+
+## Install
+
+```bash
+pip install brainstem-nwb
+# the development version:
+pip install git+https://github.com/brainstem-org/brainstem-nwb
+```
+
+Python 3.10 or newer. brainstem-nwb reads the BrainSTEM data model from the server at run time, so the
+first export needs network access to www.brainstem.org (or your `--url`); the model is then cached (see
+[installation](docs/install.md#the-brainstem-data-model)). Installing also installs [pynwb](https://github.com/NeurodataWithoutBorders/pynwb),
+[NWB Inspector](https://github.com/NeurodataWithoutBorders/nwbinspector) and the
+[BrainSTEM Python API tools](https://github.com/brainstem-org/brainstem_python_api_tools).
+
+## Quick start
+
+From a project export downloaded in BrainSTEM (Project → Download → JSON):
+
+```bash
+brainstem-nwb export --export-file brainstem-export.json.gz --output ./nwb/
+```
+
+Or fetch the export directly (needs read access to the project):
+
+```bash
+brainstem login                     # once, from the BrainSTEM API tools
+brainstem-nwb export --project PROJECT_UUID --output ./nwb/
+brainstem-nwb export --session SESSION_UUID --output ./nwb/   # one session's export
+```
+
+From Python:
+
+```python
+from brainstem_nwb import export_project
+
+result = export_project(project_id="PROJECT_UUID", output_dir="./nwb")
+# or: export_project(export_file="brainstem-export.json.gz", output_dir="./nwb")
+print(result.exit_code, result.manifest.summary)
+```
+
+The output directory then contains:
+
+```
+nwb/
+  sub-<subject>/sub-<subject>_ses-<session>.nwb
+  brainstem-nwb-manifest.json      what was exported, from which records, with every warning
+  validation-report.json / .txt    pynwb, NWB Inspector (DANDI configuration) and BrainSTEM checks
+  source/brainstem-export-….json.gz   the export the files were built from (when fetched)
+```
+
+Other commands:
+- **Dry run:** `brainstem-nwb manifest --export-file … -o manifest.json` writes only the manifest, so you can inspect what would be written.
+- **One session:** `brainstem-nwb export … --session NAME`.
+- **Validate existing files:** `brainstem-nwb validate ./nwb/`.
+
+Some NWB fields are not in a BrainSTEM export, so they are never guessed: experimenter, lab and institution. Supply them in a configuration file:
+
+```toml
+# brainstem-nwb.toml  (use with --config brainstem-nwb.toml)
+[nwbfile]
+experimenter = ["Last, First"]
+lab = "My Lab"
+institution = "My University"
+```
+
+A fully reproducible example is in [examples/synthetic-ephys](examples/synthetic-ephys/).
+
+## Principles
+
+- **Nothing is invented.** No time series, electrode positions or other observations are generated from metadata. Fields BrainSTEM does not have stay empty, and the manifest records why. Each value is marked present, missing, unknown, not applicable or not exported.
+- **Nothing is silently dropped.** Every field of every exported record is accounted for:
+  - mapped to NWB,
+  - used to link records,
+  - or kept in the manifest, with the reason.
+
+  The [mapping specification](docs/mapping-spec.md) is generated from the code.
+- **Traceable.** Every file carries a provenance block naming its BrainSTEM records, the exporter and mapping versions, and the checksum of the source export. Its NWB identifier is derived from the content, so re-exporting unchanged records gives the same identifier.
+- **Core NWB by default.** With `--extension brainstem`, every BrainSTEM record is also stored losslessly in the file as `ndx-brainstem` tables. See [extensions](docs/extensions.md).
+
+## Documentation
+
+- [Installation and authentication](docs/install.md)
+- [Quick start and CLI](docs/cli.md) · [Python API](docs/python-api.md)
+- [Mapping specification](docs/mapping-spec.md) · [Warning codes](docs/warnings.md)
+- [Conversion manifest](docs/manifest.md) · [Validation](docs/validation.md)
+- [Supported modalities](docs/modalities.md) · [Troubleshooting](docs/troubleshooting.md)
+- [Architecture](docs/design/architecture.md) · [Design decisions](docs/design/decisions.md)
+- [Contributing](CONTRIBUTING.md) · [Changelog](CHANGELOG.md)
+
+## License
+
+The code is released under the [MIT License](LICENSE).
+
+The BrainSTEM data model and its JSON schemas are not part of this package and are not covered by that
+license: they are © BrainSTEM, all rights reserved ([terms](https://www.brainstem.org/reference/license.md)).
+brainstem-nwb reads them from the BrainSTEM server as an API client. `--embed-schemas` embeds the schemas of a
+file's records unchanged, as a BrainSTEM export does, with BrainSTEM's notice next to them.

brainstem_nwb-0.1.0/README.md

@@ -0,0 +1,116 @@
+# brainstem-nwb
+
+Export experimental metadata from [BrainSTEM](https://www.brainstem.org) to
+[Neurodata Without Borders (NWB)](https://www.nwb.org) files.
+
+`brainstem-nwb` reads a BrainSTEM project export and writes one NWB file per session and subject,
+following NWB and DANDI conventions. Each file holds what BrainSTEM knows about the experiment:
+- session and subject
+- surgeries, viruses and drugs
+- devices, electrode groups and optogenetic sites
+- acquisitions, behaviors, manipulations and epochs
+
+Alongside the files it writes a versioned **conversion manifest** and a **validation report**.
+
+This release exports **metadata only**. Measured data (recordings, video, imaging) is not converted
+yet; the files describe it and point to it. Raw-data conversion with
+[NeuroConv](https://github.com/catalystneuro/neuroconv), starting with flat binary extracellular
+recordings, is designed and planned (see [docs/design/phase4-raw-data.md](docs/design/phase4-raw-data.md)).
+
+## Install
+
+```bash
+pip install brainstem-nwb
+# the development version:
+pip install git+https://github.com/brainstem-org/brainstem-nwb
+```
+
+Python 3.10 or newer. brainstem-nwb reads the BrainSTEM data model from the server at run time, so the
+first export needs network access to www.brainstem.org (or your `--url`); the model is then cached (see
+[installation](docs/install.md#the-brainstem-data-model)). Installing also installs [pynwb](https://github.com/NeurodataWithoutBorders/pynwb),
+[NWB Inspector](https://github.com/NeurodataWithoutBorders/nwbinspector) and the
+[BrainSTEM Python API tools](https://github.com/brainstem-org/brainstem_python_api_tools).
+
+## Quick start
+
+From a project export downloaded in BrainSTEM (Project → Download → JSON):
+
+```bash
+brainstem-nwb export --export-file brainstem-export.json.gz --output ./nwb/
+```
+
+Or fetch the export directly (needs read access to the project):
+
+```bash
+brainstem login                     # once, from the BrainSTEM API tools
+brainstem-nwb export --project PROJECT_UUID --output ./nwb/
+brainstem-nwb export --session SESSION_UUID --output ./nwb/   # one session's export
+```
+
+From Python:
+
+```python
+from brainstem_nwb import export_project
+
+result = export_project(project_id="PROJECT_UUID", output_dir="./nwb")
+# or: export_project(export_file="brainstem-export.json.gz", output_dir="./nwb")
+print(result.exit_code, result.manifest.summary)
+```
+
+The output directory then contains:
+
+```
+nwb/
+  sub-<subject>/sub-<subject>_ses-<session>.nwb
+  brainstem-nwb-manifest.json      what was exported, from which records, with every warning
+  validation-report.json / .txt    pynwb, NWB Inspector (DANDI configuration) and BrainSTEM checks
+  source/brainstem-export-….json.gz   the export the files were built from (when fetched)
+```
+
+Other commands:
+- **Dry run:** `brainstem-nwb manifest --export-file … -o manifest.json` writes only the manifest, so you can inspect what would be written.
+- **One session:** `brainstem-nwb export … --session NAME`.
+- **Validate existing files:** `brainstem-nwb validate ./nwb/`.
+
+Some NWB fields are not in a BrainSTEM export, so they are never guessed: experimenter, lab and institution. Supply them in a configuration file:
+
+```toml
+# brainstem-nwb.toml  (use with --config brainstem-nwb.toml)
+[nwbfile]
+experimenter = ["Last, First"]
+lab = "My Lab"
+institution = "My University"
+```
+
+A fully reproducible example is in [examples/synthetic-ephys](examples/synthetic-ephys/).
+
+## Principles
+
+- **Nothing is invented.** No time series, electrode positions or other observations are generated from metadata. Fields BrainSTEM does not have stay empty, and the manifest records why. Each value is marked present, missing, unknown, not applicable or not exported.
+- **Nothing is silently dropped.** Every field of every exported record is accounted for:
+  - mapped to NWB,
+  - used to link records,
+  - or kept in the manifest, with the reason.
+
+  The [mapping specification](docs/mapping-spec.md) is generated from the code.
+- **Traceable.** Every file carries a provenance block naming its BrainSTEM records, the exporter and mapping versions, and the checksum of the source export. Its NWB identifier is derived from the content, so re-exporting unchanged records gives the same identifier.
+- **Core NWB by default.** With `--extension brainstem`, every BrainSTEM record is also stored losslessly in the file as `ndx-brainstem` tables. See [extensions](docs/extensions.md).
+
+## Documentation
+
+- [Installation and authentication](docs/install.md)
+- [Quick start and CLI](docs/cli.md) · [Python API](docs/python-api.md)
+- [Mapping specification](docs/mapping-spec.md) · [Warning codes](docs/warnings.md)
+- [Conversion manifest](docs/manifest.md) · [Validation](docs/validation.md)
+- [Supported modalities](docs/modalities.md) · [Troubleshooting](docs/troubleshooting.md)
+- [Architecture](docs/design/architecture.md) · [Design decisions](docs/design/decisions.md)
+- [Contributing](CONTRIBUTING.md) · [Changelog](CHANGELOG.md)
+
+## License
+
+The code is released under the [MIT License](LICENSE).
+
+The BrainSTEM data model and its JSON schemas are not part of this package and are not covered by that
+license: they are © BrainSTEM, all rights reserved ([terms](https://www.brainstem.org/reference/license.md)).
+brainstem-nwb reads them from the BrainSTEM server as an API client. `--embed-schemas` embeds the schemas of a
+file's records unchanged, as a BrainSTEM export does, with BrainSTEM's notice next to them.

brainstem_nwb-0.1.0/docs/cli.md

@@ -0,0 +1,115 @@
+# Command-line interface
+
+This page is the reference for every `brainstem-nwb` command and option. `brainstem-nwb --help` and `brainstem-nwb COMMAND --help` show the same options.
+
+## `export`
+
+Converts a whole project, or some of its sessions, into metadata-only NWB files.
+
+```bash
+brainstem-nwb export (--project UUID | --export-file FILE) --output DIR [options]
+```
+
+### Where the records come from
+
+| Option | Meaning |
+|---|---|
+| `--project UUID` | Fetch the project's export from BrainSTEM |
+| `--export-file FILE` | Use a downloaded export (`.json` or `.json.gz`, project or session export) |
+| `--session ID\|NAME` | Export only this session; repeat for more. A single `--session UUID` without `--project` or `--export-file` fetches that session's export |
+| `--token`, `--url`, `--headless`, `--no-device-flow`, `--no-background` | Fetching and authentication; see [install.md](install.md) |
+| `--no-save-export` | Do not keep a copy of the fetched export in `DIR/source/` |
+
+### What is written
+
+| Option | Meaning |
+|---|---|
+| `--config FILE.toml` | Experimenter, lab, institution and subject options ([configuration](#configuration)) |
+| `--dry-run` | Write only `brainstem-nwb-manifest.json` |
+| `--allow-no-subject` | Export sessions that have no subject, without an NWB Subject |
+| `--no-infer-subjects` | When a session lists no subjects, do not take them from its behaviors, procedures and interfaces |
+| `--overwrite` | Replace existing files whose content differs. Files that are identical are kept and reported as `up_to_date` |
+| `--reproducible` | Use the export time as the creation time, so reruns give identical manifests |
+| `--extension brainstem\|localization` | Also write NWB extensions: every BrainSTEM record as `ndx-brainstem` tables, and coordinates as `ndx-anatomical-localization` tables. Repeatable. See [extensions](extensions.md) |
+| `--embed-schemas` | With `--extension brainstem`: fetch the BrainSTEM JSON schemas the records use from the server's public schema service (`--url`) and embed them |
+| `--herd` | Also write ontology terms (NCBITaxon, RRIDs, atlas regions) as a HERD zip next to each file |
+| `--fail-fast` | Stop at the first unit that fails. By default the remaining units are still exported |
+
+### Validation and output
+
+| Option | Meaning |
+|---|---|
+| `--no-validate` | Skip validation |
+| `--inspector-config dandi\|none` | NWB Inspector configuration. Default: `dandi` |
+| `--fail-on error\|warning` | Which validation severity gives exit code 4 |
+| `--json` | Print the summary as JSON |
+
+## `manifest`
+
+The same as `export --dry-run`, but the manifest can be written to a single named file:
+
+```bash
+brainstem-nwb manifest --export-file export.json -o manifest.json
+```
+
+## `validate`
+
+Validates NWB files, or a directory of them. When the files are in an export directory, they are also checked against the manifest there.
+
+```bash
+brainstem-nwb validate ./nwb/ [--manifest FILE] [--report-dir DIR] [--inspector-config dandi|none] [--fail-on warning]
+```
+
+## `fetch`
+
+Downloads a project or session export to a file, so you can convert it later or offline.
+
+```bash
+brainstem-nwb fetch --project UUID -o export.json.gz
+brainstem-nwb fetch --session UUID -o session-export.json.gz
+```
+
+## `data-model`
+
+```bash
+brainstem-nwb data-model [--url URL] [--refresh]
+```
+
+Reads the BrainSTEM data model from the server (or the fresh cached copy), caches it, and shows the server, the data model generation and whether this release supports it, the export format, its checksum and the cache directory. Run it once before working offline. See [the BrainSTEM data model](install.md#the-brainstem-data-model).
+
+## `mapping` and `schema`
+
+| Command | Prints |
+|---|---|
+| `brainstem-nwb mapping` | The mapping specification (Markdown). `--format json` prints the rules instead; `--warnings` prints the warning codes |
+| `brainstem-nwb schema manifest` | The JSON Schema of the manifest |
+| `brainstem-nwb schema report` | The JSON Schema of the validation report |
+
+## Exit codes
+
+| Code | Meaning |
+|---|---|
+| 0 | Everything was exported and validated without errors |
+| 1 | Partial export: some sessions could not be exported, or some files failed (see the manifest) |
+| 2 | Usage error, invalid configuration, or unreadable or unsupported export file |
+| 3 | Authentication or retrieval failure, or the data model could not be read and is not cached |
+| 4 | Files were written, but validation found errors (or warnings, with `--fail-on warning`) |
+
+If several apply, the highest code is returned.
+
+## Configuration
+
+Some NWB fields have no source in a BrainSTEM export. They can be set in a TOML file:
+
+```toml
+[nwbfile]
+experimenter = ["Last, First", "Other, Person"]
+experimenter_source = "config"    # or "session_creator": the user who created the session record, when no
+                                  # experimenter is configured (BrainSTEM records who created a session, not who ran it)
+lab = "My Lab"
+institution = "My University"
+
+[subject]
+label_field = "name"              # or "name_used_in_storage": used for subject_id and file names
+species_format = "binomial"       # or "ncbitaxon": NCBI Taxonomy IRI, preferred by DANDI
+```

brainstem_nwb-0.1.0/docs/design/architecture.md

@@ -0,0 +1,51 @@
+# Architecture
+
+This page is for contributors and for anyone embedding the exporter. It explains how `brainstem-nwb` is organized and how data moves through it. To change a mapping, see [Contributing](https://github.com/brainstem-org/brainstem-nwb/blob/main/CONTRIBUTING.md); the reasons behind the main choices are in [decisions.md](decisions.md).
+
+## Layers
+
+The exporter has four layers, and each knows only its neighbours:
+
+```
+retrieval ─► normalization ─► manifest (NwbPlan) ─► NWB writing ─► validation
+```
+
+| Layer | Package | Input → output | Knows about |
+|---|---|---|---|
+| Retrieval | `retrieval/` | project id or file → `ProjectExport` | the BrainSTEM API tools, the export format |
+| Normalization | `store/`, `normalize/` | `ProjectExport` → `RecordStore`, `ExportUnit`s | BrainSTEM records and links |
+| Mapping | `mapping/` | `ExportUnit` → manifest `Unit` with an `NwbPlan` | BrainSTEM and NWB concepts, not HTTP or HDF5 |
+| NWB writing | `nwb/` | `NwbPlan` → `.nwb` file | pynwb only, not BrainSTEM |
+| Validation | `validation/` | `.nwb` (+ manifest unit) → report | pynwb, NWB Inspector, the manifest |
+
+`export.py` puts the layers together. `cli.py` and the functions in `__init__.py` are thin wrappers around it.
+
+## Data flow
+
+1. **Retrieval.** `load_export()` reads a project export (format 2.x) from a file or a dict. `fetch_export()` gets one through `BrainstemClient.export_project()`. The export is the **only** source of BrainSTEM records. This keeps an export reproducible: the export file, the exporter version and the BrainSTEM data model (whose checksum the manifest records) determine the result.
+2. **Record store.** `RecordStore` indexes records by model label and primary key, and builds a reverse index of every foreign key and many-to-many link. The export does not record which model a link points to. That comes from the **inventory**: the BrainSTEM data model, read at run time from the server's reference documents (`store/datamodel.py`) and cached per server; brainstem-nwb ships no copy of it. A link to a record that is not in the export resolves to `Ref(reason="not_in_export")`; it is not an error.
+3. **Units.** `plan_units()` decides which files to write. An NWB file holds one subject, so each session gives one unit per subject. Every record of the session is attributed to the units it belongs to, using documented rules. Each decision is recorded as an `Attribution`, and sessions that cannot be exported become `unit_errors`.
+4. **Mapping.** `map_unit()` runs the mapper modules in order: session, project, people, protocols, subject, procedures, manipulations, acquisitions, devices, electrode groups, optogenetic sites, epochs and notes. Each mapper does two things:
+   - fills the `NwbPlan`;
+   - records every intended destination as a `FieldMapping` with a value state, and emits coded warnings.
+
+   Mapper functions declare their `MappingRule`s with `@rules(...)`. The registry generates the mapping specification and backs the coverage test.
+5. **Manifest.** The per-unit results, the source description, coverage and summary make up the `Manifest`. It is written as canonical JSON (`brainstem-nwb-manifest.json`), and a dry run stops here.
+6. **Writing.** `build_nwbfile(plan)` builds a pynwb `NWBFile` from the plan alone. `write_nwbfile()` writes it to a temporary name and moves it into place.
+7. **Validation.** Each file goes through `pynwb.validate`, NWB Inspector (DANDI configuration by default) and the BrainSTEM semantic checks, which compare the file with its manifest unit. Results go to `validation-report.json` and `.txt`.
+
+## Identity and reproducibility
+
+- The NWB `identifier` is `uuid5(namespace, session id | subject id | mapping version | sha256(plan content))`.
+  - It is stable across re-exports of unchanged records.
+  - It changes when any written content changes.
+  - It ignores when the export was made.
+- `--reproducible` (or `SOURCE_DATE_EPOCH`) takes the creation times from the export, which makes the manifest byte-identical across runs.
+- HDF5 files themselves are not byte-identical (HDF5 object ids are random). Reproducibility tests therefore compare a canonical summary of each file (`nwb/canonical.py`).
+
+## Extension points
+
+- **New mapping.** Add a mapper function with `@rules(...)` and call it from `mapping/pipeline.py`. The coverage test lists the BrainSTEM fields still unaccounted for.
+- **New BrainSTEM fields or models.** The data model comes from the server, so new fields appear on their own; `tests/test_spec.py` lists the ones the mapping does not account for yet. New record types go into `EXPORT_LABELS`.
+- **Raw data (Phase 4).** Implement `raw.base.RawDataConverter` and register it, either built in or through the `brainstem_nwb.raw_converters` entry-point group. See [phase4-raw-data.md](phase4-raw-data.md).
+- **Web integration (Phase 5).** The server can call `build_manifest(export_dict)` and `export_project(export=export_dict, ...)` directly with the payload of `build_project_export`, with no HTTP involved.