aind-ibl-ephys-alignment-preprocessing 0.5.2__tar.gz → 0.7.0__tar.gz

aind_ibl_ephys_alignment_preprocessing-0.7.0/AGENTS.md

@@ -0,0 +1,7 @@
+# Agent Instructions
+
+Read [CONTRIBUTING.md](CONTRIBUTING.md) before making changes. It is the
+shared guide for development commands, architecture, and validation.
+
+Keep shared development guidance in CONTRIBUTING.md and agent-specific
+instructions here. Keep CLAUDE.md as a pointer to this file.

{aind_ibl_ephys_alignment_preprocessing-0.5.2 → aind_ibl_ephys_alignment_preprocessing-0.7.0}/CHANGELOG.md

@@ -5,6 +5,34 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## v0.7.0 (2026-10-23)
+
+### BREAKING CHANGE
+
+- registration_asset directories must carry
+ls_to_template_transform_information.json or transform_information.json, and
+template_to_ccf_asset requires a registration_asset.
+
+### Feat
+
+- write template_to_ccf_asset from the manifest mapping
+
+### Fix
+
+- refuse override transforms whose frame or template is unverified
+- keep per-brain override columns on the rows discover keeps
+
+## v0.6.0 (2026-10-22)
+
+### Feat
+
+- add per-brain template_to_ccf_asset manifest column
+- add manifest builder
+
+### Fix
+
+- match override transforms through the S3 cache prefix
+
 ## v0.5.2 (2026-09-30)
 
 ### Fix

aind_ibl_ephys_alignment_preprocessing-0.7.0/CLAUDE.md

@@ -0,0 +1,3 @@
+# Claude Instructions
+
+Read and follow [AGENTS.md](AGENTS.md).

aind_ibl_ephys_alignment_preprocessing-0.5.2/CLAUDE.md → aind_ibl_ephys_alignment_preprocessing-0.7.0/CONTRIBUTING.md

@@ -1,6 +1,7 @@
-# CLAUDE.md
+# Contributing
 
-This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+This is the main development guide for human contributors and coding agents.
+Agent-specific instructions are in [AGENTS.md](AGENTS.md).
 
 ## Commands
 
@@ -42,6 +43,12 @@ Always use `uv run` to execute commands, `uv add` to add dependencies, and `uv s
 
 ## Architecture
 
+Before changing datapackage fields, asset references, output layout, or
+channel/shank joins, read
+[the datapackage and external asset contract](docs/external_asset_references.md).
+Keep machine-specific locations in resolver configuration and preserve the
+convention-loaded ephys collection contents as well as explicit references.
+
 This is a Python package using a `src/` layout. Source code lives in `src/aind_ibl_ephys_alignment_preprocessing/`, tests in `tests/`.
 
 - Build system: hatchling
@@ -72,7 +79,7 @@ src/aind_ibl_ephys_alignment_preprocessing/
   probes.py            # Per-probe annotation loading and coordinate conversion
   ephys.py             # Sync ephys extraction wrapper (delegates to aind-ephys-ibl-gui-conversion)
   pipeline.py          # Sync orchestrator: run_pipeline()
-  manifest.py          # Output datapackage.json generation (DataPackage model, schema v1.0.0)
+  datapackage.py       # Output datapackage.json generation, asset references, schema 4.1.0
   scripts/run.py       # CLI entry point: aind-ibl-preprocess
   _async/
     concurrency.py     # IO lane throttling (per-path semaphores), EphysCoordinator, thread helpers

{aind_ibl_ephys_alignment_preprocessing-0.5.2 → aind_ibl_ephys_alignment_preprocessing-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: aind-ibl-ephys-alignment-preprocessing
-Version: 0.5.2
+Version: 0.7.0
 Summary: Library to prepare histology and ephys for the IBL ephys alignment GUI
 Project-URL: Repository, https://github.com/AllenNeuralDynamics/aind-ibl-ephys-alignment-preprocessing/
 Project-URL: Issues, https://github.com/AllenNeuralDynamics/aind-ibl-ephys-alignment-preprocessing/issues
@@ -19,6 +19,7 @@ Requires-Dist: aind-registration-utils>=0.4
 Requires-Dist: aind-s3-cache>=0.1
 Requires-Dist: aind-zarr-utils>=0.11.2
 Requires-Dist: antspyx>=0.3.0
+Requires-Dist: boto3
 Requires-Dist: filelock
 Requires-Dist: iblatlas>=0.4.0
 Requires-Dist: numpy>=1.21.0
@@ -256,7 +257,8 @@ before a column existed stays valid.
 | `probe_name` | *legacy* | Alias for `ephys_collection`. |
 | `probe_shank` | *legacy* | Alias for both `histology_shank` and `ephys_shank`. |
 | `surface_finding` | *null* | Path (relative to `data_root`) to a surface-finding file. |
-| `registration_asset` | *null* | Path (relative to `data_root`) to a registration directory holding the `ls_to_template_SyN_*` transforms to use **instead of** the stitched asset's own -- e.g. `SmartSPIM_750108_reg/ccf_Ex_639_Em_667`. Per-*brain*: replicated across rows like `mouseid`, and rows must agree. The registration channel is taken from the directory name (a leading `ccf_` is stripped), so the histology volume is built from the channel the transforms were computed from rather than the one `processing.json` names. |
+| `registration_asset` | *null* | Path relative to `data_root` to a replacement registration directory, e.g. `SmartSPIM_750108_reg/ccf_Ex_639_Em_667`. One value per brain; all nonblank rows must agree. See [Separate registration asset](#separate-registration-asset) for its contents and coordinate requirements. |
+| `template_to_ccf_asset` | *null* | Path relative to `data_root` to the template-to-CCF registration of the template `registration_asset` was registered to, when that is not the SmartSPIM LCA template. One value per brain, like `registration_asset`. See [Separate template-to-CCF registration](#separate-template-to-ccf-registration). |
 
 **Constraints:**
 - All rows must have the same `mouseid`.
@@ -281,6 +283,52 @@ mouse001,2024-06-01_rec_sorted,track_annotations_probeB_shank1,B0001,probeB,Prob
 mouse001,2024-06-01_rec_sorted,track_annotations_quad_shank3,Q0001,quad0,ProbeD,3,0
 ```
 
+#### Building a manifest from a mapping
+
+Most of a manifest repeats or follows from two choices: which sorted recording each session uses,
+and which stream each annotated track was recorded on. `aind-ibl-build-manifest` takes only those,
+in a TOML file, and derives the rest. Track names are never parsed, because annotators follow no
+naming convention.
+
+```toml
+mouseid = "837772"
+annotation = "837772-probe-annotations-CK-2026-09-29.json"  # relative to this file
+# registration_asset = "..."                                  # optional, per brain
+# template_to_ccf_asset = "..."                               # optional, per brain
+
+[[session]]
+sorted_recording = "ecephys_837772_2026-06-27_14-08-21_sorted_2026-08-27_18-59-49"
+# surface_finding = "..."                                     # optional
+
+[session.tracks]
+"732_MD_1" = "46101-1"   # one shank of a split quad-base probe
+"733_BLA" = "46810"      # a single-stream probe
+"740_PL" = { stream = "45883", shank = 1 }  # one shank of a single-stream multi-shank probe; the first shank is 1
+```
+
+```bash
+aind-ibl-build-manifest 837772.toml -o manifest_837772.csv [--data-root /data]
+```
+
+A stream is the collection parsed from the sort's `postprocessed/` analyzer names. They are listed
+from a copy mounted under `--data-root`, or else unsigned from that sort's prefix in
+`aind-open-data`. How the columns are derived:
+
+- `probe_file` is the annotation file's name without its extension.
+- **Split quad-base probe:** a stream `<probe>-<N>`, when the sort also holds another
+  `<probe>-<M>`, gets `logical_probe = <probe>`, `histology_shank = N - 1` and `ephys_shank = 0`.
+- **Any other stream** is its own `logical_probe`. Its shanks stay blank, or both take the given
+  shank's index.
+
+The build writes nothing if any of these fail, and it reports every failure:
+- every stream in each sort has a track, and every assigned stream is in its sort;
+- every track is a layer of the annotation file with at least two points;
+- no track is assigned twice;
+- a split stream has exactly one track, and the tracks sharing any other stream have distinct
+  shanks.
+
+Annotated layers left out of the mapping are printed as notes, so a missed track gets noticed.
+
 ### Neuroglancer JSON
 
 A Neuroglancer state JSON file that contains image source URIs pointing to the
@@ -291,7 +339,136 @@ and any additional channels.
 The asset directory must contain:
 - `image_tile_fusing/OMEZarr/<channel>.zarr/` -- fused OME-Zarr volumes
 - `image_atlas_alignment/<registration_channel_stem>/` -- ANTs registration
-  outputs: the light-sheet to template affine and inverse warp
+  outputs: the light-sheet to template affine and inverse warp (or supply a
+  separate registration asset as described below)
+
+### Separate registration asset
+
+Set the manifest's optional `registration_asset` column to the transform
+directory relative to `data_root`:
+
+```csv
+mouseid,sorted_recording,probe_file,histology_track_id,ephys_collection,registration_asset
+750108,recording_sorted,probeA,A0001,ProbeA,SmartSPIM_750108_reg/ccf_Ex_639_Em_667
+```
+
+Use one path per brain; conflicting nonblank values fail. An absent or blank
+column uses the stitched asset's registration.
+
+The directory must contain `ls_to_template_SyN_0GenericAffine.mat`,
+`ls_to_template_SyN_1InverseWarp.nii.gz` and a geometry sidecar (below): ANTs
+SyN outputs registering this brain's selected channel to the SmartSPIM LCA
+template, or to another template whose CCF registration
+`template_to_ccf_asset` (below) supplies.
+
+The final directory name selects the channel after stripping a leading
+`ccf_`. For this example, the stitched asset must contain
+`image_tile_fusing/OMEZarr/Ex_639_Em_667.zarr/`. Images and metadata come
+from the stitched asset; the override needs only transforms and their sidecar.
+
+Every separate registration needs a geometry sidecar beside the transforms;
+a directory without one fails. Preprocessing checks
+`ls_to_template_transform_information.json`, then `transform_information.json`.
+Its `moving_domain` must describe the actual registration input in LPS
+millimeters and match the selected volume's native or pipeline-anchored frame.
+Its `fixed_domain` must match the template the template-to-CCF chain starts
+from: the SmartSPIM LCA template unless `template_to_ccf_asset` is set. Missing
+or incompatible domains fail, since a wrong frame or template still puts the
+probes somewhere plausible.
+
+**QC:** Additional-channel CCF volumes (`emit_qc=True`) also need
+`ls_to_template_SyN_1Warp.nii.gz` in the override directory; without it the
+QC warp fails rather than falling back to the stitched asset's registration.
+
+#### Generate the sidecar from ANTs outputs
+
+Use the actual fixed template and moving histology images passed to ANTs,
+including any preprocessing of their geometry. Transform files alone cannot
+recover these domains. Headers must be 3D, axis-aligned, and in millimeters.
+
+Run `uv sync` in the preprocessing library checkout, then save this as
+`generate_registration_sidecar.py`:
+
+```python
+import sys
+from pathlib import Path
+
+import SimpleITK as sitk
+from aind_ants_transform_sidecar import dump_package, load_package
+from aind_registration_utils.domains import ImageHeader, make_syn_sidecar
+
+
+def read_header(path):
+    reader = sitk.ImageFileReader()
+    reader.SetFileName(str(path))
+    reader.ReadImageInformation()  # Read geometry without loading voxel data.
+    if reader.GetDimension() != 3:
+        raise ValueError("Registration inputs must be 3D")
+    return ImageHeader(
+        origin=reader.GetOrigin(),
+        spacing=reader.GetSpacing(),
+        direction=reader.GetDirection(),
+        size=reader.GetSize(),
+    )
+
+
+registration_dir, fixed_image, moving_image = map(Path, sys.argv[1:])
+sidecar = make_syn_sidecar(
+    fixed_image_header=read_header(fixed_image),
+    moving_image_header=read_header(moving_image),
+    affine="ls_to_template_SyN_0GenericAffine.mat",
+    warp="ls_to_template_SyN_1Warp.nii.gz",
+    inverse_warp="ls_to_template_SyN_1InverseWarp.nii.gz",
+)
+text = dump_package(sidecar)
+load_package(text)  # Validate the serialized package.
+destination = registration_dir / "ls_to_template_transform_information.json"
+destination.write_text(text + "\n")
+print(destination)
+```
+
+Pass the transform directory, fixed image, and moving image in that order:
+
+```bash
+uv run python generate_registration_sidecar.py \
+    /path/to/SmartSPIM_750108_reg/ccf_Ex_639_Em_667 \
+    /path/to/fixed_template_used_by_ants.nii.gz \
+    /path/to/moving_histology_used_by_ants.nii.gz
+```
+
+The script reads only image headers. Transform filenames are relative to the
+sidecar; rename raw ANTs outputs to the expected names. The schema requires a
+forward-warp filename, but normal preprocessing needs only the affine and
+inverse warp. The helper does not verify transform files.
+
+Generate the JSON beside the transforms before uploading the asset;
+Code Ocean's `/data` mounts are read-only.
+
+### Separate template-to-CCF registration
+
+A brain registered to a template other than the SmartSPIM LCA template (an
+iDISCO template, say) needs that template's CCF registration. Set the optional
+`template_to_ccf_asset` column to its directory relative to `data_root`, one
+path per brain like `registration_asset`:
+
+```csv
+mouseid,sorted_recording,probe_file,histology_track_id,ephys_collection,registration_asset,template_to_ccf_asset
+750107,recording_sorted,probeA,A0001,ProbeA,idisco_reg/750107,idisco_template_2026-10/template_to_ccf
+```
+
+The directory must contain `syn_0GenericAffine.mat` and
+`syn_1InverseWarp.nii.gz`, ANTs SyN outputs with CCF fixed and the template
+moving; `emit_qc=True` also needs `syn_1Warp.nii.gz`. They replace the
+pipeline's `spim_template_to_ccf_syn_*_25` transforms in the probe and QC
+chains, and the datapackage refers to them under the `template_to_ccf` asset,
+named after the directory's first component under `data_root`.
+
+The column requires a `registration_asset` with a sidecar, whose `fixed_domain`
+is checked against this template. If the directory ships its own sidecar
+(`template_to_ccf_transform_information.json` or `transform_information.json`),
+the registration's `fixed_domain` must match its `moving_domain`. Without one,
+the registration's `fixed_domain` must *not* match the SmartSPIM LCA template;
+to replace the LCA template's own CCF registration, ship the sidecar.
 
 ### Reference volumes
 
@@ -368,9 +545,13 @@ results_root/
             +-- band_corr/                    # Full matrices + row_channels.json
 ```
 
-The `datapackage.json` file contains a structured manifest of all outputs,
-including transform chain paths, histology volume paths, and per-probe
-metadata. It can be loaded back with:
+The `datapackage.json` file describes the mouse, recordings, ephys collections,
+transform chain, histology, channel tables, and track mappings. It references
+each ephys directory; individual spike/QC files and `band_corr/` contents are
+loaded by convention rather than enumerated in the JSON. See
+[the datapackage contract](docs/external_asset_references.md) for asset
+descriptions versus runtime location resolution, Code Ocean/desktop examples,
+shank joins, and validation limits. The model can be loaded back with:
 
 ```python
 from aind_ibl_ephys_alignment_preprocessing.datapackage import load_datapackage
@@ -395,7 +576,9 @@ The pipeline performs the following steps:
    `xyz_picks.json` files for the alignment GUI.
 5. **Ephys extraction** (optional) -- Convert spike-sorted data into IBL ALF
    format.
-6. **Manifest generation** -- Write `datapackage.json` summarizing all outputs.
+6. **Manifest generation** -- Write `datapackage.json` describing explicit
+   references and ephys collection directories; preserve their conventional
+   contents, including QC files not individually listed in the JSON.
 
 ## Adapting to your own data
 
@@ -423,31 +606,8 @@ origin at bregma, in meters.
 
 ## Development
 
-To develop the code, run:
-```bash
-uv sync
-```
-
-Please test your changes using the full linting and testing suite:
-
-```bash
-./scripts/run_linters_and_checks.sh -c
-```
-
-Or run individual commands:
-```bash
-uv run --frozen ruff format          # Code formatting
-uv run --frozen ruff check           # Linting
-uv run --frozen mypy                 # Type checking
-uv run --frozen interrogate -v src   # Documentation coverage
-uv run --frozen codespell --check-filenames  # Spell checking
-uv run --frozen pytest --cov aind_ibl_ephys_alignment_preprocessing  # Tests with coverage
-```
-
-### Documentation
-```bash
-sphinx-build -b html docs/source/ docs/build/html
-```
+See [CONTRIBUTING.md](CONTRIBUTING.md) for developer setup, checks, and
+repository conventions. Coding agents should also read [AGENTS.md](AGENTS.md).
 
 ## License
 

{aind_ibl_ephys_alignment_preprocessing-0.5.2 → aind_ibl_ephys_alignment_preprocessing-0.7.0}/README.md

@@ -227,7 +227,8 @@ before a column existed stays valid.
 | `probe_name` | *legacy* | Alias for `ephys_collection`. |
 | `probe_shank` | *legacy* | Alias for both `histology_shank` and `ephys_shank`. |
 | `surface_finding` | *null* | Path (relative to `data_root`) to a surface-finding file. |
-| `registration_asset` | *null* | Path (relative to `data_root`) to a registration directory holding the `ls_to_template_SyN_*` transforms to use **instead of** the stitched asset's own -- e.g. `SmartSPIM_750108_reg/ccf_Ex_639_Em_667`. Per-*brain*: replicated across rows like `mouseid`, and rows must agree. The registration channel is taken from the directory name (a leading `ccf_` is stripped), so the histology volume is built from the channel the transforms were computed from rather than the one `processing.json` names. |
+| `registration_asset` | *null* | Path relative to `data_root` to a replacement registration directory, e.g. `SmartSPIM_750108_reg/ccf_Ex_639_Em_667`. One value per brain; all nonblank rows must agree. See [Separate registration asset](#separate-registration-asset) for its contents and coordinate requirements. |
+| `template_to_ccf_asset` | *null* | Path relative to `data_root` to the template-to-CCF registration of the template `registration_asset` was registered to, when that is not the SmartSPIM LCA template. One value per brain, like `registration_asset`. See [Separate template-to-CCF registration](#separate-template-to-ccf-registration). |
 
 **Constraints:**
 - All rows must have the same `mouseid`.
@@ -252,6 +253,52 @@ mouse001,2024-06-01_rec_sorted,track_annotations_probeB_shank1,B0001,probeB,Prob
 mouse001,2024-06-01_rec_sorted,track_annotations_quad_shank3,Q0001,quad0,ProbeD,3,0
 ```
 
+#### Building a manifest from a mapping
+
+Most of a manifest repeats or follows from two choices: which sorted recording each session uses,
+and which stream each annotated track was recorded on. `aind-ibl-build-manifest` takes only those,
+in a TOML file, and derives the rest. Track names are never parsed, because annotators follow no
+naming convention.
+
+```toml
+mouseid = "837772"
+annotation = "837772-probe-annotations-CK-2026-09-29.json"  # relative to this file
+# registration_asset = "..."                                  # optional, per brain
+# template_to_ccf_asset = "..."                               # optional, per brain
+
+[[session]]
+sorted_recording = "ecephys_837772_2026-06-27_14-08-21_sorted_2026-08-27_18-59-49"
+# surface_finding = "..."                                     # optional
+
+[session.tracks]
+"732_MD_1" = "46101-1"   # one shank of a split quad-base probe
+"733_BLA" = "46810"      # a single-stream probe
+"740_PL" = { stream = "45883", shank = 1 }  # one shank of a single-stream multi-shank probe; the first shank is 1
+```
+
+```bash
+aind-ibl-build-manifest 837772.toml -o manifest_837772.csv [--data-root /data]
+```
+
+A stream is the collection parsed from the sort's `postprocessed/` analyzer names. They are listed
+from a copy mounted under `--data-root`, or else unsigned from that sort's prefix in
+`aind-open-data`. How the columns are derived:
+
+- `probe_file` is the annotation file's name without its extension.
+- **Split quad-base probe:** a stream `<probe>-<N>`, when the sort also holds another
+  `<probe>-<M>`, gets `logical_probe = <probe>`, `histology_shank = N - 1` and `ephys_shank = 0`.
+- **Any other stream** is its own `logical_probe`. Its shanks stay blank, or both take the given
+  shank's index.
+
+The build writes nothing if any of these fail, and it reports every failure:
+- every stream in each sort has a track, and every assigned stream is in its sort;
+- every track is a layer of the annotation file with at least two points;
+- no track is assigned twice;
+- a split stream has exactly one track, and the tracks sharing any other stream have distinct
+  shanks.
+
+Annotated layers left out of the mapping are printed as notes, so a missed track gets noticed.
+
 ### Neuroglancer JSON
 
 A Neuroglancer state JSON file that contains image source URIs pointing to the
@@ -262,7 +309,136 @@ and any additional channels.
 The asset directory must contain:
 - `image_tile_fusing/OMEZarr/<channel>.zarr/` -- fused OME-Zarr volumes
 - `image_atlas_alignment/<registration_channel_stem>/` -- ANTs registration
-  outputs: the light-sheet to template affine and inverse warp
+  outputs: the light-sheet to template affine and inverse warp (or supply a
+  separate registration asset as described below)
+
+### Separate registration asset
+
+Set the manifest's optional `registration_asset` column to the transform
+directory relative to `data_root`:
+
+```csv
+mouseid,sorted_recording,probe_file,histology_track_id,ephys_collection,registration_asset
+750108,recording_sorted,probeA,A0001,ProbeA,SmartSPIM_750108_reg/ccf_Ex_639_Em_667
+```
+
+Use one path per brain; conflicting nonblank values fail. An absent or blank
+column uses the stitched asset's registration.
+
+The directory must contain `ls_to_template_SyN_0GenericAffine.mat`,
+`ls_to_template_SyN_1InverseWarp.nii.gz` and a geometry sidecar (below): ANTs
+SyN outputs registering this brain's selected channel to the SmartSPIM LCA
+template, or to another template whose CCF registration
+`template_to_ccf_asset` (below) supplies.
+
+The final directory name selects the channel after stripping a leading
+`ccf_`. For this example, the stitched asset must contain
+`image_tile_fusing/OMEZarr/Ex_639_Em_667.zarr/`. Images and metadata come
+from the stitched asset; the override needs only transforms and their sidecar.
+
+Every separate registration needs a geometry sidecar beside the transforms;
+a directory without one fails. Preprocessing checks
+`ls_to_template_transform_information.json`, then `transform_information.json`.
+Its `moving_domain` must describe the actual registration input in LPS
+millimeters and match the selected volume's native or pipeline-anchored frame.
+Its `fixed_domain` must match the template the template-to-CCF chain starts
+from: the SmartSPIM LCA template unless `template_to_ccf_asset` is set. Missing
+or incompatible domains fail, since a wrong frame or template still puts the
+probes somewhere plausible.
+
+**QC:** Additional-channel CCF volumes (`emit_qc=True`) also need
+`ls_to_template_SyN_1Warp.nii.gz` in the override directory; without it the
+QC warp fails rather than falling back to the stitched asset's registration.
+
+#### Generate the sidecar from ANTs outputs
+
+Use the actual fixed template and moving histology images passed to ANTs,
+including any preprocessing of their geometry. Transform files alone cannot
+recover these domains. Headers must be 3D, axis-aligned, and in millimeters.
+
+Run `uv sync` in the preprocessing library checkout, then save this as
+`generate_registration_sidecar.py`:
+
+```python
+import sys
+from pathlib import Path
+
+import SimpleITK as sitk
+from aind_ants_transform_sidecar import dump_package, load_package
+from aind_registration_utils.domains import ImageHeader, make_syn_sidecar
+
+
+def read_header(path):
+    reader = sitk.ImageFileReader()
+    reader.SetFileName(str(path))
+    reader.ReadImageInformation()  # Read geometry without loading voxel data.
+    if reader.GetDimension() != 3:
+        raise ValueError("Registration inputs must be 3D")
+    return ImageHeader(
+        origin=reader.GetOrigin(),
+        spacing=reader.GetSpacing(),
+        direction=reader.GetDirection(),
+        size=reader.GetSize(),
+    )
+
+
+registration_dir, fixed_image, moving_image = map(Path, sys.argv[1:])
+sidecar = make_syn_sidecar(
+    fixed_image_header=read_header(fixed_image),
+    moving_image_header=read_header(moving_image),
+    affine="ls_to_template_SyN_0GenericAffine.mat",
+    warp="ls_to_template_SyN_1Warp.nii.gz",
+    inverse_warp="ls_to_template_SyN_1InverseWarp.nii.gz",
+)
+text = dump_package(sidecar)
+load_package(text)  # Validate the serialized package.
+destination = registration_dir / "ls_to_template_transform_information.json"
+destination.write_text(text + "\n")
+print(destination)
+```
+
+Pass the transform directory, fixed image, and moving image in that order:
+
+```bash
+uv run python generate_registration_sidecar.py \
+    /path/to/SmartSPIM_750108_reg/ccf_Ex_639_Em_667 \
+    /path/to/fixed_template_used_by_ants.nii.gz \
+    /path/to/moving_histology_used_by_ants.nii.gz
+```
+
+The script reads only image headers. Transform filenames are relative to the
+sidecar; rename raw ANTs outputs to the expected names. The schema requires a
+forward-warp filename, but normal preprocessing needs only the affine and
+inverse warp. The helper does not verify transform files.
+
+Generate the JSON beside the transforms before uploading the asset;
+Code Ocean's `/data` mounts are read-only.
+
+### Separate template-to-CCF registration
+
+A brain registered to a template other than the SmartSPIM LCA template (an
+iDISCO template, say) needs that template's CCF registration. Set the optional
+`template_to_ccf_asset` column to its directory relative to `data_root`, one
+path per brain like `registration_asset`:
+
+```csv
+mouseid,sorted_recording,probe_file,histology_track_id,ephys_collection,registration_asset,template_to_ccf_asset
+750107,recording_sorted,probeA,A0001,ProbeA,idisco_reg/750107,idisco_template_2026-10/template_to_ccf
+```
+
+The directory must contain `syn_0GenericAffine.mat` and
+`syn_1InverseWarp.nii.gz`, ANTs SyN outputs with CCF fixed and the template
+moving; `emit_qc=True` also needs `syn_1Warp.nii.gz`. They replace the
+pipeline's `spim_template_to_ccf_syn_*_25` transforms in the probe and QC
+chains, and the datapackage refers to them under the `template_to_ccf` asset,
+named after the directory's first component under `data_root`.
+
+The column requires a `registration_asset` with a sidecar, whose `fixed_domain`
+is checked against this template. If the directory ships its own sidecar
+(`template_to_ccf_transform_information.json` or `transform_information.json`),
+the registration's `fixed_domain` must match its `moving_domain`. Without one,
+the registration's `fixed_domain` must *not* match the SmartSPIM LCA template;
+to replace the LCA template's own CCF registration, ship the sidecar.
 
 ### Reference volumes
 
@@ -339,9 +515,13 @@ results_root/
             +-- band_corr/                    # Full matrices + row_channels.json
 ```
 
-The `datapackage.json` file contains a structured manifest of all outputs,
-including transform chain paths, histology volume paths, and per-probe
-metadata. It can be loaded back with:
+The `datapackage.json` file describes the mouse, recordings, ephys collections,
+transform chain, histology, channel tables, and track mappings. It references
+each ephys directory; individual spike/QC files and `band_corr/` contents are
+loaded by convention rather than enumerated in the JSON. See
+[the datapackage contract](docs/external_asset_references.md) for asset
+descriptions versus runtime location resolution, Code Ocean/desktop examples,
+shank joins, and validation limits. The model can be loaded back with:
 
 ```python
 from aind_ibl_ephys_alignment_preprocessing.datapackage import load_datapackage
@@ -366,7 +546,9 @@ The pipeline performs the following steps:
    `xyz_picks.json` files for the alignment GUI.
 5. **Ephys extraction** (optional) -- Convert spike-sorted data into IBL ALF
    format.
-6. **Manifest generation** -- Write `datapackage.json` summarizing all outputs.
+6. **Manifest generation** -- Write `datapackage.json` describing explicit
+   references and ephys collection directories; preserve their conventional
+   contents, including QC files not individually listed in the JSON.
 
 ## Adapting to your own data
 
@@ -394,31 +576,8 @@ origin at bregma, in meters.
 
 ## Development
 
-To develop the code, run:
-```bash
-uv sync
-```
-
-Please test your changes using the full linting and testing suite:
-
-```bash
-./scripts/run_linters_and_checks.sh -c
-```
-
-Or run individual commands:
-```bash
-uv run --frozen ruff format          # Code formatting
-uv run --frozen ruff check           # Linting
-uv run --frozen mypy                 # Type checking
-uv run --frozen interrogate -v src   # Documentation coverage
-uv run --frozen codespell --check-filenames  # Spell checking
-uv run --frozen pytest --cov aind_ibl_ephys_alignment_preprocessing  # Tests with coverage
-```
-
-### Documentation
-```bash
-sphinx-build -b html docs/source/ docs/build/html
-```
+See [CONTRIBUTING.md](CONTRIBUTING.md) for developer setup, checks, and
+repository conventions. Coding agents should also read [AGENTS.md](AGENTS.md).
 
 ## License