standard-transform 1.4.1__tar.gz → 2.0.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (114) hide show
  1. standard_transform-2.0.0/.claude/settings.local.json +9 -0
  2. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/.bmv-post-commit.sh +2 -0
  3. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/.github/workflows/docs.yml +49 -0
  4. {standard_transform-1.4.1 → standard_transform-2.0.0/.claude/worktrees/laplace-streamlines}/.gitignore +4 -1
  5. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/CLAUDE.md +40 -0
  6. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/MANIFEST.in +3 -0
  7. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/README.md +87 -0
  8. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/docs/concepts/depth-and-radial-distance.md +54 -0
  9. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/docs/concepts/oriented-frame.md +75 -0
  10. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/docs/concepts/streamlines-vs-field.md +76 -0
  11. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/docs/concepts/versioning.md +84 -0
  12. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/docs/datasets.md +61 -0
  13. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/docs/guides/build-a-field.md +87 -0
  14. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/docs/guides/depth-radial.md +85 -0
  15. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/docs/guides/per-neuron-streamline.md +62 -0
  16. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/docs/guides/skeletons-meshworks.md +53 -0
  17. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/docs/guides/transform-points.md +90 -0
  18. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/docs/index.md +67 -0
  19. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/docs/reference/datasets.md +15 -0
  20. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/docs/reference/streamlines.md +22 -0
  21. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/docs/reference/transforms.md +21 -0
  22. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/docs/streamline-field-method.md +142 -0
  23. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/make_streamline_grid_state.py +196 -0
  24. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/minnie_streamline_grid_link.txt +1 -0
  25. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/minnie_streamline_grid_state.json +1 -0
  26. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/mkdocs.yml +82 -0
  27. {standard_transform-1.4.1 → standard_transform-2.0.0/.claude/worktrees/laplace-streamlines}/pyproject.toml +12 -3
  28. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/scripts/build_streamline_field.py +198 -0
  29. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/scripts/compare_field_methods.py +209 -0
  30. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/scripts/tune_laplace_strength.py +101 -0
  31. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/scripts/validate_streamline_field.py +241 -0
  32. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/standard_transform/__init__.py +28 -0
  33. {standard_transform-1.4.1 → standard_transform-2.0.0/.claude/worktrees/laplace-streamlines}/standard_transform/base.py +23 -2
  34. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/standard_transform/data/minnie_streamline_field.npz +0 -0
  35. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/standard_transform/data/v1dd_streamline_field.npz +0 -0
  36. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/standard_transform/datasets.py +460 -0
  37. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/standard_transform/streamlines.py +1153 -0
  38. {standard_transform-1.4.1 → standard_transform-2.0.0/.claude/worktrees/laplace-streamlines}/standard_transform/utils.py +25 -0
  39. {standard_transform-1.4.1 → standard_transform-2.0.0/.claude/worktrees/laplace-streamlines}/streamline_generation.ipynb +14 -0
  40. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/tests/test_laplace_field.py +81 -0
  41. {standard_transform-1.4.1 → standard_transform-2.0.0/.claude/worktrees/laplace-streamlines}/tests/test_streamline.py +5 -5
  42. {standard_transform-1.4.1 → standard_transform-2.0.0/.claude/worktrees/laplace-streamlines}/tests/test_tform.py +8 -10
  43. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/tests/test_transformer.py +54 -0
  44. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/tests/test_versions.py +199 -0
  45. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/v1dd_streamline_distributed.ipynb +19 -0
  46. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/v1dd_streamline_grid_layer.json +9827 -0
  47. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/v1dd_streamline_grid_link.txt +1 -0
  48. standard_transform-2.0.0/.claude/worktrees/laplace-streamlines/v1dd_streamline_grid_state.json +1 -0
  49. standard_transform-2.0.0/.github/workflows/docs.yml +49 -0
  50. standard_transform-2.0.0/.gitignore +123 -0
  51. standard_transform-2.0.0/.pre-commit-config.yaml +13 -0
  52. standard_transform-2.0.0/CLAUDE.md +40 -0
  53. standard_transform-2.0.0/LICENSE +21 -0
  54. standard_transform-2.0.0/MANIFEST.in +3 -0
  55. standard_transform-2.0.0/PKG-INFO +131 -0
  56. standard_transform-2.0.0/README.md +87 -0
  57. standard_transform-2.0.0/docs/concepts/depth-and-radial-distance.md +54 -0
  58. standard_transform-2.0.0/docs/concepts/oriented-frame.md +75 -0
  59. standard_transform-2.0.0/docs/concepts/streamlines-vs-field.md +76 -0
  60. standard_transform-2.0.0/docs/concepts/versioning.md +105 -0
  61. standard_transform-2.0.0/docs/datasets.md +61 -0
  62. standard_transform-2.0.0/docs/design/layer-support.md +210 -0
  63. standard_transform-2.0.0/docs/guides/build-a-field.md +102 -0
  64. standard_transform-2.0.0/docs/guides/depth-radial.md +85 -0
  65. standard_transform-2.0.0/docs/guides/per-neuron-streamline.md +62 -0
  66. standard_transform-2.0.0/docs/guides/skeletons-meshworks.md +53 -0
  67. standard_transform-2.0.0/docs/guides/transform-points.md +90 -0
  68. standard_transform-2.0.0/docs/index.md +67 -0
  69. standard_transform-2.0.0/docs/reference/datasets.md +15 -0
  70. standard_transform-2.0.0/docs/reference/streamlines.md +22 -0
  71. standard_transform-2.0.0/docs/reference/transforms.md +21 -0
  72. standard_transform-2.0.0/docs/streamline-field-method.md +182 -0
  73. standard_transform-2.0.0/docs/upgrading.md +105 -0
  74. standard_transform-2.0.0/make_streamline_grid_state.py +232 -0
  75. standard_transform-2.0.0/minnie_streamline_grid_link.txt +1 -0
  76. standard_transform-2.0.0/minnie_streamline_grid_state.json +1 -0
  77. standard_transform-2.0.0/mkdocs.yml +83 -0
  78. standard_transform-2.0.0/pyproject.toml +121 -0
  79. standard_transform-2.0.0/scripts/build_streamline_field.py +265 -0
  80. standard_transform-2.0.0/scripts/compare_field_methods.py +209 -0
  81. standard_transform-2.0.0/scripts/tune_laplace_strength.py +101 -0
  82. standard_transform-2.0.0/scripts/validate_streamline_field.py +241 -0
  83. standard_transform-2.0.0/standard_transform/__init__.py +28 -0
  84. standard_transform-2.0.0/standard_transform/base.py +329 -0
  85. standard_transform-2.0.0/standard_transform/data/minnie_streamline_field.npz +0 -0
  86. standard_transform-2.0.0/standard_transform/data/minnie_um_streamline.json +1 -0
  87. standard_transform-2.0.0/standard_transform/data/v1dd_streamline_field.npz +0 -0
  88. standard_transform-2.0.0/standard_transform/data/v1dd_um_streamline.json +1 -0
  89. standard_transform-2.0.0/standard_transform/datasets.py +462 -0
  90. standard_transform-2.0.0/standard_transform/streamlines.py +1230 -0
  91. standard_transform-2.0.0/standard_transform/utils.py +89 -0
  92. standard_transform-2.0.0/streamline_generation.ipynb +285 -0
  93. standard_transform-2.0.0/tests/test_laplace_field.py +101 -0
  94. standard_transform-2.0.0/tests/test_streamline.py +53 -0
  95. standard_transform-2.0.0/tests/test_tform.py +131 -0
  96. standard_transform-2.0.0/tests/test_transformer.py +54 -0
  97. standard_transform-2.0.0/tests/test_versions.py +200 -0
  98. standard_transform-2.0.0/uv.lock +2098 -0
  99. standard_transform-2.0.0/v1dd_streamline_distributed.ipynb +19 -0
  100. standard_transform-2.0.0/v1dd_streamline_grid_layer.json +9827 -0
  101. standard_transform-2.0.0/v1dd_streamline_grid_link.txt +1 -0
  102. standard_transform-2.0.0/v1dd_streamline_grid_state.json +1 -0
  103. standard_transform-1.4.1/MANIFEST.in +0 -2
  104. standard_transform-1.4.1/PKG-INFO +0 -214
  105. standard_transform-1.4.1/README.md +0 -172
  106. standard_transform-1.4.1/standard_transform/__init__.py +0 -14
  107. standard_transform-1.4.1/standard_transform/datasets.py +0 -187
  108. standard_transform-1.4.1/standard_transform/streamlines.py +0 -349
  109. {standard_transform-1.4.1 → standard_transform-2.0.0}/.bmv-post-commit.sh +0 -0
  110. {standard_transform-1.4.1 → standard_transform-2.0.0/.claude/worktrees/laplace-streamlines}/.pre-commit-config.yaml +0 -0
  111. {standard_transform-1.4.1 → standard_transform-2.0.0/.claude/worktrees/laplace-streamlines}/LICENSE +0 -0
  112. {standard_transform-1.4.1 → standard_transform-2.0.0/.claude/worktrees/laplace-streamlines}/standard_transform/data/minnie_um_streamline.json +0 -0
  113. {standard_transform-1.4.1 → standard_transform-2.0.0/.claude/worktrees/laplace-streamlines}/standard_transform/data/v1dd_um_streamline.json +0 -0
  114. {standard_transform-1.4.1 → standard_transform-2.0.0/.claude/worktrees/laplace-streamlines}/uv.lock +0 -0
@@ -0,0 +1,9 @@
1
+ {
2
+ "permissions": {
3
+ "allow": [
4
+ "Bash(cd *)",
5
+ "Bash(uv run *)",
6
+ "Bash(git check-ignore *)"
7
+ ]
8
+ }
9
+ }
@@ -0,0 +1,2 @@
1
+ #!/bin/bash
2
+ git push --tags
@@ -0,0 +1,49 @@
1
+ name: Docs
2
+
3
+ on:
4
+ push:
5
+ branches: [main]
6
+ workflow_dispatch:
7
+
8
+ # Allow one concurrent deployment, cancelling in-progress runs for the same ref.
9
+ concurrency:
10
+ group: pages
11
+ cancel-in-progress: true
12
+
13
+ permissions:
14
+ contents: read
15
+ pages: write
16
+ id-token: write
17
+
18
+ jobs:
19
+ build:
20
+ runs-on: ubuntu-latest
21
+ steps:
22
+ - uses: actions/checkout@v4
23
+
24
+ - name: Install uv
25
+ uses: astral-sh/setup-uv@v5
26
+ with:
27
+ enable-cache: true
28
+
29
+ - name: Sync dependencies
30
+ run: uv sync
31
+
32
+ - name: Build site (strict)
33
+ run: uv run mkdocs build --strict
34
+
35
+ - name: Upload Pages artifact
36
+ uses: actions/upload-pages-artifact@v3
37
+ with:
38
+ path: site
39
+
40
+ deploy:
41
+ needs: build
42
+ runs-on: ubuntu-latest
43
+ environment:
44
+ name: github-pages
45
+ url: ${{ steps.deployment.outputs.page_url }}
46
+ steps:
47
+ - name: Deploy to GitHub Pages
48
+ id: deployment
49
+ uses: actions/deploy-pages@v4
@@ -117,4 +117,7 @@ notebooks/
117
117
  *.pkl
118
118
  *.mp4
119
119
  *.png
120
- *.tiff
120
+ *.tiff
121
+ # streamline analysis outputs (scripts)
122
+ compare_*.npz
123
+ validation_*.npz
@@ -0,0 +1,40 @@
1
+ # CLAUDE.md
2
+
3
+ This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4
+
5
+ ## What this is
6
+
7
+ `standard_transform` provides pre-baked affine transforms and curvilinear "streamlines" for orienting points from EM connectomics datasets (Minnie65 and V1dd) into a consistent frame: microns, with the y-axis running pia→white matter and the pial surface at approximately y=0. See README.md for the full user-facing API and worked examples.
8
+
9
+ ## Development commands
10
+
11
+ This project uses `uv` and [poethepoet](https://poethepoet.natn.io/) (`poe`) tasks defined in `pyproject.toml`.
12
+
13
+ - **Run tests with coverage:** `uv run poe test` (equivalent to `uv run pytest --cov=standard_transform tests`)
14
+ - **Run a single test:** `uv run pytest tests/test_tform.py::test_name` or `uv run pytest -k pattern`
15
+ - **Lint:** `uv run ruff check` — note the ruff config only selects a minimal set of error codes (`E9,F63,F7,F82`); pre-commit additionally runs import sorting (`I`) and `ruff-format`. Notebooks are excluded from ruff.
16
+ - **Version bump (also git-tags and commits):** `uv run poe bump patch|minor|major` (dry run: `uv run poe drybump patch`). Bumping updates `__init__.py` and `pyproject.toml`, runs `uv sync`, and tags `v{version}`.
17
+ - **Docs preview:** `uv run poe doc-preview` (mkdocs)
18
+ - **Profiling:** `uv run poe profile` (pyinstrument) or `uv run poe profile-all` (scalene)
19
+
20
+ Note: tests live in `tests/` (the empty `test/` dir is unused). The `build/` and `dist/` dirs contain build artifacts, not source — never edit them.
21
+
22
+ ## Architecture
23
+
24
+ The package is small and layered. `standard_transform/__init__.py` re-exports only the dataset-specific factory functions and `Dataset` instances — that is the public surface.
25
+
26
+ - **`base.py`** — the transform engine. A `TransformSequence` holds an ordered list of primitive transforms (`ScaleTransform`, `TranslateTransform`, `RotationTransform`, each with `apply`/`invert`). `apply` runs them forward; `invert` runs them in reverse order with each primitive inverted. All the input-shape polymorphism lives here: `apply`/`invert` dispatch on whether input is a `pd.Series` (of 3-vectors) vs an array; `apply_project` extracts a single axis (x/y/z ↔ 0/1/2); `apply_dataframe` accepts a column name and auto-detects split position columns; `apply_skeleton`/`apply_meshwork_vertices`/`apply_meshwork_annotations` transform MeshParty objects (imported lazily/duck-typed — MeshParty is not a dependency).
27
+
28
+ - **`streamlines.py`** — the `Streamline` class models the curvilinear depth axis. It stores streamline points in *post-transform* space and builds 1-D interpolators giving x and z as functions of depth y. Every method takes a `transform_points` flag: `True` (default) means inputs are in the original pre-transform coordinates and get transformed first. Key methods: `streamline_at` (x,z of the streamline at a depth), `radial_distance` (in-plane distance accounting for curvature), `depth_along`/`depth_between` (path length along the curve), `radial_points` (map points into cylindrical-like radial/depth coordinates), and the `transform_*` methods that straighten skeletons/meshworks along the streamline (relocating root to origin).
29
+ - `StreamlineField(Streamline)` is the spatially-varying version: instead of one curve it holds a 3D grid of local tangent vectors `(dx/dy, dz/dy)` and integrates the streamline through any anchor, so the shape varies across the volume (a field uniform in x,z reduces to a plain `Streamline`). It overrides only `__init__`, `streamline_at` (now integrates the field), and `streamline_points_tform`; everything else is inherited. `streamline_at_point(soma)` returns a plain `Streamline` for a neuron — the intended per-cell workflow (integrate once at the soma, apply that fixed curve across the arbor). Query coords are clamped into the grid, so orientation is held constant outside the sampled depth band. `to_npz`/`from_npz` persist the grid (transform not stored — reattached on load). Build with `streamline_field_from_paths`, which bins per-segment tangents, restricts to a reliable `depth_band`, weights cells by precision `count/(var+prior)`, and regularizes via a precision-weighted diffusion (`_precision_smooth`). The method rationale (how the band and precision/diffusion were chosen) is written up in README.md under "Spatially-varying streamlines".
30
+ - **The field is the default streamline.** `v1dd_streamline_nm/_vx()` (and `v1dd_ds.streamline_nm/_vx`, which are lazy properties) return the `StreamlineField` loaded from `data/v1dd_streamline_field.npz`; pass `legacy=True` to the module functions for the old hand-drawn json streamline. This was an invisible-API but breaking-results change. Both v1dd and Minnie now default to a field (each `data/<dataset>_streamline_field.npz`, registry version "2.0"); `version="1.4"` recovers each dataset's hand json. The field npz is generated offline from ~17-18k skeleton apical paths (tall single dendritic paths, tip→root, in nm), not rebuilt at runtime. Reproduce it with `uv run python scripts/build_streamline_field.py --paths <dir> --dataset v1dd|minnie` (add `--validate N` for a held-out QC deviation). Depth bands differ per dataset — chosen from where per-cell tangent std rises: v1dd `[150, 700]`, Minnie `[150, 650]` (Minnie apical data thins out below ~650µm). `make_streamline_grid_state.py` (repo root, a `uv run` script) emits a neuroglancer QC visualization of the streamline grid.
31
+
32
+ - **`datasets.py`** — dataset-specific constants and factories. Defines voxel resolutions and pia reference points, the per-dataset rotation/translation/scaling recipes (`_minnie_transforms`, `_v1dd_transforms`), and the public factory functions (`minnie_transform_nm/vx`, `v1dd_transform_nm/vx`, `*_streamline_nm/vx`). The `Dataset` class bundles a transform + streamline for each dataset; the exported singletons `minnie_ds` and `v1dd_ds` are the recommended entry points. Streamline point data is loaded from JSON in `standard_transform/data/`.
33
+
34
+ - **`utils.py`** — dataframe helpers, chiefly split-position-column detection. Supports two naming conventions: type 1 (`pt_position_x/y/z`) and type 2 (`pt_position_x_suffix`); ambiguity between them raises.
35
+
36
+ ### Conventions when extending
37
+
38
+ - Every transform primitive must implement both `apply` and `invert` so `TransformSequence.invert` round-trips.
39
+ - Transforms and streamlines come in paired `_nm` (nanometer input) and `_vx` (voxel input, with a `voxel_resolution` argument) variants; a `_vx` factory scales by resolution first, then delegates to the shared transform recipe. Follow this pattern when adding a dataset.
40
+ - Pia points are stored in nm; voxel variants derive their pia point by dividing by resolution.
@@ -0,0 +1,3 @@
1
+ include requirements.txt
2
+ include standard_transform/data/*.json
3
+ include standard_transform/data/*.npz
@@ -0,0 +1,87 @@
1
+ # standard_transform
2
+
3
+ Orient and scale points in EM connectomics datasets consistently and easily.
4
+
5
+ 📖 **Full documentation: https://caveconnectome.github.io/standard_transform/**
6
+
7
+ When working with EM data, the orientation of the dataset often does not match the
8
+ orientation you want to reason in. For cortical data you usually want "down" to mean
9
+ the direction orthogonal to the pial surface, in microns, with the pia at `y ≈ 0`.
10
+ `standard_transform` provides pre-baked affine transforms for two datasets
11
+ (Minnie65 and V1dd) that map voxel or nanometer coordinates into that consistent
12
+ oriented frame, plus **streamlines** — a curvilinear depth axis — for separating
13
+ cortical depth from lateral (radial) distance even where cortex curves.
14
+
15
+ ## Install
16
+
17
+ ```bash
18
+ pip install standard-transform
19
+ ```
20
+
21
+ ## Quickstart
22
+
23
+ ```python
24
+ from standard_transform import minnie_ds, v1dd_ds
25
+
26
+ # n x 3 points -> oriented microns (y = pia->white matter, pia at y≈0)
27
+ # resolution defaults to "nm"; pass "vx" or an [x, y, z] list for other units.
28
+ pts_um = minnie_ds.transform().apply(xyz_nm)
29
+
30
+ # just the cortical depth (microns below pia)
31
+ depth = minnie_ds.transform().apply_project("y", xyz_nm)
32
+
33
+ # radial (in-plane) distance following the local streamline
34
+ d = v1dd_ds.streamline().radial_distance(xyz0_nm, xyz1_nm)
35
+ ```
36
+
37
+ `minnie_ds` and `v1dd_ds` bundle a transform and a streamline and are the
38
+ recommended entry points. Both `.transform(resolution="nm", version=None)` and
39
+ `.streamline(resolution="nm", version=None)` take the input resolution (`"nm"`,
40
+ `"vx"`, or an `[x, y, z]` list) and an optional version.
41
+
42
+ ## What's in the box
43
+
44
+ - **Transforms** — `apply`, `apply_project`, `apply_dataframe`, and `invert`. Point
45
+ units are selected with `resolution=` (`"nm"`, `"vx"`, or `[x, y, z]`). Accept
46
+ `n x 3` arrays, pandas Series, and split `_x/_y/_z` dataframe columns.
47
+ - **Streamlines** — depth-along, radial distance, and cylindrical-like remapping
48
+ along a curvilinear pia-to-white-matter axis.
49
+
50
+ > **Note:** transforming MeshParty morphology *objects* (skeletons/meshworks) is
51
+ > deprecated and moving to [Ossify](https://csdashm.com/ossify/), where you pass the
52
+ > transformation to the object. `standard_transform` focuses on coordinate arrays and
53
+ > dataframes.
54
+
55
+ ## Streamline fields (breaking data change)
56
+
57
+ The v1dd and Minnie65 streamlines are now **data-derived, spatially-varying
58
+ `StreamlineField`s** by default, replacing the old single hand-drawn curves. This
59
+ changes computed *results* but not the interface — `StreamlineField` is a drop-in
60
+ subclass of `Streamline`. To recover the previous behavior:
61
+
62
+ ```python
63
+ from standard_transform import minnie_ds, v1dd_ds
64
+ sl = v1dd_ds.streamline() # data-derived field (latest, default)
65
+ sl_old = v1dd_ds.streamline(version="1.4") # original hand-drawn streamline
66
+ sl_m = minnie_ds.streamline() # Minnie65 field (latest, default)
67
+ ```
68
+
69
+ See the docs for the
70
+ [concept](https://caveconnectome.github.io/standard_transform/concepts/streamlines-vs-field/),
71
+ the [per-neuron workflow](https://caveconnectome.github.io/standard_transform/guides/per-neuron-streamline/),
72
+ and [how the field is built](https://caveconnectome.github.io/standard_transform/streamline-field-method/).
73
+ Older definitions stay reachable via `version=` — see
74
+ [Versioning & reproducibility](https://caveconnectome.github.io/standard_transform/concepts/versioning/).
75
+
76
+ ## Documentation
77
+
78
+ The full guide — concepts, task-oriented tutorials, the streamline-field method, and
79
+ the API reference — lives at
80
+ **https://caveconnectome.github.io/standard_transform/**.
81
+
82
+ ## Development
83
+
84
+ ```bash
85
+ uv run poe test # run tests with coverage
86
+ uv run poe doc-preview # serve the docs locally with live reload
87
+ ```
@@ -0,0 +1,54 @@
1
+ # Depth vs. radial distance in laminar cortex
2
+
3
+ ## Why a straight axis isn't enough
4
+
5
+ Mammalian cortex is strongly **laminar**: neurons, connectivity, and cell types are
6
+ organized in layers stacked from the pial surface down to white matter. Because of
7
+ that organization, two very different questions constantly come up:
8
+
9
+ - **Depth:** how far *along the pia → white matter direction* is a point? (Which
10
+ layer is it in?)
11
+ - **Radial distance:** how far apart are two points *within the laminar plane*,
12
+ i.e. tangential to the layers?
13
+
14
+ The [oriented frame](oriented-frame.md) gives you a single global depth axis (`y`).
15
+ That works where cortex is flat. But cortex **curves** — the true "down" direction
16
+ tilts and bends as you move across the volume. If you keep using one straight axis,
17
+ then "depth" and "in-plane distance" get mixed together: a point that is actually
18
+ directly *below* another (same lamina position, deeper) can look laterally
19
+ displaced, and vice versa. The error grows with curvature and with distance from
20
+ wherever the straight axis happened to be correct.
21
+
22
+ ## The streamline
23
+
24
+ A **streamline** is a curvilinear depth axis: a curve that follows the local
25
+ pia → white matter direction as it bends through the tissue. Think of it as the
26
+ path a plumb line would trace if "down" were always the cortical down at each point.
27
+
28
+ With a streamline you can measure the two quantities *separately and correctly*:
29
+
30
+ - **Depth along the streamline** — path length down the curve from the pia, rather
31
+ than a raw `y` difference. This is what `depth_along` and `depth_between` compute.
32
+ - **Radial distance** — in-plane distance measured relative to the streamline as the
33
+ `d = 0` reference, accounting for the fact that the curve is tilted. This is what
34
+ `radial_distance` computes.
35
+
36
+ You can also remap points into a cylindrical-like coordinate system — radial
37
+ distance as one axis, depth-along-the-streamline as the other — with
38
+ `radial_points`. This is the basis for "straightening" a neuron along the cortical
39
+ axis (see [Skeletons & meshworks](../guides/skeletons-meshworks.md)).
40
+
41
+ ## A worked intuition
42
+
43
+ Imagine two synapses. In raw oriented `y` they differ by 50 µm, and they also
44
+ differ in `x`. Are they at the same lamina depth but offset sideways, or genuinely
45
+ at different depths? If the cortex is curved there, the straight `y` answer is
46
+ biased. Measuring depth *along the streamline* answers the anatomical question:
47
+ how far down the cortical column each one really is.
48
+
49
+ ## Next
50
+
51
+ There are two ways a streamline can be defined: a single fixed curve applied
52
+ everywhere, or a curve that varies with location. That distinction — and why the
53
+ v1dd default recently changed — is covered in
54
+ [Streamlines vs. fields](streamlines-vs-field.md).
@@ -0,0 +1,75 @@
1
+ # The oriented frame
2
+
3
+ ## The problem
4
+
5
+ An EM dataset is a volume of tissue imaged on a microscope. Its native coordinate
6
+ system — voxels, or nanometers derived from them — is aligned to the imaging setup,
7
+ not to the brain. Two things are usually "off":
8
+
9
+ - **Units and scale.** Coordinates come in voxels (with anisotropic voxel sizes,
10
+ e.g. `[4, 4, 40]` nm for Minnie65) or in nanometers. Neither is convenient for
11
+ thinking about distances in tissue.
12
+ - **Orientation.** The pial surface — the top of cortex — is not axis-aligned. The
13
+ cortical "down" direction (pia → white matter) points at some arbitrary angle
14
+ relative to the volume's axes, and it can be slightly different from one dataset
15
+ to the next.
16
+
17
+ If you want to ask a simple anatomical question — *how deep below the pia is this
18
+ synapse?* — you cannot just read off a coordinate, because no single native axis
19
+ corresponds to depth.
20
+
21
+ ## What the transform does
22
+
23
+ `standard_transform` applies a fixed sequence of **affine operations**
24
+ (rotation → translation → scaling) that lands every dataset in the same frame:
25
+
26
+ - **microns**, so distances are directly interpretable;
27
+ - the **y-axis oriented from pia to white matter** — increasing `y` means deeper
28
+ into cortex;
29
+ - the **pial surface at `y ≈ 0`**, so `y` reads directly as cortical depth.
30
+
31
+ The x and z axes span the plane parallel to the pial surface (the laminar plane).
32
+ Because every dataset is mapped into this same convention, analyses and figures
33
+ built on top of it are comparable across datasets without per-dataset bookkeeping.
34
+
35
+ The recipe for each dataset is pre-baked — you don't fit anything. Each is a small
36
+ `TransformSequence` of primitives (a rotation derived from the dataset's measured
37
+ "up" vector, a translation that puts pia at `y = 0`, and a scaling to microns).
38
+ See [Datasets](../datasets.md) for the specific constants.
39
+
40
+ ## Input units: the `resolution` argument
41
+
42
+ You tell the transform what units your points are in with `resolution`:
43
+
44
+ - `resolution="nm"` (the default) — **nanometers**.
45
+ - `resolution="vx"` — **voxels**, using the dataset's native voxel size.
46
+ - `resolution=[x, y, z]` — an explicit voxel resolution; the transform scales to
47
+ nanometers first, then applies the shared recipe.
48
+
49
+ ```python
50
+ minnie_ds.transform() # nm (default)
51
+ minnie_ds.transform("vx") # native voxels
52
+ minnie_ds.transform([8, 8, 40]) # explicit resolution
53
+ ```
54
+
55
+ All choices produce the same oriented microns for equivalent points, so results are
56
+ unit-invariant — the only difference is what you put in.
57
+
58
+ ## It's invertible
59
+
60
+ A `TransformSequence` runs its primitives forward under `apply` and in reverse
61
+ (each inverted) under `invert`. So you can always take a point you've computed in
62
+ the oriented frame and recover its original native coordinate — useful for, say,
63
+ turning an analysis result back into a location you can look up in the dataset.
64
+
65
+ ```python
66
+ pts_um = tform.apply(pts_nm) # native → oriented microns
67
+ pts_back = tform.invert(pts_um) # oriented microns → native
68
+ ```
69
+
70
+ ## Next
71
+
72
+ The oriented frame gives you a straight, global depth axis. But cortex curves, and
73
+ a single straight axis is not enough to measure depth and lateral distance
74
+ accurately everywhere — that's what [streamlines](depth-and-radial-distance.md)
75
+ are for.
@@ -0,0 +1,76 @@
1
+ # Streamlines vs. fields
2
+
3
+ There are two ways `standard_transform` represents the curvilinear depth axis. They
4
+ share an interface but differ in how much they can adapt to the tissue.
5
+
6
+ ## A fixed `Streamline`
7
+
8
+ A `Streamline` stores **one curve** and applies its shape everywhere: to find the
9
+ streamline through any point, it simply **translates** that single curve so it
10
+ passes through the point. The shape is identical across the whole volume — only its
11
+ position shifts.
12
+
13
+ This is exactly right when the pia → white matter direction is roughly uniform
14
+ across the region you care about. It is what the original, hand-drawn streamlines
15
+ provided, and it is still a good approximation over a limited area.
16
+
17
+ Its limitation: a single curve cannot represent a *spatially-varying* axis. If the
18
+ cortical "down" direction genuinely differs between one side of the dataset and the
19
+ other (curvature, or regional alignment drift), one fixed shape is accurate on one
20
+ side and progressively wrong on the other.
21
+
22
+ ## A spatially-varying `StreamlineField`
23
+
24
+ A `StreamlineField` (a subclass of `Streamline`, so it is a drop-in replacement)
25
+ instead stores a **3D grid of local orientation vectors** — the pia → white matter
26
+ direction at every location — estimated from many neurons. The streamline through a
27
+ point is obtained by **integrating that field**, so its shape *varies across the
28
+ volume*. A field that happens to be uniform in x and z reduces exactly to a single
29
+ `Streamline`.
30
+
31
+ Because the field is estimated directly from data, it captures local curvature and
32
+ absorbs regional alignment drift that no single fixed curve can. How it is built and
33
+ validated is described in [Streamline field: method](../streamline-field-method.md).
34
+
35
+ ## What changed (and how to opt out)
36
+
37
+ As of the current release, both the v1dd and Minnie65 streamline accessors return a
38
+ `StreamlineField` **by default**:
39
+
40
+ ```python
41
+ from standard_transform import minnie_ds, v1dd_ds
42
+
43
+ sl = v1dd_ds.streamline() # data-derived tangent field (latest, default)
44
+ sl_old = v1dd_ds.streamline(version="1.4") # original hand-drawn single streamline
45
+ sl_m = minnie_ds.streamline() # Minnie65 field (latest, default)
46
+ ```
47
+
48
+ This is a **breaking change in results** — computed depths and radial distances
49
+ differ from the old hand-drawn streamline — but **not a breaking change in
50
+ interface**. Every method (`streamline_at`, `radial_distance`, `depth_along`,
51
+ `radial_points`) works unchanged, because `StreamlineField` inherits them. Pin
52
+ `version="1.4"` to recover the previous behavior exactly — see
53
+ [Versioning & reproducibility](versioning.md).
54
+
55
+ !!! note "Which datasets have a field?"
56
+ Both shipped datasets — v1dd and Minnie65 — default to a data-derived field. The
57
+ accessors always return a `Streamline`-compatible object regardless, so your code
58
+ does not need to branch. See [Datasets](../datasets.md) for the current status of
59
+ each.
60
+
61
+ ## The per-neuron workflow
62
+
63
+ The intended way to use a field for a single neuron is **not** to re-derive the
64
+ axis at every vertex. Instead, integrate the field **once at the cell body** to get
65
+ the one streamline appropriate to that soma, then apply that fixed curve across the
66
+ whole arbor:
67
+
68
+ ```python
69
+ sl = v1dd_ds.streamline().streamline_at_point(soma_xyz) # a plain Streamline
70
+ verts_straight = sl.radial_points(soma_xyz, skel.vertices)
71
+ ```
72
+
73
+ This measures the arbor *against its soma's cortical axis* without assuming the
74
+ dendrites themselves follow the field — which they often don't (for example,
75
+ inverted layer-6 apical dendrites). See the
76
+ [per-neuron guide](../guides/per-neuron-streamline.md) for the full walkthrough.
@@ -0,0 +1,84 @@
1
+ # Versioning & reproducibility
2
+
3
+ ## Outputs are tied to the package version
4
+
5
+ Everything `standard_transform` produces is defined by two things: the **rigid affine
6
+ numbers** (rotation, pia point, scaling) that build the oriented frame, and the
7
+ **streamline data**. Both are baked into the package. If either changes between
8
+ releases — as the v1dd streamline does when it moves from a hand-drawn curve to a
9
+ data-derived field — then every depth, radial distance, and transformed coordinate
10
+ changes too.
11
+
12
+ That makes results **version-dependent**: a figure made with one release is only
13
+ directly comparable to another if they used the same definitions. To keep old work
14
+ reproducible and to compare across releases, past definitions stay reachable by name.
15
+
16
+ ## Two independent version tracks
17
+
18
+ The affine transform and the streamline are versioned **separately**, because they
19
+ change on different schedules:
20
+
21
+ - the **transform** track — the affine numbers;
22
+ - the **streamline** track — the depth-axis data.
23
+
24
+ For example, in the `2.0` release the v1dd and Minnie65 streamlines change (hand-drawn
25
+ → field) but the affine numbers do not, so each transform stays at its `1.4` version
26
+ while the streamline advances to `2.0`.
27
+
28
+ ## Package-version-anchored labels
29
+
30
+ A version label **is the release in which that definition became the default**. So
31
+ `version="1.4"` means "the definition shipped as default through the 1.4 line," and
32
+ `version="2.0"` means "the definition that became default in 2.0." Asking for a label
33
+ reproduces exactly what that release produced.
34
+
35
+ | dataset | track | version | what it is |
36
+ |---|---|---|---|
37
+ | v1dd | transform | `1.4` (latest) | the affine frame (unchanged) |
38
+ | v1dd | streamline | `1.4` | original hand-drawn single streamline |
39
+ | v1dd | streamline | `2.0` (latest) | data-derived spatially-varying field |
40
+ | minnie65 | transform | `1.4` (latest) | the affine frame |
41
+ | minnie65 | streamline | `1.4` | original hand-drawn single streamline |
42
+ | minnie65 | streamline | `2.0` (latest) | data-derived spatially-varying field |
43
+
44
+ ## Pinning a version
45
+
46
+ Pass `version=` to `transform()` / `streamline()` (or the module factories). Omitting
47
+ it — or `version=None` — resolves to the latest for that track:
48
+
49
+ ```python
50
+ from standard_transform import v1dd_ds
51
+
52
+ sl_latest = v1dd_ds.streamline() # latest: the field (2.0)
53
+ sl_old = v1dd_ds.streamline(version="1.4") # reproduce the hand-drawn streamline
54
+
55
+ tf = v1dd_ds.transform(version="1.4") # pin the affine frame
56
+ ```
57
+
58
+ An unknown label raises a `ValueError` listing the valid versions.
59
+
60
+ ## Recording provenance
61
+
62
+ Every transform and streamline carries a `.version` attribute, so an analysis can
63
+ record exactly which definition produced its numbers:
64
+
65
+ ```python
66
+ tf = v1dd_ds.transform()
67
+ tf.version # -> "1.4"
68
+ v1dd_ds.streamline().version # -> "2.0"
69
+ ```
70
+
71
+ ## Inspecting what's available
72
+
73
+ `available_versions(dataset)` (also `dataset.available_versions()`) lists both tracks,
74
+ their versions, and which release introduced each, plus the current latest:
75
+
76
+ ```python
77
+ from standard_transform import available_versions
78
+
79
+ available_versions("v1dd")
80
+ # {
81
+ # "transform": {"latest": "1.4", "versions": {"1.4": "1.4"}},
82
+ # "streamline": {"latest": "2.0", "versions": {"1.4": "1.4", "2.0": "2.0"}},
83
+ # }
84
+ ```
@@ -0,0 +1,61 @@
1
+ # Datasets
2
+
3
+ A **`Dataset`** bundles a transform and a streamline together and is the easiest way
4
+ to get started. Two are provided as ready-to-use singletons: `minnie_ds` and
5
+ `v1dd_ds`.
6
+
7
+ ```python
8
+ from standard_transform import minnie_ds, v1dd_ds
9
+
10
+ v1dd_ds.transform().apply(xyz_nm) # transform points (nm default)
11
+ v1dd_ds.streamline().radial_distance(xyz0, xyz1) # streamline query
12
+ ```
13
+
14
+ Each dataset exposes two methods (plus the module-level factories `minnie_transform`,
15
+ `v1dd_transform`, `minnie_streamline`, `v1dd_streamline`, which take the same
16
+ arguments):
17
+
18
+ - `transform(resolution="nm", version=None)` — a [transform](concepts/oriented-frame.md).
19
+ - `streamline(resolution="nm", version=None)` — a
20
+ [streamline](concepts/depth-and-radial-distance.md) (built lazily and cached).
21
+
22
+ `resolution` is `"nm"` (default), `"vx"` (native voxel size), or an explicit
23
+ `[x, y, z]`. `version` pins a historical definition; see
24
+ [Versioning & reproducibility](concepts/versioning.md).
25
+
26
+ ## Minnie65
27
+
28
+ Native voxel resolution `[4, 4, 40]` nm.
29
+
30
+ - `minnie_ds.transform()` — nanometers → oriented microns (pia flat in x,z at y≈0).
31
+ - `minnie_ds.transform("vx")` — native voxels; `minnie_ds.transform([8, 8, 40])` for
32
+ a different scale.
33
+ - `minnie_ds.streamline(...)` — returns a data-derived, spatially-varying
34
+ [`StreamlineField`](concepts/streamlines-vs-field.md) **by default** (depth band
35
+ `[150, 650]` µm). Pass `version="1.4"` for the original hand-drawn single streamline.
36
+
37
+ ## V1dd
38
+
39
+ Native voxel resolution `[9, 9, 45]` nm.
40
+
41
+ - `v1dd_ds.transform(...)` — same as Minnie65 with the v1dd recipe and `[9, 9, 45]`
42
+ native voxels.
43
+ - `v1dd_ds.streamline(...)` — returns a data-derived, spatially-varying
44
+ [`StreamlineField`](concepts/streamlines-vs-field.md) **by default** (depth band
45
+ `[150, 700]` µm). Pass `version="1.4"` for the original hand-drawn single streamline.
46
+
47
+ ## Streamline field status
48
+
49
+ | dataset | default streamline |
50
+ |---|---|
51
+ | V1dd | data-derived `StreamlineField` (pass `version="1.4"` for the hand-drawn curve) |
52
+ | Minnie65 | data-derived `StreamlineField` (pass `version="1.4"` for the hand-drawn curve) |
53
+
54
+ Both datasets now default to a data-derived, spatially-varying field. The
55
+ field-building procedure is dataset-agnostic and described in
56
+ [Streamline field: method](streamline-field-method.md).
57
+
58
+ ## Identity
59
+
60
+ - `identity_transform` — returns the input unchanged (optionally axis-projected).
61
+ Useful for compatibility when a code path expects a transform object.
@@ -0,0 +1,87 @@
1
+ # Build a field for a new dataset
2
+
3
+ A [`StreamlineField`](../concepts/streamlines-vs-field.md) is estimated offline from
4
+ a collection of **pia → white matter paths** — one per neuron — and shipped as a
5
+ compressed `.npz`. This guide shows how to build one. For the reasoning behind each
6
+ step (depth band, precision, diffusion), read
7
+ [Streamline field: method](../streamline-field-method.md).
8
+
9
+ ## The input: one tall path per neuron
10
+
11
+ Each path is an ordered `n x 3` array of points running along the cortical axis —
12
+ in practice, a tall single dendritic path from a deep soma up the apical trunk to a
13
+ tip in upper cortex, ordered tip → root. Where the paths come from (skeletons,
14
+ neuroglancer annotations) does not matter. Paths are in the dataset's **native
15
+ nanometers** unless you say otherwise.
16
+
17
+ ## Build from a script (recommended)
18
+
19
+ The repo ships `scripts/build_streamline_field.py`, a `uv run` script that loads a
20
+ directory of per-neuron `.npy` files and writes the field `.npz`:
21
+
22
+ ```bash
23
+ uv run python scripts/build_streamline_field.py \
24
+ --paths /path/to/apical_paths \
25
+ --dataset v1dd
26
+ ```
27
+
28
+ Arguments:
29
+
30
+ | flag | meaning |
31
+ |---|---|
32
+ | `--paths` | directory of per-neuron `.npy` files (native nm; only x,y,z used) |
33
+ | `--dataset` | selects the nm transform and default output name (`v1dd` or `minnie`) |
34
+ | `--out` | output `.npz` path (default `standard_transform/data/<dataset>_streamline_field.npz`) |
35
+ | `--bin-size` | grid spacing X Y Z in microns (default `30 20 30`) |
36
+ | `--depth-band` | in-band depth range LO HI in microns (default `150 700`) |
37
+ | `--validate N` | hold out `N` paths, rebuild on the rest, and report held-out median lateral deviation as a QC check |
38
+
39
+ The shipped field is always built on **all** paths; `--validate` only builds a
40
+ throwaway field for quality reporting.
41
+
42
+ ## Build programmatically
43
+
44
+ To build a field in your own code, call `streamline_field_from_paths` directly:
45
+
46
+ ```python
47
+ from standard_transform import streamline_field_from_paths, v1dd_transform
48
+
49
+ field = streamline_field_from_paths(
50
+ paths, # list of n_i x 3 arrays, native nm, tip -> root
51
+ tform=v1dd_transform(), # transform into oriented microns (nm input)
52
+ bin_size=(30.0, 20.0, 30.0),
53
+ depth_band=(150.0, 700.0),
54
+ )
55
+ field.to_npz("my_field.npz")
56
+ ```
57
+
58
+ Useful parameters (see the [API reference](../reference/streamlines.md) for the full
59
+ list):
60
+
61
+ - `weights` — per-path or per-segment weights to down-weight unreliable structure
62
+ (e.g. by Strahler index / inverse branch order).
63
+ - `depth_band` — only segments in this depth range inform the field, and the grid
64
+ spans exactly this range; outside it, orientation is held constant.
65
+ - `smoothing_passes` / `smoothing_strength` — control the precision-weighted
66
+ diffusion that fills empty cells and denoises.
67
+
68
+ ## Persist and reload
69
+
70
+ ```python
71
+ field.to_npz("my_field.npz")
72
+
73
+ from standard_transform import StreamlineField, v1dd_transform
74
+ field = StreamlineField.from_npz("my_field.npz", tform=v1dd_transform())
75
+ ```
76
+
77
+ The grid is stored in **unit-agnostic post-transform microns**, so a single `.npz`
78
+ serves both the nm and voxel variants of a dataset — only the attached transform
79
+ differs. The transform is not saved in the file; you reattach one on load.
80
+
81
+ ## Wire it into a dataset
82
+
83
+ Once a field `.npz` exists in `standard_transform/data/`, add a row to the streamline
84
+ version registry in `datasets.py` pointing at it and mark it latest — the dataset's
85
+ `streamline()` then returns it by default, while the prior `version="1.4"` hand-drawn
86
+ curve stays reachable. This mirrors how v1dd is wired; see
87
+ [Versioning & reproducibility](../concepts/versioning.md).