aind-metadata-reader 0.2.0__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,396 @@
1
+ Metadata-Version: 2.5
2
+ Name: aind-metadata-reader
3
+ Version: 0.2.0
4
+ Summary: Read historical AIND metadata without rewriting source records
5
+ Project-URL: Repository, https://github.com/AllenNeuralDynamics/aind-metadata-reader/
6
+ Project-URL: Issues, https://github.com/AllenNeuralDynamics/aind-metadata-reader/issues
7
+ Project-URL: Changelog, https://github.com/AllenNeuralDynamics/aind-metadata-reader/blob/main/CHANGELOG.md
8
+ Author: Allen Institute for Neural Dynamics
9
+ Author-email: Galen Lynch <galen@galenlynch.com>
10
+ License-Expression: MIT
11
+ License-File: LICENSE
12
+ Classifier: Programming Language :: Python :: 3
13
+ Requires-Python: >=3.10
14
+ Provides-Extra: validation
15
+ Requires-Dist: jsonschema>=4.26.0; extra == 'validation'
16
+ Description-Content-Type: text/markdown
17
+
18
+ # AIND Metadata Reader
19
+
20
+ Read historical AIND metadata, inspect incomplete records, and record explicit
21
+ corrections while preserving the source. Readers support selected representations
22
+ from `aind-data-schema` and `biodata-schema` without requiring either package at
23
+ runtime. Python 3.10 and newer are supported.
24
+
25
+ This library is **experimental**. Its API may evolve before version 1.0.
26
+ The examples below run from this checkout using the included historical fixtures.
27
+
28
+ ## What the library does
29
+
30
+ Metadata changes in several ways: fields move, vocabulary grows, and concepts
31
+ change meaning. A processing input recorded as a file path conveys different
32
+ information from an input recorded as an asset reference. The reader exposes
33
+ those distinctions through small views of code, references, and coordinate frames.
34
+
35
+ Reading preserves the original values and unknown fields. Schema-invalid metadata
36
+ remains inspectable, and a malformed component does not hide usable siblings.
37
+ You can separately request a validation report or derive a corrected interpretation.
38
+ Neither operation silently replaces the source.
39
+
40
+ ```mermaid
41
+ flowchart LR
42
+ source["JSON file or mapping"] --> capture["Preserved snapshot"]
43
+ capture --> views["Views, available values, and diagnostics"]
44
+ capture --> validation["Optional validation report"]
45
+ views --> changes["Explicit correction or inference"]
46
+ changes --> derived["Derived snapshot and change manifest"]
47
+ ```
48
+
49
+ You choose the source. The library accepts local files and caller-supplied data;
50
+ it does not query DocDB, reconcile stores, or decide which record is authoritative.
51
+ Image loading, registration, and transform application stay with their existing
52
+ libraries.
53
+
54
+ ## Get started with a historical record
55
+
56
+ Prepare the checkout environment:
57
+
58
+ ```sh
59
+ uv sync
60
+ ```
61
+
62
+ Read the included processing example from `aind-data-schema` v1.4.0:
63
+
64
+ ```python
65
+ from aind_metadata_reader import ReadStatus, read_json, read_processing
66
+
67
+ record = read_json("tests/fixtures/history/v1.4.0-processing.json")
68
+ processing = read_processing(record)
69
+
70
+ print(record.schema_version) # 1.1.4: the core schema version, not the package version
71
+ for process in processing.processes:
72
+ print(process.process_type, process.code_version)
73
+ ```
74
+
75
+ The example contains three pipeline steps and two separately recorded analyses.
76
+ Their views retain the original ordering within each group and their actual source
77
+ pointers. The reader does not construct a dependency graph from that ordering.
78
+
79
+ With your own data, replace the fixture path with a processing file or an aggregate
80
+ such as `metadata.nd.json`. `read_processing` selects the processing core from an
81
+ aggregate; `read_acquisition` does the same for acquisition metadata.
82
+
83
+ The main objects have different jobs:
84
+
85
+ | Object | What it holds |
86
+ | --- | --- |
87
+ | `Snapshot` | The complete preserved payload, source provenance, and original bytes when supplied. |
88
+ | `Document` | An object selected within a snapshot. Children retain the same root snapshot. |
89
+ | `Processing`, `Process`, `Acquisition`, `CoordinateFrame` | Views of recognized concepts in a historical representation. |
90
+ | `ReadValue` | A requested interpretation, its availability, diagnostics, and raw source field. |
91
+ | `Interpretation` | Original and derived views, applied changes, retained rule decisions, and a manifest. |
92
+
93
+ For data fetched by your own client, use `Document(mapping, provenance=Source(...))`.
94
+ A `Source` can retain an S3 locator or a DocDB host, API version, database,
95
+ collection, and record ID. Supply known conversion evidence explicitly.
96
+
97
+ Inspect recorded content directly with `record.raw` or
98
+ `record.field("/processing_pipeline/pipeline_version")`. `raw` and field values
99
+ return copies. A field's `state` distinguishes `present`, `null`, and `absent`.
100
+ Pointers passed to a document are relative to that document; returned field pointers
101
+ identify the location in the complete snapshot.
102
+
103
+ ## Check what an interpretation can establish
104
+
105
+ Selecting a process by type can produce several matches. `find()` returns them
106
+ all; `select()` reports whether exactly one matches:
107
+
108
+ ```python
109
+ selected = processing.select("Image tile fusing")
110
+ if selected.status == ReadStatus.AVAILABLE:
111
+ fusion = selected.value
112
+ print(fusion.document.pointer) # /processing_pipeline/data_processes/0
113
+ print(fusion.code.version.value) # 0.1
114
+
115
+ inputs = fusion.inputs
116
+ if inputs.status == ReadStatus.AVAILABLE:
117
+ for reference in inputs.value:
118
+ print(reference.kind, reference.location.value)
119
+ else:
120
+ print(selected.status, selected.diagnostics)
121
+
122
+ assert processing.select("Analysis").status == ReadStatus.AMBIGUOUS
123
+ ```
124
+
125
+ Code version and pipeline version are separate concepts. `process.code` exposes
126
+ independent `version`, `url`, `parameters`, and `inputs` results. Input/output
127
+ references distinguish `file`, `asset`, `asset_group`, and `unknown`; asset groups
128
+ retain their members. An asset URL does not establish a file-level input.
129
+
130
+ A `ReadValue` gives you four things:
131
+
132
+ | Attribute | Meaning |
133
+ | --- | --- |
134
+ | `status` | Whether the requested interpretation is available, and why it may be unavailable. |
135
+ | `value` | The interpreted value when available; otherwise None. |
136
+ | `field` | The actual recorded field, including its raw value, pointer, and snapshot. |
137
+ | `diagnostics` | Structured explanations with codes and source locations. |
138
+
139
+ Availability describes the requested interpretation:
140
+
141
+ | Status | What it tells you |
142
+ | --- | --- |
143
+ | `available` | The reader established a value. |
144
+ | `absent` | The source field was not recorded. |
145
+ | `null` | The source explicitly recorded JSON null. |
146
+ | `invalid` | The recorded content does not fit this requested interpretation. |
147
+ | `unsupported` | The reader has no established interpretation for this representation or meaning. |
148
+ | `ambiguous` | Several candidates exist and none has been selected. |
149
+
150
+ `invalid` describes one attempted interpretation; it does not declare the entire
151
+ metadata record invalid.
152
+
153
+ Convenience accessors such as `code_version`, `parameters`, `processes`, and
154
+ `array_axes` make common cases shorter. Use the corresponding component results
155
+ when missing information matters. An empty `processes` tuple, for example, could
156
+ mean an empty recorded collection or an unsupported representation:
157
+ `process_entries.status` distinguishes those situations.
158
+
159
+ `selected.require()`, `processing.one(type)`, and `acquisition.one_frame()` explicitly
160
+ assert a requirement and can raise. Ordinary inspection uses status-bearing results.
161
+
162
+ ## Incomplete metadata stays inspectable
163
+
164
+ This illustrative record has a numeric code version and a malformed second entry:
165
+
166
+ ```python
167
+ from aind_metadata_reader import Document
168
+
169
+ incomplete = Document(
170
+ {
171
+ "schema_version": "2.0.0",
172
+ "data_processes": [
173
+ {
174
+ "process_type": "An unfamiliar processing term",
175
+ "name": "example step",
176
+ "code": {"version": 42, "parameters": {"threshold": 12}},
177
+ "output_path": "/results/image.zarr",
178
+ },
179
+ "unfinished entry",
180
+ ],
181
+ }
182
+ )
183
+ inspected = read_processing(incomplete)
184
+ first, second = inspected.process_entries.value
185
+ step = first.value
186
+
187
+ assert step.code.version.status == ReadStatus.INVALID
188
+ assert step.code.version.field.value == 42
189
+ assert step.code.parameters.value == {"threshold": 12}
190
+ assert step.output_location == "/results/image.zarr"
191
+ assert second.status == ReadStatus.INVALID
192
+ assert second.field.value == "unfinished entry"
193
+ assert inspected.raw == incomplete.raw
194
+ ```
195
+
196
+ Unfamiliar vocabulary remains recorded text. Unknown schema versions produce
197
+ unsupported interpretations with the complete raw core still accessible. The
198
+ reader does not assume compatibility from a major version or interpret a new
199
+ layout by guessing from familiar field names.
200
+
201
+ Raw capture requires a finite JSON object. Malformed JSON, duplicate object keys,
202
+ and nonfinite numbers are parsing errors; schema-invalid content within a parsed
203
+ record is retained.
204
+
205
+ ## Inspect coordinate frames separately from array dimensions
206
+
207
+ Acquisition views keep local and global coordinate systems separate. Select a
208
+ scope or a source pointer when several frames exist:
209
+
210
+ ```python
211
+ from aind_metadata_reader import read_acquisition
212
+
213
+ acquisition = read_acquisition(read_json("tests/fixtures/v2.9.0-exaspim_acquisition.json"))
214
+ selected_frame = acquisition.select_frame(scope="local")
215
+ if selected_frame.status == ReadStatus.AVAILABLE:
216
+ frame = selected_frame.value
217
+ print(frame.name)
218
+ print([(axis.name, axis.position, axis.dimension) for axis in frame.axes])
219
+ print(frame.orientation.status, frame.orientation.value)
220
+ ```
221
+
222
+ `Axis.position` follows source coordinate order. `Axis.dimension` is an image-array
223
+ dimension only where recorded. `array_mapping` reports whether a mapping is
224
+ established; mapping it to a particular Zarr array still requires evidence about
225
+ that array's dimensions. A frame can have a known anatomical orientation while
226
+ its array mapping remains unavailable.
227
+
228
+ Configured metadata supports stream and device selection. Legacy imaging metadata
229
+ exposes scope `legacy` and reports unavailable stream/device selectors. Recorded
230
+ image transforms remain accessible through `frame.configuration`; the reader does
231
+ not compose them or substitute a local frame for a global one.
232
+
233
+ ## Request validation as a separate report
234
+
235
+ Install the optional validator in the checkout with `uv sync --extra validation`.
236
+ Supply the historical schema explicitly:
237
+
238
+ ```python
239
+ from aind_metadata_reader import validate
240
+
241
+ schema = read_json("tests/fixtures/history/v1.4.0-processing-schema.json")
242
+ report = validate(
243
+ record,
244
+ schema,
245
+ dialect="https://json-schema.org/draft/2020-12/schema",
246
+ )
247
+ assert report.status == "valid"
248
+ assert processing.validation.status == "not_checked"
249
+ ```
250
+
251
+ `validate()` returns a separate report and leaves the record and its views unchanged.
252
+ Reports distinguish `not_checked`, `valid`, `invalid`, and `unavailable`. An unusable
253
+ schema, missing validator, or unresolved external reference means unavailable
254
+ validation. Each report identifies the input, schema, and chosen dialect. A dialect
255
+ must be supplied when the schema does not declare one.
256
+
257
+ This checks JSON Schema assertions using `jsonschema`. It does not insert defaults,
258
+ coerce values, fetch remote schemas, run custom upstream Pydantic validators, or
259
+ check format annotations. A successful report establishes conformance to those
260
+ assertions; scientific accuracy and source authority remain separate questions.
261
+ Reading has no mandatory validator or upstream-schema dependency.
262
+
263
+ ## Derive an attributed correction
264
+
265
+ A correction produces a new interpretation. The following example assumes a
266
+ calibration note establishes a revised spacing:
267
+
268
+ ```python
269
+ from aind_metadata_reader import Change, apply_changes
270
+
271
+ calibration_record = Document({"voxel_spacing_um": 2.0})
272
+ change = Change.propose(
273
+ calibration_record,
274
+ "/voxel_spacing_um",
275
+ 1.8,
276
+ operation_id="spacing-calibration",
277
+ rule_id="manual.voxel-spacing",
278
+ rule_revision="1",
279
+ implementation="calibration-note/1",
280
+ kind="correction",
281
+ author="experimenter",
282
+ reason="Updated calibration measurement",
283
+ evidence=("calibration-record-123",),
284
+ scope="physical spacing in micrometers",
285
+ )
286
+ result = apply_changes(calibration_record, [change], applied_at="2026-11-10T12:00:00+00:00")
287
+ assert result.original.raw == {"voxel_spacing_um": 2.0}
288
+ assert result.derived.raw == {"voxel_spacing_um": 1.8}
289
+ ```
290
+
291
+ The attribution arguments become the audit trail. Operation kinds distinguish
292
+ `correction`, `inference` (requires assumptions), and `processing_frame`
293
+ (the conventions of a historical computation). Physical corrections and historical
294
+ processing frames retain distinct identities even if their resulting values match.
295
+
296
+ Application checks the exact input capture, original field value, selected document
297
+ scope, and conflicting edits. It does not require unrelated metadata to validate.
298
+ Original snapshots stay intact, and no-op operations remain recorded.
299
+
300
+ Domain rules propose changes through `RuleEvaluation`, whose outcome is
301
+ `applicable`, `not_applicable`, or `unknown`. For example,
302
+ `smartspim.evaluate_ls17` checks known fusion versions, pyramid geometry, and
303
+ caller-supplied association and block-averaging evidence. Reading and evaluation
304
+ never apply its proposals. Unknown producer versions do not activate the correction;
305
+ no first-fixed fusion version has been established.
306
+
307
+ Pass decisions to `apply_changes(..., evaluations=(evaluation,))` to retain their
308
+ inputs, reader provenance, and links to applied operations. Unknown decisions and
309
+ proposals deliberately left unapplied can also be retained. Archive supporting
310
+ snapshots along with the target snapshot. The [API guide](docs/api.md) describes
311
+ the rule and saved-record contracts in more detail.
312
+
313
+ ## Save the inputs and replay recorded changes
314
+
315
+ Save the source archive and interpretation manifest with your analysis:
316
+
317
+ ```python
318
+ import json
319
+ from pathlib import Path
320
+ from aind_metadata_reader import Snapshot, replay
321
+
322
+ output = Path("analysis-provenance")
323
+ output.mkdir(exist_ok=True)
324
+ (output / "source.json").write_text(json.dumps(calibration_record.snapshot.archive(), indent=2))
325
+ (output / "interpretation.json").write_text(json.dumps(result.manifest(), indent=2))
326
+
327
+ restored = Document.from_snapshot(Snapshot.from_archive(read_json(output / "source.json").raw))
328
+ again = replay(restored, read_json(output / "interpretation.json").raw)
329
+ assert again.derived.snapshot.snapshot_id == result.derived.snapshot.snapshot_id
330
+ ```
331
+
332
+ Replay applies recorded replacements to matching archived inputs. It does not rerun
333
+ the latest reader or rule. Reevaluation is a separate computation using a selected
334
+ implementation and its evidence.
335
+
336
+ `content_digest` identifies parsed values; `original_digest` identifies original
337
+ bytes when retained. `snapshot_id` also incorporates source provenance, including
338
+ retrieval evidence. Derived snapshots identify their parent snapshot and
339
+ interpretation. Restore archives for exact replay: rereading a live file creates
340
+ a new capture even if its bytes are unchanged. These hashes detect mismatches;
341
+ they do not authenticate evidence.
342
+
343
+ Archives and manifests have independent format versions. New records use `/2`,
344
+ and explicit readers retain `/1` support through fixed historical fixtures.
345
+ Persisted-record loaders and replay check their contracts strictly; the records
346
+ can always be inspected separately as JSON.
347
+
348
+ ## Supported representations and current limits
349
+
350
+ These are core schema versions, independent of Python package, pipeline, and
351
+ component-code versions. A listed format describes tested interpretation
352
+ capabilities; individual records can still contain unavailable components.
353
+
354
+ | Representation | Core versions exercised | Concepts exposed |
355
+ | --- | --- | --- |
356
+ | AIND legacy acquisition | 1.0.0, 1.0.3, 1.0.4 | Recorded axes and image dimensions |
357
+ | AIND configured acquisition | 2.0.0, 2.5.3, 2.5.4 | Frames and configuration references |
358
+ | biodata acquisition | 3.0.0 | Frames and configuration references |
359
+ | AIND legacy processing | 1.1.0, 1.1.4 | Processes, analyses, and file inputs |
360
+ | AIND flat processing | 2.0.0, 2.2.5, 2.3.1, 2.3.2, 2.3.3 | Processes, pipeline links, and asset/group inputs |
361
+ | biodata processing | 3.0.0 | Processes, pipeline links, and asset/group inputs |
362
+
363
+ Unversioned legacy structures are recognized explicitly. Unversioned configured
364
+ records need a recorded caller-supplied `format_hint`; hints cannot override
365
+ contradictory declarations. Unknown versions retain their raw records and diagnostics.
366
+
367
+ Tests include tagged historical examples/schemas, public production metadata, and
368
+ paired v1/v2 DocDB representations. Constructed cases exercise documented historical
369
+ changes such as output cardinality. V3 support is provisional against upstream
370
+ examples until production outputs are available. See [fixture provenance](tests/fixtures/README.md)
371
+ and [historical evolution evidence](tests/fixtures/history/README.md).
372
+
373
+ Parsed integers retain Python precision; fractional values use binary64. Original
374
+ bytes preserve source number spellings. Content hashing uses the documented Python
375
+ JSON convention, not RFC 8785 canonicalization.
376
+
377
+ ## Development
378
+
379
+ See [CONTRIBUTING.md](CONTRIBUTING.md) for setup and checks, the
380
+ [design proposal](planning/design.md) for rationale, and
381
+ [implementation status](planning/implementation-status.md) for remaining work.
382
+ The project was rendered from `aind-python-copier-template` commit
383
+ `5dc04dcba3b3c16b8ee9c4b7c1c2314c08567bfc`. Repository setup and CI are configured. Tagged releases publish source and wheel
384
+ artifacts through the release workflow. Licensed under [MIT](LICENSE).
385
+
386
+
387
+ ## SmartSPIM Registration Inputs
388
+
389
+ `aind_metadata_reader.smartspim.registration_input(process)` returns an inspectable
390
+ `ReadValue` of file references for an `Image importing` process. It preserves
391
+ legacy file inputs and recognizes the observed
392
+ `aind-smartspim-ccf-registration` 0.0.37 convention that stores the input under
393
+ `output_parameters.input_location`. Each reference retains its source pointer.
394
+ Other producer versions need independent evidence before this convention applies;
395
+ an asset-level input is not converted into a file path. This helper reads
396
+ provenance and does not apply spatial corrections.
@@ -0,0 +1,18 @@
1
+ aind_metadata_reader/__init__.py,sha256=U_YkUIJmygiB_TwEJtYcI39SCXcDhs0ROzEvP9MzPMA,1758
2
+ aind_metadata_reader/_enum.py,sha256=geg3z1TFJjwEAzcYKXnUCx_byItU82uu1Bg4PHCEm00,295
3
+ aind_metadata_reader/_wire.py,sha256=usPHBfOTyU77higfBV-ITC6ipXnwBWqjKzOc8c-UbRY,3298
4
+ aind_metadata_reader/applicability.py,sha256=QJlPqmshnC1dAuAypxOE65Ij3wHTHj-EgdaZisv7s-c,6359
5
+ aind_metadata_reader/components.py,sha256=DM94jvMvGGEMqFnaDZZJbJxHJ58LimBSJF21CJ0IgaU,5716
6
+ aind_metadata_reader/diagnostics.py,sha256=QBiSIBQzVQzwk28XwktgkdlExEgyBKKGK-7e82GUiGk,1133
7
+ aind_metadata_reader/document.py,sha256=xuX8C9i9gWfneUXk924CNkQ-K4HW6LloxVubMEvXmaU,16856
8
+ aind_metadata_reader/formats.py,sha256=sqIsSPPWkql7KcTJq3lK1svhehTy6Jp2kvHZP25PazE,6887
9
+ aind_metadata_reader/inspection.py,sha256=R7QitrQZjUIkVsuFtVF95pWxLIOQ1ypKriggXxkBgPQ,4503
10
+ aind_metadata_reader/interpretation.py,sha256=lRLkup3NerKnkDLSEhFD1UMLKiSR-aNvrqyeBuUQqTs,20724
11
+ aind_metadata_reader/py.typed,sha256=AbpHGcgLb-kRsJGnwFEktk7uzpZOCcBY74-YBdrKVGs,1
12
+ aind_metadata_reader/readers.py,sha256=0bmR-qQ_-fm3UHZgZr_bpGO4NKNIDmAlhocEpQL4L9w,23730
13
+ aind_metadata_reader/smartspim.py,sha256=ygSeDhhLGNtQcrSlGEX6U16Z4PeFuvqu-QG2GSKwIyI,11042
14
+ aind_metadata_reader/validation.py,sha256=vnIsX_1HkacJEn_yD_uhTBMIVmURHn6h7VpBkJ3HZu4,3860
15
+ aind_metadata_reader-0.2.0.dist-info/METADATA,sha256=9kLHF9wIOiD7Nw_arJIvi8o0wDJWKHpmm6LwvcpqUUw,18120
16
+ aind_metadata_reader-0.2.0.dist-info/WHEEL,sha256=2iZrJkDIkeDh0k9SikKMqFuRP-ua6mb4Ir1AsER4BEw,87
17
+ aind_metadata_reader-0.2.0.dist-info/licenses/LICENSE,sha256=zMZxYbR1i_XxgyJP_rPSWKcxY7QtyxUOzbIpowHNjCs,1093
18
+ aind_metadata_reader-0.2.0.dist-info/RECORD,,
@@ -0,0 +1,4 @@
1
+ Wheel-Version: 1.0
2
+ Generator: hatchling 1.33.0
3
+ Root-Is-Purelib: true
4
+ Tag: py3-none-any
@@ -0,0 +1,22 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Allen Institute for Neural Dynamics
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
22
+