vector-mtrl 0.2.0__tar.gz → 0.3.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/.gitignore +3 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/CHANGELOG.md +25 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/PKG-INFO +46 -8
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/README.md +44 -6
- vector_mtrl-0.3.0/docs/user/concepts/01-samples-runs-checkpoints.md +75 -0
- vector_mtrl-0.3.0/docs/user/concepts/02-the-commit-boundary.md +76 -0
- vector_mtrl-0.3.0/docs/user/concepts/03-two-kinds-of-identity.md +75 -0
- vector_mtrl-0.3.0/docs/user/concepts/04-corrections-vs-replacements.md +66 -0
- vector_mtrl-0.3.0/docs/user/concepts/05-lineage-and-the-recorded-recipe.md +77 -0
- vector_mtrl-0.3.0/docs/user/concepts/06-revisions-the-archive-timeline.md +69 -0
- vector_mtrl-0.3.0/docs/user/concepts/07-inside-a-mtrl-archive.md +76 -0
- vector_mtrl-0.3.0/docs/user/concepts/08-trust-over-time.md +91 -0
- vector_mtrl-0.3.0/docs/user/concepts/index.md +30 -0
- vector_mtrl-0.3.0/docs/user/how-to/align-mixed-resolution-members.md +104 -0
- vector_mtrl-0.3.0/docs/user/how-to/anchor-verification-with-receipts.md +108 -0
- vector_mtrl-0.3.0/docs/user/how-to/attach-author-disclosure.md +135 -0
- vector_mtrl-0.3.0/docs/user/how-to/check-external-evidence.md +111 -0
- vector_mtrl-0.3.0/docs/user/how-to/control-the-operation-registry.md +123 -0
- vector_mtrl-0.3.0/docs/user/how-to/handle-missing-extras.md +58 -0
- vector_mtrl-0.3.0/docs/user/how-to/index.md +44 -0
- vector_mtrl-0.3.0/docs/user/how-to/inspect-a-damaged-archive.md +86 -0
- vector_mtrl-0.3.0/docs/user/how-to/prepare-an-ml-dataset.md +157 -0
- vector_mtrl-0.3.0/docs/user/how-to/record-scientific-assertions.md +106 -0
- vector_mtrl-0.3.0/docs/user/how-to/replace-a-checkpoint.md +89 -0
- vector_mtrl-0.3.0/docs/user/how-to/replay-a-committed-chain.md +116 -0
- vector_mtrl-0.3.0/docs/user/how-to/resample-with-scipy.md +96 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/docs/user/reference.md +127 -5
- vector_mtrl-0.3.0/docs/user/start-here/01-your-first-record.md +98 -0
- vector_mtrl-0.3.0/docs/user/start-here/02-save-load-verify.md +163 -0
- vector_mtrl-0.3.0/docs/user/start-here/03-fields-and-units.md +103 -0
- vector_mtrl-0.3.0/docs/user/start-here/04-transform-a-spectrum.md +106 -0
- vector_mtrl-0.3.0/docs/user/start-here/05-fix-a-mistake.md +136 -0
- vector_mtrl-0.3.0/docs/user/start-here/06-replicates-and-the-mean.md +137 -0
- vector_mtrl-0.3.0/docs/user/start-here/07-get-your-data-out.md +221 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/_archive_integrity.py +203 -18
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/_archive_loading.py +58 -18
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/_archive_serialise.py +288 -30
- vector_mtrl-0.3.0/mtrl/_assertions_io.py +410 -0
- vector_mtrl-0.3.0/mtrl/_assertions_ops.py +987 -0
- vector_mtrl-0.3.0/mtrl/_assertions_validation.py +664 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/_candidate_matching.py +1 -1
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/_optional.py +22 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/_provenance.py +1 -1
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/_transform_validation.py +2 -2
- vector_mtrl-0.3.0/mtrl/_uuid7.py +88 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/archive/reader.py +212 -1
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/archive/validation.py +37 -13
- vector_mtrl-0.3.0/mtrl/assertions/__init__.py +38 -0
- vector_mtrl-0.3.0/mtrl/assertions/_resolver.py +137 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/domain/__init__.py +20 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/domain/archive_limits.py +3 -1
- vector_mtrl-0.3.0/mtrl/domain/assertions.py +812 -0
- vector_mtrl-0.3.0/mtrl/domain/identities.py +352 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/domain/verification.py +51 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/exceptions.py +28 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/features.py +6 -1
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/frames.py +199 -3
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/grids.py +467 -38
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/integrity/digests.py +30 -10
- vector_mtrl-0.3.0/mtrl/integrity/manifest.py +1173 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/integrity/verification.py +22 -2
- vector_mtrl-0.3.0/mtrl/ml/__init__.py +67 -0
- vector_mtrl-0.3.0/mtrl/ml/_adapters.py +787 -0
- vector_mtrl-0.3.0/mtrl/ml/_canonical.py +203 -0
- vector_mtrl-0.3.0/mtrl/ml/_dataset.py +919 -0
- vector_mtrl-0.3.0/mtrl/ml/_group_keys.py +107 -0
- vector_mtrl-0.3.0/mtrl/ml/_metadata_view.py +488 -0
- vector_mtrl-0.3.0/mtrl/ml/_preprocessing.py +1772 -0
- vector_mtrl-0.3.0/mtrl/ml/_provenance.py +682 -0
- vector_mtrl-0.3.0/mtrl/ml/_source_walk.py +371 -0
- vector_mtrl-0.3.0/mtrl/ml/_specs.py +788 -0
- vector_mtrl-0.3.0/mtrl/ml/_splits.py +1962 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/models.py +46 -3
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/mtrl.py +102 -40
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/operations/_validation.py +5 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/operations/resampling.py +9 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/operations/resampling_scipy.py +8 -5
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/processing/registry.py +9 -4
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/replay.py +4 -10
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/units.py +25 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/pyproject.toml +11 -6
- vector_mtrl-0.2.0/docs/user/getting-started.md +0 -188
- vector_mtrl-0.2.0/docs/user/workflows.md +0 -576
- vector_mtrl-0.2.0/mtrl/domain/identities.py +0 -174
- vector_mtrl-0.2.0/mtrl/integrity/manifest.py +0 -631
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/LICENSE +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/NOTICE +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/__init__.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/_alignment_report.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/_archive_documents.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/_cli.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/_disclosure.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/_ingestion_shortcuts.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/_install_targets.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/_receipts.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/_record_resolution.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/_replay_plan.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/_replay_run.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/_run_management.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/_source_capture.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/_table_plan.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/_table_provenance.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/_table_values.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/_transform_apply.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/archive/__init__.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/archive/canonical.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/archive/codecs.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/archive/dtypes.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/connectors/__init__.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/connectors/protocols.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/disclosure.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/domain/committed_values.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/domain/frozen_containers.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/domain/provenance.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/domain/records.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/domain/revisions.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/domain/source.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/excel.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/exports.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/integrity/__init__.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/integrity/canonical.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/integrity/receipts.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/operations/__init__.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/operations/_identity.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/operations/arithmetic.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/operations/base.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/operations/spectra.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/processing/__init__.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/processing/_table_capture.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/processing/limits.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/protocols.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/py.typed +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/tables.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/transformations/__init__.py +0 -0
- {vector_mtrl-0.2.0 → vector_mtrl-0.3.0}/mtrl/transformations/base.py +0 -0
|
@@ -10,6 +10,31 @@ break, and every breaking change is named in the release entry.
|
|
|
10
10
|
|
|
11
11
|
## [Unreleased]
|
|
12
12
|
|
|
13
|
+
### Added
|
|
14
|
+
|
|
15
|
+
- Scientific assertions: claims, versions, subjects, evidence, and supersession as a core
|
|
16
|
+
feature, plus the portable UUIDv7 physical-sample identity `sample_uid`
|
|
17
|
+
(`Mtrl.sample_uid` and `Mtrl.assign_sample_uid()`).
|
|
18
|
+
- The `mtrl.ml` dataset builder with channel features, grid alignment, preprocessing
|
|
19
|
+
lineage, grouped and stratified splits, metadata views, and scikit-learn export.
|
|
20
|
+
- Five ergonomics wrappers: `Unit.from_raw_map()` builds a units mapping from reported
|
|
21
|
+
text in one call, `AggregatedRun.checkpoint` returns the single combined checkpoint,
|
|
22
|
+
`Mtrl.active_ref()` names the active record for a checkpoint as an exact `RecordRef`,
|
|
23
|
+
`VerifyResult.format_summary()` prints one line per independent status, and
|
|
24
|
+
`VerifyIssue.explain()` with `VerifyResult.explain_issues()` turns issues into
|
|
25
|
+
readable sentences.
|
|
26
|
+
- Public user documentation: a seven-lesson tutorial path, nine concept pages, and
|
|
27
|
+
thirteen how-to guides under `docs/user/`, plus a Starlight documentation site under
|
|
28
|
+
`docs/site/`. The core sdist ships the rooted guide tree.
|
|
29
|
+
|
|
30
|
+
### Changed
|
|
31
|
+
|
|
32
|
+
- Python support widens from 3.14-only to 3.12 through 3.x below 4.0.
|
|
33
|
+
- The source-tree version fallback names `0.3.0` when installed distribution metadata is
|
|
34
|
+
absent.
|
|
35
|
+
|
|
36
|
+
## [0.2.0] - 2026-10-22
|
|
37
|
+
|
|
13
38
|
### Changed
|
|
14
39
|
|
|
15
40
|
- Prepare `0.2.0` as the first public-core release candidate with the interp and tabular
|
|
@@ -1,12 +1,12 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: vector-mtrl
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.3.0
|
|
4
4
|
Summary: Materials research tools for spectra and archive workflows.
|
|
5
5
|
Author-email: Fuzzy Labs <info@fuzzylabs.ai>
|
|
6
6
|
License-Expression: MPL-2.0
|
|
7
7
|
License-File: LICENSE
|
|
8
8
|
License-File: NOTICE
|
|
9
|
-
Requires-Python: <4.0,>=3.
|
|
9
|
+
Requires-Python: <4.0,>=3.12
|
|
10
10
|
Requires-Dist: numpy>=2.5.2
|
|
11
11
|
Requires-Dist: pydantic>=2.13.4
|
|
12
12
|
Requires-Dist: rfc8785==0.1.4
|
|
@@ -27,6 +27,11 @@ immutable record with exact lineage.
|
|
|
27
27
|
## What you get
|
|
28
28
|
|
|
29
29
|
- Immutable committed records with stable identities and complete lineage.
|
|
30
|
+
- A portable physical sample identity (`sample_uid`) beside the human-facing `sample_id`.
|
|
31
|
+
- Scientific assertions with typed subjects, evidence links, supersession, retraction, and
|
|
32
|
+
queries at any past revision.
|
|
33
|
+
- Machine-learning dataset preparation across archives: checkpoint state selection, in-memory
|
|
34
|
+
preprocessing, grid alignment, leak-free group splits, and canonical provenance digests.
|
|
30
35
|
- Bounded `.mtrl` archives with an integrity manifest, external receipts, and forensic inspection.
|
|
31
36
|
- NumPy-only operations: identity, scale, offset, peak normalisation, zero-at-point, and linear
|
|
32
37
|
resample, beside mean and median aggregates.
|
|
@@ -39,7 +44,7 @@ immutable record with exact lineage.
|
|
|
39
44
|
|
|
40
45
|
## Install
|
|
41
46
|
|
|
42
|
-
`vector-mtrl` requires Python 3.
|
|
47
|
+
`vector-mtrl` requires Python 3.12 or newer, below Python 4. The base install depends only on
|
|
43
48
|
NumPy, Pydantic, and RFC 8785 canonical JSON. Create one environment and install the core:
|
|
44
49
|
|
|
45
50
|
```sh
|
|
@@ -110,14 +115,47 @@ Every mutation, including the transform above, creates one new revision.
|
|
|
110
115
|
The expanded guides ship in the source archive under `docs/user/`. Download the archive for a
|
|
111
116
|
release from https://pypi.org/project/vector-mtrl/#files, then read:
|
|
112
117
|
|
|
113
|
-
- `docs/user/
|
|
114
|
-
and
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
+
- `docs/user/start-here/` for the seven-lesson tutorial path: installation, first records,
|
|
119
|
+
the save, load, verify loop, fields and units, transforms, corrections, aggregates, and
|
|
120
|
+
exports.
|
|
121
|
+
- `docs/user/concepts/` for the ideas underneath, in plain language.
|
|
122
|
+
- `docs/user/how-to/` for task guides: replacements, resampling, mixed-resolution alignment,
|
|
123
|
+
replay, receipts, forensic inspection, disclosure, registry control, missing extras,
|
|
124
|
+
scientific assertions, external evidence, and machine-learning datasets.
|
|
118
125
|
- `docs/user/reference.md` for every public route, the stable typed imports, the exact
|
|
119
126
|
package-root export list, registry names, format features, and the archive safety contract.
|
|
120
127
|
|
|
128
|
+
## Scientific assertions and machine-learning datasets
|
|
129
|
+
|
|
130
|
+
`Mtrl.assign_sample_uid()` gives a container a portable physical sample identity, a UUIDv7
|
|
131
|
+
string that stays separate from the `sample_id` label. The `mtrl.assertions` module then
|
|
132
|
+
records typed property claims about the sample, one run, or one exact record:
|
|
133
|
+
|
|
134
|
+
- `SubjectRef` names the subject, and `EvidenceRef` links local records, local source files,
|
|
135
|
+
or exact records in other archives.
|
|
136
|
+
- `Mtrl.assert_property()` and `Mtrl.assert_properties()` commit claims in one atomic revision.
|
|
137
|
+
- `Mtrl.supersede_assertion()` and `Mtrl.retract_assertion()` change a claim and keep every
|
|
138
|
+
prior version in the audit history.
|
|
139
|
+
- `Mtrl.list_assertions(as_of_revision=...)` reads the claim state at any past revision.
|
|
140
|
+
- `resolve_external_evidence()` checks cross-archive evidence offline against the archives you
|
|
141
|
+
supply. Loading an archive never opens another archive.
|
|
142
|
+
|
|
143
|
+
The `mtrl.ml` package prepares training and evaluation datasets across saved archives with
|
|
144
|
+
`build_ml_dataset()` and `DatasetSpec`:
|
|
145
|
+
|
|
146
|
+
- `FeatureChannelSpec` selects committed checkpoint states and runs governed operations in
|
|
147
|
+
memory; `GridAlignmentSpec` aligns heterogeneous spectra onto one coordinate grid.
|
|
148
|
+
- `TargetSpec` extracts labels or values from active assertions, and `FilterSpec` removes
|
|
149
|
+
duplicate sources and excluded runs with machine-readable reasons.
|
|
150
|
+
- `MLDataset.split_grouped()` partitions rows by `sample_uid` or another group without
|
|
151
|
+
leakage, and the provenance manifest holds one canonical `dataset_digest`.
|
|
152
|
+
- `MLDataset.to_sklearn()` needs no scikit-learn. `to_torch_dataset()`, `to_tf_dataset()`, and
|
|
153
|
+
`log_to_mlflow()` load their frameworks lazily, and `to_pandas()` needs the `tabular` extra.
|
|
154
|
+
|
|
155
|
+
The base install imports `mtrl.assertions` and `mtrl.ml` without SciPy, pandas, or any external
|
|
156
|
+
ML framework. Read `docs/user/how-to/record-scientific-assertions.md` and
|
|
157
|
+
`docs/user/how-to/prepare-an-ml-dataset.md` for runnable examples of both workflows.
|
|
158
|
+
|
|
121
159
|
## Archive and provenance limits
|
|
122
160
|
|
|
123
161
|
MTRL reads `.mtrl` archives inside configured size and member limits, and it validates paths,
|
|
@@ -6,6 +6,11 @@ immutable record with exact lineage.
|
|
|
6
6
|
## What you get
|
|
7
7
|
|
|
8
8
|
- Immutable committed records with stable identities and complete lineage.
|
|
9
|
+
- A portable physical sample identity (`sample_uid`) beside the human-facing `sample_id`.
|
|
10
|
+
- Scientific assertions with typed subjects, evidence links, supersession, retraction, and
|
|
11
|
+
queries at any past revision.
|
|
12
|
+
- Machine-learning dataset preparation across archives: checkpoint state selection, in-memory
|
|
13
|
+
preprocessing, grid alignment, leak-free group splits, and canonical provenance digests.
|
|
9
14
|
- Bounded `.mtrl` archives with an integrity manifest, external receipts, and forensic inspection.
|
|
10
15
|
- NumPy-only operations: identity, scale, offset, peak normalisation, zero-at-point, and linear
|
|
11
16
|
resample, beside mean and median aggregates.
|
|
@@ -18,7 +23,7 @@ immutable record with exact lineage.
|
|
|
18
23
|
|
|
19
24
|
## Install
|
|
20
25
|
|
|
21
|
-
`vector-mtrl` requires Python 3.
|
|
26
|
+
`vector-mtrl` requires Python 3.12 or newer, below Python 4. The base install depends only on
|
|
22
27
|
NumPy, Pydantic, and RFC 8785 canonical JSON. Create one environment and install the core:
|
|
23
28
|
|
|
24
29
|
```sh
|
|
@@ -89,14 +94,47 @@ Every mutation, including the transform above, creates one new revision.
|
|
|
89
94
|
The expanded guides ship in the source archive under `docs/user/`. Download the archive for a
|
|
90
95
|
release from https://pypi.org/project/vector-mtrl/#files, then read:
|
|
91
96
|
|
|
92
|
-
- `docs/user/
|
|
93
|
-
and
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
+
- `docs/user/start-here/` for the seven-lesson tutorial path: installation, first records,
|
|
98
|
+
the save, load, verify loop, fields and units, transforms, corrections, aggregates, and
|
|
99
|
+
exports.
|
|
100
|
+
- `docs/user/concepts/` for the ideas underneath, in plain language.
|
|
101
|
+
- `docs/user/how-to/` for task guides: replacements, resampling, mixed-resolution alignment,
|
|
102
|
+
replay, receipts, forensic inspection, disclosure, registry control, missing extras,
|
|
103
|
+
scientific assertions, external evidence, and machine-learning datasets.
|
|
97
104
|
- `docs/user/reference.md` for every public route, the stable typed imports, the exact
|
|
98
105
|
package-root export list, registry names, format features, and the archive safety contract.
|
|
99
106
|
|
|
107
|
+
## Scientific assertions and machine-learning datasets
|
|
108
|
+
|
|
109
|
+
`Mtrl.assign_sample_uid()` gives a container a portable physical sample identity, a UUIDv7
|
|
110
|
+
string that stays separate from the `sample_id` label. The `mtrl.assertions` module then
|
|
111
|
+
records typed property claims about the sample, one run, or one exact record:
|
|
112
|
+
|
|
113
|
+
- `SubjectRef` names the subject, and `EvidenceRef` links local records, local source files,
|
|
114
|
+
or exact records in other archives.
|
|
115
|
+
- `Mtrl.assert_property()` and `Mtrl.assert_properties()` commit claims in one atomic revision.
|
|
116
|
+
- `Mtrl.supersede_assertion()` and `Mtrl.retract_assertion()` change a claim and keep every
|
|
117
|
+
prior version in the audit history.
|
|
118
|
+
- `Mtrl.list_assertions(as_of_revision=...)` reads the claim state at any past revision.
|
|
119
|
+
- `resolve_external_evidence()` checks cross-archive evidence offline against the archives you
|
|
120
|
+
supply. Loading an archive never opens another archive.
|
|
121
|
+
|
|
122
|
+
The `mtrl.ml` package prepares training and evaluation datasets across saved archives with
|
|
123
|
+
`build_ml_dataset()` and `DatasetSpec`:
|
|
124
|
+
|
|
125
|
+
- `FeatureChannelSpec` selects committed checkpoint states and runs governed operations in
|
|
126
|
+
memory; `GridAlignmentSpec` aligns heterogeneous spectra onto one coordinate grid.
|
|
127
|
+
- `TargetSpec` extracts labels or values from active assertions, and `FilterSpec` removes
|
|
128
|
+
duplicate sources and excluded runs with machine-readable reasons.
|
|
129
|
+
- `MLDataset.split_grouped()` partitions rows by `sample_uid` or another group without
|
|
130
|
+
leakage, and the provenance manifest holds one canonical `dataset_digest`.
|
|
131
|
+
- `MLDataset.to_sklearn()` needs no scikit-learn. `to_torch_dataset()`, `to_tf_dataset()`, and
|
|
132
|
+
`log_to_mlflow()` load their frameworks lazily, and `to_pandas()` needs the `tabular` extra.
|
|
133
|
+
|
|
134
|
+
The base install imports `mtrl.assertions` and `mtrl.ml` without SciPy, pandas, or any external
|
|
135
|
+
ML framework. Read `docs/user/how-to/record-scientific-assertions.md` and
|
|
136
|
+
`docs/user/how-to/prepare-an-ml-dataset.md` for runnable examples of both workflows.
|
|
137
|
+
|
|
100
138
|
## Archive and provenance limits
|
|
101
139
|
|
|
102
140
|
MTRL reads `.mtrl` archives inside configured size and member limits, and it validates paths,
|
|
@@ -0,0 +1,75 @@
|
|
|
1
|
+
# Samples, runs, and checkpoints
|
|
2
|
+
|
|
3
|
+
Think of a lab binder. You keep one binder for each physical sample you characterise. Inside
|
|
4
|
+
the binder, each tab collects everything from one measurement session. Behind each tab sit
|
|
5
|
+
printed pages: one page for each table of data you chose to save. You can always add new
|
|
6
|
+
tabs and new pages, and the binder is organised so that you can find any page again, even
|
|
7
|
+
years later.
|
|
8
|
+
|
|
9
|
+
MTRL organises your data in exactly this shape, with three levels: the sample container,
|
|
10
|
+
the instrument run, and the checkpoint.
|
|
11
|
+
|
|
12
|
+
## One application per sample
|
|
13
|
+
|
|
14
|
+
The **application** (the `Mtrl` object) is the binder. One application holds one physical
|
|
15
|
+
sample, named by its sample identifier, such as `polymer-film-01`. Everything you commit
|
|
16
|
+
to that application — every run, every table, every correction — belongs to that sample and
|
|
17
|
+
travels with it when you save the archive.
|
|
18
|
+
|
|
19
|
+
The application keeps its runs in a read-only mapping called `data`. You can look up any
|
|
20
|
+
run, but you cannot add, replace, or delete entries through the mapping itself. All changes
|
|
21
|
+
go through the application, so nothing enters the record without being validated and
|
|
22
|
+
committed. When you read the mapping back, runs appear in commit order: the order in which
|
|
23
|
+
you recorded them, which a save and reload preserves.
|
|
24
|
+
|
|
25
|
+
## One run per measurement session
|
|
26
|
+
|
|
27
|
+
An **instrument run** (the `InstrumentRun` object) is one tab in the binder. It groups the
|
|
28
|
+
metadata and the checkpoints from one measurement session. You give each run a **run name**,
|
|
29
|
+
such as `ftir-replicate-01`, and MTRL generates a separate run identifier for it. A
|
|
30
|
+
replicate measured the next day gets its own run, named `ftir-replicate-02`, even if you
|
|
31
|
+
measured the same sample in the same way.
|
|
32
|
+
|
|
33
|
+
Names and identifiers are different things, and confusing them trips many new users. A run
|
|
34
|
+
name is for people, and names are allowed to repeat: `application.get_runs_by_name()`
|
|
35
|
+
returns every run that shares one. The run identifier is generated, unique within an
|
|
36
|
+
application, and it is what identifier-based routes expect — `application.get_run()` and
|
|
37
|
+
`application.transform()` take `run.run_id`, not the name. If you try to add a second run
|
|
38
|
+
with an identifier that already exists, MTRL refuses rather than guess which run you meant.
|
|
39
|
+
|
|
40
|
+
## One checkpoint per saved table
|
|
41
|
+
|
|
42
|
+
A **checkpoint** (the `MaterialData` object) is one printed page: one stable version of
|
|
43
|
+
measurement data. In the running story, each FTIR run saves one checkpoint with the display
|
|
44
|
+
name `spectrum`. A checkpoint carries:
|
|
45
|
+
|
|
46
|
+
- its **content**, the measured values themselves;
|
|
47
|
+
- a **stage**, one of `raw`, `cleaned`, `transformed`, or `aggregated`, describing how far
|
|
48
|
+
the data has progressed from the instrument;
|
|
49
|
+
- a generated **identifier** that names this logical result;
|
|
50
|
+
- a **display name**, the human-readable label such as `spectrum`;
|
|
51
|
+
- an optional **source file type**, when the data came from an instrument file;
|
|
52
|
+
- its **lineage**: when it was created, which checkpoint it was derived from, and the
|
|
53
|
+
transformation or aggregation that produced it.
|
|
54
|
+
|
|
55
|
+
Raw checkpoints start the story, so they have no parent. A cleaned or transformed checkpoint
|
|
56
|
+
points back at the checkpoint it came from, and its lineage names the exact versions it was
|
|
57
|
+
computed from.
|
|
58
|
+
|
|
59
|
+
## Why identifiers matter more than names
|
|
60
|
+
|
|
61
|
+
One run may hold several checkpoints of the same type or the same name. A run might, for
|
|
62
|
+
example, contain more than one table labelled `spectrum`. Display names are for people;
|
|
63
|
+
they are allowed to repeat. Identifiers are for MTRL, and they never repeat. Whenever you
|
|
64
|
+
ask for a checkpoint, it is the identifier — not the name or the stage — that selects one
|
|
65
|
+
exact answer.
|
|
66
|
+
|
|
67
|
+
This is worth remembering as your binder grows. Names tell you what a page is about. The
|
|
68
|
+
identifier tells you which page it is.
|
|
69
|
+
|
|
70
|
+
## Where next
|
|
71
|
+
|
|
72
|
+
When you add a run or a checkpoint, MTRL commits it behind a boundary that freezes and
|
|
73
|
+
protects the values. Read [The commit boundary](02-the-commit-boundary.md) to see what happens
|
|
74
|
+
at that moment, or revisit [Your first record](../start-here/01-your-first-record.md) to see
|
|
75
|
+
these three levels in working code.
|
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
# The commit boundary
|
|
2
|
+
|
|
3
|
+
Imagine a photocopier standing next to a sealed archive box. When you hand in a document,
|
|
4
|
+
the machine copies every page and seals the copy in the box. Your original stays on your
|
|
5
|
+
desk, and whatever you do to it afterwards — crossing out a line, swapping a page — cannot
|
|
6
|
+
touch what is inside. Nobody can sneak a page out of the box and rewrite it either. The
|
|
7
|
+
only way to change the archived story is to file a new page, and the box keeps both.
|
|
8
|
+
|
|
9
|
+
That is what `application.add(run)` does. The moment you call it, your run crosses the
|
|
10
|
+
**commit boundary**: MTRL detaches a copy of everything the run carries and seals it into
|
|
11
|
+
the record. This page explains what happens at that moment.
|
|
12
|
+
|
|
13
|
+
## A copy, not your original
|
|
14
|
+
|
|
15
|
+
Your checkpoint content is a NumPy array sitting in your own variables. At the boundary,
|
|
16
|
+
MTRL copies that array and keeps the copy. The committed array shares no memory with your
|
|
17
|
+
buffer, so later edits to your variables cannot alter the record. If you normalise your
|
|
18
|
+
spectrum, shift the axis, or reuse the array for `ftir-replicate-02`, the committed
|
|
19
|
+
`spectrum` checkpoint of `ftir-replicate-01` stays exactly as measured.
|
|
20
|
+
|
|
21
|
+
What you read back is a read-only view of that sealed copy. You can inspect every value,
|
|
22
|
+
but you cannot write through the view. Even asking NumPy to flip the writeable flag fails,
|
|
23
|
+
because the view rests on a read-only owning copy. As part of the copy, MTRL also tidies
|
|
24
|
+
the array into a standard memory layout, so the record never depends on how your buffer
|
|
25
|
+
happened to be arranged.
|
|
26
|
+
|
|
27
|
+
## Everything else is frozen too
|
|
28
|
+
|
|
29
|
+
Arrays are not the only things sealed. Metadata, units, and transformation parameters are
|
|
30
|
+
frozen at the same boundary, and the freezing is recursive. A list nested inside a
|
|
31
|
+
dictionary inside your metadata is exactly as frozen as the top level. Nested mappings and
|
|
32
|
+
sequences become read-only structures, and the units mapping on a committed checkpoint is
|
|
33
|
+
read-only as well.
|
|
34
|
+
|
|
35
|
+
Even MTRL respects the seal. When a transformation runs, it receives its own writable copy
|
|
36
|
+
of the committed content, so no operation can scribble on the sealed storage underneath.
|
|
37
|
+
|
|
38
|
+
## The run locks its own doors
|
|
39
|
+
|
|
40
|
+
A committed run refuses direct mutation through its public methods. Calling
|
|
41
|
+
`add_checkpoint`, `replace_active_checkpoint`, or `remove_checkpoint` on a committed run
|
|
42
|
+
raises `CommittedRecordMutationError`. The run is marked as committed at the boundary, and
|
|
43
|
+
it enforces the rule itself.
|
|
44
|
+
|
|
45
|
+
This is not stubbornness. It is how the promise "every change is recorded" is kept. All
|
|
46
|
+
change goes through the application — adding a checkpoint, transforming, correcting — and
|
|
47
|
+
each successful change creates exactly one revision. The application mutates the runs it
|
|
48
|
+
owns through internal helpers, past the same transaction boundary that guards everything
|
|
49
|
+
else. You use the public door, and the public door is watched.
|
|
50
|
+
|
|
51
|
+
## Some values never make it in
|
|
52
|
+
|
|
53
|
+
The boundary is also a customs check. Some values are rejected at commit time, not at save
|
|
54
|
+
time, so you find out immediately rather than when you archive. The refused list includes
|
|
55
|
+
object-dtype arrays, unsafe structured dtypes, and dtypes outside the closed set of plain
|
|
56
|
+
numeric widths. Callables, custom objects, and other values that cannot be frozen into
|
|
57
|
+
plain data are refused too. Metadata itself must be plain data: text, booleans, numbers,
|
|
58
|
+
lists, and dictionaries with string keys.
|
|
59
|
+
|
|
60
|
+
Each refusal raises `UnsupportedCommittedValueError`. Because validation happens before
|
|
61
|
+
anything is sealed, a rejected commit leaves your record, and its revision history,
|
|
62
|
+
untouched.
|
|
63
|
+
|
|
64
|
+
## Why it matters
|
|
65
|
+
|
|
66
|
+
Every later promise MTRL makes starts here. Lineage can point at an exact version and trust
|
|
67
|
+
that it will never change underneath. Verification can check an archive and know the bytes
|
|
68
|
+
it examines are the bytes you committed. Receipts can state what was recorded and be
|
|
69
|
+
believed. None of that is possible if committed data can quietly drift, so the boundary
|
|
70
|
+
makes drift impossible first. Trust starts here.
|
|
71
|
+
|
|
72
|
+
## Where next
|
|
73
|
+
|
|
74
|
+
The boundary seals one version of a result; [two kinds of identity](03-two-kinds-of-identity.md)
|
|
75
|
+
explains how MTRL names the logical result and each sealed version of it. Or revisit
|
|
76
|
+
[your first record](../start-here/01-your-first-record.md) to watch the boundary in action.
|
|
@@ -0,0 +1,75 @@
|
|
|
1
|
+
# Two kinds of identity
|
|
2
|
+
|
|
3
|
+
Think of a book. The title, say *The Origin of Species*, names the work itself. It stays the
|
|
4
|
+
same no matter how many printings exist. Each printing, by contrast, is fixed forever: the
|
|
5
|
+
third printing of 1861 has exactly the pages it has, and no one can change a word of it now.
|
|
6
|
+
When you cite a fact from the book, you name the printing, not only the title, because only a
|
|
7
|
+
printing has exact pages. MTRL gives every result the same two kinds of identity, and for the
|
|
8
|
+
same reason.
|
|
9
|
+
|
|
10
|
+
## The two names every result carries
|
|
11
|
+
|
|
12
|
+
Every saved result in MTRL has two identifiers.
|
|
13
|
+
|
|
14
|
+
- The **CheckpointId** is the title. It names the logical result: "the spectrum". It is
|
|
15
|
+
stable, and it is the name you, your colleagues, and your transforms use when you talk
|
|
16
|
+
about the result.
|
|
17
|
+
- The **RecordId** is the printing. It names one immutable version of that result. Once a
|
|
18
|
+
record is committed, its RecordId never changes and its content never changes.
|
|
19
|
+
|
|
20
|
+
A correction makes the distinction concrete. When you corrected the saturated FTIR spectrum
|
|
21
|
+
of `polymer-film-01`, the CheckpointId stayed: it is still "the spectrum". MTRL minted a new
|
|
22
|
+
RecordId for the corrected version, and the old version kept its own RecordId forever. The
|
|
23
|
+
title endured; a new printing was bound.
|
|
24
|
+
|
|
25
|
+
## Pointing at an exact version
|
|
26
|
+
|
|
27
|
+
Because one CheckpointId can stand behind several records over time, a CheckpointId alone
|
|
28
|
+
cannot say exactly which data you mean. For that, MTRL uses a **RecordRef**: a small frozen
|
|
29
|
+
object that pairs a CheckpointId with a RecordId. It names one exact version of one logical
|
|
30
|
+
result, with no room for drift.
|
|
31
|
+
|
|
32
|
+
This is the only acceptable way to point at scientific input. New lineage must never refer
|
|
33
|
+
to a CheckpointId alone, because that would leave the version ambiguous. When a transform
|
|
34
|
+
records its source, or an aggregate records its members, each reference is a full RecordRef.
|
|
35
|
+
A `RecordRef` also refuses to be built half-hearted: it rejects an empty CheckpointId or
|
|
36
|
+
RecordId at construction, so a vague reference cannot exist.
|
|
37
|
+
|
|
38
|
+
## The identity table, in plain words
|
|
39
|
+
|
|
40
|
+
MTRL uses a handful of identities. Some are logical names for things that persist; the rest
|
|
41
|
+
are version or history identities for things that are fixed forever.
|
|
42
|
+
|
|
43
|
+
| Identity | Names | Kind |
|
|
44
|
+
| --- | --- | --- |
|
|
45
|
+
| `CheckpointId` | One logical scientific result, such as "the spectrum" | Logical name |
|
|
46
|
+
| `RunId` | One logical instrument or aggregate run, such as `ftir-replicate-01` | Logical name |
|
|
47
|
+
| `RecordId` | One immutable version of a checkpoint | Version identity |
|
|
48
|
+
| `RevisionId` | One committed view of the whole archive | History identity |
|
|
49
|
+
| `ArchiveId` | One archive history, from first commit onwards | History identity |
|
|
50
|
+
| `SampleUid` | One physical specimen or material unit | Physical identity |
|
|
51
|
+
|
|
52
|
+
Generated identities, such as fresh RecordId, RevisionId, ArchiveId, and SampleUid values,
|
|
53
|
+
are time-ordered UUIDv7 values. Each one is unique, and newer ones sort after older ones.
|
|
54
|
+
When you supply your own RunId or CheckpointId, MTRL keeps it; it only asks that logical
|
|
55
|
+
identifiers are non-empty and safe to use inside archive paths.
|
|
56
|
+
|
|
57
|
+
## Why it matters
|
|
58
|
+
|
|
59
|
+
Two practical guarantees fall out of this split.
|
|
60
|
+
|
|
61
|
+
Corrections are honest. A correction keeps the CheckpointId and mints a new RecordId, so the
|
|
62
|
+
logical name your colleagues use survives while the old version remains retrievable by its
|
|
63
|
+
own RecordId. You saw this in [Fix a mistake](../start-here/05-fix-a-mistake.md): the
|
|
64
|
+
saturated spectrum is still in the history, linked to its corrected successor.
|
|
65
|
+
|
|
66
|
+
Recipes are exact. Lineage and aggregates point at RecordRefs, not bare checkpoint names, so
|
|
67
|
+
a recorded recipe always names the exact versions it used. The mean spectrum from your three
|
|
68
|
+
replicates names each input record precisely. If one replicate is later corrected, the recipe
|
|
69
|
+
still points at the version it genuinely consumed, and nothing silently retargets.
|
|
70
|
+
|
|
71
|
+
## Where next
|
|
72
|
+
|
|
73
|
+
Corrections keep the CheckpointId, but a replacement mints a new one — the difference is in
|
|
74
|
+
[Corrections versus replacements](04-corrections-vs-replacements.md). To see both identities at
|
|
75
|
+
work in code, revisit [Fix a mistake](../start-here/05-fix-a-mistake.md).
|
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
# Corrections or replacements?
|
|
2
|
+
|
|
3
|
+
Think of a printed book again. When the publisher finds a misprint, they paste an erratum slip
|
|
4
|
+
into the same printing: the title is unchanged, and the faulty page is still there underneath for
|
|
5
|
+
anyone who looks. When the author revises the work into a second edition, that is a new print run
|
|
6
|
+
with its own identity, and the first edition stays on the shelf. MTRL makes the same two moves
|
|
7
|
+
with your data, and the archive shelves both versions, always.
|
|
8
|
+
|
|
9
|
+
## Two different repairs
|
|
10
|
+
|
|
11
|
+
A **correction** fixes one logical result. The old value was wrong — a mistake — and you are
|
|
12
|
+
putting it right. `correct_checkpoint` keeps the CheckpointId: it is still "the spectrum". It
|
|
13
|
+
mints a new RecordId for the corrected record, makes that record the active one, and links it to
|
|
14
|
+
the old record with a `correction-of` relationship. The erratum slip, in other words.
|
|
15
|
+
|
|
16
|
+
A **replacement** introduces different evidence. The old value was fine; better evidence now
|
|
17
|
+
exists. `replace_checkpoint` mints a *new* CheckpointId and a new RecordId, reusing neither
|
|
18
|
+
identity from the replaced checkpoint. The new record links to the old one with a
|
|
19
|
+
`replacement-of` relationship. From then on the active view selects the replacement and no longer
|
|
20
|
+
selects the old checkpoint. The replaced record is not deleted. It remains in the history,
|
|
21
|
+
exactly as committed. The second edition, next to the first on the shelf.
|
|
22
|
+
|
|
23
|
+
If CheckpointId, RecordId, and RecordRef are new to you, read
|
|
24
|
+
[Two kinds of identity](03-two-kinds-of-identity.md) first; the rest of this page builds on them.
|
|
25
|
+
|
|
26
|
+
## How to choose
|
|
27
|
+
|
|
28
|
+
The rule of thumb asks what the old value claims.
|
|
29
|
+
|
|
30
|
+
- **Correct** when the old value was wrong. The saturated detector clipped the peak of your FTIR
|
|
31
|
+
spectrum, so the recorded absorbance misstates the film. You corrected that record in
|
|
32
|
+
[Fix a mistake](../start-here/05-fix-a-mistake.md), and the checkpoint name "spectrum"
|
|
33
|
+
survived.
|
|
34
|
+
- **Replace** when the old value was honest but superseded. If you re-measure `polymer-film-01`
|
|
35
|
+
a year from now on a newer spectrometer, the original spectrum is not wrong. It is an accurate
|
|
36
|
+
record of that day on that instrument. The new measurement is different evidence, so it earns
|
|
37
|
+
a CheckpointId of its own.
|
|
38
|
+
|
|
39
|
+
Ask yourself which sentence you would defend in a report. "The old number was a mistake" calls
|
|
40
|
+
for a correction. "The old number was sound, and here is a better one" calls for a replacement.
|
|
41
|
+
|
|
42
|
+
## What both moves guarantee
|
|
43
|
+
|
|
44
|
+
Both operations are careful in the same ways.
|
|
45
|
+
|
|
46
|
+
Both aim at the active record. Each call takes an exact RecordRef naming its target, and that
|
|
47
|
+
target must be the currently active record for the checkpoint. Point at a stale or historical
|
|
48
|
+
version and the call fails with no state change at all: no partial write, no new revision.
|
|
49
|
+
|
|
50
|
+
Neither retargets the past. A correction does not rewrite records that used the old version, so
|
|
51
|
+
the mean spectrum of your three replicates still names the exact records it consumed, even if one
|
|
52
|
+
replicate is corrected afterwards. A replacement removes the old checkpoint from the active view,
|
|
53
|
+
but never from the history.
|
|
54
|
+
|
|
55
|
+
Neither can overwrite an identity. Adding a checkpoint whose supplied CheckpointId already
|
|
56
|
+
exists is rejected with `DuplicateCheckpointError`: new evidence always arrives under a fresh
|
|
57
|
+
identity, and nothing committed is ever changed in place.
|
|
58
|
+
|
|
59
|
+
Each success creates exactly one revision. Each failure creates none. For the exact signatures
|
|
60
|
+
and error names, see the [reference](../reference.md).
|
|
61
|
+
|
|
62
|
+
## Where next
|
|
63
|
+
|
|
64
|
+
To see how a `correction-of` link sits inside a full recorded recipe, read
|
|
65
|
+
[Lineage and the recorded recipe](05-lineage-and-the-recorded-recipe.md). To make a correction
|
|
66
|
+
yourself, revisit [Fix a mistake](../start-here/05-fix-a-mistake.md).
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
# Lineage and the recorded recipe
|
|
2
|
+
|
|
3
|
+
Think of a recipe card sealed inside the box with the cake. The card lists the ingredients, the
|
|
4
|
+
oven temperature, and even which batch of flour went in, so anyone who opens the box can see
|
|
5
|
+
exactly how the cake was made. Crucially, there is no live chef hiding inside: the card describes
|
|
6
|
+
the cooking, but it cannot cook. Every derived record in MTRL carries exactly such a card — its
|
|
7
|
+
recipe — sealed in with the result itself.
|
|
8
|
+
|
|
9
|
+
## One producing step per record
|
|
10
|
+
|
|
11
|
+
Every derived record carries a **lineage**: a record of when the checkpoint was created, which
|
|
12
|
+
checkpoint is its parent, and the one step that produced it.
|
|
13
|
+
|
|
14
|
+
A lineage records exactly one producing step: a **transformation** (an operation that turns one
|
|
15
|
+
record into another, such as normalisation) or an **aggregation** (an operation that combines
|
|
16
|
+
several records, such as a mean) — never both. A raw checkpoint, like the spectrum straight from
|
|
17
|
+
`ftir-replicate-01`, has no parent and no producing step at all.
|
|
18
|
+
|
|
19
|
+
The lineage also names its exact inputs, in order, as RecordRefs — the paired checkpoint and
|
|
20
|
+
record identities from [Two kinds of identity](03-two-kinds-of-identity.md). A transform records one
|
|
21
|
+
RecordRef for its source. An aggregate records one RecordRef for each selected member, in
|
|
22
|
+
selection order. So the normalised spectrum from lesson 4 names the exact raw record it consumed,
|
|
23
|
+
and the mean over `ftir-replicate-01` to `ftir-replicate-03` names each replicate record
|
|
24
|
+
precisely — never a bare checkpoint name that could silently drift to a newer version.
|
|
25
|
+
|
|
26
|
+
Corrections and replacements get their own slots. When a record is a correction, `correction_of`
|
|
27
|
+
holds the exact prior RecordRef; when it is a replacement, `replacement_of` does. A lineage
|
|
28
|
+
cannot hold both. These relationships are independent of the producing step. One honest limit: a correction
|
|
29
|
+
that supplies hand-built data, like the lesson 5 correction, records what it corrects but no
|
|
30
|
+
operation recipe, because no operation produced the new values. Only records an operation
|
|
31
|
+
produced carry the full recipe described below.
|
|
32
|
+
|
|
33
|
+
## The provenance envelope
|
|
34
|
+
|
|
35
|
+
The lineage says which step ran. A second, richer object says everything about how it ran: the
|
|
36
|
+
**OperationProvenance** envelope, a frozen bundle of data attached to the record. It holds:
|
|
37
|
+
|
|
38
|
+
- the qualified operation identifier and its contract version;
|
|
39
|
+
- the implementation package name and version;
|
|
40
|
+
- the effective parameters, including defaults you never typed;
|
|
41
|
+
- the ordered RecordRef inputs;
|
|
42
|
+
- the input and output schemas and units;
|
|
43
|
+
- the execution status, `success` or `failed`;
|
|
44
|
+
- the installed dependency versions the operation actually used;
|
|
45
|
+
- a canonical digest of the output payload — a fingerprint binding the recipe to the exact
|
|
46
|
+
values, shape, and dtype it produced.
|
|
47
|
+
|
|
48
|
+
## Data only, never code
|
|
49
|
+
|
|
50
|
+
The envelope's most important property is what it refuses to contain. It holds data only: no
|
|
51
|
+
import path, no callable, no pickle, no executable source. An archive is evidence, not a
|
|
52
|
+
programme — opening one can never run code. This is enforced, not merely promised. Callables,
|
|
53
|
+
custom objects, and non-string mapping keys are rejected when the envelope is built, and every
|
|
54
|
+
value is validated, detached from the caller's objects, and frozen at construction. Once sealed,
|
|
55
|
+
a recipe cannot change after the fact.
|
|
56
|
+
|
|
57
|
+
The same caution governs the future. When a newer version of MTRL writes provenance fields your
|
|
58
|
+
version does not know, loading preserves them as inert data. They are never resolved, imported,
|
|
59
|
+
registered, or executed — they ride along, untouched, for a future reader.
|
|
60
|
+
|
|
61
|
+
## Why it matters
|
|
62
|
+
|
|
63
|
+
Because the recipe travels with the record, "how did you get this number?" is always answerable
|
|
64
|
+
from the record itself. You do not need to trust a lab notebook, a colleague's memory, or a
|
|
65
|
+
script that has since been edited. The record says which operation ran, with which parameters,
|
|
66
|
+
on which exact inputs, and whether it succeeded.
|
|
67
|
+
|
|
68
|
+
The recipe is not only for reading. It is exactly what the replay feature consumes later: the
|
|
69
|
+
operation identity, the effective parameters, the exact input references, and the recorded output
|
|
70
|
+
digest to compare against. The how-to guides cover replay in detail.
|
|
71
|
+
|
|
72
|
+
## Where next
|
|
73
|
+
|
|
74
|
+
Each sealed recipe becomes one step in a longer history —
|
|
75
|
+
[Revisions: the archive timeline](06-revisions-the-archive-timeline.md) shows how those steps
|
|
76
|
+
accumulate into a verifiable archive. To record a recipe of your own, revisit
|
|
77
|
+
[Transform a spectrum](../start-here/04-transform-a-spectrum.md).
|
|
@@ -0,0 +1,69 @@
|
|
|
1
|
+
# Revisions: your archive's timeline
|
|
2
|
+
|
|
3
|
+
Think of a bound laboratory logbook. Every time something changes, you write one dated
|
|
4
|
+
entry, and each entry points back to the page before it. You never tear a page out, and a
|
|
5
|
+
spoiled entry is never written at all: if an experiment fails, there is nothing to cross
|
|
6
|
+
out. Read the book up to any page and you know exactly what the lab looked like that day.
|
|
7
|
+
MTRL keeps the same kind of book for your archive, and each entry in it is a **revision**
|
|
8
|
+
(one committed change to the archive).
|
|
9
|
+
|
|
10
|
+
## What a revision records
|
|
11
|
+
|
|
12
|
+
Every successful top-level change to the science in your archive commits exactly one
|
|
13
|
+
revision. Adding a run, adding a checkpoint, transforming, correcting, replacing, removing
|
|
14
|
+
a run, and ingesting an instrument file each write one entry. Nothing writes half an
|
|
15
|
+
entry, and nothing writes two.
|
|
16
|
+
|
|
17
|
+
A revision captures the archive's state at that moment:
|
|
18
|
+
|
|
19
|
+
- its own identity, a **RevisionId**;
|
|
20
|
+
- its parent, the immediately prior revision — the history is one straight line, never a
|
|
21
|
+
tangle of branches;
|
|
22
|
+
- the active selections: which exact record each logical name, such as the "spectrum"
|
|
23
|
+
checkpoint, pointed at when the entry was written;
|
|
24
|
+
- the ordered events of that change, and the time it was committed.
|
|
25
|
+
|
|
26
|
+
Every revision in one history shares a single **ArchiveId**, the identity of the whole
|
|
27
|
+
book. Once written, an entry is frozen: after commit, and after load, neither the revision
|
|
28
|
+
nor its events can be edited.
|
|
29
|
+
|
|
30
|
+
## What writes no entry
|
|
31
|
+
|
|
32
|
+
Plenty of operations leave the logbook untouched, because they only read what is already
|
|
33
|
+
committed. Saving the archive, verifying it, and exporting a receipt create no revision.
|
|
34
|
+
Neither does configuring the operation registry. An aggregate call on its own writes
|
|
35
|
+
nothing either: it prepares a candidate result, and the entry appears only when you add
|
|
36
|
+
that result to the archive.
|
|
37
|
+
|
|
38
|
+
A failed change also writes no entry. MTRL validates the whole change before committing,
|
|
39
|
+
so a failure rolls back completely: no new record, no change to lineage, and no new
|
|
40
|
+
revision. The revision count after a failure is the revision count before it. In logbook
|
|
41
|
+
terms, the spoiled page does not exist.
|
|
42
|
+
|
|
43
|
+
## The timeline survives saving
|
|
44
|
+
|
|
45
|
+
Saving writes the complete timeline into the `.mtrl` file, and loading restores it
|
|
46
|
+
exactly. A loaded archive does not start a new book: it adopts the saved ArchiveId,
|
|
47
|
+
restores every saved revision without inventing a synthetic one, and carries on the chain.
|
|
48
|
+
Your next change becomes a child of the loaded head, as if the archive had never been
|
|
49
|
+
closed.
|
|
50
|
+
|
|
51
|
+
## Why it matters
|
|
52
|
+
|
|
53
|
+
Because each revision holds the active selections of its moment, the timeline answers the
|
|
54
|
+
question "what did the archive look like then?". Assertion queries can report their state
|
|
55
|
+
as of an earlier revision, so you can check what the archive claimed before a correction,
|
|
56
|
+
not only what it claims now.
|
|
57
|
+
|
|
58
|
+
You have been writing this timeline since the first lesson. Recording the raw "spectrum"
|
|
59
|
+
checkpoint for `ftir-replicate-01` was one entry. The normalised spectrum was another. The
|
|
60
|
+
correction of the saturated peak in [Fix a mistake](../start-here/05-fix-a-mistake.md) was
|
|
61
|
+
one more — and every earlier entry stayed exactly as written. The exact attributes of a
|
|
62
|
+
revision are listed in the [reference](../reference.md).
|
|
63
|
+
|
|
64
|
+
## Where next
|
|
65
|
+
|
|
66
|
+
To see how revisions sit inside the file alongside records, manifests, and integrity data,
|
|
67
|
+
read [Inside a .mtrl archive](07-inside-a-mtrl-archive.md). For the difference between the
|
|
68
|
+
ArchiveId that names this whole timeline and the RevisionId that names one entry in it,
|
|
69
|
+
revisit [Two kinds of identity](03-two-kinds-of-identity.md).
|