mtlearn 1.2.0__tar.gz → 1.2.2__tar.gz

{mtlearn-1.2.0 → mtlearn-1.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mtlearn
-Version: 1.2.0
+Version: 1.2.2
 Summary: Morphological tree learning utilities with a stable morphology facade
 Author-Email: Wonder Alexandre Luz Alves <worderalexandre@gmail.com>
 License-Expression: GPL-3.0-only

{mtlearn-1.2.0 → mtlearn-1.2.2}/docs/source/concepts/attributes.md

@@ -2,7 +2,7 @@
 
 This catalog lists the scalar attributes exposed through
 `mtlearn.morphology.AttributeType` and the groups exposed through
-`mtlearn.morphology.AttributeGroup`: 132 scalar attributes and nine groups.
+`mtlearn.morphology.AttributeGroup`: 133 scalar attributes and nine groups.
 Names, groups, and definitions follow the
 [mmcfilters v5.2.0 attribute catalog](https://github.com/wonderalexandre/mmcfilters/blob/v5.2.0/docs/attribute-catalog.md).
 
@@ -49,7 +49,7 @@ below, with its group memberships. `ALL` includes every listed scalar.
 
 | Group | Meaning |
 | --- | --- |
-| `GRAY_LEVEL` | Gray-level mass, contrast, height, mean, and variance. |
+| `GRAY_LEVEL` | Node valuation, gray-level mass, contrast, height, mean, and variance. |
 | `SHAPE` | Size, bounding box, moment, bitquad, contour, and selected shape descriptors. |
 | `MOMENTS` | Central moments, Hu invariants, and moment-derived descriptors. |
 | `BOUNDARY` | Bitquad and contour descriptors. |
@@ -75,16 +75,11 @@ The catalog uses three contracts:
 | Topology/support | Uses node support, pixel coordinates, contours, or adjacency information. |
 | Tree topology | Reads only parent/child relations in the hierarchy. |
 
-Distance-transform attributes require a regular 2D image domain. They support
-max-trees, min-trees, and trees of shapes without requiring node altitudes or
-construction-adjacency metadata. Bitquad and filled-shape attributes require
-canonical 4- or 8-connectivity; trees of shapes also use node altitudes to
-select the applicable directional connectivity.
-
 ## Gray-Level Attributes
 
 | Attribute | Groups | Contract | Use |
 | --- | --- | --- | --- |
+| `GRAY_LEVEL` | `GRAY_LEVEL` | Altitude-aware | Node valuation: the gray level associated with the node. |
 | `VOLUME` | `GRAY_LEVEL` | Altitude-aware | Sum of altitude-weighted support contributions over the node subtree. |
 | `RELATIVE_VOLUME` | `GRAY_LEVEL` | Altitude-aware | Recursive contrast volume `R(n) = area(n) + sum_c [R(c) + area(c) * abs(altitude(c) - altitude(n))]` over the direct children `c`. |
 | `GRAY_LEVEL_HEIGHT` | `GRAY_LEVEL` | Altitude-aware | Maximum absolute altitude difference between the node and any node in its subtree. |
@@ -99,23 +94,16 @@ contrast, intensity, or altitude differences rather than pure geometry.
 | Attribute | Groups | Contract | Use |
 | --- | --- | --- | --- |
 | `AREA` | `SHAPE` | Topology/support | Number of pixels in the full node support. |
-| `BOX_WIDTH` | `SHAPE` | Topology/support | Width, in columns, of the smallest axis-aligned bounding box enclosing the node support. |
+| `BOUNDING_BOX_WIDTH` | `SHAPE` | Topology/support | Width, in columns, of the smallest axis-aligned bounding box enclosing the node support. |
 | `BOUNDING_BOX_HEIGHT` | `SHAPE` | Topology/support | Height, in rows, of the smallest axis-aligned bounding box enclosing the node support. |
 | `DIAGONAL_LENGTH` | `SHAPE` | Topology/support | Euclidean diagonal length of the bounding box, `sqrt(width^2 + height^2)`. |
-| `RECTANGULARITY` | `SHAPE` | Topology/support | Ratio `AREA / (BOX_WIDTH * BOUNDING_BOX_HEIGHT)`. |
+| `RECTANGULARITY` | `SHAPE` | Topology/support | Ratio `AREA / (BOUNDING_BOX_WIDTH * BOUNDING_BOX_HEIGHT)`. |
 | `RATIO_WH` | `SHAPE` | Topology/support | Bounding-box aspect ratio, `max(width, height) / min(width, height)` for non-degenerate boxes. |
 | `BOX_COLUMN_MIN` | `SHAPE` | Topology/support | Minimum image column index covered by the node support. |
 | `BOX_COLUMN_MAX` | `SHAPE` | Topology/support | Maximum image column index covered by the node support. |
 | `BOX_ROW_MIN` | `SHAPE` | Topology/support | Minimum image row index covered by the node support. |
 | `BOX_ROW_MAX` | `SHAPE` | Topology/support | Maximum image row index covered by the node support. |
 
-`AREA` is the usual attribute for size filtering. A high `AREA` value means
-that the component represented by the node covers many pixels in the original
-image domain; it is not the number of children in the tree.
-
-These attributes are the most direct tools for area openings, size priors, and
-filters that should keep or remove elongated connected components.
-
 ## Distance-Transform Attributes
 
 These attributes describe distances from support pixels to the foreground

{mtlearn-1.2.0 → mtlearn-1.2.2}/docs/source/guides/custom-attributes-filter-specs.md

@@ -10,10 +10,11 @@ For the full attribute list, see {doc}`../concepts/attributes`.
 
 | Intent | Candidate attributes |
 | --- | --- |
+| Use the gray level associated with each node | `GRAY_LEVEL` |
 | Remove small components | `AREA`, `VOLUME` |
 | Prefer contrast in the tree | `GRAY_LEVEL_HEIGHT`, `RELATIVE_VOLUME` |
 | Prefer elongated or compact shapes | `COMPACTNESS`, `ECCENTRICITY`, `RATIO_WH` |
-| Use bounding-box geometry | `BOX_WIDTH`, `BOUNDING_BOX_HEIGHT`, `RECTANGULARITY` |
+| Use bounding-box geometry | `BOUNDING_BOX_WIDTH`, `BOUNDING_BOX_HEIGHT`, `RECTANGULARITY` |
 | Use tree position | `DEPTH_NODE`, `SUBTREE_HEIGHT`, `NUM_CHILDREN_NODE` |
 | Use contours | `CONTOUR_PERIMETER`, `CONTOUR_PIXELS` |
 

{mtlearn-1.2.0 → mtlearn-1.2.2}/mtlearn/bindings/morphology/AttributeBinding.hpp

@@ -38,7 +38,7 @@ inline const std::vector<morphology::Attribute>& allAttributes()
         morphology::Attribute::GrayLevelHeight,
         morphology::Attribute::MeanGrayLevel,
         morphology::Attribute::GrayLevelVariance,
-        morphology::Attribute::BoxWidth,
+        morphology::Attribute::BoundingBoxWidth,
         morphology::Attribute::BoundingBoxHeight,
         morphology::Attribute::DiagonalLength,
         morphology::Attribute::Rectangularity,
@@ -164,6 +164,7 @@ inline const std::vector<morphology::Attribute>& allAttributes()
         morphology::Attribute::FilledCentroidDisplacementNormalized,
         morphology::Attribute::FilledCompactness,
         morphology::Attribute::FilledCircularity,
+        morphology::Attribute::GrayLevel,
     };
     return attributes;
 }
@@ -173,18 +174,29 @@ inline const std::vector<morphology::Attribute>& allAttributes()
 // facade is introduced.
 inline std::string describeAttribute(morphology::Attribute attribute)
 {
+    if (attribute == morphology::Attribute::GrayLevel) {
+        return "Gray level: Node valuation (the gray level associated with the node).";
+    }
     return mmcfilters::AttributeNames::describe(toBackend(attribute));
 }
 
+inline std::string attributeName(mmcfilters::Attribute attribute)
+{
+    if (fromBackend(attribute) == morphology::Attribute::BoundingBoxWidth) {
+        return "BOUNDING_BOX_WIDTH";
+    }
+    return mmcfilters::AttributeNames::toString(attribute);
+}
+
 // Return all descriptions keyed by backend/public attribute name. Keeping the
 // names aligned with enum values makes the Python API easy to inspect.
 inline py::dict describeAllAttributes()
 {
     py::dict descriptions;
     for (const auto attribute : allAttributes()) {
-        const auto backendAttribute = toBackend(attribute);
-        descriptions[py::str(mmcfilters::AttributeNames::toString(backendAttribute))] =
-            py::str(mmcfilters::AttributeNames::describe(backendAttribute));
+        const auto name = attribute == morphology::Attribute::GrayLevel
+            ? std::string("GRAY_LEVEL") : attributeName(toBackend(attribute));
+        descriptions[py::str(name)] = py::str(describeAttribute(attribute));
     }
     return descriptions;
 }
@@ -203,6 +215,9 @@ inline std::vector<morphology::Attribute> expandAttributeGroup(morphology::Attri
     for (const auto attribute : it->second) {
         attributes.push_back(fromBackend(attribute));
     }
+    if (group == morphology::AttributeGroup::All || group == morphology::AttributeGroup::GrayLevel) {
+        attributes.push_back(morphology::Attribute::GrayLevel);
+    }
     return attributes;
 }
 
@@ -211,7 +226,11 @@ inline std::vector<morphology::Attribute> expandAttributeGroup(morphology::Attri
 // row count for the selected node-id space.
 inline int outputSize(const morphology::WeightedTree& tree, morphology::NodeIdSpace outputSpace)
 {
-    return morphology::detail::topology(tree).getNodeIdSpaceSize(toBackend(outputSpace));
+    const auto& topology = morphology::detail::topology(tree);
+    if (outputSpace == morphology::NodeIdSpace::Higra) {
+        return topology.numPixels() + topology.numNodes();
+    }
+    return topology.getNodeIdSpaceSize(toBackend(outputSpace));
 }
 
 // mmcfilters returns an attribute-name map that is not guaranteed to iterate in
@@ -225,7 +244,7 @@ inline py::dict sortedAttributeIndex(const mmcfilters::AttributeNames& attribute
     values.reserve(attributeNames.indexMap.size());
 
     for (const auto& item : attributeNames.indexMap) {
-        keys.push_back(attributeNames.toString(item.first));
+        keys.push_back(attributeName(item.first));
         values.push_back(item.second);
     }
 
@@ -242,6 +261,25 @@ inline py::dict sortedAttributeIndex(const mmcfilters::AttributeNames& attribute
     return result;
 }
 
+template <std::floating_point Real>
+std::vector<Real> computeGrayLevel(const morphology::WeightedTree& tree, morphology::NodeIdSpace outputSpace)
+{
+    const auto& valuedTree = morphology::detail::backend(tree);
+    if (outputSpace == morphology::NodeIdSpace::Higra) {
+        const auto [parents, altitudes] = valuedTree.exportHigraHierarchy();
+        return std::vector<Real>(altitudes.begin(), altitudes.end());
+    }
+    if (outputSpace != morphology::NodeIdSpace::MorphologicalTree) {
+        throw std::invalid_argument("unknown NodeIdSpace");
+    }
+    const auto& topology = morphology::detail::topology(tree);
+    std::vector<Real> values(topology.numInternalNodeSlots(), Real{0});
+    for (const auto node : topology.aliveNodeIds()) {
+        values[node] = static_cast<Real>(valuedTree.nodeAltitude(node));
+    }
+    return values;
+}
+
 template <std::floating_point Real>
 std::pair<py::dict, py::array> computeAttributesTyped(
     morphology::WeightedTreePtr tree,
@@ -252,16 +290,57 @@ std::pair<py::dict, py::array> computeAttributesTyped(
         throw py::value_error("invalid ValuedMorphologicalTree");
     }
 
-    const auto backendAttributes = toBackend(attributes);
-    auto [attributeNames, buffer] =
-        mmcfilters::AttributeComputation::computeAttributes<Real>(
-            morphology::detail::backend(*tree),
-            backendAttributes,
-            toBackend(outputSpace));
+    bool includeGrayLevel = false;
+    std::vector<mmcfilters::AttributeOrGroup> backendAttributes;
+    backendAttributes.reserve(attributes.size());
+    for (const auto& request : attributes) {
+        if (const auto* scalar = std::get_if<morphology::Attribute>(&request)) {
+            if (*scalar == morphology::Attribute::GrayLevel) {
+                includeGrayLevel = true;
+                continue;
+            }
+        } else {
+            const auto group = std::get<morphology::AttributeGroup>(request);
+            includeGrayLevel = includeGrayLevel || group == morphology::AttributeGroup::All
+                || group == morphology::AttributeGroup::GrayLevel;
+        }
+        backendAttributes.push_back(toBackend(request));
+    }
 
-    return {
-        sortedAttributeIndex(attributeNames),
-        vectorToNumpyOwned(std::move(buffer), outputSize(*tree, outputSpace), attributeNames.NUM_ATTRIBUTES)};
+    const int rows = outputSize(*tree, outputSpace);
+    py::dict layout;
+    std::vector<Real> buffer;
+    int columns = 0;
+    if (!backendAttributes.empty() || !includeGrayLevel) {
+        auto [attributeNames, values] =
+            mmcfilters::AttributeComputation::computeAttributes<Real>(
+                morphology::detail::backend(*tree), backendAttributes);
+        if (outputSpace == morphology::NodeIdSpace::Higra) {
+            values = mmcfilters::AttributeComputation::projectNodeValuesToExportedHigra<Real>(
+                morphology::detail::backend(*tree), attributeNames, values);
+        }
+        layout = sortedAttributeIndex(attributeNames);
+        columns = attributeNames.NUM_ATTRIBUTES;
+        buffer = std::move(values);
+    }
+    if (includeGrayLevel) {
+        auto grayLevel = computeGrayLevel<Real>(*tree, outputSpace);
+        layout["GRAY_LEVEL"] = columns;
+        if (columns == 0) {
+            buffer = std::move(grayLevel);
+        } else {
+            std::vector<Real> values(static_cast<std::size_t>(rows) * (columns + 1));
+            for (int row = 0; row < rows; ++row) {
+                const auto source = static_cast<std::size_t>(row) * columns;
+                const auto destination = static_cast<std::size_t>(row) * (columns + 1);
+                std::copy_n(buffer.begin() + source, columns, values.begin() + destination);
+                values[destination + columns] = grayLevel[row];
+            }
+            buffer = std::move(values);
+        }
+        ++columns;
+    }
+    return {std::move(layout), vectorToNumpyOwned(std::move(buffer), rows, columns)};
 }
 
 // Compute several attributes or attribute groups and return the pair used by
@@ -289,12 +368,18 @@ py::array computeSingleAttributeTyped(
         throw py::value_error("invalid ValuedMorphologicalTree");
     }
 
+    if (attribute == morphology::Attribute::GrayLevel) {
+        return vectorToNumpyOwned(computeGrayLevel<Real>(*tree, outputSpace), outputSize(*tree, outputSpace));
+    }
+
     auto [attributeNames, buffer] =
         mmcfilters::AttributeComputation::computeSingleAttribute<Real>(
             morphology::detail::backend(*tree),
-            toBackend(attribute),
-            toBackend(outputSpace));
-    (void)attributeNames;
+            toBackend(attribute));
+    if (outputSpace == morphology::NodeIdSpace::Higra) {
+        buffer = mmcfilters::AttributeComputation::projectNodeValuesToExportedHigra<Real>(
+            morphology::detail::backend(*tree), attributeNames, buffer);
+    }
 
     return vectorToNumpyOwned(std::move(buffer), outputSize(*tree, outputSpace));
 }
@@ -385,14 +470,14 @@ Returns:
         attribute,
         "Type",
         py::module_local(),
-        "Scalar morphology attributes supported by the current backend.")
+        "Scalar morphology attributes supported by mtlearn.")
         .value("AREA", morphology::Attribute::Area)
         .value("VOLUME", morphology::Attribute::Volume)
         .value("RELATIVE_VOLUME", morphology::Attribute::RelativeVolume)
         .value("GRAY_LEVEL_HEIGHT", morphology::Attribute::GrayLevelHeight)
         .value("MEAN_GRAY_LEVEL", morphology::Attribute::MeanGrayLevel)
         .value("GRAY_LEVEL_VARIANCE", morphology::Attribute::GrayLevelVariance)
-        .value("BOX_WIDTH", morphology::Attribute::BoxWidth)
+        .value("BOUNDING_BOX_WIDTH", morphology::Attribute::BoundingBoxWidth)
         .value("BOUNDING_BOX_HEIGHT", morphology::Attribute::BoundingBoxHeight)
         .value("RECTANGULARITY", morphology::Attribute::Rectangularity)
         .value("DIAGONAL_LENGTH", morphology::Attribute::DiagonalLength)
@@ -518,7 +603,13 @@ Returns:
         .value("FILLED_CENTROID_DISPLACEMENT_NORMALIZED", morphology::Attribute::FilledCentroidDisplacementNormalized)
         .value("FILLED_COMPACTNESS", morphology::Attribute::FilledCompactness)
         .value("FILLED_CIRCULARITY", morphology::Attribute::FilledCircularity)
+        .value("GRAY_LEVEL", morphology::Attribute::GrayLevel)
         .export_values();
+
+    attribute.attr("GRAY_LEVEL") = attribute.attr("Group").attr("GRAY_LEVEL");
+
+    attribute.attr("Type").attr("BOX_WIDTH") = attribute.attr("Type").attr("BOUNDING_BOX_WIDTH");
+    attribute.attr("BOX_WIDTH") = attribute.attr("BOUNDING_BOX_WIDTH");
 }
 
 } // namespace morphology_pybind

{mtlearn-1.2.0 → mtlearn-1.2.2}/mtlearn/bindings/morphology/BindingSupport.hpp

@@ -266,21 +266,21 @@ inline mmcfilters::AttributeGroup toBackend(morphology::AttributeGroup group)
 inline mmcfilters::Attribute toBackend(morphology::Attribute attribute)
 {
     switch (attribute) {
+    case morphology::Attribute::GrayLevel:
+        throw std::invalid_argument("GRAY_LEVEL is computed from the node valuation by mtlearn");
     case morphology::Attribute::Area:
         return mmcfilters::Attribute::Area;
     case morphology::Attribute::Volume:
         return mmcfilters::Attribute::Volume;
     case morphology::Attribute::RelativeVolume:
         return mmcfilters::Attribute::RelativeVolume;
-    // The backend dropped the plain node-altitude attribute, so the facade no
-    // longer publishes LEVEL/ALTITUDE either.
     case morphology::Attribute::GrayLevelHeight:
         return mmcfilters::Attribute::GrayLevelHeight;
     case morphology::Attribute::MeanGrayLevel:
         return mmcfilters::Attribute::MeanGrayLevel;
     case morphology::Attribute::GrayLevelVariance:
         return mmcfilters::Attribute::GrayLevelVariance;
-    case morphology::Attribute::BoxWidth:
+    case morphology::Attribute::BoundingBoxWidth:
         return mmcfilters::Attribute::BoxWidth;
     case morphology::Attribute::BoundingBoxHeight:
         return mmcfilters::Attribute::BoundingBoxHeight;
@@ -551,7 +551,7 @@ inline morphology::Attribute fromBackend(mmcfilters::Attribute attribute)
     MTLEARN_FROM_BACKEND_ATTRIBUTE(GrayLevelHeight, GrayLevelHeight);
     MTLEARN_FROM_BACKEND_ATTRIBUTE(MeanGrayLevel, MeanGrayLevel);
     MTLEARN_FROM_BACKEND_ATTRIBUTE(GrayLevelVariance, GrayLevelVariance);
-    MTLEARN_FROM_BACKEND_ATTRIBUTE(BoxWidth, BoxWidth);
+    MTLEARN_FROM_BACKEND_ATTRIBUTE(BoxWidth, BoundingBoxWidth);
     MTLEARN_FROM_BACKEND_ATTRIBUTE(BoundingBoxHeight, BoundingBoxHeight);
     MTLEARN_FROM_BACKEND_ATTRIBUTE(DiagonalLength, DiagonalLength);
     MTLEARN_FROM_BACKEND_ATTRIBUTE(Rectangularity, Rectangularity);

{mtlearn-1.2.0 → mtlearn-1.2.2}/mtlearn/include/mtlearn/morphology.hpp

@@ -61,7 +61,7 @@ enum class NodeIdSpace {
     Higra,              ///< Rows are indexed by exported Higra hierarchy ids.
 };
 
-/// Public attribute identifiers supported by the current morphology backend.
+/// Public attribute identifiers supported by the morphology facade.
 ///
 /// Keep this enum synchronized with the conversion table in
 /// `bindings/morphology/BindingSupport.hpp` and the Python exposure in
@@ -73,7 +73,8 @@ enum class Attribute {
     GrayLevelHeight,                  ///< Gray-level height.
     MeanGrayLevel,                   ///< Mean gray-level value.
     GrayLevelVariance,               ///< Gray-level variance.
-    BoxWidth,                    ///< Bounding-box width.
+    BoundingBoxWidth,            ///< Bounding-box width.
+    BoxWidth = BoundingBoxWidth,
     BoundingBoxHeight,                   ///< Bounding-box height.
     DiagonalLength,              ///< Bounding-box diagonal length.
     Rectangularity,               ///< Component rectangularity.
@@ -199,14 +200,15 @@ enum class Attribute {
     FilledCentroidDisplacementNormalized,
     FilledCompactness,
     FilledCircularity,
+    GrayLevel,                  ///< Node valuation (gray level).
 };
 
-/// Attribute groups expanded by the backend attribute computer.
+/// Attribute groups expanded by the morphology facade.
 ///
 /// Groups are part of the facade because Python notebooks and future C++
 /// consumers should not depend on backend attribute-group types directly.
 enum class AttributeGroup {
-    All,           ///< All public scalar attributes supported by the backend.
+    All,           ///< All public scalar attributes supported by the facade.
     GrayLevel,    ///< Gray-level attributes.
     Shape,         ///< Shape attributes.
     Moments,       ///< Moment-based attributes.

{mtlearn-1.2.0 → mtlearn-1.2.2}/mtlearn/python/mtlearn/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '1.2.0'
-__version_tuple__ = version_tuple = (1, 2, 0)
+__version__ = version = '1.2.2'
+__version_tuple__ = version_tuple = (1, 2, 2)
 
-__commit_id__ = commit_id = 'g1ddf531d3'
+__commit_id__ = commit_id = 'g218d9dab2'

{mtlearn-1.2.0 → mtlearn-1.2.2}/mtlearn/python/mtlearn/layers/cfp/normalization/stats_serializer.py

@@ -30,10 +30,15 @@ class StatsSerializer:
     ) -> dict[str, dict[str, Any]]:
         """Move serialized statistics to ``device`` for a CFP layer."""
         device = torch.device(device)
-        return {
-            str(key): {
+        result = {}
+        for key, stats in serialized.items():
+            key = str(key)
+            if key.endswith("::BOX_WIDTH"):
+                key = key.removesuffix("::BOX_WIDTH") + "::BOUNDING_BOX_WIDTH"
+            if key in result:
+                raise ValueError(f"duplicate dataset statistics for {key}")
+            result[key] = {
                 name: value.to(device) if torch.is_tensor(value) else value
                 for name, value in stats.items()
             }
-            for key, stats in serialized.items()
-        }
+        return result

{mtlearn-1.2.0 → mtlearn-1.2.2}/mtlearn/python/mtlearn/layers/cfp/preparation/_disk_preparation.py

@@ -4,7 +4,7 @@ from pathlib import Path
 import shutil
 
 from ..storage import DiskStore
-from ._identity import canonical_json, preprocessor_config, validate_identity, tree_config
+from ._identity import canonical_json, compatibility_config, preprocessor_config, validate_identity, tree_config
 from ._persistent_preparation import statistics_contract, statistics_identity
 from .preparation_result import DiskPreparationResult
 
@@ -78,7 +78,9 @@ def _header(store, preprocessor, manifest, source_version, preprocessing_version
                 "preprocessing_version": preprocessing_version, "split": split}
     if count is not None:
         expected["sample_count"] = count
-    differences = [name for name, value in expected.items() if row[name] != value]
+    differences = [name for name, value in expected.items()
+                   if (compatibility_config(json.loads(row[name])) != compatibility_config(json.loads(value))
+                       if name == "config" else row[name] != value)]
     if differences:
         raise ValueError(f"Manifest {manifest!r} is incompatible: {', '.join(differences)} changed. "
                          "Use the matching contract or a new manifest name; cache metadata must not be rewritten.")

{mtlearn-1.2.0 → mtlearn-1.2.2}/mtlearn/python/mtlearn/layers/cfp/preparation/_identity.py

@@ -1,11 +1,8 @@
 """Versioned, primitive-only identities for local persistent preparation."""
-from functools import lru_cache
 import hashlib
 import json
-from pathlib import Path
 
 from .... import morphology
-from ...._native import load_bindings
 from ..specs import FeatureSpec, TreeSpec
 
 FORMAT_VERSION = 4
@@ -24,12 +21,20 @@ def digest_file(path):
     return digest.hexdigest()
 
 
-@lru_cache(maxsize=1)
 def implementation_identity():
-    # Binary identity is deliberately conservative across backend builds. The
-    # semantic version must change when Python preparation/quantization changes.
-    return {"semantics": PREPARATION_SEMANTICS,
-            "native_sha256": digest_file(Path(load_bindings().__file__))}
+    return {"semantics": PREPARATION_SEMANTICS}
+
+
+def compatible_implementation(implementation):
+    return (isinstance(implementation, dict)
+            and set(implementation) in ({"semantics"}, {"semantics", "native_sha256"})
+            and implementation["semantics"] == PREPARATION_SEMANTICS)
+
+
+def compatibility_config(config):
+    if not isinstance(config, dict) or not compatible_implementation(config.get("implementation")):
+        raise ValueError("Incompatible persistent preparation implementation or format.")
+    return dict(config, implementation=implementation_identity())
 
 
 def tree_config(spec):
@@ -55,7 +60,8 @@ def preprocessor_config(preprocessor):
     from .cfp_preprocessor import CFPPreprocessor
     if type(preprocessor) is not CFPPreprocessor or preprocessor.morphology is not morphology:
         raise ValueError("Persistent preparation currently requires the standard CFPPreprocessor implementation.")
-    return {"format_version": FORMAT_VERSION, "implementation": dict(implementation_identity()),
+    implementation = getattr(preprocessor, "_implementation", implementation_identity())
+    return {"format_version": FORMAT_VERSION, "implementation": dict(implementation),
             "attribute_dtype": preprocessor.attribute_dtype.name,
             "trees": [{"tree": tree_config(spec),
                        "attributes": [a.name for a in preprocessor.features[key].attributes]}
@@ -64,14 +70,16 @@ def preprocessor_config(preprocessor):
 
 def preprocessor_from_config(config):
     from .cfp_preprocessor import CFPPreprocessor
-    if config["format_version"] != FORMAT_VERSION or config["implementation"] != implementation_identity():
+    if config["format_version"] != FORMAT_VERSION or not compatible_implementation(config["implementation"]):
         raise ValueError("Incompatible persistent preparation implementation or format.")
     trees, features = {}, {}
     for entry in config["trees"]:
         tree = tree_from_config(entry["tree"])
         trees[tree.cache_key()] = tree
         features[tree.cache_key()] = FeatureSpec(tuple(getattr(morphology.AttributeType, a) for a in entry["attributes"]))
-    return CFPPreprocessor(tree_specs=trees, features=features, attribute_dtype=config["attribute_dtype"])
+    preprocessor = CFPPreprocessor(tree_specs=trees, features=features, attribute_dtype=config["attribute_dtype"])
+    preprocessor._implementation = dict(config["implementation"])
+    return preprocessor
 
 
 def image_identity(preprocessor, image, tree_key, *, pixel_digest=None):
@@ -91,7 +99,7 @@ def validate_identity(identity):
     expected = {"format_version", "implementation", "pixels_sha256", "shape", "tree", "attributes", "attribute_dtype"}
     if set(identity) != expected or identity["format_version"] != FORMAT_VERSION:
         raise ValueError("Unsupported preparation identity format.")
-    if identity["implementation"] != implementation_identity():
+    if not compatible_implementation(identity["implementation"]):
         raise ValueError("Preparation backend/semantics do not match this runtime.")
     digest = identity["pixels_sha256"]
     if not isinstance(digest, str) or len(digest) != 64 or any(c not in "0123456789abcdef" for c in digest):

{mtlearn-1.2.0 → mtlearn-1.2.2}/mtlearn/python/mtlearn/layers/cfp/preparation/_persistent_preparation.py

@@ -10,7 +10,7 @@ from ..._helpers import to_numpy_u8
 from ..normalization import AttributeNormalizer
 from ..normalization.statistics_snapshot import StatisticsSnapshot
 from ..runtime.cache_input_contract import validate_cfp_cache_batch_x
-from ._identity import canonical_json, image_identity, identity_key, preprocessor_config, preprocessor_from_config
+from ._identity import canonical_json, compatibility_config, image_identity, identity_key, preprocessor_config, preprocessor_from_config
 from .preparation_result import PreparationResult
 from ._preparation_progress import emit
 
@@ -63,13 +63,19 @@ def _begin_manifest(store, name, preprocessor, count, source_version, preprocess
     if any(not isinstance(v, str) or not v for v in (name, source_version, preprocessing_version)):
         raise ValueError("Persistent preparation requires manifest, source_version and preprocessing_version strings.")
     config = canonical_json(preprocessor_config(preprocessor))
-    values = (config, source_version, preprocessing_version, split, count)
     row = store._db.execute("SELECT * FROM manifests WHERE name=?", (name,)).fetchone()
+    if row is not None:
+        previous = json.loads(row["config"])
+        if compatibility_config(previous) == compatibility_config(json.loads(config)):
+            preprocessor = preprocessor_from_config(previous)
+            config = row["config"]
+    values = (config, source_version, preprocessing_version, split, count)
     if row is not None and tuple(row[k] for k in ("config", "source_version", "preprocessing_version", "split", "sample_count")) != values:
         raise ValueError("Manifest contract/source version changed; use a new manifest name to reuse compatible content safely.")
     with store._db:
         store._db.execute("""INSERT INTO manifests VALUES (?,?,?,?,?,?,'preparing',NULL)
             ON CONFLICT(name) DO UPDATE SET state='preparing',error=NULL""", (name, *values))
+    return preprocessor
 
 
 def _bind_sample(store, name, position, sample_id, bindings, shape):
@@ -146,7 +152,7 @@ def prepare_source(preprocessor, source, *, store=None, manifest=None, source_ve
     persistent = getattr(store, "persistent", False)
     if persistent:
         store._check(write=True)
-        _begin_manifest(store, manifest, preprocessor, count, source_version, preprocessing_version, split)
+        preprocessor = _begin_manifest(store, manifest, preprocessor, count, source_version, preprocessing_version, split)
         # A resumed pass must revalidate durable files, even if this process has
         # already borrowed valid CPU buffers from a now damaged/missing file.
         store.clear()

{mtlearn-1.2.0 → mtlearn-1.2.2}/mtlearn/python/mtlearn/layers/cfp/storage/_manifest.py

@@ -2,7 +2,7 @@
 import json
 import sqlite3
 
-from ..preparation._identity import FORMAT_VERSION, canonical_json, implementation_identity
+from ..preparation._identity import FORMAT_VERSION, canonical_json, implementation_identity, compatible_implementation
 
 SCHEMA = """
 CREATE TABLE metadata (key TEXT PRIMARY KEY, value TEXT NOT NULL);
@@ -55,7 +55,10 @@ def connect(path, *, readonly):
                 db.rollback()
                 raise
         row = db.execute("SELECT value FROM metadata WHERE key='contract'").fetchone()
-        if row is None or json.loads(row[0]) != expected:
+        contract = json.loads(row[0]) if row is not None else None
+        if (not isinstance(contract, dict) or set(contract) != set(expected)
+                or contract["format_version"] != FORMAT_VERSION
+                or not compatible_implementation(contract["implementation"])):
             raise ValueError("DiskStore format/backend is incompatible; use a separate store directory.")
     except BaseException:
         db.close()

mtlearn-1.2.2/mtlearn/tests/python/cfp_reference_names.py

@@ -0,0 +1,12 @@
+import re
+
+
+def canonical_reference_names(value):
+    if isinstance(value, str):
+        return re.sub(r"(?<![A-Z0-9_])BOX_WIDTH(?![A-Z0-9_])", "BOUNDING_BOX_WIDTH", value)
+    if isinstance(value, dict):
+        return {canonical_reference_names(key): canonical_reference_names(item)
+                for key, item in value.items()}
+    if isinstance(value, (tuple, list)):
+        return type(value)(canonical_reference_names(item) for item in value)
+    return value

{mtlearn-1.2.0 → mtlearn-1.2.2}/mtlearn/tests/python/test_cfp_cache_baseline.py

@@ -11,6 +11,8 @@ from pathlib import Path
 import pytest
 import torch
 
+from cfp_reference_names import canonical_reference_names
+
 import mtlearn
 from mtlearn.layers import ConnectedFilterPreprocessingLayer, load_checkpoint
 
@@ -25,7 +27,7 @@ CASE_NAMES = MANIFEST['cases']
 
 @pytest.fixture(scope='module')
 def reference():
-    return torch.load(FIXTURES / 'baseline.pt', map_location='cpu', weights_only=True)
+    return canonical_reference_names(torch.load(FIXTURES / 'baseline.pt', map_location='cpu', weights_only=True))
 
 
 def assert_nested_close(actual, expected):