bw-processing 1.2__tar.gz → 1.3__tar.gz

{bw_processing-1.2/src/bw_processing.egg-info → bw_processing-1.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: bw_processing
-Version: 1.2
+Version: 1.3
 Summary: Foo
 Author-email: Chris Mutel <cmutel@gmail.com>
 Maintainer-email: Chris Mutel <cmutel@gmail.com>
@@ -84,6 +84,7 @@ The [Brightway LCA framework](https://brightway.dev/) has stored data used in co
 * **Dynamic data sources**. Instead of requiring that data for matrix construction be present and savedd on disk, it can now be generated dynamically, either through code running locally or on another computer system. This is a big step towards embeddding life cycle assessment in a web of environmental models.
 * **Use [fsspec](https://filesystem-spec.readthedocs.io/en/latest/) for file IO**. The use of this library allows for data packages to be stored on your local computer, or on [many logical or virtual file systems](https://docs.pyfilesystem.org/en/latest/guide.html).
 * **Simpler handling of numeric values whose sign should be flipped**. Sometimes it is more convenient to specify positive numbers in dataset definitions, even though such numbers should be negative when inserted into the resulting matrices. For example, in the technosphere matrix in life cycle assessment, products produced are positive and products consumed are negative, though both values are given as positive in datasets. Brightway used to use a type mapping dictionary to indicate which values in a matrix should have their sign flipped after insertion. Such mapping dictionaries are brittle and inelegant. `bw_processing` uses an optional boolean vector, called `flip`, to indicate if any values should be flipped.
+* **Per-exchange multiplicative scaling**. An optional float vector, called `scale`, can be attached to any resource group. Each element is a multiplicative factor applied to the corresponding data value — whether static or sampled stochastically — before it is inserted into the matrix. Typical uses are allocation factors and unit conversions. A value of `1.0` leaves the data unchanged.
 * **Separation of uncertainty distribution parameters from other data**. Fitting data to a [probability density function](https://en.wikipedia.org/wiki/Probability_density_function) (PDF), or an estimate of such a PDF, is only one approach to quantitative uncertainty analysis. We would like to support other approaches, including [direct sampling from real data](https://github.com/PascalLesage/presamples/). Therefore, uncertainty distribution parameters are stored separately,  only loaded if needed, and are only one way to express quantitative uncertainty.
 
 ## Concepts
@@ -96,6 +97,18 @@ Data objects can be vectors or arrays. Vectors will always produce the same matr
 
 ### Vectors versus arrays
 
+Vectors and arrays differ in how many possible values they provide per matrix cell.
+
+A **vector** provides one value per cell. Every time a vector resource is used, it produces the same result. This is the standard case for deterministic LCA calculations.
+
+An **array** provides multiple possible values per cell, stored as columns of a 2-D numpy array. Each time the data package is iterated, a different column is selected and inserted into the matrix. This is used for:
+
+* **Monte Carlo analysis** — columns hold independently sampled values drawn from the uncertainty distributions.
+* **Scenario analysis** — each column represents a predefined scenario, such as a different technology mix or policy assumption.
+* **Presamples** — a generalization of the [presamples library](https://github.com/PascalLesage/presamples/), where pre-drawn samples are stored for reproducibility.
+
+Which column is selected on each iteration is controlled by the `sequential` and `combinatorial` policies; see [Policies](#policies).
+
 ### Persistent versus dynamic
 
 Persistent data is fixed, and can be completely loaded into memory and used directly or written to disk. Dynamic data is only resolved as the data is used, during matrix construction and iteration. Dynamic data is provided by *interfaces* - Python code that either generates the data, or wraps data coming from other software. There are many possible use cases for data interfaces, including:
@@ -187,6 +200,36 @@ print(data_obj.url)
 >>> "example.com"
 ```
 
+### Scale arrays
+
+Any resource group (persistent or dynamic, vector or array) can carry an optional `scale_array`: a one-dimensional float array of the same length as `indices_array`. Each element is a multiplicative factor applied to the corresponding data value before it is inserted into the matrix. The factor is applied to both static and stochastically-sampled values. A value of `1.0` leaves the data unchanged.
+
+Typical use cases:
+
+* **Allocation factors** — when a process produces multiple products, the exchange amounts must be partitioned between them. Storing the allocation coefficients as a `scale_array` keeps them alongside the data they modify without requiring a separate processing step.
+* **Unit conversions** — when source data is expressed in a unit that differs from the matrix convention, a constant conversion factor can be stored as a `scale_array` rather than baked into every data value.
+
+```python
+import numpy as np
+from bw_processing import create_datapackage
+from bw_processing.constants import INDICES_DTYPE
+
+dp = create_datapackage()
+indices_array = np.array([(1, 4), (2, 5), (3, 6)], dtype=INDICES_DTYPE)
+data_array = np.array([100.0, 200.0, 300.0])
+scale_array = np.array([0.6, 1.0, 0.4])  # e.g. allocation factors
+
+dp.add_persistent_vector(
+    matrix="technosphere",
+    name="my-process",
+    indices_array=indices_array,
+    data_array=data_array,
+    scale_array=scale_array,
+)
+```
+
+The stored resource has `kind="scale"` and can be retrieved via `dp.get_resource("my-process.scale")`. The `scale_array` must be a float dtype (`float32` or `float64`); passing an integer array raises `WrongDatatype`.
+
 ### Policies
 
 Data package policies define how the data should be used. Policies apply to the entire data package; you may wish to adjust what is stored in which data packages to get the effect you desire.
@@ -219,7 +262,7 @@ Please make sure you understand how `combinatorial` and `sequential` interact! T
 
 ## Install
 
-Install using pip or conda (channel `cmutel`). Depends on `numpy` and `pandas` (for reading and writing CSVs).
+Install using pip or conda (channel `conda-forge`). Depends on `numpy` and `pandas` (for reading and writing CSVs).
 
 Has no explicit or implicit dependence on any other part of Brightway.
 

{bw_processing-1.2 → bw_processing-1.3}/README.md

@@ -42,6 +42,7 @@ The [Brightway LCA framework](https://brightway.dev/) has stored data used in co
 * **Dynamic data sources**. Instead of requiring that data for matrix construction be present and savedd on disk, it can now be generated dynamically, either through code running locally or on another computer system. This is a big step towards embeddding life cycle assessment in a web of environmental models.
 * **Use [fsspec](https://filesystem-spec.readthedocs.io/en/latest/) for file IO**. The use of this library allows for data packages to be stored on your local computer, or on [many logical or virtual file systems](https://docs.pyfilesystem.org/en/latest/guide.html).
 * **Simpler handling of numeric values whose sign should be flipped**. Sometimes it is more convenient to specify positive numbers in dataset definitions, even though such numbers should be negative when inserted into the resulting matrices. For example, in the technosphere matrix in life cycle assessment, products produced are positive and products consumed are negative, though both values are given as positive in datasets. Brightway used to use a type mapping dictionary to indicate which values in a matrix should have their sign flipped after insertion. Such mapping dictionaries are brittle and inelegant. `bw_processing` uses an optional boolean vector, called `flip`, to indicate if any values should be flipped.
+* **Per-exchange multiplicative scaling**. An optional float vector, called `scale`, can be attached to any resource group. Each element is a multiplicative factor applied to the corresponding data value — whether static or sampled stochastically — before it is inserted into the matrix. Typical uses are allocation factors and unit conversions. A value of `1.0` leaves the data unchanged.
 * **Separation of uncertainty distribution parameters from other data**. Fitting data to a [probability density function](https://en.wikipedia.org/wiki/Probability_density_function) (PDF), or an estimate of such a PDF, is only one approach to quantitative uncertainty analysis. We would like to support other approaches, including [direct sampling from real data](https://github.com/PascalLesage/presamples/). Therefore, uncertainty distribution parameters are stored separately,  only loaded if needed, and are only one way to express quantitative uncertainty.
 
 ## Concepts
@@ -54,6 +55,18 @@ Data objects can be vectors or arrays. Vectors will always produce the same matr
 
 ### Vectors versus arrays
 
+Vectors and arrays differ in how many possible values they provide per matrix cell.
+
+A **vector** provides one value per cell. Every time a vector resource is used, it produces the same result. This is the standard case for deterministic LCA calculations.
+
+An **array** provides multiple possible values per cell, stored as columns of a 2-D numpy array. Each time the data package is iterated, a different column is selected and inserted into the matrix. This is used for:
+
+* **Monte Carlo analysis** — columns hold independently sampled values drawn from the uncertainty distributions.
+* **Scenario analysis** — each column represents a predefined scenario, such as a different technology mix or policy assumption.
+* **Presamples** — a generalization of the [presamples library](https://github.com/PascalLesage/presamples/), where pre-drawn samples are stored for reproducibility.
+
+Which column is selected on each iteration is controlled by the `sequential` and `combinatorial` policies; see [Policies](#policies).
+
 ### Persistent versus dynamic
 
 Persistent data is fixed, and can be completely loaded into memory and used directly or written to disk. Dynamic data is only resolved as the data is used, during matrix construction and iteration. Dynamic data is provided by *interfaces* - Python code that either generates the data, or wraps data coming from other software. There are many possible use cases for data interfaces, including:
@@ -145,6 +158,36 @@ print(data_obj.url)
 >>> "example.com"
 ```
 
+### Scale arrays
+
+Any resource group (persistent or dynamic, vector or array) can carry an optional `scale_array`: a one-dimensional float array of the same length as `indices_array`. Each element is a multiplicative factor applied to the corresponding data value before it is inserted into the matrix. The factor is applied to both static and stochastically-sampled values. A value of `1.0` leaves the data unchanged.
+
+Typical use cases:
+
+* **Allocation factors** — when a process produces multiple products, the exchange amounts must be partitioned between them. Storing the allocation coefficients as a `scale_array` keeps them alongside the data they modify without requiring a separate processing step.
+* **Unit conversions** — when source data is expressed in a unit that differs from the matrix convention, a constant conversion factor can be stored as a `scale_array` rather than baked into every data value.
+
+```python
+import numpy as np
+from bw_processing import create_datapackage
+from bw_processing.constants import INDICES_DTYPE
+
+dp = create_datapackage()
+indices_array = np.array([(1, 4), (2, 5), (3, 6)], dtype=INDICES_DTYPE)
+data_array = np.array([100.0, 200.0, 300.0])
+scale_array = np.array([0.6, 1.0, 0.4])  # e.g. allocation factors
+
+dp.add_persistent_vector(
+    matrix="technosphere",
+    name="my-process",
+    indices_array=indices_array,
+    data_array=data_array,
+    scale_array=scale_array,
+)
+```
+
+The stored resource has `kind="scale"` and can be retrieved via `dp.get_resource("my-process.scale")`. The `scale_array` must be a float dtype (`float32` or `float64`); passing an integer array raises `WrongDatatype`.
+
 ### Policies
 
 Data package policies define how the data should be used. Policies apply to the entire data package; you may wish to adjust what is stored in which data packages to get the effect you desire.
@@ -177,7 +220,7 @@ Please make sure you understand how `combinatorial` and `sequential` interact! T
 
 ## Install
 
-Install using pip or conda (channel `cmutel`). Depends on `numpy` and `pandas` (for reading and writing CSVs).
+Install using pip or conda (channel `conda-forge`). Depends on `numpy` and `pandas` (for reading and writing CSVs).
 
 Has no explicit or implicit dependence on any other part of Brightway.
 

{bw_processing-1.2 → bw_processing-1.3}/src/bw_processing/__init__.py

@@ -29,7 +29,7 @@ __all__ = (
     "UndefinedInterface",
 )
 
-__version__ = "1.2"
+__version__ = "1.3"
 
 
 from bw_processing.array_creation import create_array, create_structured_array

{bw_processing-1.2 → bw_processing-1.3}/src/bw_processing/constants.py

@@ -35,12 +35,15 @@ DEFAULT_LICENSES = [
 
 
 class MatrixSerializeFormat(str, Enum):
-    """
-    Enum with the serializing formats for the vectors and matrices.
+    """Serialization format used when writing numpy arrays to disk.
+
+    Because this is a ``str`` enum, values can be compared directly to strings.
+    The default is ``NUMPY``.  ``PARQUET`` requires the optional ``pyarrow``
+    dependency.
     """
 
-    NUMPY = "numpy"  # numpy .npy format
-    PARQUET = "parquet"  # Apache .parquet format
+    NUMPY = "numpy"
+    PARQUET = "parquet"
 
 
 # FILE EXTENSIONS

{bw_processing-1.2 → bw_processing-1.3}/src/bw_processing/datapackage.py

@@ -384,6 +384,17 @@ class Datapackage(DatapackageBase):
         self.data = []
 
     def finalize_serialization(self) -> None:
+        """Write the metadata file and close the filesystem.
+
+        Must be called once after all resources have been added, before the
+        datapackage can be loaded again from disk. Dehydrates any interface
+        resources (replaces them with ``UndefinedInterface``) prior to writing.
+
+        Raises:
+            Closed: Datapackage has already been finalized.
+            ValueError: Datapackage uses an in-memory filesystem, which cannot
+                be serialized.
+        """
         if self._finalized:
             raise Closed("Datapackage already finalized")
         elif isinstance(self.fs, DictFS):
@@ -485,11 +496,20 @@ class Datapackage(DatapackageBase):
         data_array: Optional[np.ndarray] = None,
         flip_array: Optional[np.ndarray] = None,
         distributions_array: Optional[np.ndarray] = None,
+        scale_array: Optional[np.ndarray] = None,
         keep_proxy: bool = False,
         matrix_serialize_format_type: Optional[MatrixSerializeFormat] = None,
         **kwargs,
     ) -> None:
-        """ """
+        """Add a persistent vector resource group to the datapackage.
+
+        ``scale_array`` is an optional 1-D float array of the same length as
+        ``indices_array``.  Each element is a multiplicative factor applied to
+        the corresponding data value — whether static or stochastic — before
+        the value is inserted into the matrix.  Typical uses are allocation
+        factors and unit conversions.  A value of ``1.0`` leaves the data
+        unchanged.
+        """
         self._prepare_modifications()
 
         # Check lengths
@@ -580,6 +600,15 @@ class Datapackage(DatapackageBase):
                     meta_type="generic",
                     **kwargs,
                 )
+        if scale_array is not None:
+            self._add_scale_array_resource(
+                scale_array=scale_array,
+                indices_array=indices_array,
+                name=name,
+                keep_proxy=keep_proxy,
+                matrix_serialize_format_type=matrix_serialize_format_type,
+                **kwargs,
+            )
 
     def add_persistent_array(
         self,
@@ -589,11 +618,20 @@ class Datapackage(DatapackageBase):
         indices_array: np.ndarray,
         name: Optional[str] = None,
         flip_array: Optional[np.ndarray] = None,
+        scale_array: Optional[np.ndarray] = None,
         keep_proxy: bool = False,
         matrix_serialize_format_type: Optional[MatrixSerializeFormat] = None,
         **kwargs,
     ) -> None:
-        """ """
+        """Add a persistent array resource group to the datapackage.
+
+        ``scale_array`` is an optional 1-D float array of the same length as
+        ``indices_array``.  Each element is a multiplicative factor applied to
+        the corresponding data value — whether static or stochastic — before
+        the value is inserted into the matrix.  Typical uses are allocation
+        factors and unit conversions.  A value of ``1.0`` leaves the data
+        unchanged.
+        """
         self._prepare_modifications()
 
         kwargs.update({"matrix": matrix, "category": "array", "nrows": len(indices_array)})
@@ -660,9 +698,23 @@ class Datapackage(DatapackageBase):
                     meta_type="generic",
                     **kwargs,
                 )
+        if scale_array is not None:
+            self._add_scale_array_resource(
+                scale_array=scale_array,
+                indices_array=indices_array,
+                name=name,
+                keep_proxy=keep_proxy,
+                matrix_serialize_format_type=matrix_serialize_format_type,
+                **kwargs,
+            )
 
     def write_modified(self):
-        """Write the data in modified files to the filesystem (if allowed)."""
+        """Flush modified resources back to the filesystem.
+
+        After directly editing data arrays in ``self.data``, call this method
+        to persist the changes. Clears the internal ``_modified`` set on
+        completion. Does nothing if no resources have been marked as modified.
+        """
         for index in self._modified:
             # get resource
             resource = self.resources[index]
@@ -683,7 +735,7 @@ class Datapackage(DatapackageBase):
                     if kind == "indices":
                         meta_object = "vector"
                         meta_type = "indices"
-                    elif kind == "flip":
+                    elif kind in ("flip", "scale"):
                         meta_object = "vector"
                         meta_type = "generic"
                     elif kind == "distributions":
@@ -716,6 +768,39 @@ class Datapackage(DatapackageBase):
 
         self._modified = set()
 
+    def _add_scale_array_resource(
+        self,
+        *,
+        scale_array: np.ndarray,
+        indices_array: np.ndarray,
+        name: str,
+        keep_proxy: bool,
+        matrix_serialize_format_type: Optional[MatrixSerializeFormat],
+        **kwargs,
+    ) -> None:
+        scale_array = load_bytes(scale_array)
+        if not np.issubdtype(scale_array.dtype, np.floating):
+            raise WrongDatatype(
+                "`scale_array` dtype is {}, but must be a float dtype".format(scale_array.dtype)
+            )
+        elif scale_array.shape != indices_array.shape:
+            raise ShapeMismatch(
+                "`scale_array` shape ({}) doesn't match `indices_array` ({}).".format(
+                    scale_array.shape, indices_array.shape
+                )
+            )
+        self._add_numpy_array_resource(
+            array=scale_array,
+            group=name,
+            name=name + ".scale",
+            kind="scale",
+            keep_proxy=keep_proxy,
+            matrix_serialize_format_type=matrix_serialize_format_type,
+            meta_object="vector",
+            meta_type="generic",
+            **kwargs,
+        )
+
     def _add_numpy_array_resource(
         self,
         *,
@@ -798,10 +883,36 @@ class Datapackage(DatapackageBase):
         indices_array: np.ndarray,  # Not interface
         name: Optional[str] = None,
         flip_array: Optional[np.ndarray] = None,  # Not interface
+        scale_array: Optional[np.ndarray] = None,  # Not interface
         keep_proxy: bool = False,
         matrix_serialize_format_type: Optional[MatrixSerializeFormat] = None,
         **kwargs,
     ) -> None:
+        """Add a dynamic vector resource group to the datapackage.
+
+        The matrix values are provided at runtime by ``interface`` rather than
+        stored on disk. ``interface`` must implement ``__next__()`` and return a
+        1-D numpy array of length ``len(indices_array)`` each time it is called.
+
+        The ``indices_array``, optional ``flip_array``, and optional
+        ``scale_array`` are static and are stored as normal numpy resources.
+
+        Args:
+            matrix: Name of the target matrix.
+            interface: Object implementing the dynamic-vector interface
+                (``__next__()``).
+            indices_array: Structured numpy array with dtype ``INDICES_DTYPE``
+                mapping each data value to a matrix cell.
+            name: Optional resource group name; auto-generated if omitted.
+            flip_array: Optional boolean array; where ``True`` the value is
+                multiplied by ``-1`` before insertion.
+            scale_array: Optional 1-D float array of multiplicative factors
+                applied before matrix insertion.
+            keep_proxy: If ``True``, store a proxy rather than the raw array
+                for on-disk resources.
+            matrix_serialize_format_type: Override the instance-level
+                serialization format for static arrays in this group.
+        """
         self._prepare_modifications()
 
         kwargs.update({"matrix": matrix, "category": "vector", "nrows": len(indices_array)})
@@ -843,6 +954,15 @@ class Datapackage(DatapackageBase):
                     meta_type="generic",
                     **kwargs,
                 )
+        if scale_array is not None:
+            self._add_scale_array_resource(
+                scale_array=scale_array,
+                indices_array=indices_array,
+                name=name,
+                keep_proxy=keep_proxy,
+                matrix_serialize_format_type=matrix_serialize_format_type,
+                **kwargs,
+            )
 
         self.data.append(interface)
         resource = {
@@ -862,11 +982,38 @@ class Datapackage(DatapackageBase):
         indices_array: np.ndarray,  # Not interface
         name: Optional[str] = None,
         flip_array: Optional[np.ndarray] = None,
+        scale_array: Optional[np.ndarray] = None,  # Not interface
         keep_proxy: bool = False,
         matrix_serialize_format_type: Optional[MatrixSerializeFormat] = None,
         **kwargs,
     ) -> None:
-        """`interface` must support the presamples API."""
+        """Add a dynamic array resource group to the datapackage.
+
+        The matrix values are provided at runtime by ``interface``, which must
+        implement the presamples array API: a ``.shape`` property returning
+        ``(nrows, ncols)`` and ``.__getitem__(args)`` returning the 1-D column
+        array for ``args[1]``.  ``ncols`` may be ``None`` for an infinite
+        interface.
+
+        The ``indices_array``, optional ``flip_array``, and optional
+        ``scale_array`` are static and are stored as normal numpy resources.
+
+        Args:
+            matrix: Name of the target matrix.
+            interface: Object implementing the dynamic-array interface
+                (``.shape`` and ``.__getitem__``).
+            indices_array: Structured numpy array with dtype ``INDICES_DTYPE``
+                mapping each data value to a matrix cell.
+            name: Optional resource group name; auto-generated if omitted.
+            flip_array: Optional boolean array; where ``True`` the value is
+                multiplied by ``-1`` before insertion.
+            scale_array: Optional 1-D float array of multiplicative factors
+                applied before matrix insertion.
+            keep_proxy: If ``True``, store a proxy rather than the raw array
+                for on-disk resources.
+            matrix_serialize_format_type: Override the instance-level
+                serialization format for static arrays in this group.
+        """
         self._prepare_modifications()
 
         if isinstance(flip_array, np.ndarray) and not flip_array.sum():
@@ -911,6 +1058,15 @@ class Datapackage(DatapackageBase):
                     meta_type="generic",
                     **kwargs,
                 )
+        if scale_array is not None:
+            self._add_scale_array_resource(
+                scale_array=scale_array,
+                indices_array=indices_array,
+                name=name,
+                keep_proxy=keep_proxy,
+                matrix_serialize_format_type=matrix_serialize_format_type,
+                **kwargs,
+            )
 
         self.data.append(interface)
         resource = {

{bw_processing-1.2 → bw_processing-1.3}/src/bw_processing/io_helpers.py

@@ -27,6 +27,19 @@ except ImportError:
 
 
 def generic_directory_filesystem(*, dirpath: Path) -> DirFileSystem:
+    """Return a ``DirFileSystem`` rooted at ``dirpath``, creating it if needed.
+
+    Args:
+        dirpath: Path to the target directory. Created if it does not exist;
+            its parent must already exist.
+
+    Returns:
+        A ``fsspec`` ``DirFileSystem`` pointing at ``dirpath``.
+
+    Raises:
+        ValueError: Parent directory does not exist.
+        AssertionError: ``dirpath`` is not a ``pathlib.Path``.
+    """
     assert isinstance(dirpath, Path), "`dirpath` must be a `pathlib.Path` instance"
     if not dirpath.is_dir():
         if not dirpath.parent.is_dir():
@@ -43,6 +56,28 @@ def generic_zipfile_filesystem(
     compression: int = zipfile.ZIP_DEFLATED,
     compresslevel: Optional[int] = None,
 ) -> ZipFileSystem:
+    """Return a ``ZipFileSystem`` for ``dirpath / filename``.
+
+    Args:
+        dirpath: Directory that contains (or will contain) the zip file.
+            Must already exist.
+        filename: Name of the zip file (e.g. ``"my-dp.zip"``).
+        write: If ``True`` (default), open for writing; otherwise open for
+            reading.
+        compression: ``zipfile`` compression constant.  Defaults to
+            ``zipfile.ZIP_DEFLATED`` which gives good compression speed.
+            Pass ``zipfile.ZIP_STORED`` for no compression or
+            ``zipfile.ZIP_LZMA`` for maximum compression.
+        compresslevel: Compression level passed to ``zipfile.ZipFile``;
+            ``None`` uses the default for the chosen algorithm.
+
+    Returns:
+        A ``fsspec`` ``ZipFileSystem``.
+
+    Raises:
+        ValueError: ``dirpath`` does not exist.
+        AssertionError: ``dirpath`` is not a ``pathlib.Path``.
+    """
     assert isinstance(dirpath, Path), "`dirpath` must be a `pathlib.Path` instance"
     if not dirpath.is_dir():
         raise ValueError("Destination directory `{}` doesn't exist".format(dirpath))

{bw_processing-1.2 → bw_processing-1.3}/src/bw_processing/proxies.py

@@ -5,6 +5,14 @@ class UndefinedInterface:
 
 
 class Proxy:
+    """Deferred file-read wrapper returned when ``proxy=True`` is passed to ``file_reader``.
+
+    Stores the reader function and its arguments without executing them. The
+    actual data is loaded the first time the proxy is called (i.e. when
+    ``get_resource`` resolves it). The file or buffer is rewound to position 0
+    before each call so that repeated calls return the same data.
+    """
+
     def __init__(self, func, label, kwargs):
         self.func = func
         self.label = label

{bw_processing-1.2 → bw_processing-1.3}/src/bw_processing/unique_fields.py

@@ -60,6 +60,23 @@ def greedy_set_cover(data, exclude=None, raise_error=True):
 
 
 def as_unique_attributes_dataframe(df, exclude=None, include=None, raise_error=False):
+    """Return a copy of ``df`` keeping only the columns needed to uniquely identify each row.
+
+    Columns are selected using the greedy set-cover heuristic (see
+    :func:`greedy_set_cover`). Columns listed in ``include`` are always kept,
+    even if they are not required for uniqueness.
+
+    Args:
+        df: Input pandas DataFrame.
+        exclude: Column names to skip during the uniqueness search.
+        include: Column names to always retain in the output.
+        raise_error: If ``True``, raise ``NonUnique`` when no unique set can be
+            found; otherwise, all columns are kept silently.
+
+    Returns:
+        A DataFrame with only the uniqueness-covering (and always-included)
+        columns.
+    """
     assert isinstance(df, pd.DataFrame)
     include = greedy_set_cover(
         df.reset_index().to_dict("records"), exclude=exclude, raise_error=raise_error

{bw_processing-1.2 → bw_processing-1.3/src/bw_processing.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: bw_processing
-Version: 1.2
+Version: 1.3
 Summary: Foo
 Author-email: Chris Mutel <cmutel@gmail.com>
 Maintainer-email: Chris Mutel <cmutel@gmail.com>
@@ -84,6 +84,7 @@ The [Brightway LCA framework](https://brightway.dev/) has stored data used in co
 * **Dynamic data sources**. Instead of requiring that data for matrix construction be present and savedd on disk, it can now be generated dynamically, either through code running locally or on another computer system. This is a big step towards embeddding life cycle assessment in a web of environmental models.
 * **Use [fsspec](https://filesystem-spec.readthedocs.io/en/latest/) for file IO**. The use of this library allows for data packages to be stored on your local computer, or on [many logical or virtual file systems](https://docs.pyfilesystem.org/en/latest/guide.html).
 * **Simpler handling of numeric values whose sign should be flipped**. Sometimes it is more convenient to specify positive numbers in dataset definitions, even though such numbers should be negative when inserted into the resulting matrices. For example, in the technosphere matrix in life cycle assessment, products produced are positive and products consumed are negative, though both values are given as positive in datasets. Brightway used to use a type mapping dictionary to indicate which values in a matrix should have their sign flipped after insertion. Such mapping dictionaries are brittle and inelegant. `bw_processing` uses an optional boolean vector, called `flip`, to indicate if any values should be flipped.
+* **Per-exchange multiplicative scaling**. An optional float vector, called `scale`, can be attached to any resource group. Each element is a multiplicative factor applied to the corresponding data value — whether static or sampled stochastically — before it is inserted into the matrix. Typical uses are allocation factors and unit conversions. A value of `1.0` leaves the data unchanged.
 * **Separation of uncertainty distribution parameters from other data**. Fitting data to a [probability density function](https://en.wikipedia.org/wiki/Probability_density_function) (PDF), or an estimate of such a PDF, is only one approach to quantitative uncertainty analysis. We would like to support other approaches, including [direct sampling from real data](https://github.com/PascalLesage/presamples/). Therefore, uncertainty distribution parameters are stored separately,  only loaded if needed, and are only one way to express quantitative uncertainty.
 
 ## Concepts
@@ -96,6 +97,18 @@ Data objects can be vectors or arrays. Vectors will always produce the same matr
 
 ### Vectors versus arrays
 
+Vectors and arrays differ in how many possible values they provide per matrix cell.
+
+A **vector** provides one value per cell. Every time a vector resource is used, it produces the same result. This is the standard case for deterministic LCA calculations.
+
+An **array** provides multiple possible values per cell, stored as columns of a 2-D numpy array. Each time the data package is iterated, a different column is selected and inserted into the matrix. This is used for:
+
+* **Monte Carlo analysis** — columns hold independently sampled values drawn from the uncertainty distributions.
+* **Scenario analysis** — each column represents a predefined scenario, such as a different technology mix or policy assumption.
+* **Presamples** — a generalization of the [presamples library](https://github.com/PascalLesage/presamples/), where pre-drawn samples are stored for reproducibility.
+
+Which column is selected on each iteration is controlled by the `sequential` and `combinatorial` policies; see [Policies](#policies).
+
 ### Persistent versus dynamic
 
 Persistent data is fixed, and can be completely loaded into memory and used directly or written to disk. Dynamic data is only resolved as the data is used, during matrix construction and iteration. Dynamic data is provided by *interfaces* - Python code that either generates the data, or wraps data coming from other software. There are many possible use cases for data interfaces, including:
@@ -187,6 +200,36 @@ print(data_obj.url)
 >>> "example.com"
 ```
 
+### Scale arrays
+
+Any resource group (persistent or dynamic, vector or array) can carry an optional `scale_array`: a one-dimensional float array of the same length as `indices_array`. Each element is a multiplicative factor applied to the corresponding data value before it is inserted into the matrix. The factor is applied to both static and stochastically-sampled values. A value of `1.0` leaves the data unchanged.
+
+Typical use cases:
+
+* **Allocation factors** — when a process produces multiple products, the exchange amounts must be partitioned between them. Storing the allocation coefficients as a `scale_array` keeps them alongside the data they modify without requiring a separate processing step.
+* **Unit conversions** — when source data is expressed in a unit that differs from the matrix convention, a constant conversion factor can be stored as a `scale_array` rather than baked into every data value.
+
+```python
+import numpy as np
+from bw_processing import create_datapackage
+from bw_processing.constants import INDICES_DTYPE
+
+dp = create_datapackage()
+indices_array = np.array([(1, 4), (2, 5), (3, 6)], dtype=INDICES_DTYPE)
+data_array = np.array([100.0, 200.0, 300.0])
+scale_array = np.array([0.6, 1.0, 0.4])  # e.g. allocation factors
+
+dp.add_persistent_vector(
+    matrix="technosphere",
+    name="my-process",
+    indices_array=indices_array,
+    data_array=data_array,
+    scale_array=scale_array,
+)
+```
+
+The stored resource has `kind="scale"` and can be retrieved via `dp.get_resource("my-process.scale")`. The `scale_array` must be a float dtype (`float32` or `float64`); passing an integer array raises `WrongDatatype`.
+
 ### Policies
 
 Data package policies define how the data should be used. Policies apply to the entire data package; you may wish to adjust what is stored in which data packages to get the effect you desire.
@@ -219,7 +262,7 @@ Please make sure you understand how `combinatorial` and `sequential` interact! T
 
 ## Install
 
-Install using pip or conda (channel `cmutel`). Depends on `numpy` and `pandas` (for reading and writing CSVs).
+Install using pip or conda (channel `conda-forge`). Depends on `numpy` and `pandas` (for reading and writing CSVs).
 
 Has no explicit or implicit dependence on any other part of Brightway.
 

{bw_processing-1.2 → bw_processing-1.3}/tests/test_datapackage.py

@@ -8,7 +8,7 @@ import pytest
 from fsspec.implementations.zip import ZipFileSystem
 from morefs.dict import DictFS
 
-from bw_processing import create_datapackage, load_datapackage, simple_graph
+from bw_processing import create_datapackage, load_datapackage, simple_graph, MatrixSerializeFormat
 from bw_processing.constants import INDICES_DTYPE, UNCERTAINTY_DTYPE, MAX_SIGNED_64BIT_INT, MAX_SIGNED_32BIT_INT
 from bw_processing.errors import NonUnique, PotentialInconsistency, ShapeMismatch, WrongDatatype
 from bw_processing.io_helpers import generic_directory_filesystem, generic_zipfile_filesystem
@@ -457,6 +457,218 @@ def test_add_dynamic_vector_flip_shapemistmatch():
         )
 
 
+def test_add_persistent_vector_scale_array():
+    dp = create_datapackage()
+    data_array = np.array([2.0, 7.0, 12.0])
+    scale_array = np.array([0.5, 1.0, 2.0])
+    indices_array = np.array([(1, 4), (2, 5), (3, 6)], dtype=INDICES_DTYPE)
+    dp.add_persistent_vector(
+        matrix="sa_matrix",
+        data_array=data_array,
+        name="sa-data-vector",
+        indices_array=indices_array,
+        scale_array=scale_array,
+    )
+    assert "sa-data-vector.scale" in [o["name"] for o in dp.resources]
+    data, meta = dp.get_resource("sa-data-vector.scale")
+    assert meta["kind"] == "scale"
+    assert np.allclose(data, scale_array)
+
+
+def test_add_persistent_vector_scale_dtype():
+    dp = create_datapackage()
+    data_array = np.array([2.0, 7.0, 12.0])
+    scale_array = np.array([1, 2, 3])  # integer dtype
+    indices_array = np.array([(1, 4), (2, 5), (3, 6)], dtype=INDICES_DTYPE)
+    with pytest.raises(WrongDatatype):
+        dp.add_persistent_vector(
+            matrix="sa_matrix",
+            data_array=data_array,
+            name="sa-data-vector",
+            indices_array=indices_array,
+            scale_array=scale_array,
+        )
+
+
+def test_add_persistent_vector_scale_shapemismatch():
+    dp = create_datapackage()
+    data_array = np.array([2.0, 7.0, 12.0])
+    scale_array = np.array([0.5, 1.0, 2.0, 3.0])
+    indices_array = np.array([(1, 4), (2, 5), (3, 6)], dtype=INDICES_DTYPE)
+    with pytest.raises(ShapeMismatch):
+        dp.add_persistent_vector(
+            matrix="sa_matrix",
+            data_array=data_array,
+            name="sa-data-vector",
+            indices_array=indices_array,
+            scale_array=scale_array,
+        )
+
+
+def test_add_persistent_array_scale_array():
+    dp = create_datapackage()
+    data_array = np.arange(12, dtype=float).reshape(3, 4)
+    scale_array = np.array([0.5, 1.0, 2.0])
+    indices_array = np.array([(1, 4), (2, 5), (3, 6)], dtype=INDICES_DTYPE)
+    dp.add_persistent_array(
+        matrix="sa_matrix",
+        data_array=data_array,
+        name="sa-data-vector",
+        indices_array=indices_array,
+        scale_array=scale_array,
+    )
+    assert "sa-data-vector.scale" in [o["name"] for o in dp.resources]
+    data, meta = dp.get_resource("sa-data-vector.scale")
+    assert meta["kind"] == "scale"
+    assert np.allclose(data, scale_array)
+
+
+def test_add_persistent_array_scale_dtype():
+    dp = create_datapackage()
+    data_array = np.arange(12, dtype=float).reshape(3, 4)
+    scale_array = np.array([1, 2, 3])
+    indices_array = np.array([(1, 4), (2, 5), (3, 6)], dtype=INDICES_DTYPE)
+    with pytest.raises(WrongDatatype):
+        dp.add_persistent_array(
+            matrix="sa_matrix",
+            data_array=data_array,
+            name="sa-data-vector",
+            indices_array=indices_array,
+            scale_array=scale_array,
+        )
+
+
+def test_add_persistent_array_scale_shapemismatch():
+    dp = create_datapackage()
+    data_array = np.arange(12, dtype=float).reshape(3, 4)
+    scale_array = np.array([0.5, 1.0, 2.0, 3.0])
+    indices_array = np.array([(1, 4), (2, 5), (3, 6)], dtype=INDICES_DTYPE)
+    with pytest.raises(ShapeMismatch):
+        dp.add_persistent_array(
+            matrix="sa_matrix",
+            data_array=data_array,
+            name="sa-data-vector",
+            indices_array=indices_array,
+            scale_array=scale_array,
+        )
+
+
+def test_add_dynamic_vector_scale_array():
+    dp = create_datapackage()
+    scale_array = np.array([0.5, 1.0, 2.0])
+    indices_array = np.array([(1, 4), (2, 5), (3, 6)], dtype=INDICES_DTYPE)
+    dp.add_dynamic_vector(
+        matrix="sa_matrix",
+        interface=Dummy(),
+        name="sa-data-vector",
+        indices_array=indices_array,
+        scale_array=scale_array,
+    )
+    assert "sa-data-vector.scale" in [o["name"] for o in dp.resources]
+    data, meta = dp.get_resource("sa-data-vector.scale")
+    assert meta["kind"] == "scale"
+    assert np.allclose(data, scale_array)
+
+
+def test_add_dynamic_vector_scale_dtype():
+    dp = create_datapackage()
+    scale_array = np.array([1, 2, 3])
+    indices_array = np.array([(1, 4), (2, 5), (3, 6)], dtype=INDICES_DTYPE)
+    with pytest.raises(WrongDatatype):
+        dp.add_dynamic_vector(
+            matrix="sa_matrix",
+            interface=Dummy(),
+            name="sa-data-vector",
+            indices_array=indices_array,
+            scale_array=scale_array,
+        )
+
+
+def test_add_dynamic_vector_scale_shapemismatch():
+    dp = create_datapackage()
+    scale_array = np.array([0.5, 1.0, 2.0, 3.0])
+    indices_array = np.array([(1, 4), (2, 5), (3, 6)], dtype=INDICES_DTYPE)
+    with pytest.raises(ShapeMismatch):
+        dp.add_dynamic_vector(
+            matrix="sa_matrix",
+            interface=Dummy(),
+            name="sa-data-vector",
+            indices_array=indices_array,
+            scale_array=scale_array,
+        )
+
+
+def test_add_dynamic_array_scale_array():
+    dp = create_datapackage()
+    scale_array = np.array([0.5, 1.0, 2.0])
+    indices_array = np.array([(1, 4), (2, 5), (3, 6)], dtype=INDICES_DTYPE)
+    dp.add_dynamic_array(
+        matrix="sa_matrix",
+        interface=Dummy(),
+        name="sa-data-vector",
+        indices_array=indices_array,
+        scale_array=scale_array,
+    )
+    assert "sa-data-vector.scale" in [o["name"] for o in dp.resources]
+    data, meta = dp.get_resource("sa-data-vector.scale")
+    assert meta["kind"] == "scale"
+    assert np.allclose(data, scale_array)
+
+
+def test_add_dynamic_array_scale_dtype():
+    dp = create_datapackage()
+    scale_array = np.array([1, 2, 3])
+    indices_array = np.array([(1, 4), (2, 5), (3, 6)], dtype=INDICES_DTYPE)
+    with pytest.raises(WrongDatatype):
+        dp.add_dynamic_array(
+            matrix="sa_matrix",
+            interface=Dummy(),
+            name="sa-data-vector",
+            indices_array=indices_array,
+            scale_array=scale_array,
+        )
+
+
+def test_add_dynamic_array_scale_shapemismatch():
+    dp = create_datapackage()
+    scale_array = np.array([0.5, 1.0, 2.0, 3.0])
+    indices_array = np.array([(1, 4), (2, 5), (3, 6)], dtype=INDICES_DTYPE)
+    with pytest.raises(ShapeMismatch):
+        dp.add_dynamic_array(
+            matrix="sa_matrix",
+            interface=Dummy(),
+            name="sa-data-vector",
+            indices_array=indices_array,
+            scale_array=scale_array,
+        )
+
+
+def test_scale_array_parquet_roundtrip(tmp_path):
+    scale_array = np.array([0.5, 1.0, 2.0])
+    indices_array = np.array([(1, 4), (2, 5), (3, 6)], dtype=INDICES_DTYPE)
+    data_array = np.array([100.0, 200.0, 300.0])
+
+    dp = create_datapackage(
+        fs=generic_directory_filesystem(dirpath=tmp_path),
+        name="scale-parquet-test",
+    )
+    dp.add_persistent_vector(
+        matrix="sa_matrix",
+        data_array=data_array,
+        name="sa-data-vector",
+        indices_array=indices_array,
+        scale_array=scale_array,
+        matrix_serialize_format_type=MatrixSerializeFormat.PARQUET,
+    )
+    dp.finalize_serialization()
+
+    dp2 = load_datapackage(generic_directory_filesystem(dirpath=tmp_path))
+    assert "sa-data-vector.scale" in [o["name"] for o in dp2.resources]
+    loaded, meta = dp2.get_resource("sa-data-vector.scale")
+    assert meta["kind"] == "scale"
+    assert np.allclose(loaded, scale_array)
+
+
 def test_simple_graph():
     data = {
         "some_matrix": [(1, 2, 3.14), (4, 5, 17, True), (8, 9, 11.11, False)],