bw-processing 1.3__tar.gz → 1.5__tar.gz

{bw_processing-1.3/src/bw_processing.egg-info → bw_processing-1.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: bw_processing
-Version: 1.3
+Version: 1.5
 Summary: Foo
 Author-email: Chris Mutel <cmutel@gmail.com>
 Maintainer-email: Chris Mutel <cmutel@gmail.com>
@@ -21,6 +21,7 @@ Requires-Python: >=3.9
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: fsspec
+Requires-Dist: jsonschema>=4.0
 Requires-Dist: morefs
 Requires-Dist: numpy
 Requires-Dist: pandas
@@ -67,6 +68,13 @@ Library for storing numeric data for use in matrix-based calculations. Designed
 
 - [Background](#background)
 - [Concepts](#concepts)
+  - [Data packages](#data-packages)
+  - [Vectors versus arrays](#vectors-versus-arrays)
+  - [Persistent versus dynamic](#persistent-versus-dynamic)
+  - [Scale arrays](#scale-arrays)
+  - [Parameter arrays for sensitivity analysis](#parameter-arrays-for-sensitivity-analysis)
+  - [NaN as a sentinel value](#nan-as-a-sentinel-value)
+  - [Policies](#policies)
 - [Install](#install)
 - [Usage](#usage)
 - [Contributing](#contributing)
@@ -85,6 +93,7 @@ The [Brightway LCA framework](https://brightway.dev/) has stored data used in co
 * **Use [fsspec](https://filesystem-spec.readthedocs.io/en/latest/) for file IO**. The use of this library allows for data packages to be stored on your local computer, or on [many logical or virtual file systems](https://docs.pyfilesystem.org/en/latest/guide.html).
 * **Simpler handling of numeric values whose sign should be flipped**. Sometimes it is more convenient to specify positive numbers in dataset definitions, even though such numbers should be negative when inserted into the resulting matrices. For example, in the technosphere matrix in life cycle assessment, products produced are positive and products consumed are negative, though both values are given as positive in datasets. Brightway used to use a type mapping dictionary to indicate which values in a matrix should have their sign flipped after insertion. Such mapping dictionaries are brittle and inelegant. `bw_processing` uses an optional boolean vector, called `flip`, to indicate if any values should be flipped.
 * **Per-exchange multiplicative scaling**. An optional float vector, called `scale`, can be attached to any resource group. Each element is a multiplicative factor applied to the corresponding data value — whether static or sampled stochastically — before it is inserted into the matrix. Typical uses are allocation factors and unit conversions. A value of `1.0` leaves the data unchanged.
+* **Recording independent variables for sensitivity analysis**. An optional `params_array` can be attached to any resource group to record the values of model parameters (independent variables) that were used to generate the data. For array resources each column of `params_array` corresponds to the same column in `data_array`, making it straightforward to correlate inputs with outputs for sensitivity analysis methods such as Morris or Sobol.
 * **Separation of uncertainty distribution parameters from other data**. Fitting data to a [probability density function](https://en.wikipedia.org/wiki/Probability_density_function) (PDF), or an estimate of such a PDF, is only one approach to quantitative uncertainty analysis. We would like to support other approaches, including [direct sampling from real data](https://github.com/PascalLesage/presamples/). Therefore, uncertainty distribution parameters are stored separately,  only loaded if needed, and are only one way to express quantitative uncertainty.
 
 ## Concepts
@@ -230,6 +239,151 @@ dp.add_persistent_vector(
 
 The stored resource has `kind="scale"` and can be retrieved via `dp.get_resource("my-process.scale")`. The `scale_array` must be a float dtype (`float32` or `float64`); passing an integer array raises `WrongDatatype`.
 
+### Parameter arrays for sensitivity analysis
+
+Any resource group can carry an optional `params_array` that records the values of the independent variables (model parameters) used to generate the data. This is the foundation for global sensitivity analysis workflows such as [Morris screening](https://en.wikipedia.org/wiki/Morris_method) or [Sobol indices](https://en.wikipedia.org/wiki/Variance-based_sensitivity_analysis).
+
+**Shape conventions:**
+
+| Resource type | `data_array` shape | `params_array` shape |
+|---|---|---|
+| persistent / dynamic vector | `(n_exchanges,)` | `(n_params,)` |
+| persistent array | `(n_exchanges, n_scenarios)` | `(n_params, n_scenarios)` |
+| dynamic array | `(n_exchanges, n_scenarios)` | `(n_params, n_scenarios)` — column count not validated against interface |
+
+Column `j` of `params_array` describes the parameter configuration that produced column `j` of `data_array`. The `params_array` must be a float dtype.
+
+#### Basic usage
+
+```python
+import numpy as np
+from bw_processing import create_datapackage, INDICES_DTYPE
+
+dp = create_datapackage()
+indices = np.array([(1, 4), (2, 5)], dtype=INDICES_DTYPE)
+
+# Store the parameter values alongside the data
+dp.add_persistent_vector(
+    matrix="technosphere",
+    name="my-process",
+    indices_array=indices,
+    data_array=np.array([100.0, 200.0]),
+    params_array=np.array([25.0, 1.013]),  # temperature (°C), pressure (atm)
+)
+```
+
+#### Adding labels
+
+`param_labels` is an optional list of label objects (strings or dicts) whose length must match `params_array.shape[0]`. When provided, a companion `name.param_labels.json` file is written inside the same resource group, containing a `"values"` list and an optional `"schema"` (a [JSON Schema](https://json-schema.org/) document). Pass `param_label_schema=StringLabelSchema()` for plain-string labels, or a `ParamLabelSchema` for structured dict labels.
+
+```python
+import numpy as np
+from bw_processing import create_datapackage, INDICES_DTYPE, StringLabelSchema
+
+dp = create_datapackage()
+dp.add_persistent_vector(
+    matrix="technosphere",
+    name="my-process",
+    indices_array=np.array([(1, 4)], dtype=INDICES_DTYPE),
+    data_array=np.array([100.0]),
+    params_array=np.array([25.0, 1.013]),
+    param_labels=["temperature", "pressure"],
+    param_label_schema=StringLabelSchema(),
+)
+```
+
+#### Structured labels with a schema
+
+Use `ParamLabelSchema` and `ParamLabelField` when labels are structured objects, and pass a `param_label_schema` to validate each label at write time:
+
+```python
+import numpy as np
+from bw_processing import (
+    create_datapackage,
+    INDICES_DTYPE,
+    ParamLabelField,
+    ParamLabelSchema,
+)
+
+dp = create_datapackage()
+
+schema = ParamLabelSchema(
+    fields=[
+        ParamLabelField(name="name",      type="string"),
+        ParamLabelField(name="database",  type="string"),
+        ParamLabelField(name="year",      type="integer", required=False),
+    ],
+    description="Brightway activity reference",
+)
+
+param_labels = [
+    {"name": "electricity", "database": "ecoinvent", "year": 2020},
+    {"name": "heat",        "database": "ecoinvent"},
+]
+
+dp.add_persistent_array(
+    matrix="technosphere",
+    name="sa-run",
+    indices_array=np.array([(1, 4), (2, 5)], dtype=INDICES_DTYPE),
+    data_array=np.array([[10.0, 20.0, 30.0],   # 2 exchanges × 3 scenarios
+                         [40.0, 50.0, 60.0]]),
+    params_array=np.array([[25.0, 30.0, 35.0],  # temperature: 2 params × 3 scenarios
+                            [1.0,  1.1,  1.2]]),
+    param_labels=param_labels,
+    param_label_schema=schema,  # validates every label against the JSON Schema on write
+)
+```
+
+`ParamLabelField.type` accepts the standard JSON Schema type names: `"string"`, `"integer"`, `"number"`, `"boolean"`. For plain-string labels you can also pass `param_label_schema=StringLabelSchema()` to make the schema explicit.
+
+#### Retrieving params and labels
+
+```python
+params_data, _  = dp.get_resource("sa-run.params")       # numpy array
+labels_data, _  = dp.get_resource("sa-run.param_labels")  # {"schema": ..., "values": [...]}
+
+# Reconstruct the schema dataclass from the stored JSON Schema:
+from bw_processing import schema_from_json_schema
+schema = schema_from_json_schema(labels_data["schema"])
+# → ParamLabelSchema(fields=[...])
+```
+
+#### Dependency
+
+`params_array` validation uses the [`jsonschema`](https://python-jsonschema.readthedocs.io/) library, which is a required dependency of `bw_processing`.
+
+### NaN as a sentinel value
+
+A `NaN` value in a data vector or array is treated by `matrix_utils` as **"no data insertion"** — that element is skipped when the matrix is built or rebuilt, leaving the corresponding matrix cell at whatever value was written by an earlier package. This convention makes it straightforward to define scenario or override packages: set an element to `NaN` to inherit the base value, or to a real number to override it.
+
+```python
+import numpy as np
+from bw_processing import create_datapackage, INDICES_DTYPE
+
+# Base package — sets matrix[0,0]=5 and matrix[1,1]=7
+dp_base = create_datapackage()
+dp_base.add_persistent_vector(
+    matrix="foo",
+    name="base",
+    indices_array=np.array([(0, 0), (1, 1)], dtype=INDICES_DTYPE),
+    data_array=np.array([5.0, 7.0]),
+)
+
+# Scenario package — overrides matrix[1,1] but leaves matrix[0,0] untouched
+dp_scenario = create_datapackage()
+dp_scenario.add_persistent_vector(
+    matrix="foo",
+    name="scenario",
+    indices_array=np.array([(0, 0), (1, 1)], dtype=INDICES_DTYPE),
+    data_array=np.array([np.nan, 99.0]),
+)
+# When both packages are passed to MappedMatrix:
+# matrix[0,0] == 5.0  (NaN in scenario → base value preserved)
+# matrix[1,1] == 99.0 (non-NaN in scenario → override applied)
+```
+
+Note that `NaN` skipping is implemented in `matrix_utils`, not in `bw_processing` itself. `bw_processing` stores and retrieves the `NaN` values faithfully; the skip logic runs at matrix construction time.
+
 ### Policies
 
 Data package policies define how the data should be used. Policies apply to the entire data package; you may wish to adjust what is stored in which data packages to get the effect you desire.

{bw_processing-1.3 → bw_processing-1.5}/README.md

@@ -25,6 +25,13 @@ Library for storing numeric data for use in matrix-based calculations. Designed
 
 - [Background](#background)
 - [Concepts](#concepts)
+  - [Data packages](#data-packages)
+  - [Vectors versus arrays](#vectors-versus-arrays)
+  - [Persistent versus dynamic](#persistent-versus-dynamic)
+  - [Scale arrays](#scale-arrays)
+  - [Parameter arrays for sensitivity analysis](#parameter-arrays-for-sensitivity-analysis)
+  - [NaN as a sentinel value](#nan-as-a-sentinel-value)
+  - [Policies](#policies)
 - [Install](#install)
 - [Usage](#usage)
 - [Contributing](#contributing)
@@ -43,6 +50,7 @@ The [Brightway LCA framework](https://brightway.dev/) has stored data used in co
 * **Use [fsspec](https://filesystem-spec.readthedocs.io/en/latest/) for file IO**. The use of this library allows for data packages to be stored on your local computer, or on [many logical or virtual file systems](https://docs.pyfilesystem.org/en/latest/guide.html).
 * **Simpler handling of numeric values whose sign should be flipped**. Sometimes it is more convenient to specify positive numbers in dataset definitions, even though such numbers should be negative when inserted into the resulting matrices. For example, in the technosphere matrix in life cycle assessment, products produced are positive and products consumed are negative, though both values are given as positive in datasets. Brightway used to use a type mapping dictionary to indicate which values in a matrix should have their sign flipped after insertion. Such mapping dictionaries are brittle and inelegant. `bw_processing` uses an optional boolean vector, called `flip`, to indicate if any values should be flipped.
 * **Per-exchange multiplicative scaling**. An optional float vector, called `scale`, can be attached to any resource group. Each element is a multiplicative factor applied to the corresponding data value — whether static or sampled stochastically — before it is inserted into the matrix. Typical uses are allocation factors and unit conversions. A value of `1.0` leaves the data unchanged.
+* **Recording independent variables for sensitivity analysis**. An optional `params_array` can be attached to any resource group to record the values of model parameters (independent variables) that were used to generate the data. For array resources each column of `params_array` corresponds to the same column in `data_array`, making it straightforward to correlate inputs with outputs for sensitivity analysis methods such as Morris or Sobol.
 * **Separation of uncertainty distribution parameters from other data**. Fitting data to a [probability density function](https://en.wikipedia.org/wiki/Probability_density_function) (PDF), or an estimate of such a PDF, is only one approach to quantitative uncertainty analysis. We would like to support other approaches, including [direct sampling from real data](https://github.com/PascalLesage/presamples/). Therefore, uncertainty distribution parameters are stored separately,  only loaded if needed, and are only one way to express quantitative uncertainty.
 
 ## Concepts
@@ -188,6 +196,151 @@ dp.add_persistent_vector(
 
 The stored resource has `kind="scale"` and can be retrieved via `dp.get_resource("my-process.scale")`. The `scale_array` must be a float dtype (`float32` or `float64`); passing an integer array raises `WrongDatatype`.
 
+### Parameter arrays for sensitivity analysis
+
+Any resource group can carry an optional `params_array` that records the values of the independent variables (model parameters) used to generate the data. This is the foundation for global sensitivity analysis workflows such as [Morris screening](https://en.wikipedia.org/wiki/Morris_method) or [Sobol indices](https://en.wikipedia.org/wiki/Variance-based_sensitivity_analysis).
+
+**Shape conventions:**
+
+| Resource type | `data_array` shape | `params_array` shape |
+|---|---|---|
+| persistent / dynamic vector | `(n_exchanges,)` | `(n_params,)` |
+| persistent array | `(n_exchanges, n_scenarios)` | `(n_params, n_scenarios)` |
+| dynamic array | `(n_exchanges, n_scenarios)` | `(n_params, n_scenarios)` — column count not validated against interface |
+
+Column `j` of `params_array` describes the parameter configuration that produced column `j` of `data_array`. The `params_array` must be a float dtype.
+
+#### Basic usage
+
+```python
+import numpy as np
+from bw_processing import create_datapackage, INDICES_DTYPE
+
+dp = create_datapackage()
+indices = np.array([(1, 4), (2, 5)], dtype=INDICES_DTYPE)
+
+# Store the parameter values alongside the data
+dp.add_persistent_vector(
+    matrix="technosphere",
+    name="my-process",
+    indices_array=indices,
+    data_array=np.array([100.0, 200.0]),
+    params_array=np.array([25.0, 1.013]),  # temperature (°C), pressure (atm)
+)
+```
+
+#### Adding labels
+
+`param_labels` is an optional list of label objects (strings or dicts) whose length must match `params_array.shape[0]`. When provided, a companion `name.param_labels.json` file is written inside the same resource group, containing a `"values"` list and an optional `"schema"` (a [JSON Schema](https://json-schema.org/) document). Pass `param_label_schema=StringLabelSchema()` for plain-string labels, or a `ParamLabelSchema` for structured dict labels.
+
+```python
+import numpy as np
+from bw_processing import create_datapackage, INDICES_DTYPE, StringLabelSchema
+
+dp = create_datapackage()
+dp.add_persistent_vector(
+    matrix="technosphere",
+    name="my-process",
+    indices_array=np.array([(1, 4)], dtype=INDICES_DTYPE),
+    data_array=np.array([100.0]),
+    params_array=np.array([25.0, 1.013]),
+    param_labels=["temperature", "pressure"],
+    param_label_schema=StringLabelSchema(),
+)
+```
+
+#### Structured labels with a schema
+
+Use `ParamLabelSchema` and `ParamLabelField` when labels are structured objects, and pass a `param_label_schema` to validate each label at write time:
+
+```python
+import numpy as np
+from bw_processing import (
+    create_datapackage,
+    INDICES_DTYPE,
+    ParamLabelField,
+    ParamLabelSchema,
+)
+
+dp = create_datapackage()
+
+schema = ParamLabelSchema(
+    fields=[
+        ParamLabelField(name="name",      type="string"),
+        ParamLabelField(name="database",  type="string"),
+        ParamLabelField(name="year",      type="integer", required=False),
+    ],
+    description="Brightway activity reference",
+)
+
+param_labels = [
+    {"name": "electricity", "database": "ecoinvent", "year": 2020},
+    {"name": "heat",        "database": "ecoinvent"},
+]
+
+dp.add_persistent_array(
+    matrix="technosphere",
+    name="sa-run",
+    indices_array=np.array([(1, 4), (2, 5)], dtype=INDICES_DTYPE),
+    data_array=np.array([[10.0, 20.0, 30.0],   # 2 exchanges × 3 scenarios
+                         [40.0, 50.0, 60.0]]),
+    params_array=np.array([[25.0, 30.0, 35.0],  # temperature: 2 params × 3 scenarios
+                            [1.0,  1.1,  1.2]]),
+    param_labels=param_labels,
+    param_label_schema=schema,  # validates every label against the JSON Schema on write
+)
+```
+
+`ParamLabelField.type` accepts the standard JSON Schema type names: `"string"`, `"integer"`, `"number"`, `"boolean"`. For plain-string labels you can also pass `param_label_schema=StringLabelSchema()` to make the schema explicit.
+
+#### Retrieving params and labels
+
+```python
+params_data, _  = dp.get_resource("sa-run.params")       # numpy array
+labels_data, _  = dp.get_resource("sa-run.param_labels")  # {"schema": ..., "values": [...]}
+
+# Reconstruct the schema dataclass from the stored JSON Schema:
+from bw_processing import schema_from_json_schema
+schema = schema_from_json_schema(labels_data["schema"])
+# → ParamLabelSchema(fields=[...])
+```
+
+#### Dependency
+
+`params_array` validation uses the [`jsonschema`](https://python-jsonschema.readthedocs.io/) library, which is a required dependency of `bw_processing`.
+
+### NaN as a sentinel value
+
+A `NaN` value in a data vector or array is treated by `matrix_utils` as **"no data insertion"** — that element is skipped when the matrix is built or rebuilt, leaving the corresponding matrix cell at whatever value was written by an earlier package. This convention makes it straightforward to define scenario or override packages: set an element to `NaN` to inherit the base value, or to a real number to override it.
+
+```python
+import numpy as np
+from bw_processing import create_datapackage, INDICES_DTYPE
+
+# Base package — sets matrix[0,0]=5 and matrix[1,1]=7
+dp_base = create_datapackage()
+dp_base.add_persistent_vector(
+    matrix="foo",
+    name="base",
+    indices_array=np.array([(0, 0), (1, 1)], dtype=INDICES_DTYPE),
+    data_array=np.array([5.0, 7.0]),
+)
+
+# Scenario package — overrides matrix[1,1] but leaves matrix[0,0] untouched
+dp_scenario = create_datapackage()
+dp_scenario.add_persistent_vector(
+    matrix="foo",
+    name="scenario",
+    indices_array=np.array([(0, 0), (1, 1)], dtype=INDICES_DTYPE),
+    data_array=np.array([np.nan, 99.0]),
+)
+# When both packages are passed to MappedMatrix:
+# matrix[0,0] == 5.0  (NaN in scenario → base value preserved)
+# matrix[1,1] == 99.0 (non-NaN in scenario → override applied)
+```
+
+Note that `NaN` skipping is implemented in `matrix_utils`, not in `bw_processing` itself. `bw_processing` stores and retrieves the `NaN` values faithfully; the skip logic runs at matrix construction time.
+
 ### Policies
 
 Data package policies define how the data should be used. Policies apply to the entire data package; you may wish to adjust what is stored in which data packages to get the effect you desire.

{bw_processing-1.3 → bw_processing-1.5}/pyproject.toml

@@ -32,6 +32,7 @@ dependencies = [
     # dependencies as strings with quotes, e.g. "foo"
     # You can add version requirements
     "fsspec",
+    "jsonschema>=4.0",
     "morefs",
     "numpy",
     "pandas",

{bw_processing-1.3 → bw_processing-1.5}/src/bw_processing/__init__.py

@@ -1,5 +1,6 @@
 __all__ = (
     "__version__",
+    "AnyLabelSchema",
     "as_unique_attributes",
     "as_unique_attributes_dataframe",
     "clean_datapackage_name",
@@ -16,24 +17,36 @@ __all__ = (
     "generic_zipfile_filesystem",
     "INDICES_DTYPE",
     "load_datapackage",
+    "ArrayEntry",
     "MatrixEntry",
     "MatrixName",
     "MatrixSerializeFormat",
     "md5",
     "merge_datapackages_with_mask",
+    "ParamLabelField",
+    "ParamLabelSchema",
     "reindex",
     "reset_index",
     "safe_filename",
+    "schema_from_json_schema",
     "simple_graph",
+    "StringLabelSchema",
     "UNCERTAINTY_DTYPE",
     "UndefinedInterface",
 )
 
-__version__ = "1.3"
+__version__ = "1.5"
 
 
 from bw_processing.array_creation import create_array, create_structured_array
 from bw_processing.constants import DEFAULT_LICENSES, INDICES_DTYPE, UNCERTAINTY_DTYPE, MatrixSerializeFormat
+from bw_processing.param_labels import (
+    AnyLabelSchema,
+    ParamLabelField,
+    ParamLabelSchema,
+    StringLabelSchema,
+    schema_from_json_schema,
+)
 from bw_processing.datapackage import (
     Datapackage,
     DatapackageBase,
@@ -46,7 +59,7 @@ from bw_processing.examples import examples_dir
 from bw_processing.filesystem import clean_datapackage_name, md5, safe_filename
 from bw_processing.indexing import reindex, reset_index
 from bw_processing.io_helpers import generic_directory_filesystem, generic_zipfile_filesystem
-from bw_processing.matrix_entry import MatrixEntry, MatrixName, create_datapackage_from_entries
+from bw_processing.matrix_entry import ArrayEntry, MatrixEntry, MatrixName, create_datapackage_from_entries
 from bw_processing.merging import merge_datapackages_with_mask
 from bw_processing.proxies import UndefinedInterface
 from bw_processing.unique_fields import as_unique_attributes, as_unique_attributes_dataframe

{bw_processing-1.3 → bw_processing-1.5}/src/bw_processing/array_creation.py

@@ -21,46 +21,45 @@ def chunked(iterable, chunk_size):
     return iter(lambda: list(itertools.islice(iterable, chunk_size)), [])
 
 
-def create_chunked_structured_array(iterable, dtype, bucket_size=20000):
-    """Create a numpy structured array from an iterable of indeterminate length.
+def create_chunked(iterable, dtype, ncols=None, bucket_size=500):
+    """Create a numpy array from an iterable of indeterminate length.
+
+    Needed when we can't determine the length of the iterable ahead of time
+    (e.g. for a generator or a database cursor), so can't create the complete
+    array in memory in one step.
 
-    Needed when we can't determine the length of the iterable ahead of time (e.g. for a generator or a database cursor), so can't create the complete array in memory in on step
+    Creates a list of arrays with ``bucket_size`` rows until ``iterable`` is
+    exhausted, then concatenates them along axis 0.
 
-    Creates a list of arrays with ``bucket_size`` rows until ``iterable`` is exhausted, then concatenates them.
+    Pass ``ncols`` for a plain 2D array; omit it for a 1D structured array.
 
     Args:
         iterable: Iterable of data used to populate the array.
-        dtype: Numpy dtype of the created array
-        format_function: If provided, this function will be called on each row of ``iterable`` before insertion in the array.
+        dtype: Numpy dtype of the created array.
+        ncols: Number of columns; if None, a 1D structured array is created.
         bucket_size: Number of rows in each intermediate array.
 
-    Returns:.
-        Returns the created array. Will return a zero-length array if ``iterable`` has no data.
-
+    Returns:
+        The created array. Returns a zero-length array if ``iterable`` has no data.
     """
+    bucket_shape = (bucket_size,) if ncols is None else (bucket_size, ncols)
+    empty_shape = (0,) if ncols is None else (0, ncols)
     arrays = []
-    array = np.zeros(bucket_size, dtype=dtype)
-
+    array = np.zeros(bucket_shape, dtype=dtype)
     for chunk in chunked(iterable, bucket_size):
         for i, row in enumerate(chunk):
             array[i] = row
         if i < bucket_size - 1:
-            array = array[: i + 1]
-            arrays.append(array)
+            # .copy() releases the oversized bucket buffer immediately rather
+            # than keeping it alive as a view until the final concatenation.
+            arrays.append(array[: i + 1].copy())
         else:
             arrays.append(array)
-            array = np.zeros(bucket_size, dtype=dtype)
-
-    # Empty iterable - create zero-length array
+            array = np.zeros(bucket_shape, dtype=dtype)
+    # Empty iterable - create zero-length array.
     # Needed because we return iterators for SQL databases
-    # but don't know if e.g. sometime a database has
-    # no biosphere exchanges
-    if arrays:
-        array = np.hstack(arrays)
-    else:
-        array = np.zeros(0, dtype=dtype)
-
-    return array
+    # but don't know if e.g. sometimes a database has no biosphere exchanges.
+    return np.concatenate(arrays, axis=0) if arrays else np.zeros(empty_shape, dtype=dtype)
 
 
 def create_structured_array(iterable, dtype, nrows=None, sort=False, sort_fields=None):
@@ -80,7 +79,7 @@ def create_structured_array(iterable, dtype, nrows=None, sort=False, sort_fields
             array[i] = tuple(row)
 
     else:
-        array = create_chunked_structured_array(iterable, dtype)
+        array = create_chunked(iterable, dtype)
 
     if sort:
         sort_fields = sort_fields or ()
@@ -93,44 +92,6 @@ def create_structured_array(iterable, dtype, nrows=None, sort=False, sort_fields
     return array
 
 
-def create_chunked_array(iterable, ncols, dtype=np.float32, bucket_size=500):
-    """Create a numpy array from an iterable of indeterminate length.
-
-    Needed when we can't determine the length of the iterable ahead of time (e.g. for a generator or a database cursor), so can't create the complete array in memory in on step
-
-    Creates a list of arrays with ``bucket_size`` rows until ``iterable`` is exhausted, then concatenates them.
-
-    Args:
-        iterable: Iterable of data used to populate the array.
-        ncols: Number of columns in the created array.
-        dtype: Numpy dtype of the created array
-        bucket_size: Number of rows in each intermediate array.
-
-    Returns:.
-        Returns the created array. Will return a zero-length array if ``iterable`` has no data.
-
-    """
-    arrays = []
-    array = np.zeros((bucket_size, ncols), dtype=dtype)
-
-    for chunk in chunked(iterable, bucket_size):
-        for i, row in enumerate(chunk):
-            array[i, :] = row
-        if i < bucket_size - 1:
-            array = array[: i + 1, :]
-            arrays.append(array)
-        else:
-            arrays.append(array)
-            array = np.zeros((bucket_size, ncols), dtype=dtype)
-
-    if arrays:
-        array = np.hstack(arrays)
-    else:
-        array = np.zeros((0, ncols), dtype=dtype)
-
-    return array
-
-
 def create_array(iterable, nrows=None, dtype=np.float32):
     """Create a numpy array data ``iterable``. Returns a filepath of a created file (if ``filepath`` is provided, or the array.
 
@@ -152,7 +113,10 @@ def create_array(iterable, nrows=None, dtype=np.float32):
             array[i, :] = tuple(row)
 
     else:
-        ncols, data = get_ncols(iterable)
-        array = create_chunked_array(data, ncols, dtype)
+        try:
+            ncols, data = get_ncols(iterable)
+        except StopIteration:
+            return np.zeros((0, 0), dtype=dtype)
+        array = create_chunked(data, dtype, ncols=ncols)
 
     return array