technologydata 0.1.0__tar.gz → 0.2.1__tar.gz

{technologydata-0.1.0 → technologydata-0.2.1}/CITATIONS.cff

@@ -6,7 +6,7 @@ cff-version: 1.2.0
 message: "If you use this package, we suggest the following way of citing it."
 title: "technologydata: Data for Energy Systems Models"
 repository: https://github.com/open-energy-transition/technology-data
-version: 0.1.0 # Don't touch, will be updated by the release script
+version: 0.2.1 # Don't touch, will be updated by the release script
 license: MIT
 authors:
   - family-names: Hampp

{technologydata-0.1.0 → technologydata-0.2.1}/MANIFEST.in

@@ -1,6 +1,7 @@
 include src/technologydata/utils/*.txt
-include src/technologydata/package_data/dea_energy_storage/*.json
-include src/technologydata/package_data/raw/*
+include src/technologydata/parsers/dea_energy_storage/*.json
+include src/technologydata/parsers/manual_input_usa/*.json
+include src/technologydata/parsers/raw/*
 include test/test_data/currency_conversion/WB_CNY_2020/*
 include test/test_data/currency_conversion/WB_EUR_2020/*
 include test/test_data/currency_conversion/WB_USD_2020/*

{technologydata-0.1.0 → technologydata-0.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: technologydata
-Version: 0.1.0
+Version: 0.2.1
 Summary: Package for providing common data assumptions for energy system modelling on techno-economics and macro-economics.
 Author-email: Contributors to technologydata <johannes.hampp@openenergytransition.org>, Contributors to technologydata <fabrizio.finozzi.business@gmail.com>
 Project-URL: Homepage, https://github.com/open-energy-transition/technology-data

{technologydata-0.1.0 → technologydata-0.2.1}/REUSE.toml

@@ -8,22 +8,21 @@ path = ["uv.lock", "*.yaml", "docs/**", "*.md", "*.in"]
 SPDX-FileCopyrightText = "technologydata contributors"
 SPDX-License-Identifier = "MIT"
 
-
 [[annotations]]
-path = ["test/test_data/**",]
+path = ["test/test_data/**"]
 SPDX-FileCopyrightText = "technologydata contributors"
 SPDX-License-Identifier = "CC-BY-4.0"
 
 [[annotations]]
 path = [
-    "src/technologydata/package_data/dea_energy_storage/*.json", "src/technologydata/package_data/schemas/*.json"
+    "src/technologydata/parsers/dea_energy_storage/*.json", "src/technologydata/parsers/schemas/*.json", "src/technologydata/parsers/raw/manual_input_usa.csv", "src/technologydata/parsers/manual_input_usa/*.json",
 ]
 SPDX-FileCopyrightText = "technologydata contributors"
 SPDX-License-Identifier = "CC-BY-4.0"
 
 [[annotations]]
 path = [
-    "src/technologydata/package_data/raw/Technology_datasheet_for_energy_storage.xlsx"
+    "src/technologydata/parsers/raw/Technology_datasheet_for_energy_storage.xlsx"
 ]
 SPDX-FileCopyrightText = "The Danish Energy Agency"
 SPDX-License-Identifier = "CC-BY-4.0"

{technologydata-0.1.0 → technologydata-0.2.1}/docs/examples/dea_storage.md

@@ -2,11 +2,11 @@
 
 ## Overview
 
-The Danish Energy Agency (DEA) data parser `dea_energy_storage.py` demonstrates a full data-cleaning and transformation pipeline for converting raw tabular data into the `technologydata` schema files `technologies.json` and `sources.json`. The parser is implemented in `src/technologydata/package_data/dea_energy_storage/dea_energy_storage.py`.
+The Danish Energy Agency (DEA) data parser `dea_energy_storage.py` demonstrates a full data-cleaning and transformation pipeline for converting raw tabular data into the `technologydata` schema files `technologies.json` and `sources.json`. The parser is implemented in `src/technologydata/parsers/dea_energy_storage/dea_energy_storage.py`.
 
 ## Dataset Description
 
-The original dataset is available at this [link](https://ens.dk/media/6589/download). A full description of the dataset is available at this [link](https://ens.dk/media/6588/download). The raw source file is included in the repository at `src/technologydata/package_data/raw/Technology_datasheet_for_energy_storage.xlsx`.
+The original dataset is available at this [link](https://ens.dk/media/6589/download). A full description of the dataset is available at this [link](https://ens.dk/media/6588/download). The raw source file is included in the repository at `src/technologydata/parsers/raw/Technology_datasheet_for_energy_storage.xlsx`.
 
 The dataset is in Excel format, and it includes, under the data sheet `alldata_flat`, a flat table of technology parameters for a range of energy storage technologies. Columns include `Technology`, `ws`, `par` (parameter name), `val` (value), `unit`, `year`, `est` (case/estimate), `priceyear`, plus metadata columns such as `cat`, `ref`, `note`. Rows are individual parameter records (parameter value + unit + context) for technologies and estimation cases.
 
@@ -25,7 +25,7 @@ Function `parse_input_arguments()` defines and parses the command-line arguments
 
 ### Read the raw data
 
-The script reads the raw data available at `src/technologydata/package_data/raw/Technology_datasheet_for_energy_storage.xlsx`, under sheet `alldata_flat`, in a `pandas` dataframe. It uses `pandas.read_excel(..., engine=calamine, dtype=str)`. All entries are handled as strings initially.
+The script reads the raw data available at `src/technologydata/parsers/raw/Technology_datasheet_for_energy_storage.xlsx`, under sheet `alldata_flat`, in a `pandas` dataframe. It uses `pandas.read_excel(..., engine=calamine, dtype=str)`. All entries are handled as strings initially.
 
 ### Data cleaning, validation and dealing with missing/null values
 
@@ -64,7 +64,7 @@ Function `build_technology_collection()`:
 - for each group, builds a dictionary of `Parameter` objects (each with `magnitude`, `units`, `sources`, `provenance`).
 - creates a `Technology` object for each group, with `name` = `ws`, `detailed_technology` = `Technology`, `year`=`year`, `region` = `EU`, `case` = `est` and collects them into a `TechnologyCollection` object.
 - writes the `TechnologyCollection` object to a `technologies.json`.
-- if `--export_schema` is used, schema files produced during export are moved to the sub-folder `src/technologydata/package_data/schemas`.
+- if `--export_schema` is used, schema files produced during export are moved to the sub-folder `src/technologydata/parsers/schemas`.
 
 ## Running the parser
 
@@ -72,13 +72,13 @@ Function `build_technology_collection()`:
 
 From repository root:
 
-- Basic run: `python src/technologydata/package_data/dea_energy_storage/dea_energy_storage.py`
+- Basic run: `python src/technologydata/parsers/dea_energy_storage/dea_energy_storage.py`
 - Example with options: `--num_digits 3 --store_source --filter_params --export_schema`
 
 ### Outputs
 
 The parser generates the following outputs:
 
-- `src/technologydata/package_data/dea_energy_storage/technologies.json`.
-- `src/technologydata/package_data/dea_energy_storage/sources.json`.
-- Optional schema files moved to `src/technologydata/package_data/schemas` when `--export_schema` is used.
+- `src/technologydata/parsers/dea_energy_storage/technologies.json`.
+- `src/technologydata/parsers/dea_energy_storage/sources.json`.
+- Optional schema files moved to `src/technologydata/parsers/schemas` when `--export_schema` is used.

technologydata-0.2.1/docs/examples/manual_input_usa.md

@@ -0,0 +1,78 @@
+# Manual Input USA Parser Documentation
+
+## Overview
+
+The Manual Input USA data parser `manual_input_usa.py` demonstrates a data-cleaning and transformation pipeline for converting manually curated, USA-specific tabular data into the `technologydata` schema files `technologies.json` and `sources.json`. The parser is implemented in `src/technologydata/parsers/manual_input_usa/manual_input_usa.py`.
+
+## Dataset Description
+
+The original dataset is a manually curated CSV file containing USA-specific technology parameters available at this [link](https://github.com/PyPSA/technology-data/blob/v0.13.4/inputs/US/manual_input_usa.csv). The raw source file is included in the repository at `src/technologydata/parsers/raw/manual_input_usa.csv`.
+
+The dataset is in CSV format and includes a flat table of technology parameters for various energy technologies relevant to the USA context. Columns include `technology`, `parameter`, `year`, `value`, `unit`, `currency_year`, `source`, `further_description`, `financial_case`, and `scenario`. Rows are individual parameter records (parameter value + unit + context) for technologies with different scenarios and financial cases.
+
+## Parser description
+
+The parser is articulated in the following steps.
+
+### Command line argument parsing
+
+Function `CommonsParser.parse_input_arguments()` defines and parses the command-line arguments:
+
+- `--num_digits` (int, default 4) — number of decimals used when rounding numeric values. The default value is 4.
+- `--store_source` (boolean flag) — whether to store the source on the Wayback Machine. The default value is `false`.
+
+### Read the raw data
+
+The script reads the raw data available at `src/technologydata/parsers/raw/manual_input_usa.csv` in a `pandas` dataframe. It uses `pandas.read_csv(..., dtype=str, na_values="None")`. All entries are handled as strings initially except for the `value` column which is converted to float.
+
+### Data cleaning, validation and dealing with missing/null values
+
+The data cleaning and validation happens with the following steps.
+
+Function `extract_units_carriers_heating_value()` extracts standardized units, carriers, and heating values from input unit strings. This function maps complex unit representations to simplified unit, carrier, and heating value combinations using a predefined dictionary of special patterns. Examples include:
+
+- `USD_2022/MW_FT` → unit: `USD_2022/MW`, carrier: `1/FT`, heating_value: `1/LHV`
+- `MWh_H2/MWh_FT` → unit: `MWh/MWh`, carrier: `H2/FT`, heating_value: `LHV`
+- `MWh_el/MWh_FT` → unit: `MWh/MWh`, carrier: `el/FT`, heating_value: `LHV`
+- `t_CO2/MWh_FT` → unit: `t/MWh`, carrier: `CO2/FT`, heating_value: `LHV`
+- `USD_2022/kWh_H2` → unit: `USD_2022/kWh`, carrier: `1/H2`, heating_value: `LHV`
+- `USD_2023/t_CO2/h` → unit: `USD_2023/t/h`, carrier: `1/CO2`, heating_value: `None`
+- `MWh_el/t_CO2` → unit: `MWh/t`, carrier: `el/CO2`, heating_value: `LHV`
+- `MWh_th/t_CO2` → unit: `MWh/t`, carrier: `thermal/CO2`, heating_value: `LHV`
+
+The parser also fills missing values in the `scenario` column with `"not_available"`.
+
+The parser applies the following unit conversions:
+
+- Convert `per unit` to `%` and multiply the corresponding `value` by 100.0, rounding to `num_digits` decimals.
+
+Function `Commons.update_unit_with_currency_year(unit, currency_year)` appends `currency_year` information to currency units when present. This is because `technologydata` follows the currency pattern `\b(?P<cu_iso3>[A-Z]{3})_(?P<year>\d{4})\b`, as for example `USD_2022`.
+
+### Populate and export the source and technology collections
+
+Function `build_technology_collection()`:
+
+- if `store_source` is set, constructs a `Source` object for the manual input USA dataset, calls `ensure_in_wayback()` and writes `sources.json`; otherwise reads an existing `sources.json`.
+- groups the cleaned DataFrame by `scenario`, `year`, `technology`.
+- for each group, builds a dictionary of `Parameter` objects (each with `magnitude`, `sources`, and optionally `carrier`, `heating_value`, `units`, `note`).
+- captures the `financial_case` value from rows within each group to combine with `scenario`.
+- creates a `case` value by combining `scenario` and `financial_case` in the format `"{scenario} - {financial_case}"` when `financial_case` is present; otherwise uses `scenario` alone.
+- creates a `Technology` object for each group, with `name` = `technology`, `detailed_technology` = `technology`, `year` = `year`, `region` = `USA`, `case` = combined case value, and collects them into a `TechnologyCollection` object.
+- writes the `TechnologyCollection` object to a `technologies.json`.
+
+## Running the parser
+
+### Execution instructions
+
+From repository root:
+
+- Basic run: `python src/technologydata/parsers/manual_input_usa/manual_input_usa.py`
+- Example with options: `--num_digits 3 --store_source`
+
+### Outputs
+
+The parser generates the following outputs:
+
+- `src/technologydata/parsers/manual_input_usa/technologies.json`.
+- `src/technologydata/parsers/manual_input_usa/sources.json`.
+- Optional schema files moved to `src/technologydata/parsers/schemas` when `--export_schema` is used.

{technologydata-0.1.0 → technologydata-0.2.1}/docs/index.md

@@ -2,6 +2,7 @@
   <img src="assets/logo/technology_data_logo.png" alt="technologydata Header Logo" width="400"/>
 </p>
 
+[![PyPI version](https://img.shields.io/pypi/v/technologydata.svg)](https://pypi.python.org/pypi/technologydata)
 [![License](https://img.shields.io/pypi/l/pypsa.svg)](https://github.com/PyPSA/pypsa?tab=MIT-1-ov-file)
 [![Discord](https://img.shields.io/discord/911692131440148490?logo=discord)](https://discord.gg/T7YZbnVU)
 

{technologydata-0.1.0 → technologydata-0.2.1}/mkdocs.yaml

@@ -32,6 +32,7 @@ nav:
 
 - Examples:
   - Danish Energy Agency Parser: examples/dea_storage.md
+  - Manual Input USA Parser: examples/manual_input_usa.md
 
 - API Reference:
   - Data Package: api/datapackage.md

technologydata-0.2.1/src/technologydata/parsers/commons.py

@@ -0,0 +1,136 @@
+# SPDX-FileCopyrightText: technologydata contributors
+#
+# SPDX-License-Identifier: MIT
+
+"""Classes for Commons methods for the data parsers."""
+
+import argparse
+from typing import Annotated, Any
+
+import pydantic
+from pydantic import BaseModel, ConfigDict
+
+
+class ArgumentConfig(BaseModel):
+    """
+    Pydantic model for defining argument configurations.
+
+    Allows flexible configuration of command-line arguments with type checking
+    and validation.
+    """
+
+    name: Annotated[str, pydantic.Field(description="Name of the argument config")]
+    arg_type: Annotated[
+        type | None,
+        pydantic.Field(
+            description="The type to which the command-line argument should be converted."
+        ),
+    ] = None
+    default: Annotated[
+        Any | None, pydantic.Field(description="Default value of the argument config")
+    ] = None
+    help: Annotated[
+        str | None,
+        pydantic.Field(description="A brief description of what the argument does."),
+    ] = None
+    action: Annotated[
+        str | None,
+        pydantic.Field(
+            description="Specification of how the command-line arguments should be handled"
+        ),
+    ] = None
+    required: Annotated[
+        bool, pydantic.Field(description="Flag to check whether field is mondatory")
+    ] = False
+
+    # Allow extra fields for maximum flexibility
+    model_config = ConfigDict(extra="allow")
+
+
+class CommonsParser:
+    """Commons methods for the data parsers."""
+
+    @staticmethod
+    @pydantic.validate_call
+    def parse_input_arguments(
+        additional_arguments: list[ArgumentConfig] | None = None,
+        description: str = "Flexible command line argument parser",
+    ) -> argparse.Namespace:
+        """
+        Parse command line arguments with robust configuration.
+
+        Parameters
+        ----------
+        additional_arguments : Optional[List[ArgumentConfig]]
+            A list of ArgumentConfig objects defining extra arguments.
+        description : str
+            Description for the argument parser. Defaults to a generic message.
+
+        Returns
+        -------
+        argparse.Namespace
+            Parsed command line arguments
+
+        Examples
+        --------
+        >>> extra_args = [
+        ...     ArgumentConfig(
+        ...         name="--input_file",
+        ...         arg_type=str,
+        ...         required=True,
+        ...         help="Path to input CSV file"
+        ...     ),
+        ...     ArgumentConfig(
+        ...         name="--verbose",
+        ...         action="store_true",
+        ...         help="Enable verbose output"
+        ...     )
+        ... ]
+        >>> args = CommonsParser.parse_input_arguments(additional_arguments=extra_args)
+
+        """
+        # Create parser with provided or default description
+        parser = argparse.ArgumentParser(
+            description=description,
+            formatter_class=argparse.RawTextHelpFormatter,
+        )
+
+        # Default arguments
+        default_args = [
+            ArgumentConfig(
+                name="--num_digits",
+                arg_type=int,
+                default=4,
+                help="Number of significant digits to round the values.",
+            ),
+            ArgumentConfig(
+                name="--store_source",
+                action="store_true",
+                help="Store_source, store the source object on the wayback machine. Default: false",
+            ),
+        ]
+
+        # Combine default and additional arguments
+        all_arguments = default_args + (additional_arguments or [])
+
+        # Add arguments to parser (Option 1)
+        for arg_config in all_arguments:
+            # Convert Pydantic model to argparse-compatible dictionary
+            arg_dict = {
+                k: v
+                for k, v in arg_config.model_dump().items()
+                if v is not None and k != "name"
+            }
+
+            if arg_dict.get("arg_type") is not None:
+                arg_dict["type"] = arg_dict.pop("arg_type")
+
+            print("arg_dict", arg_dict)
+
+            # Add argument to parser
+            parser.add_argument(arg_config.name, **arg_dict)
+
+        # Parse arguments
+        args = parser.parse_args()
+
+        return args

{technologydata-0.1.0/src/technologydata/package_data → technologydata-0.2.1/src/technologydata/parsers}/dea_energy_storage/dea_energy_storage.py

@@ -7,19 +7,19 @@ Data parser for the DEA energy storage data set.
 
 How to run:
     From the repository root, execute:
-        python src/technologydata/package_data/dea_energy_storage/dea_energy_storage.py
+        python src/technologydata/parsers/dea_energy_storage/dea_energy_storage.py
 
 Configuration options (command-line arguments):
     --num_digits <int>         Number of significant digits to round the values. Default: 4
     --store_source             Store the source object on the Wayback Machine. Default: False
     --filter_params            Filter the parameters stored to technologies.json. Default: False
+    --export_schema            Export the Source/TechnologyCollection schemas. Default: False
 
 Example:
-    python src/technologydata/package_data/dea_energy_storage/dea_energy_storage.py --num_digits 3 --store_source --filter_params
+    python src/technologydata/parsers/dea_energy_storage/dea_energy_storage.py --num_digits 3 --store_source --filter_params
 
 """
 
-import argparse
 import logging
 import pathlib
 import re
@@ -36,6 +36,7 @@ from technologydata import (
     Technology,
     TechnologyCollection,
 )
+from technologydata.parsers.commons import ArgumentConfig, CommonsParser
 
 path_cwd = pathlib.Path.cwd()
 
@@ -478,60 +479,26 @@ def build_technology_collection(
     return TechnologyCollection(technologies=list_techs)
 
 
-@pydantic.validate_call
-def parse_input_arguments() -> argparse.Namespace:
-    """
-    Parse command line arguments.
+if __name__ == "__main__":
+    # Parse input arguments
 
-    Returns
-    -------
-    argparse.Namespace
-        Parsed command line arguments containing:
-        - Number of significant digits
-        - Store source flag
+    additional_input_args = [
+        ArgumentConfig(
+            name="--filter_params",
+            action="store_true",
+            help="filter_params. Filter the parameters stored to technologies.json. Default: false",
+        ),
+        ArgumentConfig(
+            name="--export_schema",
+            action="store_true",
+            help="export_schema. Export the Source/TechnologyCollection schemas. Default: false",
+        ),
+    ]
 
-    """
-    # Create the parser
-    parser = argparse.ArgumentParser(
+    input_args = CommonsParser.parse_input_arguments(
+        additional_arguments=additional_input_args,
         description="Parse the DEA technology storage dataset",
-        formatter_class=argparse.RawTextHelpFormatter,
-    )
-
-    # Define arguments
-    parser.add_argument(
-        "--num_digits",
-        type=int,
-        default=4,
-        help="Name of significant digits to round the values. ",
     )
-
-    parser.add_argument(
-        "--store_source",
-        action="store_true",
-        help="store_source, store the source object on the wayback machine. Default: false",
-    )
-
-    parser.add_argument(
-        "--filter_params",
-        action="store_true",
-        help="filter_params. Filter the parameters stored to technologies.json. Default: false",
-    )
-
-    parser.add_argument(
-        "--export_schema",
-        action="store_true",
-        help="export_schema. Export the Source/TechnologyCollection schemas. Default: false",
-    )
-
-    # Parse arguments
-    args = parser.parse_args()
-
-    return args
-
-
-if __name__ == "__main__":
-    # Parse input arguments
-    input_args = parse_input_arguments()
     logger.info("Command line arguments parsed.")
 
     # Read the raw data
@@ -539,7 +506,7 @@ if __name__ == "__main__":
         path_cwd,
         "src",
         "technologydata",
-        "package_data",
+        "parsers",
         "raw",
         "Technology_datasheet_for_energy_storage.xlsx",
     )
@@ -651,7 +618,7 @@ if __name__ == "__main__":
         path_cwd,
         "src",
         "technologydata",
-        "package_data",
+        "parsers",
         "dea_energy_storage",
     )
     output_technologies_path = pathlib.Path(
@@ -676,7 +643,7 @@ if __name__ == "__main__":
     if input_args.export_schema:
         # Move schema files if they exist
         schema_folder = pathlib.Path(
-            path_cwd, "src", "technologydata", "package_data", "schemas"
+            path_cwd, "src", "technologydata", "parsers", "schemas"
         )
         sources_schema = pathlib.Path(dea_storage_path, "sources.schema.json")
         technologies_schema = pathlib.Path(dea_storage_path, "technologies.schema.json")