openbb-us-eia 1.0.0__tar.gz

openbb_us_eia-1.0.0/PKG-INFO

@@ -0,0 +1,140 @@
+Metadata-Version: 2.1
+Name: openbb-us-eia
+Version: 1.0.0
+Summary: The U.S. Energy Information Administration is committed to its free and open data by making it available through an Application Programming Interface (API) and its open data tools. See https://www.eia.gov/opendata/ for more information.
+License: AGPL-3.0-only
+Author: OpenBB Team
+Author-email: hello@openbb.co
+Requires-Python: >=3.9,<4.0
+Classifier: License :: OSI Approved :: GNU Affero General Public License v3
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: async-lru (>=2.0.4,<3.0.0)
+Requires-Dist: openbb-core (>=1.3.6,<2.0.0)
+Requires-Dist: openpyxl (>=3.1.5,<4.0.0)
+Requires-Dist: xlrd (>=2.0.1,<3.0.0)
+Description-Content-Type: text/markdown
+
+# OpenBB EIA Provider Extension
+
+This module integrates the [EIA](https://eia.gov) data provider into the OpenBB Platform.
+
+## Installation
+
+### PyPI
+
+```sh
+pip install openbb-us-eia
+```
+
+### From Source
+
+After cloning the main repository, navigate into this folder and enter:
+
+```sh
+pip install .
+```
+
+To install in editable mode:
+
+```sh
+pip install -e .
+```
+
+## Authorization
+
+Functions calling the EIA's API require free registration and an API key, obtained [here](https://www.eia.gov/opendata/register.php).
+
+### OpenBB Hub
+
+Add the key as "eia_api_key" in the OpenBB Hub Credentials page, [here](https://my.openbb.co/app/platform/credentials)
+
+### `user_settings.json`
+
+Add it to the credentials section of `~/.openbb_platform/user_settings.json`
+
+```json
+{
+    "credentials": {
+        "eia_api_key": "REPLACE_WITH_YOUR_KEY"
+    }
+}
+```
+
+### Current Python Session
+
+The credential can be added for the current session only, after importing the OpenBB package.
+
+```python
+from openbb import obb
+
+obb.user.credentials.eia_api_key = "REPLACE_WITH_YOUR_KEY"
+```
+
+## Coverage
+
+### Endpoints
+
+- `obb.commodity.petroleum_status_report` (API key not required.)
+- `obb.commodity.short_term_energy_outlook` (API key required.)
+
+### Weekly Petroluem Status Report
+
+The WPSR is comprised of thirteen (excludes discontinued series) high-level categories with each containing a subset of tables. Data is from the static Excel files published [here](https://www.eia.gov/petroleum/supply/weekly/), and each file represents a single category.
+
+All data from a single category is returned by supplying "all" to the `table` parameter of the WPSR endpoint.
+
+Tables from the WPSR are returned in a flat format in the same order as presented in the Excel files. The response is suitable for pivot tables and SQL storage.
+
+Category choices are defined as:
+
+    balance_sheet
+    inputs_and_production
+    refiner_and_blender_net_production
+    crude_petroleum_stocks
+    gasoline_fuel_stocks
+    total_gasoline_by_sub_padd
+    distillate_fuel_oil_stocks
+    imports
+    imports_by_country
+    weekly_estimates
+    spot_prices_crude_gas_heating
+    spot_prices_diesel_jet_fuel_propane
+    retail_prices
+
+### Short Term Energy Outlook
+
+The Short Term Energy Outlook (STEO) is curated by table, and relies on the EIA V2 API. Tables are defined by their alphanumeric code, and return in the same format as the WPSR tables.
+
+    01: US Energy Markets Summary
+    02: Nominal Energy Prices
+    03a: World Petroleum and Other Liquid Fuels Production, Consumption, and Inventories
+    03b: Non-OPEC Petroleum and Other Liquid Fuels Production
+    03c: World Petroleum and Other Liquid Fuels Production
+    03d: World Crude Oil Production
+    03e: World Petroleum and Other Liquid Fuels Consumption
+    04a: US Petroleum and Other Liquid Fuels Supply, Consumption, and Inventories
+    04b: US Hydrocarbon Gas Liquids (HGL) and Petroleum Refinery Balances
+    04c: US Regional Motor Gasoline Prices and Inventories
+    04d: US Biofuel Supply, Consumption, and Inventories
+    05a: US Natural Gas Supply, Consumption, and Inventories
+    05b: US Regional Natural Gas Prices
+    06: US Coal Supply, Consumption, and Inventories
+    07a: US Electricity Industry Overview
+    07b: US Regional Electricity Retail Sales
+    07c: US Regional Electricity Prices
+    07d1: US Regional Electricity Generation, Electric Power Sector
+    07d2: US Regional Electricity Generation, Electric Power Sector, continued
+    07e: US Electricity Generating Capacity
+    08: US Renewable Energy Consumption
+    09a: US Macroeconomic Indicators and CO2 Emissions
+    09b: US Regional Macroeconomic Data
+    09c: US Regional Weather Data
+    10a: Drilling Productivity Metrics
+    10b: Crude Oil and Natural Gas Production from Shale and Tight Formations
+
+    A "symbol" parameter allows lookup by individual series ID(s) within the dataset.

openbb_us_eia-1.0.0/README.md

@@ -0,0 +1,119 @@
+# OpenBB EIA Provider Extension
+
+This module integrates the [EIA](https://eia.gov) data provider into the OpenBB Platform.
+
+## Installation
+
+### PyPI
+
+```sh
+pip install openbb-us-eia
+```
+
+### From Source
+
+After cloning the main repository, navigate into this folder and enter:
+
+```sh
+pip install .
+```
+
+To install in editable mode:
+
+```sh
+pip install -e .
+```
+
+## Authorization
+
+Functions calling the EIA's API require free registration and an API key, obtained [here](https://www.eia.gov/opendata/register.php).
+
+### OpenBB Hub
+
+Add the key as "eia_api_key" in the OpenBB Hub Credentials page, [here](https://my.openbb.co/app/platform/credentials)
+
+### `user_settings.json`
+
+Add it to the credentials section of `~/.openbb_platform/user_settings.json`
+
+```json
+{
+    "credentials": {
+        "eia_api_key": "REPLACE_WITH_YOUR_KEY"
+    }
+}
+```
+
+### Current Python Session
+
+The credential can be added for the current session only, after importing the OpenBB package.
+
+```python
+from openbb import obb
+
+obb.user.credentials.eia_api_key = "REPLACE_WITH_YOUR_KEY"
+```
+
+## Coverage
+
+### Endpoints
+
+- `obb.commodity.petroleum_status_report` (API key not required.)
+- `obb.commodity.short_term_energy_outlook` (API key required.)
+
+### Weekly Petroluem Status Report
+
+The WPSR is comprised of thirteen (excludes discontinued series) high-level categories with each containing a subset of tables. Data is from the static Excel files published [here](https://www.eia.gov/petroleum/supply/weekly/), and each file represents a single category.
+
+All data from a single category is returned by supplying "all" to the `table` parameter of the WPSR endpoint.
+
+Tables from the WPSR are returned in a flat format in the same order as presented in the Excel files. The response is suitable for pivot tables and SQL storage.
+
+Category choices are defined as:
+
+    balance_sheet
+    inputs_and_production
+    refiner_and_blender_net_production
+    crude_petroleum_stocks
+    gasoline_fuel_stocks
+    total_gasoline_by_sub_padd
+    distillate_fuel_oil_stocks
+    imports
+    imports_by_country
+    weekly_estimates
+    spot_prices_crude_gas_heating
+    spot_prices_diesel_jet_fuel_propane
+    retail_prices
+
+### Short Term Energy Outlook
+
+The Short Term Energy Outlook (STEO) is curated by table, and relies on the EIA V2 API. Tables are defined by their alphanumeric code, and return in the same format as the WPSR tables.
+
+    01: US Energy Markets Summary
+    02: Nominal Energy Prices
+    03a: World Petroleum and Other Liquid Fuels Production, Consumption, and Inventories
+    03b: Non-OPEC Petroleum and Other Liquid Fuels Production
+    03c: World Petroleum and Other Liquid Fuels Production
+    03d: World Crude Oil Production
+    03e: World Petroleum and Other Liquid Fuels Consumption
+    04a: US Petroleum and Other Liquid Fuels Supply, Consumption, and Inventories
+    04b: US Hydrocarbon Gas Liquids (HGL) and Petroleum Refinery Balances
+    04c: US Regional Motor Gasoline Prices and Inventories
+    04d: US Biofuel Supply, Consumption, and Inventories
+    05a: US Natural Gas Supply, Consumption, and Inventories
+    05b: US Regional Natural Gas Prices
+    06: US Coal Supply, Consumption, and Inventories
+    07a: US Electricity Industry Overview
+    07b: US Regional Electricity Retail Sales
+    07c: US Regional Electricity Prices
+    07d1: US Regional Electricity Generation, Electric Power Sector
+    07d2: US Regional Electricity Generation, Electric Power Sector, continued
+    07e: US Electricity Generating Capacity
+    08: US Renewable Energy Consumption
+    09a: US Macroeconomic Indicators and CO2 Emissions
+    09b: US Regional Macroeconomic Data
+    09c: US Regional Weather Data
+    10a: Drilling Productivity Metrics
+    10b: Crude Oil and Natural Gas Production from Shale and Tight Formations
+
+    A "symbol" parameter allows lookup by individual series ID(s) within the dataset.

openbb_us_eia-1.0.0/openbb_us_eia/__init__.py

@@ -0,0 +1,25 @@
+"""OpenBB EIA Provider Module."""
+
+from openbb_core.provider.abstract.provider import Provider
+from openbb_us_eia.models.petroleum_status_report import EiaPetroleumStatusReportFetcher
+from openbb_us_eia.models.short_term_energy_outlook import (
+    EiaShortTermEnergyOutlookFetcher,
+)
+
+eia_provider = Provider(
+    name="eia",
+    website="https://eia.gov/",
+    description="The U.S. Energy Information Administration is committed to its free and open data"
+    + " by making it available through an Application Programming Interface (API) and its open data tools."
+    + " See https://www.eia.gov/opendata/ for more information.",
+    credentials=[
+        "api_key"
+    ],  # This is not required for the Weekly Petroleum Status Report
+    fetcher_dict={
+        "PetroleumStatusReport": EiaPetroleumStatusReportFetcher,
+        "ShortTermEnergyOutlook": EiaShortTermEnergyOutlookFetcher,
+    },
+    repr_name="U.S. Energy Information Administration (EIA) Open Data and API",
+    instructions="""Credentials are required for functions calling the EIA's API.
+    Register for a free key here: https://www.eia.gov/opendata/register.php""",
+)

openbb_us_eia-1.0.0/openbb_us_eia/models/__init__.py

@@ -0,0 +1 @@
+"""OpenBB EIA Provider Models."""

openbb_us_eia-1.0.0/openbb_us_eia/models/petroleum_status_report.py

@@ -0,0 +1,241 @@
+"""EIA Weekly Petroleum Status Report model."""
+
+# pylint: disable=unused-argument
+
+from typing import Any, Optional
+
+from openbb_core.app.model.abstract.error import OpenBBError
+from openbb_core.provider.abstract.fetcher import Fetcher
+from openbb_core.provider.standard_models.petroleum_status_report import (
+    PetroleumStatusReportData,
+    PetroleumStatusReportQueryParams,
+)
+from openbb_core.provider.utils.errors import EmptyDataError
+from openbb_us_eia.utils.constants import (
+    WpsrCategoryChoices,
+    WpsrCategoryType,
+    WpsrFileMap,
+    WpsrTableChoices,
+    WpsrTableMap,
+)
+from pydantic import Field
+
+WpsrTableChoicesString = "\n        ".join(WpsrTableChoices)
+
+
+class EiaPetroleumStatusReportQueryParams(PetroleumStatusReportQueryParams):
+    """EIA Petroleum Status Report Query Parameters.
+
+    Source: https://www.eia.gov/petroleum/supply/weekly/
+    """
+
+    __json_schema_extra__ = {
+        "category": {
+            "multiiple_items_allowed": False,
+            "choices": WpsrCategoryChoices,
+        },
+        "table": {
+            "multiple_items_allowed": True,
+            "choices": WpsrTableChoices,
+        },
+    }
+
+    category: WpsrCategoryType = Field(
+        default="balance_sheet",
+        description="The group of data to be returned. The default is the balance sheet.",
+    )
+    table: Optional[str] = Field(
+        default=None,
+        description="The specific table element within the category to be returned,"
+        + " default is 'stocks', if the category is 'weekly_estimates', else 'all'."
+        + "\n    Note: Choices represent all available tables from the entire collection and are not all"
+        + " available for every category."
+        + "\n    Invalid choices will raise a ValidationError with a message"
+        + " indicating the valid choices for the selected category."
+        + "\n    Choices are:"
+        + f"\n        {WpsrTableChoicesString}\n   ",
+    )
+    use_cache: bool = Field(
+        default=True,
+        description="Subsequent requests for the same source data are cached for the session using ALRU cache.",
+    )
+
+
+class EiaPetroleumStatusReportData(PetroleumStatusReportData):
+    """EIA Petroleum Status Report Data Model."""
+
+
+class EiaPetroleumStatusReportFetcher(
+    Fetcher[EiaPetroleumStatusReportQueryParams, list[EiaPetroleumStatusReportData]]
+):
+    """EIA Petroleum Status Report Fetcher."""
+
+    require_credentials = False
+
+    @staticmethod
+    def transform_query(params: dict[str, Any]) -> EiaPetroleumStatusReportQueryParams:
+        """Transform the query parameters."""
+        # pylint: disable=import-outside-toplevel
+        from warnings import warn
+
+        category = params.get("category", "balance_sheet")
+        tables = WpsrTableMap.get(category, {})
+        _table = params.get("table", "")
+
+        if not _table:
+            _table = "stocks" if category == "weekly_estimates" else "all"
+
+        _tables = _table.split(",")
+
+        if len(_tables) == 1 and _tables[0] == "all" and category == "weekly_estimates":
+            raise OpenBBError(
+                ValueError(
+                    f"'all' is not a supported choice for {category}. Please choose from: {list(tables)}"
+                )
+            )
+
+        if "all" in _tables and len(_tables) > 1:
+            _tables.remove("all")
+            warn("'all' cannot be used with other table choices. Ignoring 'all'.")
+
+        for table in _tables:
+            if table != "all" and table not in tables:
+                raise OpenBBError(
+                    ValueError(
+                        f"Invalid table choice: {table}. Valid choices for {category}: {list(tables)}"
+                    )
+                )
+
+        params["table"] = ",".join(_tables)
+
+        return EiaPetroleumStatusReportQueryParams(**params)
+
+    @staticmethod
+    async def aextract_data(
+        query: EiaPetroleumStatusReportQueryParams,
+        credentials: Optional[dict[str, Any]],
+        **kwargs: Any,
+    ) -> dict:
+        """Extract the data from the EIA website."""
+        # pylint: disable=import-outside-toplevel
+        from openbb_us_eia.utils.helpers import download_excel_file
+
+        url = WpsrFileMap.get(query.category, "balance_sheet")
+
+        try:
+            results = await download_excel_file(url, query.use_cache)
+        except OpenBBError as e:
+            raise OpenBBError(f"Error extracting data -> {e}") from e
+
+        return {"file": results}
+
+    @staticmethod
+    def transform_data(
+        query: EiaPetroleumStatusReportQueryParams,
+        data: dict,
+        **kwargs: Any,
+    ) -> list[EiaPetroleumStatusReportData]:
+        """Transform the data."""
+        # pylint: disable=import-outside-toplevel
+        import concurrent.futures  # noqa
+        import re
+        from functools import lru_cache
+        from numpy import nan
+        from pandas import Categorical, ExcelFile, concat, read_excel
+        from warnings import warn
+
+        category = query.category
+
+        _tables = (
+            query.table.split(",")  # type: ignore
+            if query.table
+            else ["stocks"] if category == "weekly_estimates" else ["all"]
+        )
+        all_tables = list(WpsrTableMap[category])
+        tables = all_tables if "all" in _tables else _tables
+
+        file = data.get("file")
+
+        if not isinstance(file, ExcelFile):
+            raise OpenBBError(
+                TypeError(f"Expected an ExcelFile object, got {type(file)} instead.")
+            )
+
+        dfs: list = []
+
+        def replace_data_strings(text):
+            """Replace the table strings with sortable numbers."""
+            pattern = r"Data (\d):"
+
+            def replacer(match):
+                """Replace the matched string with a sortable number."""
+                return f"Data 0{match.group(1)}:"
+
+            return re.sub(pattern, replacer, text)
+
+        @lru_cache(maxsize=128)
+        def read_excel_file(file, category, table):
+            """Read the ExcelFile for the sheet name and flatten the table."""
+            sheet_name = WpsrTableMap[category][table]
+            table_name = read_excel(file, sheet_name, header=None, nrows=1).iloc[0, 1]
+            table_name = replace_data_strings(table_name)
+            df = read_excel(file, sheet_name, header=[1, 2], nrows=3)
+            symbols = df.columns.get_level_values(0).tolist()
+            titles = [
+                d.replace(".1", "") for d in df.columns.get_level_values(1).tolist()
+            ]
+            title_map = dict(zip(symbols, titles))
+            df = read_excel(file, sheet_name, header=None, skiprows=3)
+            df.columns = [d.replace("Sourcekey", "date") for d in symbols]
+            df = df.melt(
+                id_vars="date",
+                value_vars=[d for d in df.columns if d != "date"],
+                var_name="symbol",
+            ).dropna()
+            df = df.reset_index(drop=True)
+            df.loc[:, "title"] = df.symbol.map(title_map)
+            df.loc[:, "unit"] = df.title.map(lambda x: x.split(" (")[-1].split(")")[0])
+            units = [f"({d})" for d in df.unit.unique().tolist()]
+            for unit in units:
+                df.title = df.title.str.replace(unit, "", regex=False).str.strip()
+            df.loc[:, "table"] = table_name
+            df["order"] = df.groupby("date").cumcount() + 1
+            df = df[["date", "table", "symbol", "order", "title", "value", "unit"]]
+            df.symbol = Categorical(df.symbol, categories=symbols, ordered=True)
+            df = df.sort_values(["date", "symbol"])
+            df.date = df.date.dt.date
+
+            if query.start_date:
+                df = df[df.date >= query.start_date]
+
+            if query.end_date:
+                df = df[df.date <= query.end_date]
+
+            df = df.reset_index(drop=True)
+
+            if len(df) > 0:
+                dfs.append(df)
+            else:
+                warn(f"No data for table: {table}")
+
+        try:
+            with concurrent.futures.ThreadPoolExecutor() as executor:
+                executor.map(
+                    lambda table: read_excel_file(file, category, table), tables
+                )
+
+            results = concat(dfs)
+
+            if len(results) < 1:
+                raise EmptyDataError("The data is empty.")
+
+            results = results.sort_values(by=["date", "table", "order"]).replace(
+                {nan: None}
+            )
+
+            return [
+                EiaPetroleumStatusReportData.model_validate(d)
+                for d in results.to_dict(orient="records")
+            ]
+        except Exception as e:  # pylint: disable=broad-except
+            raise OpenBBError(f"Error transforming the data -> {e}") from e