dsgrid-toolkit 0.3.0__tar.gz

dsgrid_toolkit-0.3.0/LICENSE

@@ -0,0 +1,29 @@
+BSD 3-Clause License
+
+Copyright (c) 2021, Alliance for Sustainable Energy, LLC
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

dsgrid_toolkit-0.3.0/PKG-INFO

@@ -0,0 +1,193 @@
+Metadata-Version: 2.4
+Name: dsgrid-toolkit
+Version: 0.3.0
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: BSD License
+Classifier: Natural Language :: English
+Classifier: Programming Language :: Python :: 3.11
+Requires-Dist: chronify~=0.6.0
+Requires-Dist: click>=8.2,<9
+Requires-Dist: dash
+Requires-Dist: dash-bootstrap-components
+Requires-Dist: duckdb>=1,<2
+Requires-Dist: fastapi
+Requires-Dist: json5
+Requires-Dist: networkx
+Requires-Dist: pandas
+Requires-Dist: prettytable
+Requires-Dist: pyarrow
+Requires-Dist: pydantic~=2.11.6
+Requires-Dist: requests
+Requires-Dist: rich-click
+Requires-Dist: semver
+Requires-Dist: sqlalchemy>=2,<3
+Requires-Dist: uvicorn
+Requires-Dist: tzdata
+Requires-Dist: httpx ; extra == 'dev'
+Requires-Dist: pytest ; extra == 'dev'
+Requires-Dist: pytest-cov ; extra == 'dev'
+Requires-Dist: pre-commit ; extra == 'dev'
+Requires-Dist: devtools ; extra == 'dev'
+Requires-Dist: flake8 ; extra == 'dev'
+Requires-Dist: mypy ; extra == 'dev'
+Requires-Dist: pyarrow ; extra == 'dev'
+Requires-Dist: maturin ; extra == 'dev'
+Requires-Dist: furo ; extra == 'doc'
+Requires-Dist: ghp-import ; extra == 'doc'
+Requires-Dist: numpydoc ; extra == 'doc'
+Requires-Dist: pandas-stubs ; extra == 'doc'
+Requires-Dist: ruff ; extra == 'doc'
+Requires-Dist: sphinx~=7.2 ; extra == 'doc'
+Requires-Dist: sphinx-click~=5.0 ; extra == 'doc'
+Requires-Dist: sphinx-copybutton~=0.5.2 ; extra == 'doc'
+Requires-Dist: sphinx-tabs~=3.4 ; extra == 'doc'
+Requires-Dist: sphinx-argparse~=0.4.0 ; extra == 'doc'
+Requires-Dist: sphinxcontrib-programoutput ; extra == 'doc'
+Requires-Dist: autodoc-pydantic[erdantic]~=2.0 ; extra == 'doc'
+Requires-Dist: twine ; extra == 'release'
+Requires-Dist: setuptools ; extra == 'release'
+Requires-Dist: wheel ; extra == 'release'
+Requires-Dist: chronify[spark] ; extra == 'spark'
+Requires-Dist: pyspark==4.0.0 ; extra == 'spark'
+Requires-Dist: thrift ; extra == 'spark'
+Requires-Dist: thrift-sasl ; extra == 'spark'
+Provides-Extra: dev
+Provides-Extra: doc
+Provides-Extra: release
+Provides-Extra: spark
+License-File: LICENSE
+Summary: Python API for accessing demand-side grid model (dsgrid) datasets
+Keywords: dsgrid
+Author-email: Elaine Hale <elaine.hale@nrel.gov>, Lixi Liu <lixi.liu@nrel.gov>, Meghan Mooney <meghan.mooney@nrel.gov>, Daniel Thom <daniel.thom@nrel.gov>
+Maintainer-email: Elaine Hale <elaine.hale@nrel.gov>
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
+Project-URL: Documentation, https://dsgrid.github.io/dsgrid/
+Project-URL: GitHub, https://github.com/dsgrid/dsgrid
+Project-URL: Homepage, https://www.nrel.gov/analysis/dsgrid
+
+# dsgrid
+[![Documentation](https://img.shields.io/badge/docs-ready-blue.svg)](https://dsgrid.github.io/dsgrid)
+[![codecov](https://codecov.io/gh/dsgrid/dsgrid/branch/main/graph/badge.svg?token=W0441C9XAL)](https://codecov.io/gh/dsgrid/dsgrid)
+
+Python API for contributing to and accessing demand-side grid model (dsgrid) projects and datasets.
+
+⚠️ **dsgrid is under active development and does not yet have a formal package release.** Details listed here are subject to change. Please reach out to the dsgrid coordination team with any questions or other feedback. ⚠️
+
+[Install](#install) | [Usage](#usage) | [Uninstall](#uninstall)
+
+## Install
+
+[Virtual environment](#virtual-environment) | [Dependencies](#dependencies) | [from PIPY/pip](#from-pipypip) | [from pip+git](#from-pipgit) | [from cloned repository](#from-cloned-repository)
+
+### Virtual environment
+
+Create a virtual environment in which to install dsgrid. Anaconda or miniconda is recommended.
+
+```
+conda create -n dsgrid python=3.11
+conda activate dsgrid
+```
+
+### Dependencies
+
+dsgrid uses [Apache Spark](#https://spark.apache.org/) to manage big data. There are no separate installation steps for Apache Spark beyond installing the dsgrid package and installing:
+
+```
+pip install "dsgrid-toolkit[spark]"
+```
+
+Otherwise installing the pyspark Python dependency handles it.
+
+However, you should be aware that Apache Spark's Microsoft Windows support is poor and essentially limited to local mode. That is, if you use dsgrid on a Windows machine you should not attempt to install a full version of Spark nor expect to run on a Spark cluster. As such, we recommend limiting dsgrid use on Windows to browsing the registry, registering and submitting small- to medium-sized datasets, or development work with small test projects. Full dsgrid functionality with large projects requires additional computational resources, e.g., high performance or cloud computing, typically on a Linux operating system.
+
+#### Additional Notes
+- If pyspark complains about not finding Python, you may need to locate your python executable file (python.exe on Windows), copy it, and rename the copy to python3 (python3.exe on Windows)
+
+Spark requires Java 8 or later with the `JAVA_HOME` environment variable set to the Java installation directory.
+
+On Linux you can install OpenJDK with conda:
+```
+conda install openjdk
+```
+
+Windows install instructions are below.
+
+#### Windows
+
+To install Apache Spark on Windows, follow [these instructions](https://towardsdatascience.com/installing-apache-pyspark-on-windows-10-f5f0c506bea1).
+
+### From PIPY/pip
+
+pip install dsgrid-toolkit
+
+or
+
+pip install "dsgrid-toolkit[spark]"
+
+### From pip+git
+
+**With ssh keys:**
+```
+pip install git+ssh://git@github.com/dsgrid/dsgrid.git@main
+
+# or
+
+pip install git+ssh://git@github.com/dsgrid/dsgrid.git@develop
+```
+
+**From http:**
+```
+pip install git+https://github.com/dsgrid/dsgrid.git@main
+
+# or
+
+pip install git+https://github.com/dsgrid/dsgrid.git@develop
+```
+
+### From Cloned Repository
+
+First, clone the repository and change into the `dsgrid` directory. For example:
+
+```
+cd ~                                       # or other directory where you put repositories
+git clone git@github.com:dsgrid/dsgrid.git # or the http address
+cd dsgrid
+```
+
+Then install the pacakge using the pip `-e` flag to directly use the files in the
+cloned repository.
+
+**Users:**
+```
+pip install -e .
+```
+
+**Developers:**
+```
+pip install -e '.[dev,spark]'
+```
+
+## Usage
+
+dsgrid is primarily a command-line interface (CLI) tool. To see the available commands:
+```
+dsgrid --help
+```
+
+## Uninstall
+
+```
+pip uninstall dsgrid
+```
+
+If you are using a conda environment
+```
+conda deactivate
+```
+
+## Software Record
+
+dsgrid is developed under NREL Software Record SWR-21-52, "demand-side grid model".
+

dsgrid_toolkit-0.3.0/README.md

@@ -0,0 +1,123 @@
+# dsgrid
+[![Documentation](https://img.shields.io/badge/docs-ready-blue.svg)](https://dsgrid.github.io/dsgrid)
+[![codecov](https://codecov.io/gh/dsgrid/dsgrid/branch/main/graph/badge.svg?token=W0441C9XAL)](https://codecov.io/gh/dsgrid/dsgrid)
+
+Python API for contributing to and accessing demand-side grid model (dsgrid) projects and datasets.
+
+⚠️ **dsgrid is under active development and does not yet have a formal package release.** Details listed here are subject to change. Please reach out to the dsgrid coordination team with any questions or other feedback. ⚠️
+
+[Install](#install) | [Usage](#usage) | [Uninstall](#uninstall)
+
+## Install
+
+[Virtual environment](#virtual-environment) | [Dependencies](#dependencies) | [from PIPY/pip](#from-pipypip) | [from pip+git](#from-pipgit) | [from cloned repository](#from-cloned-repository)
+
+### Virtual environment
+
+Create a virtual environment in which to install dsgrid. Anaconda or miniconda is recommended.
+
+```
+conda create -n dsgrid python=3.11
+conda activate dsgrid
+```
+
+### Dependencies
+
+dsgrid uses [Apache Spark](#https://spark.apache.org/) to manage big data. There are no separate installation steps for Apache Spark beyond installing the dsgrid package and installing:
+
+```
+pip install "dsgrid-toolkit[spark]"
+```
+
+Otherwise installing the pyspark Python dependency handles it.
+
+However, you should be aware that Apache Spark's Microsoft Windows support is poor and essentially limited to local mode. That is, if you use dsgrid on a Windows machine you should not attempt to install a full version of Spark nor expect to run on a Spark cluster. As such, we recommend limiting dsgrid use on Windows to browsing the registry, registering and submitting small- to medium-sized datasets, or development work with small test projects. Full dsgrid functionality with large projects requires additional computational resources, e.g., high performance or cloud computing, typically on a Linux operating system.
+
+#### Additional Notes
+- If pyspark complains about not finding Python, you may need to locate your python executable file (python.exe on Windows), copy it, and rename the copy to python3 (python3.exe on Windows)
+
+Spark requires Java 8 or later with the `JAVA_HOME` environment variable set to the Java installation directory.
+
+On Linux you can install OpenJDK with conda:
+```
+conda install openjdk
+```
+
+Windows install instructions are below.
+
+#### Windows
+
+To install Apache Spark on Windows, follow [these instructions](https://towardsdatascience.com/installing-apache-pyspark-on-windows-10-f5f0c506bea1).
+
+### From PIPY/pip
+
+pip install dsgrid-toolkit
+
+or
+
+pip install "dsgrid-toolkit[spark]"
+
+### From pip+git
+
+**With ssh keys:**
+```
+pip install git+ssh://git@github.com/dsgrid/dsgrid.git@main
+
+# or
+
+pip install git+ssh://git@github.com/dsgrid/dsgrid.git@develop
+```
+
+**From http:**
+```
+pip install git+https://github.com/dsgrid/dsgrid.git@main
+
+# or
+
+pip install git+https://github.com/dsgrid/dsgrid.git@develop
+```
+
+### From Cloned Repository
+
+First, clone the repository and change into the `dsgrid` directory. For example:
+
+```
+cd ~                                       # or other directory where you put repositories
+git clone git@github.com:dsgrid/dsgrid.git # or the http address
+cd dsgrid
+```
+
+Then install the pacakge using the pip `-e` flag to directly use the files in the
+cloned repository.
+
+**Users:**
+```
+pip install -e .
+```
+
+**Developers:**
+```
+pip install -e '.[dev,spark]'
+```
+
+## Usage
+
+dsgrid is primarily a command-line interface (CLI) tool. To see the available commands:
+```
+dsgrid --help
+```
+
+## Uninstall
+
+```
+pip uninstall dsgrid
+```
+
+If you are using a conda environment
+```
+conda deactivate
+```
+
+## Software Record
+
+dsgrid is developed under NREL Software Record SWR-21-52, "demand-side grid model".

dsgrid_toolkit-0.3.0/dsgrid/__init__.py

@@ -0,0 +1,22 @@
+import datetime as dt
+import warnings
+
+from dsgrid.dsgrid_rc import DsgridRuntimeConfig
+from dsgrid.utils.timing import timer_stats_collector  # noqa: F401
+
+__title__ = "dsgrid"
+__description__ = (
+    "Python API for registring and accessing demand-side grid model (dsgrid) datasets"
+)
+__url__ = "https://github.com/dsgrid/dsgrid"
+__version__ = "0.3.0"
+__author__ = "NREL"
+__maintainer_email__ = "elaine.hale@nrel.gov"
+__license__ = "BSD-3"
+__copyright__ = "Copyright {}, The Alliance for Sustainable Energy, LLC".format(
+    dt.date.today().year
+)
+
+warnings.filterwarnings("ignore", module="duckdb_engine")
+
+runtime_config = DsgridRuntimeConfig.load()

dsgrid_toolkit-0.3.0/dsgrid/api/api_manager.py

@@ -0,0 +1,179 @@
+import logging
+import threading
+from datetime import datetime
+from pathlib import Path
+
+from dsgrid.exceptions import DSGValueNotStored
+from dsgrid.registry.registry_manager import RegistryManager
+from dsgrid.utils.files import load_data
+from .models import StoreModel, AsyncTaskModel, AsyncTaskStatus, AsyncTaskType
+
+
+MAX_CONCURRENT_ASYNC_TASKS = 4
+
+logger = logging.getLogger(__name__)
+
+
+class ApiManager:
+    """Manages API requests"""
+
+    def __init__(
+        self,
+        home_dir: str | Path,
+        registry_manager: RegistryManager,
+        max_concurrent_async_tasks=MAX_CONCURRENT_ASYNC_TASKS,
+    ):
+        self._home_dir = Path(home_dir)
+        self._store = Store.load(self._home_dir)
+        self._lock = threading.RLock()
+        self._max_concurrent_async_tasks = max_concurrent_async_tasks
+        self._cached_projects = {}
+        self._registry_mgr = registry_manager
+
+    def can_start_new_async_task(self):
+        self._lock.acquire()
+        try:
+            return len(self._store.data.outstanding_async_tasks) < self._max_concurrent_async_tasks
+        finally:
+            self._lock.release()
+
+    def initialize_async_task(self, task_type: AsyncTaskType) -> int:
+        self._lock.acquire()
+        try:
+            num_outstanding = len(self._store.data.outstanding_async_tasks)
+            # TODO: implement queueing so that we don't return an error
+            if num_outstanding > self._max_concurrent_async_tasks:
+                msg = f"Too many async tasks are already running: {num_outstanding}"
+                raise Exception(msg)
+            async_task_id = self._get_next_async_task_id()
+            task = AsyncTaskModel(
+                async_task_id=async_task_id,
+                task_type=task_type,
+                status=AsyncTaskStatus.IN_PROGRESS,
+                start_time=datetime.now(),
+            )
+            self._store.data.async_tasks[async_task_id] = task
+            self._store.data.outstanding_async_tasks.add(async_task_id)
+            self._store.persist()
+        finally:
+            self._lock.release()
+
+        logger.info("Initialized async_task_id=%s", async_task_id)
+        return async_task_id
+
+    def clear_completed_async_tasks(self):
+        self._lock.acquire()
+        try:
+            to_remove = [
+                x.async_task_id
+                for x in self._store.data.async_tasks
+                if x.status == AsyncTaskStatus.COMPLETE
+            ]
+            for async_task_id in to_remove:
+                self._store.data.async_tasks.pop(async_task_id)
+            self._store.persist()
+            logger.info("Cleared %d completed tasks", len(to_remove))
+        finally:
+            self._lock.release()
+
+    def get_async_task_status(self, async_task_id):
+        """Return the status of the async ID."""
+        self._lock.acquire()
+        try:
+            return self._store.data.async_tasks[async_task_id]
+        finally:
+            self._lock.release()
+
+    def complete_async_task(self, async_task_id, return_code: int, result=None):
+        """Complete an asynchronous operation."""
+        self._lock.acquire()
+        try:
+            task = self._store.data.async_tasks[async_task_id]
+            task.status = AsyncTaskStatus.COMPLETE
+            task.return_code = return_code
+            task.completion_time = datetime.now()
+            self._store.data.outstanding_async_tasks.remove(async_task_id)
+            if result is not None:
+                task.result = result
+            self._store.persist()
+        finally:
+            self._lock.release()
+
+        logger.info("Completed async_task_id=%s", async_task_id)
+
+    def list_async_tasks(self, async_task_ids=None, status=None) -> list[AsyncTaskModel]:
+        """Return async tasks.
+
+        Parameters
+        ----------
+        async_task_ids : list | None
+            IDs of tasks for which to return status. If not set, return all statuses.
+        status : AsyncTaskStatus | None
+            If set, filter tasks by this status.
+
+        """
+        self._lock.acquire()
+        try:
+            if async_task_ids is not None:
+                diff = set(async_task_ids).difference(self._store.data.async_tasks.keys())
+                if diff:
+                    msg = f"async_task_ids={diff} are not stored"
+                    raise DSGValueNotStored(msg)
+            tasks = (
+                self._store.data.async_tasks.keys() if async_task_ids is None else async_task_ids
+            )
+            return [
+                self._store.data.async_tasks[x]
+                for x in tasks
+                if status is None or self._store.data.async_tasks[x].status == status
+            ]
+        finally:
+            self._lock.release()
+
+    def _get_next_async_task_id(self) -> int:
+        self._lock.acquire()
+        try:
+            next_id = self._store.data.next_async_task_id
+            self._store.data.next_async_task_id += 1
+            self._store.persist()
+        finally:
+            self._lock.release()
+
+        return next_id
+
+    def get_project(self, project_id):
+        """Load a Project and cache it for future calls.
+        Loading is slow and the Project isn't being changed by this API.
+        """
+        self._lock.acquire()
+        try:
+            project = self._cached_projects.get(project_id)
+            if project is not None:
+                return project
+            project = self._registry_mgr.project_manager.load_project(project_id)
+            self._cached_projects[project_id] = project
+            return project
+        finally:
+            self._lock.release()
+
+
+class Store:
+    STORE_FILENAME = "api_server_store.json"
+
+    def __init__(self, store_file: Path, data: StoreModel):
+        self._store_file = store_file
+        self.data = data
+
+    @classmethod
+    def load(cls, path: Path):
+        # TODO: use MongoDB or some other db
+        store_file = path / cls.STORE_FILENAME
+        if store_file.exists():
+            logger.info("Load from existing store: %s", store_file)
+            store_data = load_data(store_file)
+            return cls(store_file, StoreModel(**store_data))
+        logger.info("Create new store: %s", store_file)
+        return cls(store_file, StoreModel())
+
+    def persist(self):
+        self._store_file.write_text(self.data.model_dump_json(indent=2))