nci-cidc-api-modules 1.0.0rc0__tar.gz → 1.0.1__tar.gz

{nci_cidc_api_modules-1.0.0rc0 → nci_cidc_api_modules-1.0.1}/PKG-INFO

@@ -1,12 +1,29 @@
 Metadata-Version: 2.1
 Name: nci_cidc_api_modules
-Version: 1.0.0rc0
+Version: 1.0.1
 Summary: SQLAlchemy data models and configuration tools used in the NCI CIDC API
 Home-page: https://github.com/NCI-CIDC/cidc-api-gae
 License: MIT license
-Requires-Python: >=3.6
+Requires-Python: >=3.9
 Description-Content-Type: text/markdown
 License-File: LICENSE
+Requires-Dist: werkzeug==2.0.3
+Requires-Dist: flask~=2.0.3
+Requires-Dist: flask-migrate==2.5.2
+Requires-Dist: flask-sqlalchemy~=2.4.0
+Requires-Dist: sqlalchemy~=1.3.0
+Requires-Dist: marshmallow==3.19.0
+Requires-Dist: marshmallow-sqlalchemy==0.22.3
+Requires-Dist: google-cloud-storage~=1.35.0
+Requires-Dist: google-cloud-secret-manager==2.16.2
+Requires-Dist: google-cloud-pubsub==0.43.0
+Requires-Dist: google-cloud-bigquery==3.3.5
+Requires-Dist: google-api-python-client==2.64.0
+Requires-Dist: pandas<2,>=1
+Requires-Dist: python-dotenv==0.10.3
+Requires-Dist: requests==2.22.0
+Requires-Dist: jinja2~=3.0.3
+Requires-Dist: nci-cidc-schemas~=0.26.28
 
 # NCI CIDC API <!-- omit in TOC -->
 
@@ -27,10 +44,11 @@ The next generation of the CIDC API, reworked to use Google Cloud-managed servic
   - [Deploying by hand](#deploying-by-hand)
 - [Connecting to the API](#connecting-to-the-api)
 - [Provisioning the system from scratch](#provisioning-the-system-from-scratch)
+- [Docker Compose](#setting-up-docker-compose)
 
 ## Install Python dependencies
 
-Python versions tested include 3.7, 3.8, and 3.9. The current App Engine is using version 3.7 (see [app.prod.yaml](./app.prod.yaml)). You can use https://github.com/pyenv/pyenv to manage your python versions. Homebrew will also work, but you will have to be specific when you install packages with pip outside of virtual environments. On that note, it is recommended that you install your python dependencies in an isolated environment. For example,
+Python versions tested include 3.9 and 3.10. The current App Engine is using version 3.9 (see [app.prod.yaml](./app.prod.yaml)). You can use https://github.com/pyenv/pyenv to manage your python versions. Homebrew will also work, but you will have to be specific when you install packages with pip outside of virtual environments. On that note, it is recommended that you install your python dependencies in an isolated environment. For example,
 
 ```bash
 # make a virtual environment in the current direcory called "venv"
@@ -284,6 +302,33 @@ To connect to the production API locally, follow the same procedure, but instead
 
 For an overview of how to set up the CIDC API service from scratch, see the step-by-step guide in `PROVISION.md`.
 
+## Setting up Docker Compose
+
+If you would like to run this project as a docker container.  We have dockerized the cidc-api-gae and cidc-ui so that you don't have to install all the requirements above.  Included in the docker-compose file are postgres:14 with data and test user login, bigquery-emulator, fake-gcs-server with buckets and data to match postgres, and gcs-oauth2-emulator to generate faked presigned urls.
+
+**_NOTE:_** You must have docker installed and have this repository and cidc-ui in the same directory (~/git/cidc/cidc-ui and ~/git/cidc/cidc-api-gae), or you can download each and build the image with the command `docker build .`
+
+**_NOTE:_** Having issues with the cidc-ui docker container. You'll have to start that manually using the instructions in the repo.
+
+**_NOTE:_** You can't use Docker while simultaneously running your NIH VPN. This is due to a quirk with self-hosting a google secrets bucket. More work is required to make the docker containers work while the VPN is on.
+
+This repo has hot code reloading. However, you will need to build the image again if there is an update to python libraries. Make sure you don't use a cached image when rebuilding.
+
+Make sure you add this line to your /etc/hosts file: ```127.0.0.1       host.docker.internal```
+
+To run everything simply run the following commands:
+```bash
+vim .env # uncomment the docker section in the .env file.  Comment out any overlaping variable defintions(POSTGRES)
+cp ~/.config/gcloud/application_default_credentials.json .
+cd docker
+docker compose up
+```
+**_NOTE:_** You still need to install and signin to gcloud CLI. The application_default_credentials.json should be under the cidc-api-gae directory next to the Dockerfile. We have mocked most of the connection points to GCP but at startup it still checks for a valid user account. This is very similar behavior to aws's localstack. It requires a realistic token at the start even though it doesn't make connections to aws.
+
+**_TODO:_** The application_default_credentials.json I think could be faked and pointed to the gcs-oauth2-emulator for startup.  In this case a gcloud cli wouldnt be needed at all and a faked application_default_credentials.json could be uploaded under the docker folder.
+
+
+
 ## JIRA Integration
 
 To set-up the git hook for JIRA integration, run:

nci_cidc_api_modules-1.0.0rc0/nci_cidc_api_modules.egg-info/PKG-INFO → nci_cidc_api_modules-1.0.1/README.md

@@ -1,13 +1,3 @@
-Metadata-Version: 2.1
-Name: nci-cidc-api-modules
-Version: 1.0.0rc0
-Summary: SQLAlchemy data models and configuration tools used in the NCI CIDC API
-Home-page: https://github.com/NCI-CIDC/cidc-api-gae
-License: MIT license
-Requires-Python: >=3.6
-Description-Content-Type: text/markdown
-License-File: LICENSE
-
 # NCI CIDC API <!-- omit in TOC -->
 
 The next generation of the CIDC API, reworked to use Google Cloud-managed services. This API is built with the Flask REST API framework backed by Google Cloud SQL, running on Google App Engine.
@@ -27,10 +17,11 @@ The next generation of the CIDC API, reworked to use Google Cloud-managed servic
   - [Deploying by hand](#deploying-by-hand)
 - [Connecting to the API](#connecting-to-the-api)
 - [Provisioning the system from scratch](#provisioning-the-system-from-scratch)
+- [Docker Compose](#setting-up-docker-compose)
 
 ## Install Python dependencies
 
-Python versions tested include 3.7, 3.8, and 3.9. The current App Engine is using version 3.7 (see [app.prod.yaml](./app.prod.yaml)). You can use https://github.com/pyenv/pyenv to manage your python versions. Homebrew will also work, but you will have to be specific when you install packages with pip outside of virtual environments. On that note, it is recommended that you install your python dependencies in an isolated environment. For example,
+Python versions tested include 3.9 and 3.10. The current App Engine is using version 3.9 (see [app.prod.yaml](./app.prod.yaml)). You can use https://github.com/pyenv/pyenv to manage your python versions. Homebrew will also work, but you will have to be specific when you install packages with pip outside of virtual environments. On that note, it is recommended that you install your python dependencies in an isolated environment. For example,
 
 ```bash
 # make a virtual environment in the current direcory called "venv"
@@ -284,6 +275,33 @@ To connect to the production API locally, follow the same procedure, but instead
 
 For an overview of how to set up the CIDC API service from scratch, see the step-by-step guide in `PROVISION.md`.
 
+## Setting up Docker Compose
+
+If you would like to run this project as a docker container.  We have dockerized the cidc-api-gae and cidc-ui so that you don't have to install all the requirements above.  Included in the docker-compose file are postgres:14 with data and test user login, bigquery-emulator, fake-gcs-server with buckets and data to match postgres, and gcs-oauth2-emulator to generate faked presigned urls.
+
+**_NOTE:_** You must have docker installed and have this repository and cidc-ui in the same directory (~/git/cidc/cidc-ui and ~/git/cidc/cidc-api-gae), or you can download each and build the image with the command `docker build .`
+
+**_NOTE:_** Having issues with the cidc-ui docker container. You'll have to start that manually using the instructions in the repo.
+
+**_NOTE:_** You can't use Docker while simultaneously running your NIH VPN. This is due to a quirk with self-hosting a google secrets bucket. More work is required to make the docker containers work while the VPN is on.
+
+This repo has hot code reloading. However, you will need to build the image again if there is an update to python libraries. Make sure you don't use a cached image when rebuilding.
+
+Make sure you add this line to your /etc/hosts file: ```127.0.0.1       host.docker.internal```
+
+To run everything simply run the following commands:
+```bash
+vim .env # uncomment the docker section in the .env file.  Comment out any overlaping variable defintions(POSTGRES)
+cp ~/.config/gcloud/application_default_credentials.json .
+cd docker
+docker compose up
+```
+**_NOTE:_** You still need to install and signin to gcloud CLI. The application_default_credentials.json should be under the cidc-api-gae directory next to the Dockerfile. We have mocked most of the connection points to GCP but at startup it still checks for a valid user account. This is very similar behavior to aws's localstack. It requires a realistic token at the start even though it doesn't make connections to aws.
+
+**_TODO:_** The application_default_credentials.json I think could be faked and pointed to the gcs-oauth2-emulator for startup.  In this case a gcloud cli wouldnt be needed at all and a faked application_default_credentials.json could be uploaded under the docker folder.
+
+
+
 ## JIRA Integration
 
 To set-up the git hook for JIRA integration, run:

{nci_cidc_api_modules-1.0.0rc0 → nci_cidc_api_modules-1.0.1}/cidc_api/config/db.py

@@ -52,7 +52,7 @@ def get_sqlalchemy_database_uri(testing: bool = False) -> str:
                 "host": f'{socket_dir}{environ.get("CLOUD_SQL_INSTANCE_NAME")}'
             }
         else:
-            raise Exception(
+            raise RuntimeError(
                 "Either POSTGRES_URI or CLOUD_SQL_INSTANCE_NAME must be defined to connect "
                 + "to a database."
             )

{nci_cidc_api_modules-1.0.0rc0 → nci_cidc_api_modules-1.0.1}/cidc_api/config/secrets.py

@@ -11,8 +11,8 @@ def get_secrets_manager(is_testing=False):
 
         # If we're testing, we shouldn't need access to secrets in Secret Manager
         return MagicMock()
-    else:
-        return GoogleSecretManager()
+
+    return GoogleSecretManager()
 
 
 class GoogleSecretManager:

{nci_cidc_api_modules-1.0.0rc0 → nci_cidc_api_modules-1.0.1}/cidc_api/config/settings.py

@@ -21,10 +21,9 @@ DEV_USE_GCS = environ.get("DEV_USE_GCS") == "True"
 assert ENV in (
     "dev",
     "dev-int",
-    "featuredev",
     "staging",
     "prod",
-), "ENV environment variable must be set to 'dev', 'dev-int', 'featuredev', 'staging', or 'prod'"
+), "ENV environment variable must be set to 'dev', 'dev-int', 'staging', or 'prod'"
 DEBUG = environ.get("DEBUG") == "True"
 assert ENV == "dev" if DEBUG else True, "DEBUG mode is only allowed when ENV='dev'"
 TESTING = environ.get("TESTING") == "True"

{nci_cidc_api_modules-1.0.0rc0 → nci_cidc_api_modules-1.0.1}/cidc_api/csms/auth.py

@@ -1,9 +1,9 @@
 __all__ = ["get_token", "get_with_authorization", "get_with_paging"]
-import os
 
-os.environ["TZ"] = "UTC"
+import os
 from datetime import datetime, timedelta
 from typing import Any, Dict, Iterator
+
 import requests
 
 from ..config.settings import (
@@ -13,7 +13,9 @@ from ..config.settings import (
     CSMS_TOKEN_URL,
 )
 
+os.environ["TZ"] = "UTC"
 
+TIMEOUT_IN_SECONDS = 20
 _TOKEN, _TOKEN_EXPIRY = None, datetime.now()
 
 
@@ -29,13 +31,14 @@ def get_token():
                     "client_id": CSMS_CLIENT_ID,
                     "client_secret": CSMS_CLIENT_SECRET,
                 },
+                timeout=TIMEOUT_IN_SECONDS,
             ).json(),
             datetime.now(),
         )
 
         # res definition from https://developer.okta.com/docs/reference/api/oidc/#response-example-error-7
         if "errorCode" in res:
-            raise Exception(res["errorCode"] + ": " + res.get("errorSummary"))
+            raise RuntimeError(res["errorCode"] + ": " + res.get("errorSummary"))
 
         _TOKEN = res["access_token"]
         _TOKEN_EXPIRY = time + timedelta(seconds=res["expires_in"])
@@ -54,7 +57,11 @@ def get_with_authorization(url: str, **kwargs) -> requests.Response:
     kwargs["headers"] = headers
     if not url.startswith(CSMS_BASE_URL):
         url = CSMS_BASE_URL + url
-    return requests.get(url, **kwargs)
+    return requests.get(
+        url,
+        **kwargs,
+        timeout=TIMEOUT_IN_SECONDS,
+    )
 
 
 def get_with_paging(
@@ -88,7 +95,7 @@ def get_with_paging(
         else:
             limit = 1
 
-    kwargs.update(dict(limit=limit, offset=offset))
+    kwargs.update({"limit": limit, "offset": offset})
 
     res = get_with_authorization(url, params=kwargs)
     while res.status_code < 300 and len(res.json().get("data", [])) > 0:
@@ -96,5 +103,5 @@ def get_with_paging(
         yield from res.json()["data"]
         kwargs["offset"] += 1  # get the next page
         res = get_with_authorization(url, params=kwargs)
-    else:
-        res.raise_for_status()
+
+    res.raise_for_status()