nmdc-runtime 2.0.1__tar.gz → 2.1.1__tar.gz

{nmdc_runtime-2.0.1 → nmdc_runtime-2.1.1}/.env.example

@@ -1,5 +1,7 @@
 # To load this in a shell session:
-# export $(grep -v '^#' .env | xargs)
+# set -a # automatically export all variables
+# source .env
+# set +a
 
 MONGO_HOST=mongodb://mongo:27017
 MONGO_USERNAME=admin

{nmdc_runtime-2.0.1 → nmdc_runtime-2.1.1}/.env.test

@@ -1,5 +1,7 @@
 # To load this in a shell session:
-# export $(grep -v '^#' .env.test | xargs)
+# set -a # automatically export all variables
+# source .env.test
+# set +a
 
 MONGO_HOST=mongodb://mongo:27017
 MONGO_USERNAME=admin

nmdc_runtime-2.1.1/.github/pull_request_template.md

@@ -0,0 +1,108 @@
+<!--┌─────────────────────────────────────────────────────────────────────────┐
+    │ 1. Summary (required)                                                   │
+    │                                                                         │
+    │ Summarize the changes you made on this branch. This is typically a more │
+    │ detailed restatement of the PR title.                                   │
+    │                                                                         │
+    │ Example: "In this branch, I updated the `/studies/{study_id}` endpoint  │
+    │           so it returns an HTTP 404 response when the specified study   │
+    │           does not exist."                                              │
+    └─────────────────────────────────────────────────────────────────────────┘-->
+
+In this branch, I...
+
+### Details
+
+<!--┌─────────────────────────────────────────────────────────────────────────┐
+    │ 2. Details (optional)                                                   │
+    │                                                                         │
+    │ Provide additional information you think readers will find useful.      │
+    │ Readers include PR reviewers, release note authors, app debuggers, and  │
+    │ your future self. Additional information might include motivation,      │
+    │ rationale, and a description of how things used to be.                  │
+    │                                                                         │
+    │ Example: "It previously returned an HTTP 404 response and an empty      │
+    │           JSON object."                                                 │
+    └─────────────────────────────────────────────────────────────────────────┘-->
+
+...
+
+### Related issue(s)
+
+<!--┌─────────────────────────────────────────────────────────────────────────┐
+    │ 3. Related issue(s) (optional)                                          │
+    │                                                                         │
+    │ Link to any GitHub issue(s) this branch was designed to resolve.        │
+    │                                                                         │
+    │ Example: "Fixes #12345"                                                 │
+    └─────────────────────────────────────────────────────────────────────────┘-->
+
+...
+
+### Related subsystem(s)
+
+<!--┌─────────────────────────────────────────────────────────────────────────┐
+    │ 4. Related subsystem(s) (required)                                      │
+    │                                                                         │
+    │ Mark the checkbox next to each subsystem related to the changes in this │
+    │ branch. This information might influence who you request reviews from.  │
+    │                                                                         │
+    │ Example: If you modified the `/studies/{study_id}` API endpoint,        │
+    │          mark the checkbox next to "Runtime API (except the Minter)".   │
+    └─────────────────────────────────────────────────────────────────────────┘-->
+
+- [ ] Runtime API (except the Minter)
+- [ ] Minter
+- [ ] Dagster
+- [ ] Project documentation (in the `docs` directory)
+- [ ] MongoDB migrations
+- [ ] Other
+
+### Testing
+
+<!--┌─────────────────────────────────────────────────────────────────────────┐
+    │ 5. Testing (required)                                                   │
+    │                                                                         │
+    │ Indicate whether you have already tested the changes this branch        │
+    │ contains; and, if so, how someone other than you can test them. That    │
+    │ may involve attaching example files or ad hoc test instructions.        │
+    │                                                                         │
+    │ Example: "I tested these changes by adding a pytest test that ensures   │
+    │           the database does not contain a Study whose ID is `foo`,      │
+    │           then submits an HTTP request to `/studies/foo` and confirms   │
+    │           the response status is 404."                                  │
+    └─────────────────────────────────────────────────────────────────────────┘-->
+
+- [ ] I tested these changes (explain below)
+- [ ] I did not test these changes
+
+I tested these changes by...
+
+### Documentation
+
+<!--┌─────────────────────────────────────────────────────────────────────────┐
+    │ 6. Documentation (required)                                             │
+    │                                                                         │
+    │ Indicate whether, in this branch, you have updated all documentation    │
+    │ that would otherwise become inaccurate if this branch were to be        │
+    │ merged in.                                                              │
+    └─────────────────────────────────────────────────────────────────────────┘-->
+
+- [ ] I **have not checked** for relevant documentation yet (e.g. in the `docs` directory)
+- [ ] I have **updated** all relevant documentation so it will remain accurate
+- [ ] Other (explain below)
+
+### Maintainability
+
+<!--┌─────────────────────────────────────────────────────────────────────────┐
+    │ 7. Maintainability (required)                                           │
+    │                                                                         │
+    │ Indicate whether you have done each of these things that can make code  │
+    │ easier to maintain, whether by your teammates or by your future self.   │
+    └─────────────────────────────────────────────────────────────────────────┘-->
+
+- [ ] Every Python function I defined includes a docstring _(test functions are exempt from this)_
+- [ ] Every Python function parameter I introduced includes a type hint (e.g. `study_id: str`)
+- [ ] All "to do" or "fix me" Python comments I added begin with either `# TODO` or `# FIXME`
+- [ ] I used `black` to format all the Python files I created/modified
+- [ ] The PR title is in the imperative mood (e.g. "Do X") and not the declarative mood (e.g. "Does X" or "Did X")

{nmdc_runtime-2.0.1/nmdc_runtime.egg-info → nmdc_runtime-2.1.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: nmdc_runtime
-Version: 2.0.1
+Version: 2.1.1
 Summary: A runtime system for NMDC data management and orchestration
 Home-page: https://github.com/microbiomedata/nmdc-runtime
 Author: Donny Winston
@@ -114,7 +114,9 @@ cp .env.example .env
 Create environment variables in your shell session, based upon the contents of the `.env` file.
 
 ```shell
-export $(grep -v '^#' .env | xargs)
+set -a # automatically export all variables
+source .env
+set +a
 ```
 
 If you are connecting to resources that require an SSH tunnel—for example, a MongoDB server that is only accessible on the NERSC network—set up the SSH tunnel.

{nmdc_runtime-2.0.1 → nmdc_runtime-2.1.1}/README.md

@@ -100,7 +100,9 @@ cp .env.example .env
 Create environment variables in your shell session, based upon the contents of the `.env` file.
 
 ```shell
-export $(grep -v '^#' .env | xargs)
+set -a # automatically export all variables
+source .env
+set +a
 ```
 
 If you are connecting to resources that require an SSH tunnel—for example, a MongoDB server that is only accessible on the NERSC network—set up the SSH tunnel.

{nmdc_runtime-2.0.1 → nmdc_runtime-2.1.1}/demo/metadata_migration/notebooks/.notebook.env.example

@@ -8,13 +8,19 @@ PATH_TO_MONGORESTORE_BINARY = "__REPLACE_ME__"  # e.g. "/Users/Alice/Downloads/m
 PATH_TO_MONGOSH_BINARY = "__REPLACE_ME__"  # e.g. "/Users/Alice/Downloads/mongosh-1.10.6-darwin-x64/bin/mongosh"
 
 # Connection parameters for the Origin Mongo server (typically a remote serve).
+# Note: The Origin Mongo server is the one that contains the database you want
+#       to migrate. It is where the "E" and "L" in "ETL" will take place.
 ORIGIN_MONGO_HOST="__REPLACE_ME__"
 ORIGIN_MONGO_PORT="__REPLACE_ME__"
 ORIGIN_MONGO_USERNAME="__REPLACE_ME__"
 ORIGIN_MONGO_PASSWORD="__REPLACE_ME__"
+ORIGIN_MONGO_DATABASE_NAME="__REPLACE_ME__"  # e.g. "nmdc"
 
 # Connection parameters for the Transformer Mongo server (typically a local server).
+# Note: The Transformer Mongo server is the one you want to use to perform the
+#       database transformations. It is where the "T" in "ETL" will take place.
 TRANSFORMER_MONGO_HOST="__REPLACE_ME__"
 TRANSFORMER_MONGO_PORT="__REPLACE_ME__"
 TRANSFORMER_MONGO_USERNAME="__REPLACE_ME__"
 TRANSFORMER_MONGO_PASSWORD="__REPLACE_ME__"
+TRANSFORMER_MONGO_DATABASE_NAME="__REPLACE_ME__"  # e.g. "nmdc_transformer"

{nmdc_runtime-2.0.1 → nmdc_runtime-2.1.1}/demo/metadata_migration/notebooks/helpers.py

@@ -2,6 +2,7 @@ from pathlib import Path
 from typing import Dict, Optional, List
 import logging
 from datetime import datetime
+from functools import cache
 
 from dotenv import dotenv_values
 from linkml_runtime import SchemaView
@@ -13,6 +14,34 @@ DATABASE_CLASS_NAME = "Database"
 class Config:
     """Wrapper class for configuration values related to database migration."""
 
+    @staticmethod
+    def make_mongo_cli_base_options(
+        mongo_host: str,
+        mongo_port: str,
+        mongo_username: Optional[str],
+        mongo_password: Optional[str],
+    ) -> str:
+        r"""Compile common Mongo CLI options into a reusable string."""
+
+        # Always include the CLI options that specify the server host and port.
+        base_options: List[str] = [
+            f"--host='{mongo_host}'",
+            f"--port='{mongo_port}'",
+        ]
+
+        # If the server uses authentication, include authentication-related CLI options.
+        server_uses_authentication: bool = mongo_username not in ["", None]
+        if server_uses_authentication:
+            base_options.extend(
+                [
+                    f"--username='{mongo_username}'",
+                    f"--password='{mongo_password}'",
+                    f"--authenticationDatabase='admin'",
+                ]
+            )
+
+        return " ".join(base_options)
+
     def parse_and_validate_notebook_config_file(
         self, notebook_config_file_path: str
     ) -> Dict[str, str]:
@@ -27,7 +56,9 @@ class Config:
 
         # Validate the dump folder paths.
         origin_dump_folder_path = notebook_config["PATH_TO_ORIGIN_MONGO_DUMP_FOLDER"]
-        transformer_dump_folder_path = notebook_config["PATH_TO_TRANSFORMER_MONGO_DUMP_FOLDER"]
+        transformer_dump_folder_path = notebook_config[
+            "PATH_TO_TRANSFORMER_MONGO_DUMP_FOLDER"
+        ]
         if not Path(origin_dump_folder_path).parent.is_dir():
             raise FileNotFoundError(
                 f"Parent folder of {origin_dump_folder_path} (origin Mongo dump folder path) not found."
@@ -44,7 +75,9 @@ class Config:
         if not Path(mongodump_path).is_file():
             raise FileNotFoundError(f"mongodump binary not found at: {mongodump_path}")
         if not Path(mongorestore_path).is_file():
-            raise FileNotFoundError(f"mongorestore binary not found at: {mongorestore_path}")
+            raise FileNotFoundError(
+                f"mongorestore binary not found at: {mongorestore_path}"
+            )
         if not Path(mongosh_path).is_file():
             raise FileNotFoundError(f"mongosh binary not found at: {mongosh_path}")
 
@@ -52,11 +85,33 @@ class Config:
         origin_mongo_port = notebook_config["ORIGIN_MONGO_PORT"]
         origin_mongo_username = notebook_config["ORIGIN_MONGO_USERNAME"]
         origin_mongo_password = notebook_config["ORIGIN_MONGO_PASSWORD"]
+        origin_mongo_database_name = notebook_config["ORIGIN_MONGO_DATABASE_NAME"]
 
         transformer_mongo_host = notebook_config["TRANSFORMER_MONGO_HOST"]
         transformer_mongo_port = notebook_config["TRANSFORMER_MONGO_PORT"]
         transformer_mongo_username = notebook_config["TRANSFORMER_MONGO_USERNAME"]
         transformer_mongo_password = notebook_config["TRANSFORMER_MONGO_PASSWORD"]
+        transformer_mongo_database_name = notebook_config[
+            "TRANSFORMER_MONGO_DATABASE_NAME"
+        ]
+
+        # Validate the database names.
+        if origin_mongo_database_name.strip() == "":
+            raise ValueError(f"Origin database name cannot be empty")
+        if transformer_mongo_database_name.strip() == "":
+            raise ValueError(f"Transformer database name cannot be empty")
+        if all(
+            [
+                origin_mongo_host == transformer_mongo_host,
+                origin_mongo_port == transformer_mongo_port,
+                origin_mongo_database_name == transformer_mongo_database_name,
+            ]
+        ):
+            # Note: We don't allow the use of the origin database as the transformer,
+            #       because that would prevent us from easily aborting the migration.
+            raise ValueError(
+                f"The origin and transformer cannot both be the same database"
+            )
 
         return dict(
             origin_dump_folder_path=origin_dump_folder_path,
@@ -68,30 +123,40 @@ class Config:
             origin_mongo_port=origin_mongo_port,
             origin_mongo_username=origin_mongo_username,
             origin_mongo_password=origin_mongo_password,
+            origin_mongo_database_name=origin_mongo_database_name,
             transformer_mongo_host=transformer_mongo_host,
             transformer_mongo_port=transformer_mongo_port,
             transformer_mongo_username=transformer_mongo_username,
             transformer_mongo_password=transformer_mongo_password,
+            transformer_mongo_database_name=transformer_mongo_database_name,
         )
 
     def __init__(self, notebook_config_file_path: str = "./.notebook.env") -> None:
         # Parse and validate the notebook config file.
-        notebook_config = self.parse_and_validate_notebook_config_file(notebook_config_file_path)
+        notebook_config = self.parse_and_validate_notebook_config_file(
+            notebook_config_file_path
+        )
         self.mongodump_path = notebook_config["mongodump_path"]
         self.mongorestore_path = notebook_config["mongorestore_path"]
         self.mongosh_path = notebook_config["mongosh_path"]
         self.origin_dump_folder_path = notebook_config["origin_dump_folder_path"]
-        self.transformer_dump_folder_path = notebook_config["transformer_dump_folder_path"]
+        self.transformer_dump_folder_path = notebook_config[
+            "transformer_dump_folder_path"
+        ]
 
         # Parse the Mongo connection parameters.
         self.origin_mongo_host = notebook_config["origin_mongo_host"]
         self.origin_mongo_port = notebook_config["origin_mongo_port"]
         self.origin_mongo_username = notebook_config["origin_mongo_username"]
         self.origin_mongo_password = notebook_config["origin_mongo_password"]
+        self.origin_mongo_database_name = notebook_config["origin_mongo_database_name"]
         self.transformer_mongo_host = notebook_config["transformer_mongo_host"]
         self.transformer_mongo_port = notebook_config["transformer_mongo_port"]
         self.transformer_mongo_username = notebook_config["transformer_mongo_username"]
         self.transformer_mongo_password = notebook_config["transformer_mongo_password"]
+        self.transformer_mongo_database_name = notebook_config[
+            "transformer_mongo_database_name"
+        ]
 
 
 def setup_logger(
@@ -128,6 +193,9 @@ def get_collection_names_from_schema(schema_view: SchemaView) -> List[str]:
     Returns the names of the slots of the `Database` class that describe database collections.
 
     :param schema_view: A `SchemaView` instance
+
+    Source: This function was copied from https://github.com/microbiomedata/refscan/blob/main/refscan/lib/helpers.py
+            with permission from its author.
     """
     collection_names = []
 
@@ -144,3 +212,50 @@ def get_collection_names_from_schema(schema_view: SchemaView) -> List[str]:
         collection_names = list(set(collection_names))
 
     return collection_names
+
+
+@cache  # memoizes the decorated function
+def translate_class_uri_into_schema_class_name(
+    schema_view: SchemaView, class_uri: str
+) -> Optional[str]:
+    r"""
+    Returns the name of the schema class that has the specified value as its `class_uri`.
+
+    Example: "nmdc:Biosample" (a `class_uri` value) -> "Biosample" (a class name)
+
+    References:
+    - https://linkml.io/linkml/developers/schemaview.html#linkml_runtime.utils.schemaview.SchemaView.all_classes
+    - https://linkml.io/linkml/code/metamodel.html#linkml_runtime.linkml_model.meta.ClassDefinition.class_uri
+
+    Source: This function was copied from https://github.com/microbiomedata/refscan/blob/main/refscan/lib/helpers.py
+            with permission from its author.
+    """
+    schema_class_name = None
+    all_class_definitions_in_schema = schema_view.all_classes()
+    for class_name, class_definition in all_class_definitions_in_schema.items():
+        if class_definition.class_uri == class_uri:
+            schema_class_name = class_definition.name
+            break
+    return schema_class_name
+
+
+def derive_schema_class_name_from_document(
+    schema_view: SchemaView, document: dict
+) -> Optional[str]:
+    r"""
+    Returns the name of the schema class, if any, of which the specified document claims to represent an instance.
+
+    This function is written under the assumption that the document has a `type` field whose value is the `class_uri`
+    belonging to the schema class of which the document represents an instance. Slot definition for such a field:
+    https://github.com/microbiomedata/berkeley-schema-fy24/blob/fc2d9600/src/schema/basic_slots.yaml#L420-L436
+
+    Source: This function was copied from https://github.com/microbiomedata/refscan/blob/main/refscan/lib/helpers.py
+            with permission from its author.
+    """
+    schema_class_name = None
+    if "type" in document and isinstance(document["type"], str):
+        class_uri = document["type"]
+        schema_class_name = translate_class_uri_into_schema_class_name(
+            schema_view, class_uri
+        )
+    return schema_class_name