nmdc-runtime 1.10.0__tar.gz → 2.0.1__tar.gz

{nmdc_runtime-1.10.0 → nmdc_runtime-2.0.1}/.env.example

@@ -40,4 +40,9 @@ NERSC_USERNAME=replaceme
 ORCID_NMDC_CLIENT_ID=replaceme
 ORCID_NMDC_CLIENT_SECRET=replaceme
 
+# Base URL (without a trailing slash) at which the Runtime can access an instance of ORCID.
+# Note: For the production instance of ORCID, use: https://orcid.org (default)
+#       For the sandbox instance of ORCID, use: https://sandbox.orcid.org
+ORCID_BASE_URL=https://orcid.org
+
 INFO_BANNER_INNERHTML='Announcement: Something important is about to happen. If you have questions, please contact <a href="mailto:support@microbiomedata.org">support@microbiomedata.org</a>.'

{nmdc_runtime-1.10.0 → nmdc_runtime-2.0.1}/.github/workflows/python-app.yml

@@ -19,6 +19,9 @@ on:
       - '**.Dockerfile'
       - '**.py'
       - 'requirements/main.txt'
+  # Allow developers to trigger this workflow manually via the "Actions" page on GitHub.
+  # Reference: https://docs.github.com/en/actions/managing-workflow-runs-and-deployments/managing-workflow-runs/manually-running-a-workflow
+  workflow_dispatch: { }
 
 
 jobs:

{nmdc_runtime-1.10.0/nmdc_runtime.egg-info → nmdc_runtime-2.0.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: nmdc_runtime
-Version: 1.10.0
+Version: 2.0.1
 Summary: A runtime system for NMDC data management and orchestration
 Home-page: https://github.com/microbiomedata/nmdc-runtime
 Author: Donny Winston

nmdc_runtime-1.10.0/demo/metadata_migration/notebooks/migrate_10_5_6_to_10_8_0.ipynb → nmdc_runtime-2.0.1/demo/metadata_migration/notebooks/migrate_10_4_0_to_10_9_1.ipynb

@@ -3,13 +3,13 @@
   {
    "metadata": {},
    "cell_type": "markdown",
-   "source": "# Migrate MongoDB database from `nmdc-schema` `v10.5.6` to `v10.8.0`",
+   "source": "# Migrate MongoDB database from `nmdc-schema` `v10.4.0` to `v10.9.1`",
    "id": "d05efc6327778f9c"
   },
   {
    "metadata": {},
    "cell_type": "markdown",
-   "source": "There are no migrators associated with any schema changes between schema versions `v10.5.6` and `v10.8.0`. So, this notebook is a \"no op\" (i.e. \"no operation\").",
+   "source": "There are no migrators associated with any schema changes between schema versions `v10.4.0` and `v10.9.1`. So, this notebook is a \"no op\" (i.e. \"no operation\").",
    "id": "b99d5924e825b9a2"
   },
   {

nmdc_runtime-1.10.0/demo/metadata_migration/notebooks/migrate_10_8_0_to_11_0_0.ipynb → nmdc_runtime-2.0.1/demo/metadata_migration/notebooks/migrate_10_9_1_to_11_0_0.ipynb

@@ -4,9 +4,14 @@
    "cell_type": "markdown",
    "id": "initial_id",
    "metadata": {
-    "collapsed": true
+    "collapsed": true,
+    "jupyter": {
+     "outputs_hidden": true
+    }
    },
-   "source": "# Migrate MongoDB database from `nmdc-schema` `v10.8.0` to `v11.0.0`"
+   "source": [
+    "# Migrate MongoDB database from `nmdc-schema` `v10.9.1` to `v11.0.0`"
+   ]
   },
   {
    "cell_type": "markdown",
@@ -15,7 +20,7 @@
    "source": [
     "## Introduction\n",
     "\n",
-    "This notebook will be used to migrate the database from `nmdc-schema` `v10.8.0` ([released](https://github.com/microbiomedata/nmdc-schema/releases/tag/v10.8.0) August 21, 2024) to `v11.0.0` (i.e. the initial version of the so-called \"Berkeley schema\").\n",
+    "This notebook will be used to migrate the database from `nmdc-schema` `v10.9.1` ([released](https://github.com/microbiomedata/nmdc-schema/releases/tag/v10.9.1) October 7, 2024) to `v11.0.0` (i.e. the initial version of the so-called \"Berkeley schema\").\n",
     "\n",
     "Unlike previous migrators, this one does not pick and choose which collections it will dump. There are two reasons for this: (1) migrators no longer have a dedicated `self.agenda` dictionary that indicates all the collections involved in the migration; and (2) this migration is the first one that involves creating, renaming, and dropping any collections; none of which are things that the old `self.agenda`-based system was designed to handle. So, instead of picking and choosing collections, this migrator **dumps them all.**"
    ]
@@ -61,32 +66,12 @@
    "outputs": [],
    "execution_count": null
   },
-  {
-   "cell_type": "markdown",
-   "id": "22f5c78f",
-   "metadata": {},
-   "source": [
-    "2. Delete **obsolete dumps** from previous notebooks runs.\n",
-    "    1. This is so the dumps you generate below will not be merged with any unrelated ones."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "id": "c70b6715",
-   "metadata": {},
-   "source": [
-    "!rm -rf {cfg.origin_dump_folder_path}\n",
-    "!rm -rf {cfg.transformer_dump_folder_path}"
-   ],
-   "outputs": [],
-   "execution_count": null
-  },
   {
    "cell_type": "markdown",
    "id": "6cd05ccb",
    "metadata": {},
    "source": [
-    "3. Create and populate a **notebook configuration file** named `.notebook.env`.\n",
+    "2. Create and populate a **notebook configuration file** named `.notebook.env`.\n",
     "    1. You can use `.notebook.env.example` as a template.\n",
     "    2. The \"origin\" MongoDB server is the one that contains the database you want to migrate.\n",
     "    3. The \"transformer\" MongoDB server is the one you want to use to perform the data transformations."
@@ -124,12 +109,16 @@
    "cell_type": "code",
    "id": "e25a0af308c3185b",
    "metadata": {
-    "collapsed": false
+    "collapsed": false,
+    "jupyter": {
+     "outputs_hidden": false
+    },
+    "scrolled": true
    },
    "source": [
     "%pip install --upgrade pip\n",
     "%pip install -r requirements.txt\n",
-    "%pip install nmdc-schema==11.0.0rc20"
+    "%pip install nmdc-schema==11.0.0"
    ],
    "outputs": [],
    "execution_count": null
@@ -158,7 +147,7 @@
     "# Third-party packages:\n",
     "import pymongo\n",
     "from jsonschema import Draft7Validator as JSONSchemaValidator\n",
-    "from nmdc_schema.nmdc_data import get_nmdc_jsonschema_dict, SchemaVariantIdentifier, get_nmdc_jsonschema\n",
+    "from nmdc_schema.nmdc_data import get_nmdc_jsonschema_dict, SchemaVariantIdentifier, get_nmdc_schema_definition\n",
     "from nmdc_schema.migrators.adapters.mongo_adapter import MongoAdapter\n",
     "from linkml_runtime import SchemaView\n",
     "\n",
@@ -220,14 +209,14 @@
    "source": [
     "# Mongo client for \"origin\" MongoDB server.\n",
     "origin_mongo_client = pymongo.MongoClient(host=cfg.origin_mongo_host, \n",
-    "                                          port=cfg.origin_mongo_port,\n",
+    "                                          port=int(cfg.origin_mongo_port),\n",
     "                                          username=cfg.origin_mongo_username,\n",
     "                                          password=cfg.origin_mongo_password,\n",
     "                                          directConnection=True)\n",
     "\n",
     "# Mongo client for \"transformer\" MongoDB server.\n",
     "transformer_mongo_client = pymongo.MongoClient(host=cfg.transformer_mongo_host, \n",
-    "                                               port=cfg.transformer_mongo_port,\n",
+    "                                               port=int(cfg.transformer_mongo_port),\n",
     "                                               username=cfg.transformer_mongo_username,\n",
     "                                               password=cfg.transformer_mongo_password,\n",
     "                                               directConnection=True)\n",
@@ -291,7 +280,10 @@
    "cell_type": "markdown",
    "id": "bc387abc62686091",
    "metadata": {
-    "collapsed": false
+    "collapsed": false,
+    "jupyter": {
+     "outputs_hidden": false
+    }
    },
    "source": [
     "### Create JSON Schema validator\n",
@@ -303,7 +295,10 @@
    "cell_type": "code",
    "id": "5c982eb0c04e606d",
    "metadata": {
-    "collapsed": false
+    "collapsed": false,
+    "jupyter": {
+     "outputs_hidden": false
+    }
    },
    "source": [
     "nmdc_jsonschema: dict = get_nmdc_jsonschema_dict(variant=SchemaVariantIdentifier.nmdc_materialized_patterns)\n",
@@ -320,22 +315,27 @@
    "execution_count": null
   },
   {
-   "metadata": {},
    "cell_type": "markdown",
+   "id": "e7e8befb362a1670",
+   "metadata": {},
    "source": [
     "### Create SchemaView\n",
     "\n",
     "In this step, you'll instantiate a `SchemaView` that is bound to the destination schema. \n",
     "\n",
     "- Reference: https://linkml.io/linkml/developers/schemaview.html"
-   ],
-   "id": "e7e8befb362a1670"
+   ]
   },
   {
-   "metadata": {},
    "cell_type": "code",
-   "source": "schema_view = SchemaView(get_nmdc_jsonschema())",
    "id": "625a6e7df5016677",
+   "metadata": {},
+   "source": [
+    "schema_view = SchemaView(get_nmdc_schema_definition())\n",
+    "\n",
+    "# As a sanity test, confirm we can use the `SchemaView` instance to access a schema class.\n",
+    "schema_view.get_class(class_name=\"Database\")[\"name\"]"
+   ],
    "outputs": [],
    "execution_count": null
   },
@@ -379,6 +379,27 @@
    "outputs": [],
    "execution_count": null
   },
+  {
+   "cell_type": "markdown",
+   "id": "7f9c87de6fb8530c",
+   "metadata": {},
+   "source": [
+    "### Delete obsolete dumps from previous migrations\n",
+    "\n",
+    "Delete any existing dumps before we create new ones in this notebook. This is so the dumps you generate with this notebook do not get merged with any unrelated ones."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "id": "6a949d0fcb4b6fa0",
+   "metadata": {},
+   "source": [
+    "!rm -rf {cfg.origin_dump_folder_path}\n",
+    "!rm -rf {cfg.transformer_dump_folder_path}"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
   {
    "cell_type": "markdown",
    "id": "b7799910b6b0715d",
@@ -394,7 +415,9 @@
   {
    "cell_type": "code",
    "id": "da530d6754c4f6fe",
-   "metadata": {},
+   "metadata": {
+    "scrolled": true
+   },
    "source": [
     "# Dump all collections from the \"origin\" database.\n",
     "shell_command = f\"\"\"\n",
@@ -427,7 +450,9 @@
   {
    "cell_type": "code",
    "id": "79bd888e82d52a93",
-   "metadata": {},
+   "metadata": {
+    "scrolled": true
+   },
    "source": [
     "# Restore the dumped collections to the \"transformer\" MongoDB server.\n",
     "shell_command = f\"\"\"\n",
@@ -466,7 +491,9 @@
   {
    "cell_type": "code",
    "id": "9c89c9dd3afe64e2",
-   "metadata": {},
+   "metadata": {
+    "scrolled": true
+   },
    "source": [
     "# Instantiate a MongoAdapter bound to the \"transformer\" database.\n",
     "adapter = MongoAdapter(\n",
@@ -516,7 +543,7 @@
     "for collection_name in ordered_collection_names:\n",
     "    collection = transformer_mongo_client[\"nmdc\"][collection_name]\n",
     "    num_documents_in_collection = collection.count_documents({})\n",
-    "    print(f\"Validating collection {collection_name} ({num_documents_in_collection} documents)\")\n",
+    "    print(f\"Validating collection {collection_name} ({num_documents_in_collection} documents)\", end=\"\\t\")  # no newline\n",
     "\n",
     "    for document in collection.find():\n",
     "        # Validate the transformed document.\n",
@@ -533,7 +560,9 @@
     "        #\n",
     "        document_without_underscore_id_key = {key: value for key, value in document.items() if key != \"_id\"}\n",
     "        root_to_validate = dict([(collection_name, [document_without_underscore_id_key])])\n",
-    "        nmdc_jsonschema_validator.validate(root_to_validate)  # raises exception if invalid"
+    "        nmdc_jsonschema_validator.validate(root_to_validate)  # raises exception if invalid\n",
+    "\n",
+    "    print(f\"Done\")"
    ],
    "outputs": [],
    "execution_count": null
@@ -551,7 +580,9 @@
   {
    "cell_type": "code",
    "id": "db6e432d",
-   "metadata": {},
+   "metadata": {
+    "scrolled": true
+   },
    "source": [
     "# Dump the database from the \"transformer\" MongoDB server.\n",
     "shell_command = f\"\"\"\n",
@@ -575,7 +606,10 @@
    "cell_type": "markdown",
    "id": "997fcb281d9d3222",
    "metadata": {
-    "collapsed": false
+    "collapsed": false,
+    "jupyter": {
+     "outputs_hidden": false
+    }
    },
    "source": [
     "### Create a bookkeeper\n",
@@ -656,7 +690,9 @@
   {
    "cell_type": "code",
    "id": "1dfbcf0a",
-   "metadata": {},
+   "metadata": {
+    "scrolled": true
+   },
    "source": [
     "# Load the transformed collections into the origin server, replacing any same-named ones that are there.\n",
     "shell_command = f\"\"\"\n",
@@ -683,7 +719,10 @@
    "cell_type": "markdown",
    "id": "ca5ee89a79148499",
    "metadata": {
-    "collapsed": false
+    "collapsed": false,
+    "jupyter": {
+     "outputs_hidden": false
+    }
    },
    "source": [
     "### Indicate that the migration is complete\n",
@@ -695,7 +734,10 @@
    "cell_type": "code",
    "id": "d1eaa6c92789c4f3",
    "metadata": {
-    "collapsed": false
+    "collapsed": false,
+    "jupyter": {
+     "outputs_hidden": false
+    }
    },
    "source": [
     "bookkeeper.record_migration_event(migrator=migrator, event=MigrationEvent.MIGRATION_COMPLETED)"
@@ -732,11 +774,19 @@
    ],
    "outputs": [],
    "execution_count": null
+  },
+  {
+   "cell_type": "code",
+   "id": "037db214-ea76-46bf-bb6a-bf1ff9b28a72",
+   "metadata": {},
+   "source": [],
+   "outputs": [],
+   "execution_count": null
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },

nmdc_runtime-2.0.1/demo/metadata_migration/notebooks/migrate_11_0_0_to_11_0_1.ipynb

@@ -0,0 +1,50 @@
+{
+ "cells": [
+  {
+   "metadata": {},
+   "cell_type": "markdown",
+   "source": "# Migrate MongoDB database from `nmdc-schema` `v11.0.0` to `v11.0.1`",
+   "id": "d05efc6327778f9c"
+  },
+  {
+   "metadata": {},
+   "cell_type": "markdown",
+   "source": "There are no migrators associated with any schema changes between schema versions `v11.0.0` and `v11.0.1`. So, this notebook is a \"no op\" (i.e. \"no operation\").",
+   "id": "b99d5924e825b9a2"
+  },
+  {
+   "cell_type": "code",
+   "id": "initial_id",
+   "metadata": {
+    "collapsed": true,
+    "jupyter": {
+     "is_executing": true
+    }
+   },
+   "source": "# no op",
+   "outputs": [],
+   "execution_count": null
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}