nmdc-runtime 2.9.0__py3-none-any.whl → 2.11.0__py3-none-any.whl

nmdc_runtime-2.9.0.dist-info/METADATA

@@ -1,214 +0,0 @@
-Metadata-Version: 2.4
-Name: nmdc_runtime
-Version: 2.9.0
-Summary: A runtime system for NMDC data management and orchestration
-Home-page: https://github.com/microbiomedata/nmdc-runtime
-Author: Donny Winston
-Author-email: donny@polyneme.xyz
-Classifier: Development Status :: 3 - Alpha
-Classifier: Programming Language :: Python :: 3
-Classifier: License :: OSI Approved :: Apache Software License
-Requires-Python: >=3.10
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Dynamic: author
-Dynamic: author-email
-Dynamic: classifier
-Dynamic: description
-Dynamic: description-content-type
-Dynamic: home-page
-Dynamic: license-file
-Dynamic: requires-python
-Dynamic: summary
-
-A runtime system for NMDC data management and orchestration.
-
-## Service Status
-
-http://nmdcstatus.polyneme.xyz/
-
-## How It Fits In
-
-* [issues](https://github.com/microbiomedata/issues)  
-tracks issues related to NMDC, which may necessitate work across multiple repos.
-  
-* [nmdc-schema](https://github.com/microbiomedata/nmdc-schema/)
-houses the LinkML schema specification, as well as generated artifacts (e.g. JSON Schema).
-
-* [nmdc-server](https://github.com/microbiomedata/nmdc-server)
-houses code specific to the data portal -- its database, back-end API, and front-end application.
-
-* Workflows — documented in the [workflows](https://docs.microbiomedata.org/workflows/) section of the NMDC documentation website — take source data and produce computed data.
-
-* This repo (nmdc-runtime)
-   * houses code that takes source data and computed data, and transforms it
-     to broadly accommodate downstream applications such as the data portal
-   * manages execution of the above (i.e., lightweight data transformations) and also
-     of computationally- and data-intensive workflows performed at other sites,
-     ensuring that claimed jobs have access to needed configuration and data resources.
-
-## Data exports
-
-The NMDC metadata as of 2021-10 is available here:
-
-https://drs.microbiomedata.org/ga4gh/drs/v1/objects/sys086d541
-
-The link returns a [GA4GH DRS API bundle object record](https://ga4gh.github.io/data-repository-service-schemas/preview/release/drs-1.0.0/docs/#_drs_datatypes), with the NMDC metadata collections (study_set, biosample_set, etc.) as contents, each a DRS API blob object.
-
-For example the blob for the study_set collection export, named "study_set.jsonl.gz", is listed with DRS API ID "sys0xsry70". Thus, it is retrievable via
-
-https://drs.microbiomedata.org/ga4gh/drs/v1/objects/sys0xsry70
-
-The returned blob object record lists https://nmdc-runtime.files.polyneme.xyz/nmdcdb-mongoexport/2021-10-14/study_set.jsonl.gz as the url for an access method.
-
-The 2021-10 exports are currently all accessible at `https://nmdc-runtime.files.polyneme.xyz/nmdcdb-mongoexport/2021-10-14/${COLLECTION_NAME}.jsonl.gz`, but the DRS API indirection allows these links to change in the future, for mirroring via other URLs, etc. So, the DRS API links should be the links you share.
-
-## Overview
-
-The runtime features:
-
-1. [Dagster](https://docs.dagster.io/concepts) orchestration:
-    - dagit - a web UI to monitor and manage the running system.
-    - dagster-daemon - a service that triggers pipeline runs based on time or external state.
-    - PostgresSQL database - for storing run history, event logs, and scheduler state.
-    - workspace code
-      - Code to run is loaded into a Dagster `workspace`. This code is loaded from
-         one or more dagster `repositories`. Each Dagster `repository` may be run with a different
-         Python virtual environment if need be, and may be loaded from a local Python file or
-         `pip install`ed from an external source. In our case, each Dagster `repository` is simply
-         loaded from a Python file local to the nmdc-runtime GitHub repository, and all code is
-         run in the same Python environment.
-      - A Dagster repository consists of `solids` and `pipelines`,
-         and optionally `schedules` and `sensors`.
-         - `solids` represent individual units of computation
-         - `pipelines` are built up from solids
-         - `schedules` trigger recurring pipeline runs based on time
-         - `sensors` trigger pipeline runs based on external state
-      - Each `pipeline` can declare dependencies on any runtime `resources` or additional
-         configuration. There are MongoDB `resources` defined, as well as `preset`
-         configuration definitions for both "dev" and "prod" `modes`. The `preset`s tell Dagster to
-         look to a set of known environment variables to load resources configurations, depending on
-         the `mode`.
-   
-2. A MongoDB database supporting write-once, high-throughput internal
-data storage by the nmdc-runtime FastAPI instance.
-   
-3. A [FastAPI](https://fastapi.tiangolo.com/) service to interface with the orchestrator and
-database, as a hub for data management and workflow automation.
-
-## Local Development
-
-Ensure Docker (and Docker Compose) are installed; and the Docker engine is running.
-
-```shell
-docker --version
-docker compose version
-docker info
-```
-
-Ensure the permissions of `./.docker/mongoKeyFile` are such that only the file's owner can read or write the file.
-
-```shell
-chmod 600 ./.docker/mongoKeyFile
-```
-
-Ensure you have a `.env` file for the Docker services to source from. You may copy `.env.example` to
-`.env` (which is gitignore'd) to get started.
-
-```shell
-cp .env.example .env
-```
-
-Create environment variables in your shell session, based upon the contents of the `.env` file.
-
-```shell
-set -a # automatically export all variables
-source .env
-set +a
-```
-
-If you are connecting to resources that require an SSH tunnel—for example, a MongoDB server that is only accessible on 
-the NERSC network—set up the SSH tunnel.
-
-The following command could be useful to you, either directly or as a template (see `Makefile`).
-
-```shell
-make nersc-mongo-tunnels
-```
-
-Finally, spin up the Docker Compose stack.
-
-```bash
-make up-dev
-```
-
-Docker Compose is used to start local MongoDB and PostgresSQL (used by Dagster) instances, as well
-as a Dagster web server (dagit) and daemon (dagster-daemon).
-
-The Dagit web server is viewable at http://127.0.0.1:3000/. 
-
-The FastAPI service is viewable at http://127.0.0.1:8000/ -- e.g., rendered documentation at
-http://127.0.0.1:8000/redoc/.
-
-
-*  NOTE: Any time you add or change requirements in requirements/main.in or requirements/dev.in, you must run:
-```
-pip-compile --build-isolation --allow-unsafe --resolver=backtracking --strip-extras --output-file requirements/[main|dev].txt requirements/[main|dev].in
-```
-to generate main.txt and dev.txt files respectively. main.in is kind of like a poetry dependency stanza, dev.in is kind 
-of like poetry dev.dependencies stanza. main.txt and dev.txt are kind of like poetry.lock files to specify the exact 
-versions of dependencies to use. main.txt and dev.txt are combined in the docker compose build process to create the 
-final requirements.txt file and import the dependencies into the Docker image.
-
-
-```bash
-
-## Local Testing
-
-Tests can be found in `tests` and are run with the following commands:
-
-```bash
-make up-test
-make test
-
-# Run a Specific test file eg. tests/test_api/test_endpoints.py
-make test ARGS="tests/test_api/test_endpoints.py"
-```
-docker compose --file docker-compose.test.yml run test
-
-As you create Dagster solids and pipelines, add tests in `tests/` to check that your code behaves as
-desired and does not break over time.
-
-[For hints on how to write tests for solids and pipelines in Dagster, see their documentation
-tutorial on Testing](https://docs.dagster.io/guides/test/unit-testing-assets-and-ops).
-
-### RAM usage
-
-The `dagster-daemon` and `dagster-dagit` containers can consume a lot of RAM. If tests are failing and the console of
-the `test` container shows "Error 137," here is something you can try as a workaround: In Docker Desktop, go to 
-"Settings > Resources > Advanced," and increase the memory limit. One of our team members has
-found **12 GB** to be sufficient for running the tests.
-
-> Dedicating 12 GB of RAM to Docker may be prohibitive for some prospective developers.
-> There is an open [issue](https://github.com/microbiomedata/nmdc-runtime/issues/928) about the memory requirement.
-
-## Publish to PyPI
-
-This repository contains a GitHub Actions workflow that publishes a Python package to [PyPI](https://pypi.org/project/nmdc-runtime/).
-
-You can also _manually_ publish the Python package to PyPI by issuing the following commands in the root directory of the repository:
-
-```
-rm -rf dist
-python -m build
-twine upload dist/*
-```
-
-## Links
-
-Here are links related to this repository:
-
-- Production API server: https://api.microbiomedata.org
-- PyPI package: https://pypi.org/project/nmdc-runtime
-- DockerHub image (API server): https://hub.docker.com/r/microbiomedata/nmdc-runtime-fastapi
-- DockerHub image (Dagster): https://hub.docker.com/r/microbiomedata/nmdc-runtime-dagster

nmdc_runtime-2.9.0.dist-info/RECORD

@@ -1,84 +0,0 @@
-nmdc_runtime/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/config.py,sha256=CW6LnN8Idsbra_mZnHU-kcWsYBZWbgivqVEp8rpOMi4,1989
-nmdc_runtime/containers.py,sha256=8m_S1wiFu8VOWvY7tyqzf-02X9gXY83YGc8FgjWzLGA,418
-nmdc_runtime/main.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/mongo_util.py,sha256=7NRvqFE8W2CUcpcXAA4KElUACIdAkBehZ9TBG4k7zNE,3000
-nmdc_runtime/util.py,sha256=Rw-OiQDHrz4cNX3ZdC-cgfHYUMq1qsk-_Mv81UrDlC8,19823
-nmdc_runtime/client/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/core/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/core/db/Database.py,sha256=WamgBUbq85A7-fr3p5B9Tk92U__yPdr9pBb4zyQok-4,377
-nmdc_runtime/core/db/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/core/exceptions/__init__.py,sha256=s486odD0uhUuk9K7M5_NISOgRrUE5RNnDJSypA2Qe_I,520
-nmdc_runtime/core/exceptions/base.py,sha256=G5azYv0FJvbjrpQtK90BkM-KK2f534szdwrHj9N-SNo,1343
-nmdc_runtime/core/exceptions/token.py,sha256=7iTdfRQjfijDExd6-KJBjN7t0BGI_Kc1F6Lc-d0AsE8,293
-nmdc_runtime/domain/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/domain/users/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/domain/users/queriesInterface.py,sha256=0DjOehnsA5oKADmRKh8NTool2zoQZaejFigXHuUGoOg,476
-nmdc_runtime/domain/users/userSchema.py,sha256=eVpsB5aSbT89XjPh2_m7ao8XyyinEC94hpZQIouV4uk,758
-nmdc_runtime/domain/users/userService.py,sha256=b-HD7N-wWQyAux_iZsXMBFrz5_j9ygRc3qsJlm-vQGI,428
-nmdc_runtime/infrastructure/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/infrastructure/database/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/infrastructure/database/db.py,sha256=djdqVxXvvJWtJUj4yariINcOuYOkQ_OiAYI_jGqOtM8,32
-nmdc_runtime/infrastructure/database/models/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/infrastructure/database/models/user.py,sha256=gOZVsQ9uZ_JlPKUcNOiJpj4_CQ9p2BpCaegcPiJQETs,188
-nmdc_runtime/lib/__init__.py,sha256=AbpHGcgLb-kRsJGnwFEktk7uzpZOCcBY74-YBdrKVGs,1
-nmdc_runtime/lib/extract_nmdc_data.py,sha256=xDFPoYsgkauN48R4v-tJIF0cP_p3J-sBjnyHd0InD9Y,1177
-nmdc_runtime/lib/load_nmdc_data.py,sha256=KO2cIqkY3cBCVcFIwsGokZNOKntOejZVG8ecq43NjFM,3934
-nmdc_runtime/lib/nmdc_dataframes.py,sha256=rVTczY2Jey1yE3x3nZ-RTgtdc2XkzLtKhB_PM3FIb-E,28849
-nmdc_runtime/lib/nmdc_etl_class.py,sha256=tVh3rKVMkBHQE65_LhKeIjCsaCZQk_HJzbc9K4xUNCs,13522
-nmdc_runtime/lib/transform_nmdc_data.py,sha256=hij4lR3IMQRJQdL-rsP_I-m_WyFPsBMchV2MNFUkh0M,39906
-nmdc_runtime/minter/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/minter/bootstrap.py,sha256=5Ej6pJVBRryRIi0ZwEloY78Zky7iE2okF6tPwRI2axM,822
-nmdc_runtime/minter/config.py,sha256=gsXZropDeeTO5tmLAtRuoocwqL3HgfgqVAENyCbX-Gc,2739
-nmdc_runtime/minter/adapters/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/minter/adapters/repository.py,sha256=I-jmGP38-9kPhkogrwUht_Ir0CfHA9_5ZImw5I_wbcw,8323
-nmdc_runtime/minter/domain/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/minter/domain/model.py,sha256=WMOuKub3dVzkOt_EZSRDLeTsJPqFbKx01SMQ53TOlDU,1416
-nmdc_runtime/minter/entrypoints/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/minter/entrypoints/fastapi_app.py,sha256=JC4thvzfFwRc1mhWQ-kHy3yvs0SYxF6ktE7LXNCwqlI,4031
-nmdc_runtime/site/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/site/graphs.py,sha256=CWbLLtoaakmNgSoaQWylXvcOY6qS7qwkTexEUDiMNfM,18295
-nmdc_runtime/site/ops.py,sha256=y6bBJhAytrSqt0COkOqXVKgfSGVdgQ7uByUP8S-zUB4,63935
-nmdc_runtime/site/repository.py,sha256=g0bZytvCrUjLpWuvkAzzmI16mChsrYPbWcvVFPNZFnM,47687
-nmdc_runtime/site/resources.py,sha256=dLNtNa4FfSKN_6b21eItn-i8e0ZHyveoBsexl2I6zmo,20144
-nmdc_runtime/site/util.py,sha256=h70UJCT9g-I63EJn0drZjv1iaQ8LHJTbG29R9kqJ04c,1821
-nmdc_runtime/site/backup/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/site/backup/nmdcdb_mongodump.py,sha256=H5uosmEiXwLwklJrYJWrNhb_Nuf_ew8dBpZLl6_dYhs,2699
-nmdc_runtime/site/backup/nmdcdb_mongoexport.py,sha256=XIFI_AI3zl0dFr-ELOEmwvT41MyRKBGFaAT3RcamTNE,4166
-nmdc_runtime/site/backup/nmdcdb_mongoimport.py,sha256=k6w5yscMNYoMBVkaAA9soWS0Dj2CB0FRBSFlifRO3Ro,1739
-nmdc_runtime/site/changesheets/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/site/changesheets/base.py,sha256=lZT6WCsEBl-FsTr7Ki8_ploT93uMiVyIWWKM36aOrRk,3171
-nmdc_runtime/site/drsobjects/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/site/drsobjects/ingest.py,sha256=pcMP69WSzFHFqHB9JIL55ePFhilnCLRc2XHCQ97w1Ik,3107
-nmdc_runtime/site/drsobjects/registration.py,sha256=D1T3QUuxEOxqKZIvB5rkb_6ZxFZiA-U9SMPajyeWC2Y,3572
-nmdc_runtime/site/export/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/site/export/ncbi_xml.py,sha256=iZQHBr3LL5Q32I2L_Xpfp9n4ZtgAz_MwrlxIF5do7Pw,29715
-nmdc_runtime/site/export/ncbi_xml_utils.py,sha256=RnoAW0HQwBG6JR63d9muI18RIC114wnX3iYPqOllw44,10700
-nmdc_runtime/site/export/study_metadata.py,sha256=yR5pXL6JG8d7cAtqcF-60Hp7bLD3dJ0Rut4AtYc0tXA,4844
-nmdc_runtime/site/normalization/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/site/normalization/gold.py,sha256=iISDD4qs4d6uLhv631WYNeQVOzY5DO201ZpPtxHdkVk,1311
-nmdc_runtime/site/repair/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/site/repair/database_updater.py,sha256=a6POYZcLEl0JvnuWxPjaOJtwZjkJhhvvUg1ABhnBiP8,21268
-nmdc_runtime/site/translation/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/site/translation/emsl.py,sha256=-aCTJTSCNaK-Koh8BE_4fTf5nyxP1KkquR6lloLEJl0,1245
-nmdc_runtime/site/translation/gold.py,sha256=R3W99sdQb7Pgu_esN7ruIC-tyREQD_idJ4xCzkqWuGw,1622
-nmdc_runtime/site/translation/gold_translator.py,sha256=n7PrAyZb6ODG1uaZ0cay91DygAHIefOL2qXLuukOyIM,33075
-nmdc_runtime/site/translation/jgi.py,sha256=qk878KhIw674TkrVfbl2x1QJrKi3zlvE0vesIpe9slM,876
-nmdc_runtime/site/translation/neon_benthic_translator.py,sha256=8_QF75Gf-dc2xVeO6jzTmdDrlGdh1-QrLJKG2SwUhCA,23797
-nmdc_runtime/site/translation/neon_soil_translator.py,sha256=IMeq4ABgWaSUbB_gmG8vBCMeynQSlbCUw9p2be6o8kE,38620
-nmdc_runtime/site/translation/neon_surface_water_translator.py,sha256=Js8_r6vHBW8b-_BpFySTUuYOFe7r51k8HwaNCQ7nAAg,30587
-nmdc_runtime/site/translation/neon_utils.py,sha256=d00o7duKKugpLHmsEifNbp4WjeC4GOqcgw0b5qlCg4I,5549
-nmdc_runtime/site/translation/submission_portal_translator.py,sha256=d5ycQhd-I07iUeuqN0vcHvMkOHqrwB67j2Q64aFkKBw,44147
-nmdc_runtime/site/translation/translator.py,sha256=V6Aq0y03LoQ4LTL2iHDHxGTh_eMjOmDJJSwNHSrp2wo,837
-nmdc_runtime/site/translation/util.py,sha256=w_l3SiExGsl6cXRqto0a_ssDmHkP64ITvrOVfPxmNpY,4366
-nmdc_runtime/site/validation/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/site/validation/emsl.py,sha256=OG20mv_3E2rkQqTQtYO0_SVRqFb-Z_zKCiAVbty6Wl0,671
-nmdc_runtime/site/validation/gold.py,sha256=Z5ZzYdjERbrJ2Tu06d0TDTBSfwaFdL1Z23Rl-YkZ2Ow,803
-nmdc_runtime/site/validation/jgi.py,sha256=LdJfhqBVHWCDp0Kzyk8eJZMwEI5NQ-zuTda31BcGwOA,1299
-nmdc_runtime/site/validation/util.py,sha256=GGbMDSwR090sr_E_fHffCN418gpYESaiot6XghS7OYk,3349
-nmdc_runtime-2.9.0.dist-info/licenses/LICENSE,sha256=VWiv65r7gHGjgtr3jMJYVmQny5GRpQ6H-W9sScb1x70,2408
-nmdc_runtime-2.9.0.dist-info/METADATA,sha256=4NgNI-Et3t1WLDfZPbSFT18JnMBVEuSCoFAZbm_V0xk,8953
-nmdc_runtime-2.9.0.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
-nmdc_runtime-2.9.0.dist-info/entry_points.txt,sha256=JxdvOnvxHK_8046cwlvE30s_fV0-k-eTpQtkKYA69nQ,224
-nmdc_runtime-2.9.0.dist-info/top_level.txt,sha256=b0K1s09L_iHH49ueBKaLrB5-lh6cyrSv9vL6x4Qvyz8,13
-nmdc_runtime-2.9.0.dist-info/RECORD,,

nmdc_runtime-2.9.0.dist-info/top_level.txt

@@ -1 +0,0 @@
-nmdc_runtime