nmdc-runtime 2.6.0__py3-none-any.whl → 2.12.0__py3-none-any.whl

nmdc_runtime-2.6.0.dist-info/METADATA

@@ -1,199 +0,0 @@
-Metadata-Version: 2.4
-Name: nmdc_runtime
-Version: 2.6.0
-Summary: A runtime system for NMDC data management and orchestration
-Home-page: https://github.com/microbiomedata/nmdc-runtime
-Author: Donny Winston
-Author-email: donny@polyneme.xyz
-Classifier: Development Status :: 3 - Alpha
-Classifier: Programming Language :: Python :: 3
-Classifier: License :: OSI Approved :: Apache Software License
-Requires-Python: >=3.10
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Dynamic: author
-Dynamic: author-email
-Dynamic: classifier
-Dynamic: description
-Dynamic: description-content-type
-Dynamic: home-page
-Dynamic: license-file
-Dynamic: requires-python
-Dynamic: summary
-
-A runtime system for NMDC data management and orchestration.
-
-## Service Status
-
-http://nmdcstatus.polyneme.xyz/
-
-## How It Fits In
-
-* [issues](https://github.com/microbiomedata/issues)  
-tracks issues related to NMDC, which may necessitate work across multiple repos.
-  
-* [nmdc-schema](https://github.com/microbiomedata/nmdc-schema/)
-houses the LinkML schema specification, as well as generated artifacts (e.g. JSON Schema).
-
-* [nmdc-server](https://github.com/microbiomedata/nmdc-server)
-houses code specific to the data portal -- its database, back-end API, and front-end application.
-
-* Workflows — documented in the [workflows](https://docs.microbiomedata.org/workflows/) section of the NMDC documentation website — take source data and produce computed data.
-
-* This repo (nmdc-runtime)
-   * houses code that takes source data and computed data, and transforms it
-     to broadly accommodate downstream applications such as the data portal
-   * manages execution of the above (i.e., lightweight data transformations) and also
-     of computationally- and data-intensive workflows performed at other sites,
-     ensuring that claimed jobs have access to needed configuration and data resources.
-
-## Data exports
-
-The NMDC metadata as of 2021-10 is available here:
-
-https://drs.microbiomedata.org/ga4gh/drs/v1/objects/sys086d541
-
-The link returns a [GA4GH DRS API bundle object record](https://ga4gh.github.io/data-repository-service-schemas/preview/release/drs-1.0.0/docs/#_drs_datatypes), with the NMDC metadata collections (study_set, biosample_set, etc.) as contents, each a DRS API blob object.
-
-For example the blob for the study_set collection export, named "study_set.jsonl.gz", is listed with DRS API ID "sys0xsry70". Thus, it is retrievable via
-
-https://drs.microbiomedata.org/ga4gh/drs/v1/objects/sys0xsry70
-
-The returned blob object record lists https://nmdc-runtime.files.polyneme.xyz/nmdcdb-mongoexport/2021-10-14/study_set.jsonl.gz as the url for an access method.
-
-The 2021-10 exports are currently all accessible at `https://nmdc-runtime.files.polyneme.xyz/nmdcdb-mongoexport/2021-10-14/${COLLECTION_NAME}.jsonl.gz`, but the DRS API indirection allows these links to change in the future, for mirroring via other URLs, etc. So, the DRS API links should be the links you share.
-
-## Overview
-
-The runtime features:
-
-1. [Dagster](https://docs.dagster.io/concepts) orchestration:
-    - dagit - a web UI to monitor and manage the running system.
-    - dagster-daemon - a service that triggers pipeline runs based on time or external state.
-    - PostgresSQL database - for storing run history, event logs, and scheduler state.
-    - workspace code
-      - Code to run is loaded into a Dagster `workspace`. This code is loaded from
-         one or more dagster `repositories`. Each Dagster `repository` may be run with a different
-         Python virtual environment if need be, and may be loaded from a local Python file or
-         `pip install`ed from an external source. In our case, each Dagster `repository` is simply
-         loaded from a Python file local to the nmdc-runtime GitHub repository, and all code is
-         run in the same Python environment.
-      - A Dagster repository consists of `solids` and `pipelines`,
-         and optionally `schedules` and `sensors`.
-         - `solids` represent individual units of computation
-         - `pipelines` are built up from solids
-         - `schedules` trigger recurring pipeline runs based on time
-         - `sensors` trigger pipeline runs based on external state
-      - Each `pipeline` can declare dependencies on any runtime `resources` or additional
-         configuration. There are MongoDB `resources` defined, as well as `preset`
-         configuration definitions for both "dev" and "prod" `modes`. The `preset`s tell Dagster to
-         look to a set of known environment variables to load resources configurations, depending on
-         the `mode`.
-   
-2. A MongoDB database supporting write-once, high-throughput internal
-data storage by the nmdc-runtime FastAPI instance.
-   
-3. A [FastAPI](https://fastapi.tiangolo.com/) service to interface with the orchestrator and
-database, as a hub for data management and workflow automation.
-
-## Local Development
-
-Ensure Docker (and Docker Compose) are installed; and the Docker engine is running.
-
-```shell
-docker --version
-docker compose version
-docker info
-```
-
-Ensure the permissions of `./.docker/mongoKeyFile` are such that only the file's owner can read or write the file.
-
-```shell
-chmod 600 ./.docker/mongoKeyFile
-```
-
-Ensure you have a `.env` file for the Docker services to source from. You may copy `.env.example` to
-`.env` (which is gitignore'd) to get started.
-
-```shell
-cp .env.example .env
-```
-
-Create environment variables in your shell session, based upon the contents of the `.env` file.
-
-```shell
-set -a # automatically export all variables
-source .env
-set +a
-```
-
-If you are connecting to resources that require an SSH tunnel—for example, a MongoDB server that is only accessible on the NERSC network—set up the SSH tunnel.
-
-The following command could be useful to you, either directly or as a template (see `Makefile`).
-
-```shell
-make nersc-mongo-tunnels
-```
-
-Finally, spin up the Docker Compose stack.
-
-```bash
-make up-dev
-```
-
-Docker Compose is used to start local MongoDB and PostgresSQL (used by Dagster) instances, as well
-as a Dagster web server (dagit) and daemon (dagster-daemon).
-
-The Dagit web server is viewable at http://127.0.0.1:3000/. 
-
-The FastAPI service is viewable at http://127.0.0.1:8000/ -- e.g., rendered documentation at
-http://127.0.0.1:8000/redoc/.
-
-## Local Testing
-
-Tests can be found in `tests` and are run with the following commands:
-
-```bash
-make up-test
-make test
-
-# Run a Specific test file eg. tests/test_api/test_endpoints.py
-make test ARGS="tests/test_api/test_endpoints.py"
-```
-
-As you create Dagster solids and pipelines, add tests in `tests/` to check that your code behaves as
-desired and does not break over time.
-
-[For hints on how to write tests for solids and pipelines in Dagster, see their documentation
-tutorial on Testing](https://docs.dagster.io/tutorial/testable).
-
-### RAM usage
-
-The `dagster-daemon` and `dagster-dagit` containers can consume a lot of RAM. If tests are failing and the console of
-the `test` container shows "Error 137," here is something you can try as a workaround: In Docker Desktop, go to 
-"Settings > Resources > Advanced," and increase the memory limit. One of our team members has
-found **12 GB** to be sufficient for running the tests.
-
-> Dedicating 12 GB of RAM to Docker may be prohibitive for some prospective developers.
-> There is an open [issue](https://github.com/microbiomedata/nmdc-runtime/issues/928) about the memory requirement.
-
-## Publish to PyPI
-
-This repository contains a GitHub Actions workflow that publishes a Python package to [PyPI](https://pypi.org/project/nmdc-runtime/).
-
-You can also _manually_ publish the Python package to PyPI by issuing the following commands in the root directory of the repository:
-
-```
-rm -rf dist
-python -m build
-twine upload dist/*
-```
-
-## Links
-
-Here are links related to this repository:
-
-- Production API server: https://api.microbiomedata.org
-- PyPI package: https://pypi.org/project/nmdc-runtime
-- DockerHub image (API server): https://hub.docker.com/r/microbiomedata/nmdc-runtime-fastapi
-- DockerHub image (Dagster): https://hub.docker.com/r/microbiomedata/nmdc-runtime-dagster

nmdc_runtime-2.6.0.dist-info/RECORD

@@ -1,83 +0,0 @@
-nmdc_runtime/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/config.py,sha256=qyV_To6t--DQUpYJ3SrE6sZlxuVXLPmx2dVtZV-3l-c,33
-nmdc_runtime/containers.py,sha256=8m_S1wiFu8VOWvY7tyqzf-02X9gXY83YGc8FgjWzLGA,418
-nmdc_runtime/main.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/util.py,sha256=HzQsNMYG6Pb-IuBEE9HBzX_lNkII7jiNe65UFk34ZYA,31414
-nmdc_runtime/client/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/core/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/core/db/Database.py,sha256=WamgBUbq85A7-fr3p5B9Tk92U__yPdr9pBb4zyQok-4,377
-nmdc_runtime/core/db/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/core/exceptions/__init__.py,sha256=s486odD0uhUuk9K7M5_NISOgRrUE5RNnDJSypA2Qe_I,520
-nmdc_runtime/core/exceptions/base.py,sha256=G5azYv0FJvbjrpQtK90BkM-KK2f534szdwrHj9N-SNo,1343
-nmdc_runtime/core/exceptions/token.py,sha256=7iTdfRQjfijDExd6-KJBjN7t0BGI_Kc1F6Lc-d0AsE8,293
-nmdc_runtime/domain/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/domain/users/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/domain/users/queriesInterface.py,sha256=0DjOehnsA5oKADmRKh8NTool2zoQZaejFigXHuUGoOg,476
-nmdc_runtime/domain/users/userSchema.py,sha256=eVpsB5aSbT89XjPh2_m7ao8XyyinEC94hpZQIouV4uk,758
-nmdc_runtime/domain/users/userService.py,sha256=b-HD7N-wWQyAux_iZsXMBFrz5_j9ygRc3qsJlm-vQGI,428
-nmdc_runtime/infrastructure/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/infrastructure/database/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/infrastructure/database/db.py,sha256=djdqVxXvvJWtJUj4yariINcOuYOkQ_OiAYI_jGqOtM8,32
-nmdc_runtime/infrastructure/database/models/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/infrastructure/database/models/user.py,sha256=gOZVsQ9uZ_JlPKUcNOiJpj4_CQ9p2BpCaegcPiJQETs,188
-nmdc_runtime/lib/__init__.py,sha256=AbpHGcgLb-kRsJGnwFEktk7uzpZOCcBY74-YBdrKVGs,1
-nmdc_runtime/lib/extract_nmdc_data.py,sha256=xDFPoYsgkauN48R4v-tJIF0cP_p3J-sBjnyHd0InD9Y,1177
-nmdc_runtime/lib/load_nmdc_data.py,sha256=KO2cIqkY3cBCVcFIwsGokZNOKntOejZVG8ecq43NjFM,3934
-nmdc_runtime/lib/nmdc_dataframes.py,sha256=rVTczY2Jey1yE3x3nZ-RTgtdc2XkzLtKhB_PM3FIb-E,28849
-nmdc_runtime/lib/nmdc_etl_class.py,sha256=tVh3rKVMkBHQE65_LhKeIjCsaCZQk_HJzbc9K4xUNCs,13522
-nmdc_runtime/lib/transform_nmdc_data.py,sha256=hij4lR3IMQRJQdL-rsP_I-m_WyFPsBMchV2MNFUkh0M,39906
-nmdc_runtime/minter/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/minter/bootstrap.py,sha256=5Ej6pJVBRryRIi0ZwEloY78Zky7iE2okF6tPwRI2axM,822
-nmdc_runtime/minter/config.py,sha256=gsXZropDeeTO5tmLAtRuoocwqL3HgfgqVAENyCbX-Gc,2739
-nmdc_runtime/minter/adapters/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/minter/adapters/repository.py,sha256=I-jmGP38-9kPhkogrwUht_Ir0CfHA9_5ZImw5I_wbcw,8323
-nmdc_runtime/minter/domain/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/minter/domain/model.py,sha256=WMOuKub3dVzkOt_EZSRDLeTsJPqFbKx01SMQ53TOlDU,1416
-nmdc_runtime/minter/entrypoints/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/minter/entrypoints/fastapi_app.py,sha256=JC4thvzfFwRc1mhWQ-kHy3yvs0SYxF6ktE7LXNCwqlI,4031
-nmdc_runtime/site/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/site/graphs.py,sha256=DoKK6B6xkSwRcY5PVVo6jV_IA4HI5qL8xW9_n94jVfQ,15990
-nmdc_runtime/site/ops.py,sha256=atZNkU5mzRRqTnaW39fvq7gVO2sKSH8ztVOp8_dOLbU,48048
-nmdc_runtime/site/repository.py,sha256=nHu1skayyTjJWwGEf5eToX02cgBNTG_kdSluzJZ6rJc,43695
-nmdc_runtime/site/resources.py,sha256=sqtRWb4ewU61U-JZTphsC4wBvYT5B0wj33WU70vjq_k,19677
-nmdc_runtime/site/util.py,sha256=h70UJCT9g-I63EJn0drZjv1iaQ8LHJTbG29R9kqJ04c,1821
-nmdc_runtime/site/backup/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/site/backup/nmdcdb_mongodump.py,sha256=H5uosmEiXwLwklJrYJWrNhb_Nuf_ew8dBpZLl6_dYhs,2699
-nmdc_runtime/site/backup/nmdcdb_mongoexport.py,sha256=XIFI_AI3zl0dFr-ELOEmwvT41MyRKBGFaAT3RcamTNE,4166
-nmdc_runtime/site/backup/nmdcdb_mongoimport.py,sha256=k6w5yscMNYoMBVkaAA9soWS0Dj2CB0FRBSFlifRO3Ro,1739
-nmdc_runtime/site/changesheets/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/site/changesheets/base.py,sha256=lZT6WCsEBl-FsTr7Ki8_ploT93uMiVyIWWKM36aOrRk,3171
-nmdc_runtime/site/drsobjects/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/site/drsobjects/ingest.py,sha256=pcMP69WSzFHFqHB9JIL55ePFhilnCLRc2XHCQ97w1Ik,3107
-nmdc_runtime/site/drsobjects/registration.py,sha256=D1T3QUuxEOxqKZIvB5rkb_6ZxFZiA-U9SMPajyeWC2Y,3572
-nmdc_runtime/site/export/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/site/export/ncbi_xml.py,sha256=4RqaT6qs1LDSiDDfF-JNZL5gOel8m65oCOelfr0blXs,26209
-nmdc_runtime/site/export/ncbi_xml_utils.py,sha256=X35zbkxBxEyCnA9peY9YBAa_0oeoWy3DQEXoAXmc6vg,10100
-nmdc_runtime/site/export/study_metadata.py,sha256=yR5pXL6JG8d7cAtqcF-60Hp7bLD3dJ0Rut4AtYc0tXA,4844
-nmdc_runtime/site/normalization/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/site/normalization/gold.py,sha256=iISDD4qs4d6uLhv631WYNeQVOzY5DO201ZpPtxHdkVk,1311
-nmdc_runtime/site/repair/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/site/repair/database_updater.py,sha256=eTNAPtgAc_xQodADBfgomwow9-14j5rBqQWF8R7BheY,11525
-nmdc_runtime/site/translation/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/site/translation/emsl.py,sha256=-aCTJTSCNaK-Koh8BE_4fTf5nyxP1KkquR6lloLEJl0,1245
-nmdc_runtime/site/translation/gold.py,sha256=R3W99sdQb7Pgu_esN7ruIC-tyREQD_idJ4xCzkqWuGw,1622
-nmdc_runtime/site/translation/gold_translator.py,sha256=HGbWeuxppqlVfU8F5oKTYIDoC6qaftugJeWFIALB9XE,32720
-nmdc_runtime/site/translation/jgi.py,sha256=qk878KhIw674TkrVfbl2x1QJrKi3zlvE0vesIpe9slM,876
-nmdc_runtime/site/translation/neon_benthic_translator.py,sha256=VxN7yCziQE-ZP9mtrzqI-yaS9taEgTy0EnIEattYeKo,23727
-nmdc_runtime/site/translation/neon_soil_translator.py,sha256=Rol0g67nVBGSBySUzpfdW4Fwes7bKtvnlv2g5cB0aTI,38550
-nmdc_runtime/site/translation/neon_surface_water_translator.py,sha256=k06eULMTYx0sQ00UlyeNJvCJMcX-neClnES1G6zpPKg,30517
-nmdc_runtime/site/translation/neon_utils.py,sha256=d00o7duKKugpLHmsEifNbp4WjeC4GOqcgw0b5qlCg4I,5549
-nmdc_runtime/site/translation/submission_portal_translator.py,sha256=UEeqlkz_YGqcnx8vomFysetOlXxDu23q0Ryr93SZy78,41684
-nmdc_runtime/site/translation/translator.py,sha256=V6Aq0y03LoQ4LTL2iHDHxGTh_eMjOmDJJSwNHSrp2wo,837
-nmdc_runtime/site/translation/util.py,sha256=w_l3SiExGsl6cXRqto0a_ssDmHkP64ITvrOVfPxmNpY,4366
-nmdc_runtime/site/validation/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
-nmdc_runtime/site/validation/emsl.py,sha256=OG20mv_3E2rkQqTQtYO0_SVRqFb-Z_zKCiAVbty6Wl0,671
-nmdc_runtime/site/validation/gold.py,sha256=Z5ZzYdjERbrJ2Tu06d0TDTBSfwaFdL1Z23Rl-YkZ2Ow,803
-nmdc_runtime/site/validation/jgi.py,sha256=LdJfhqBVHWCDp0Kzyk8eJZMwEI5NQ-zuTda31BcGwOA,1299
-nmdc_runtime/site/validation/util.py,sha256=GGbMDSwR090sr_E_fHffCN418gpYESaiot6XghS7OYk,3349
-nmdc_runtime-2.6.0.dist-info/licenses/LICENSE,sha256=VWiv65r7gHGjgtr3jMJYVmQny5GRpQ6H-W9sScb1x70,2408
-nmdc_runtime-2.6.0.dist-info/METADATA,sha256=RK075FB7BHmZL0fm3elHYjBV7YB4hsG-pQRHfgCUm0g,8155
-nmdc_runtime-2.6.0.dist-info/WHEEL,sha256=ck4Vq1_RXyvS4Jt6SI0Vz6fyVs4GWg7AINwpsaGEgPE,91
-nmdc_runtime-2.6.0.dist-info/entry_points.txt,sha256=JxdvOnvxHK_8046cwlvE30s_fV0-k-eTpQtkKYA69nQ,224
-nmdc_runtime-2.6.0.dist-info/top_level.txt,sha256=b0K1s09L_iHH49ueBKaLrB5-lh6cyrSv9vL6x4Qvyz8,13
-nmdc_runtime-2.6.0.dist-info/RECORD,,

nmdc_runtime-2.6.0.dist-info/top_level.txt

@@ -1 +0,0 @@
-nmdc_runtime