lamindb 0.69.1__tar.gz → 0.69.3__tar.gz

{lamindb-0.69.1 → lamindb-0.69.3}/.github/workflows/build.yml

@@ -29,11 +29,9 @@ jobs:
         with:
           submodules: recursive
           fetch-depth: 0
-      - uses: actions/setup-python@v4
+      - uses: actions/setup-python@v5
         with:
-          python-version: "3.10"
-          cache: "pip"
-          cache-dependency-path: ".github/workflows/build.yml"
+          python-version: "3.11"
       - name: cache pre-commit
         uses: actions/cache@v3
         with:
@@ -100,11 +98,9 @@ jobs:
           aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
           aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
           aws-region: us-east-1
-      - uses: actions/setup-python@v4
+      - uses: actions/setup-python@v5
         with:
           python-version: "3.10"
-          cache: "pip"
-          cache-dependency-path: ".github/workflows/build.yml"
       - run: pip install -U laminci
       - run: nox -s "install(group='docs')"
       - uses: actions/download-artifact@v2
@@ -124,14 +120,13 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
-      - uses: actions/setup-python@v4
+      - uses: actions/setup-python@v5
         with:
-          python-version: 3.9
-          cache: "pip"
-          cache-dependency-path: ".github/workflows/build.yml"
+          python-version: "3.11"
       - run: |
-          pip install coverage[toml]
-          pip install --no-deps .
+          python -m pip install -U uv
+          uv pip install --system coverage[toml]
+          uv pip install --system --no-deps .
       - uses: actions/download-artifact@v2
       - name: run coverage
         run: |

{lamindb-0.69.1 → lamindb-0.69.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: lamindb
-Version: 0.69.1
+Version: 0.69.3
 Summary: A data framework for biology.
 Author-email: Lamin Labs <open-source@lamin.ai>
 Requires-Python: >=3.8
@@ -9,10 +9,10 @@ Classifier: Programming Language :: Python :: 3.8
 Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
-Requires-Dist: lnschema_core==0.64.0
-Requires-Dist: lamindb_setup==0.67.1
-Requires-Dist: lamin_utils==0.13.0
-Requires-Dist: lamin_cli==0.10.1
+Requires-Dist: lnschema_core==0.64.1
+Requires-Dist: lamindb_setup==0.68.1
+Requires-Dist: lamin_utils==0.13.1
+Requires-Dist: lamin_cli==0.10.2
 Requires-Dist: rapidfuzz
 Requires-Dist: pyarrow
 Requires-Dist: typing_extensions!=4.6.0
@@ -26,7 +26,7 @@ Requires-Dist: urllib3<2 ; extra == "aws"
 Requires-Dist: aiobotocore[boto3]>=2.5.4,<3.0.0 ; extra == "aws"
 Requires-Dist: s3fs==2023.12.2 ; extra == "aws"
 Requires-Dist: fsspec[s3]==2023.12.2 ; extra == "aws"
-Requires-Dist: bionty==0.42.2 ; extra == "bionty"
+Requires-Dist: bionty==0.42.4 ; extra == "bionty"
 Requires-Dist: pandas<2 ; extra == "dev"
 Requires-Dist: pre-commit ; extra == "dev"
 Requires-Dist: nox ; extra == "dev"
@@ -64,7 +64,7 @@ Provides-Extra: zarr
 - Track data lineage across notebooks & pipelines.
 - Integrate registries for experimental metadata & in-house ontologies.
 - Validate, standardize & annotate.
-- Collaborate across a mesh of distributed LaminDB instances.
+- Collaborate across distributed LaminDB instances.
 
 ## Documentation
 

{lamindb-0.69.1 → lamindb-0.69.3}/README.md

@@ -8,7 +8,7 @@
 - Track data lineage across notebooks & pipelines.
 - Integrate registries for experimental metadata & in-house ontologies.
 - Validate, standardize & annotate.
-- Collaborate across a mesh of distributed LaminDB instances.
+- Collaborate across distributed LaminDB instances.
 
 ## Documentation
 

lamindb-0.69.1/docs/annotate.ipynb → lamindb-0.69.3/docs/annotate-for-developers.ipynb

@@ -4,7 +4,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Annotate data\n",
+    "# Annotate data for developers\n",
     "\n",
     "While data is the primary information or raw facts that are collected and stored, metadata is the supporting information that provides context and meaning to that data.\n",
     "\n",

lamindb-0.69.3/docs/annotate.ipynb

@@ -0,0 +1,556 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "1d2cd769",
+   "metadata": {},
+   "source": [
+    "# Annotate data\n",
+    "\n",
+    "This guide shows how to annotate data, including defining clear validation criteria, validating & curating metadata, registering annotated artifacts, within a few minutes.\n",
+    "\n",
+    "By the end, you'll have validated data objects empowered by LaminDB registries.\n",
+    "\n",
+    ":::{dropdown} What does \"validating a categorical variable based on registries\" mean?\n",
+    "\n",
+    "The records in your LaminDB instance define the validated reference values for any entity managed in your schema.\n",
+    "\n",
+    "Validated categorical values are stored in a field of a registry; a column of the registry table.\n",
+    "\n",
+    "The default field to label an entity record is the `name` field.\n",
+    "\n",
+    "For instance, if \"Experiment 1\" has been registered as the `name` of a `ULabel` record, it is a validated value for field `ULabel.name`.\n",
+    "\n",
+    "{class}`~lamindb.core.CanValidate` methods {meth}`~lamindb.core.CanValidate.validate`, {meth}`~lamindb.core.CanValidate.inspect`, {meth}`~lamindb.core.CanValidate.standardize`, {meth}`~lamindb.core.Registry.from_values` take 2 important parameters: `values` and `field`. The parameter `values` takes an iterable of input categorical values, and the parameter `field` takes a typed field of a registry.\n",
+    "\n",
+    ":::\n",
+    "\n",
+    "```{toctree}\n",
+    ":maxdepth: 1\n",
+    ":hidden:\n",
+    "\n",
+    "can-validate\n",
+    "annotate-for-developers\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "553332e1",
+   "metadata": {},
+   "source": [
+    "## Set up"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "74008c61",
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "!lamin init --storage ./test-annotate --schema bionty"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a691aae0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import lamindb as ln\n",
+    "import bionty as bt\n",
+    "import pandas as pd\n",
+    "import anndata as ad\n",
+    "\n",
+    "ln.settings.verbosity = \"hint\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "65c71f2f",
+   "metadata": {},
+   "source": [
+    "## A DataFrame with labels"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "52bbbac2",
+   "metadata": {},
+   "source": [
+    "Let's start with a DataFrame object that we'd like to validate and curate:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6e313088",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df = pd.DataFrame({\n",
+    "    \"cell_type\": [\"cerebral pyramidal neuron\", \"astrocyte\", \"oligodendrocyte\"],\n",
+    "    \"assay_ontology_id\": [\"EFO:0008913\", \"EFO:0008913\", \"EFO:0008913\"],\n",
+    "    \"donor\": [\"D0001\", \"D0002\", \"DOOO3\"],\n",
+    "})\n",
+    "df"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "946a3371",
+   "metadata": {},
+   "source": [
+    "## Validate and curate metadata"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "abc3aafa",
+   "metadata": {},
+   "source": [
+    "Define validation criteria for the columns:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cfb3492d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "fields = {\n",
+    "    \"cell_type\": bt.CellType.name,\n",
+    "    \"assay_ontology_id\": bt.ExperimentalFactor.ontology_id,\n",
+    "    \"donor\": ln.ULabel.name,\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "acdb45d8",
+   "metadata": {},
+   "source": [
+    "Validate the Pandas DataFrame:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "15137652",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "annotate = ln.Annotate.from_df(df, fields=fields)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "24e00633",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "validated = annotate.validate()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f75d3f15",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "validated"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "df078eb3",
+   "metadata": {},
+   "source": [
+    "## Validate using registries in another instance\n",
+    "\n",
+    "Sometimes you want to validate against existing registries others might have created.\n",
+    "\n",
+    "Here we use the [cellxgene instance](https://lamin.ai/laminlabs/cellxgene) registries to curate against. You will notice more terms are validated than above.\n",
+    "\n",
+    "This allows us to register values that are currently missing in our instance from the [cellxgene instance](https://lamin.ai/laminlabs/cellxgene) directly.\n",
+    "By having our own registry but also validating against the [cellxgene instance](https://lamin.ai/laminlabs/cellxgene), we enable the addition of new registry values while keeping the [cellxgene instance](https://lamin.ai/laminlabs/cellxgene) focused on the [cellxgene schema](https://github.com/chanzuckerberg/single-cell-curation/tree/main/schema)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a230b1e9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "annotate = ln.Annotate.from_df(\n",
+    "    df, \n",
+    "    fields=fields, \n",
+    "    using=\"laminlabs/cellxgene\",  # pass the instance slug\n",
+    "    )\n",
+    "annotate.validate()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7c157df6",
+   "metadata": {},
+   "source": [
+    "## Register new metadata labels\n",
+    "\n",
+    "Following the suggestions above to register labels that aren't present in the current instance:\n",
+    "\n",
+    "(Note that our current instance is empty. Once you filled up the registries, registering new labels won't be frequently needed)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "943e0e7a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "annotate.update_registry(\"cell_type\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2dcd5a3d",
+   "metadata": {},
+   "source": [
+    "Fix typo and register again:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ea6b82b8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# use a lookup object to get the correct spelling of categories from public reference\n",
+    "# pass \"public\" to use the public reference\n",
+    "lookup = annotate.lookup()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "24a56369",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "lookup"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4c98914a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "cell_types = lookup[\"cell_type\"]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c49dc7eb",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "cell_types.cerebral_cortex_pyramidal_neuron"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b06c7875",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# fix the typo\n",
+    "df[\"cell_type\"] = df[\"cell_type\"].replace({\"cerebral pyramidal neuron\": cell_types.cerebral_cortex_pyramidal_neuron.name})\n",
+    "\n",
+    "annotate.update_registry(\"cell_type\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "13cec5d4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "annotate.update_registry(\"donor\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b182af1c",
+   "metadata": {},
+   "source": [
+    "To register non-validated terms, pass `validated_only=False`:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b393accf",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "annotate.update_registry(\"donor\", validated_only=False)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1f3cd3a9",
+   "metadata": {},
+   "source": [
+    "Let's validate it again:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1a90599b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "validated = annotate.validate()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "55b9b30e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "validated"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b9d09a10",
+   "metadata": {},
+   "source": [
+    "## Validate an AnnData object\n",
+    "\n",
+    "We offer an AnnData specific annotate that is aware of the variables in addition to the observations DataFrame.\n",
+    "\n",
+    "Here we specify which `var_fields` and `obs_fields` to validate against."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "15d7b858",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df.index = [\"obs1\", \"obs2\", \"obs3\"]\n",
+    "\n",
+    "X = pd.DataFrame({\"TCF7\": [1, 2, 3], \"PDCD1\": [4, 5, 6], \"CD3E\": [7, 8, 9], \"CD4\": [10, 11, 12], \"CD8A\": [13, 14, 15]}, index=[\"obs1\", \"obs2\", \"obs3\"])\n",
+    "\n",
+    "adata = ad.AnnData(X=X, obs=df)\n",
+    "adata"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "19e7b8ca",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "annotate = ln.Annotate.from_anndata(\n",
+    "    adata, \n",
+    "    obs_fields=fields, \n",
+    "    var_field=bt.Gene.symbol, # specify the field for the var\n",
+    "    organism=\"human\",\n",
+    "    )"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e8d8cee5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "annotate.validate()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7877c058",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "annotate.update_registry(\"all\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0e139bda",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "annotate.validate()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "38a30170",
+   "metadata": {},
+   "source": [
+    "## Register file"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a814ef37",
+   "metadata": {},
+   "source": [
+    "The validated object can be subsequently registered as an {class}`~lamindb.Artifact` in your LaminDB instance:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a70fe66d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ln.transform.stem_uid = \"WOK3vP0bNGLx\"\n",
+    "ln.transform.version = \"0\"\n",
+    "ln.track()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f12ec345",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "artifact = annotate.register_artifact(description=\"test AnnData\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4fc354e4",
+   "metadata": {},
+   "source": [
+    "View the registered artifact with metadata:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5e901343",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "artifact.describe()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1aa82222",
+   "metadata": {},
+   "source": [
+    "## Register collection"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3cf67f90",
+   "metadata": {},
+   "source": [
+    "Register a new collection for the registered artifact:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8940e83c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# register a new collection\n",
+    "collection = annotate.register_collection(\n",
+    "    artifact,  # registered artifact above, can also pass a list of artifacts\n",
+    "    name=\"Experiment X in brain\",  # title of the publication\n",
+    "    description=\"10.1126/science.xxxxx\",  # DOI of the publication\n",
+    "    reference=\"E-MTAB-xxxxx\", # accession number (e.g. GSE#, E-MTAB#, etc.)\n",
+    "    reference_type=\"ArrayExpress\") # source type (e.g. GEO, ArrayExpress, SRA, etc.)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "867e52b8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "collection.artifact"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6e963036",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "artifact.collection"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.13"
+  },
+  "nbproject": {
+   "id": "WOK3vP0bNGLx",
+   "parent": null,
+   "pypackage": null,
+   "time_init": "2023-12-20T04:27:49.536493+00:00",
+   "user_handle": null,
+   "user_id": null,
+   "user_name": null,
+   "version": "0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}