lamindb 0.69.2__tar.gz → 0.69.3__tar.gz

{lamindb-0.69.2 → lamindb-0.69.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: lamindb
-Version: 0.69.2
+Version: 0.69.3
 Summary: A data framework for biology.
 Author-email: Lamin Labs <open-source@lamin.ai>
 Requires-Python: >=3.8
@@ -10,7 +10,7 @@ Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Requires-Dist: lnschema_core==0.64.1
-Requires-Dist: lamindb_setup==0.68.0
+Requires-Dist: lamindb_setup==0.68.1
 Requires-Dist: lamin_utils==0.13.1
 Requires-Dist: lamin_cli==0.10.2
 Requires-Dist: rapidfuzz
@@ -26,7 +26,7 @@ Requires-Dist: urllib3<2 ; extra == "aws"
 Requires-Dist: aiobotocore[boto3]>=2.5.4,<3.0.0 ; extra == "aws"
 Requires-Dist: s3fs==2023.12.2 ; extra == "aws"
 Requires-Dist: fsspec[s3]==2023.12.2 ; extra == "aws"
-Requires-Dist: bionty==0.42.2 ; extra == "bionty"
+Requires-Dist: bionty==0.42.4 ; extra == "bionty"
 Requires-Dist: pandas<2 ; extra == "dev"
 Requires-Dist: pre-commit ; extra == "dev"
 Requires-Dist: nox ; extra == "dev"

lamindb-0.69.2/docs/annotate.ipynb → lamindb-0.69.3/docs/annotate-for-developers.ipynb

@@ -4,7 +4,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Annotate data\n",
+    "# Annotate data for developers\n",
     "\n",
     "While data is the primary information or raw facts that are collected and stored, metadata is the supporting information that provides context and meaning to that data.\n",
     "\n",

lamindb-0.69.3/docs/annotate.ipynb

@@ -0,0 +1,556 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "1d2cd769",
+   "metadata": {},
+   "source": [
+    "# Annotate data\n",
+    "\n",
+    "This guide shows how to annotate data, including defining clear validation criteria, validating & curating metadata, registering annotated artifacts, within a few minutes.\n",
+    "\n",
+    "By the end, you'll have validated data objects empowered by LaminDB registries.\n",
+    "\n",
+    ":::{dropdown} What does \"validating a categorical variable based on registries\" mean?\n",
+    "\n",
+    "The records in your LaminDB instance define the validated reference values for any entity managed in your schema.\n",
+    "\n",
+    "Validated categorical values are stored in a field of a registry; a column of the registry table.\n",
+    "\n",
+    "The default field to label an entity record is the `name` field.\n",
+    "\n",
+    "For instance, if \"Experiment 1\" has been registered as the `name` of a `ULabel` record, it is a validated value for field `ULabel.name`.\n",
+    "\n",
+    "{class}`~lamindb.core.CanValidate` methods {meth}`~lamindb.core.CanValidate.validate`, {meth}`~lamindb.core.CanValidate.inspect`, {meth}`~lamindb.core.CanValidate.standardize`, {meth}`~lamindb.core.Registry.from_values` take 2 important parameters: `values` and `field`. The parameter `values` takes an iterable of input categorical values, and the parameter `field` takes a typed field of a registry.\n",
+    "\n",
+    ":::\n",
+    "\n",
+    "```{toctree}\n",
+    ":maxdepth: 1\n",
+    ":hidden:\n",
+    "\n",
+    "can-validate\n",
+    "annotate-for-developers\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "553332e1",
+   "metadata": {},
+   "source": [
+    "## Set up"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "74008c61",
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "!lamin init --storage ./test-annotate --schema bionty"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a691aae0",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import lamindb as ln\n",
+    "import bionty as bt\n",
+    "import pandas as pd\n",
+    "import anndata as ad\n",
+    "\n",
+    "ln.settings.verbosity = \"hint\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "65c71f2f",
+   "metadata": {},
+   "source": [
+    "## A DataFrame with labels"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "52bbbac2",
+   "metadata": {},
+   "source": [
+    "Let's start with a DataFrame object that we'd like to validate and curate:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6e313088",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df = pd.DataFrame({\n",
+    "    \"cell_type\": [\"cerebral pyramidal neuron\", \"astrocyte\", \"oligodendrocyte\"],\n",
+    "    \"assay_ontology_id\": [\"EFO:0008913\", \"EFO:0008913\", \"EFO:0008913\"],\n",
+    "    \"donor\": [\"D0001\", \"D0002\", \"DOOO3\"],\n",
+    "})\n",
+    "df"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "946a3371",
+   "metadata": {},
+   "source": [
+    "## Validate and curate metadata"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "abc3aafa",
+   "metadata": {},
+   "source": [
+    "Define validation criteria for the columns:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cfb3492d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "fields = {\n",
+    "    \"cell_type\": bt.CellType.name,\n",
+    "    \"assay_ontology_id\": bt.ExperimentalFactor.ontology_id,\n",
+    "    \"donor\": ln.ULabel.name,\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "acdb45d8",
+   "metadata": {},
+   "source": [
+    "Validate the Pandas DataFrame:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "15137652",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "annotate = ln.Annotate.from_df(df, fields=fields)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "24e00633",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "validated = annotate.validate()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f75d3f15",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "validated"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "df078eb3",
+   "metadata": {},
+   "source": [
+    "## Validate using registries in another instance\n",
+    "\n",
+    "Sometimes you want to validate against existing registries others might have created.\n",
+    "\n",
+    "Here we use the [cellxgene instance](https://lamin.ai/laminlabs/cellxgene) registries to curate against. You will notice more terms are validated than above.\n",
+    "\n",
+    "This allows us to register values that are currently missing in our instance from the [cellxgene instance](https://lamin.ai/laminlabs/cellxgene) directly.\n",
+    "By having our own registry but also validating against the [cellxgene instance](https://lamin.ai/laminlabs/cellxgene), we enable the addition of new registry values while keeping the [cellxgene instance](https://lamin.ai/laminlabs/cellxgene) focused on the [cellxgene schema](https://github.com/chanzuckerberg/single-cell-curation/tree/main/schema)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a230b1e9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "annotate = ln.Annotate.from_df(\n",
+    "    df, \n",
+    "    fields=fields, \n",
+    "    using=\"laminlabs/cellxgene\",  # pass the instance slug\n",
+    "    )\n",
+    "annotate.validate()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7c157df6",
+   "metadata": {},
+   "source": [
+    "## Register new metadata labels\n",
+    "\n",
+    "Following the suggestions above to register labels that aren't present in the current instance:\n",
+    "\n",
+    "(Note that our current instance is empty. Once you filled up the registries, registering new labels won't be frequently needed)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "943e0e7a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "annotate.update_registry(\"cell_type\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2dcd5a3d",
+   "metadata": {},
+   "source": [
+    "Fix typo and register again:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ea6b82b8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# use a lookup object to get the correct spelling of categories from public reference\n",
+    "# pass \"public\" to use the public reference\n",
+    "lookup = annotate.lookup()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "24a56369",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "lookup"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "4c98914a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "cell_types = lookup[\"cell_type\"]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c49dc7eb",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "cell_types.cerebral_cortex_pyramidal_neuron"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b06c7875",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# fix the typo\n",
+    "df[\"cell_type\"] = df[\"cell_type\"].replace({\"cerebral pyramidal neuron\": cell_types.cerebral_cortex_pyramidal_neuron.name})\n",
+    "\n",
+    "annotate.update_registry(\"cell_type\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "13cec5d4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "annotate.update_registry(\"donor\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b182af1c",
+   "metadata": {},
+   "source": [
+    "To register non-validated terms, pass `validated_only=False`:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b393accf",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "annotate.update_registry(\"donor\", validated_only=False)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1f3cd3a9",
+   "metadata": {},
+   "source": [
+    "Let's validate it again:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "1a90599b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "validated = annotate.validate()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "55b9b30e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "validated"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b9d09a10",
+   "metadata": {},
+   "source": [
+    "## Validate an AnnData object\n",
+    "\n",
+    "We offer an AnnData specific annotate that is aware of the variables in addition to the observations DataFrame.\n",
+    "\n",
+    "Here we specify which `var_fields` and `obs_fields` to validate against."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "15d7b858",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df.index = [\"obs1\", \"obs2\", \"obs3\"]\n",
+    "\n",
+    "X = pd.DataFrame({\"TCF7\": [1, 2, 3], \"PDCD1\": [4, 5, 6], \"CD3E\": [7, 8, 9], \"CD4\": [10, 11, 12], \"CD8A\": [13, 14, 15]}, index=[\"obs1\", \"obs2\", \"obs3\"])\n",
+    "\n",
+    "adata = ad.AnnData(X=X, obs=df)\n",
+    "adata"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "19e7b8ca",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "annotate = ln.Annotate.from_anndata(\n",
+    "    adata, \n",
+    "    obs_fields=fields, \n",
+    "    var_field=bt.Gene.symbol, # specify the field for the var\n",
+    "    organism=\"human\",\n",
+    "    )"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e8d8cee5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "annotate.validate()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "7877c058",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "annotate.update_registry(\"all\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0e139bda",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "annotate.validate()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "38a30170",
+   "metadata": {},
+   "source": [
+    "## Register file"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a814ef37",
+   "metadata": {},
+   "source": [
+    "The validated object can be subsequently registered as an {class}`~lamindb.Artifact` in your LaminDB instance:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "a70fe66d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ln.transform.stem_uid = \"WOK3vP0bNGLx\"\n",
+    "ln.transform.version = \"0\"\n",
+    "ln.track()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f12ec345",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "artifact = annotate.register_artifact(description=\"test AnnData\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "4fc354e4",
+   "metadata": {},
+   "source": [
+    "View the registered artifact with metadata:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5e901343",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "artifact.describe()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1aa82222",
+   "metadata": {},
+   "source": [
+    "## Register collection"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3cf67f90",
+   "metadata": {},
+   "source": [
+    "Register a new collection for the registered artifact:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "8940e83c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# register a new collection\n",
+    "collection = annotate.register_collection(\n",
+    "    artifact,  # registered artifact above, can also pass a list of artifacts\n",
+    "    name=\"Experiment X in brain\",  # title of the publication\n",
+    "    description=\"10.1126/science.xxxxx\",  # DOI of the publication\n",
+    "    reference=\"E-MTAB-xxxxx\", # accession number (e.g. GSE#, E-MTAB#, etc.)\n",
+    "    reference_type=\"ArrayExpress\") # source type (e.g. GEO, ArrayExpress, SRA, etc.)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "867e52b8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "collection.artifact"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "6e963036",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "artifact.collection"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.13"
+  },
+  "nbproject": {
+   "id": "WOK3vP0bNGLx",
+   "parent": null,
+   "pypackage": null,
+   "time_init": "2023-12-20T04:27:49.536493+00:00",
+   "user_handle": null,
+   "user_id": null,
+   "user_name": null,
+   "version": "0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

lamindb-0.69.2/docs/validate.ipynb → lamindb-0.69.3/docs/can-validate.ipynb

@@ -21,9 +21,7 @@
     ":::{dropdown} How do I validate based on a public ontology?\n",
     "\n",
     "{class}`~lamindb.core.CanValidate` methods validate against the registries in your LaminDB instance.\n",
-    "\n",
     "In {doc}`./bio-registries`, you'll see how to extend standard validation to validation against _public references_ using a `ReferenceTable` ontology object: `public = Registry.public()`.\n",
-    "\n",
     "By default, {meth}`~lamindb.core.Registry.from_values` considers a match in a public reference a validated value for any {mod}`bionty` entity.\n",
     "\n",
     ":::\n",
@@ -31,7 +29,6 @@
     ":::{dropdown} What to do for non-validated values?\n",
     "\n",
     "Be aware when you are working with a _freshly initialized instance_: nothing is validated as no records have yet been registered.\n",
-    "\n",
     "Run `inspect` to get instructions of how to register non-validated values. You may need to standardize your values, fix typos or simply register them.\n",
     "\n",
     ":::"
@@ -201,7 +198,6 @@
     "## Registry-based validation\n",
     "\n",
     "{meth}`~lamindb.core.CanValidate.validate` validates passed values against reference values in a registry.\n",
-    "\n",
     "It returns a boolean vector indicating whether a value has an exact match in the reference values."
    ]
   },
@@ -211,13 +207,7 @@
    "source": [
     "### Using dedicated registries\n",
     "\n",
-    "For instance, {mod}`bionty` types basic biological entities: every entity has its own registry, a Python class."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
+    "For instance, {mod}`bionty` types basic biological entities: every entity has its own registry, a Python class.\n",
     "By default, the first string field is used for validation. For {class}`~bionty.Disease`, it's `name`:"
    ]
   },
@@ -366,9 +356,7 @@
    "metadata": {},
    "source": [
     "When calling `File.from_...` and `Collection.from_...`, features are automatically validated.\n",
-    "\n",
     "Validated features are grouped in \"feature sets\" indexed by \"slots\".\n",
-    "\n",
     "For a basic example, see {doc}`/tutorial2`.\n",
     "\n",
     "For an overview of data formats used to model different data types, see {doc}`docs:by-datatype`."

{lamindb-0.69.2 → lamindb-0.69.3}/docs/changelog.md

@@ -1,14 +1,15 @@
 # Changelog
 
-
-
 ```{eval-rst}
 .. role:: small
 ```
 
-
 ## 0.69
 
+### 0.69.3 {small}`2024-03-28`
+
+- ✨ Introduce annotation flow via `Annotate.from_df` and `Annotate.from_anndata` [PR 1](https://github.com/laminlabs/lamindb/pull/1524) [2](https://github.com/laminlabs/lamindb/pull/1526) [3](https://github.com/laminlabs/lamindb/pull/1528) [@sunnyosun](https://github.com/sunnyosun)
+
 ### 0.69.2 {small}`2024-03-26`
 
 - ✨ Stage collections [PR](https://github.com/laminlabs/lamindb/pull/1521) [@Koncopd](https://github.com/Koncopd)

{lamindb-0.69.2 → lamindb-0.69.3}/docs/guide.md

@@ -16,9 +16,10 @@ tutorial2
 setup
 query-search
 track
-validate
-bio-registries
 annotate
+can-validate
+annotate-for-developers
+bio-registries
 schemas
 transfer
 ```