lamindb 0.69.4__tar.gz → 0.69.6__tar.gz

{lamindb-0.69.4 → lamindb-0.69.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: lamindb
-Version: 0.69.4
+Version: 0.69.6
 Summary: A data framework for biology.
 Author-email: Lamin Labs <open-source@lamin.ai>
 Requires-Python: >=3.8
@@ -9,10 +9,10 @@ Classifier: Programming Language :: Python :: 3.8
 Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
-Requires-Dist: lnschema_core==0.64.1
-Requires-Dist: lamindb_setup==0.68.2
+Requires-Dist: lnschema_core==0.64.4
+Requires-Dist: lamindb_setup==0.68.4
 Requires-Dist: lamin_utils==0.13.1
-Requires-Dist: lamin_cli==0.10.2
+Requires-Dist: lamin_cli==0.11.0
 Requires-Dist: rapidfuzz
 Requires-Dist: pyarrow
 Requires-Dist: typing_extensions!=4.6.0

{lamindb-0.69.4 → lamindb-0.69.6}/docs/annotate.ipynb

@@ -5,24 +5,19 @@
    "id": "1d2cd769",
    "metadata": {},
    "source": [
-    "# Annotate data\n",
+    "# Validate, standardize & annotate\n",
     "\n",
-    "This guide shows how to annotate data, including defining clear validation criteria, validating & curating metadata, registering annotated artifacts, within a few minutes.\n",
+    "We'll walk you through the following flow: \n",
+    "1. define validation criteria\n",
+    "2. validate & standardize metadata\n",
+    "3. save validated & annotated artifacts\n",
     "\n",
-    "By the end, you'll have validated data objects empowered by LaminDB registries.\n",
+    ":::{dropdown} How do we validate metadata?\n",
     "\n",
-    ":::{dropdown} What does \"validating a categorical variable based on registries\" mean?\n",
-    "\n",
-    "The records in your LaminDB instance define the validated reference values for any entity managed in your schema.\n",
-    "\n",
-    "Validated categorical values are stored in a field of a registry; a column of the registry table.\n",
-    "\n",
-    "The default field to label an entity record is the `name` field.\n",
+    "Registries in your database define the \"truth\" for metadata.\n",
     "\n",
     "For instance, if \"Experiment 1\" has been registered as the `name` of a `ULabel` record, it is a validated value for field `ULabel.name`.\n",
     "\n",
-    "{class}`~lamindb.core.CanValidate` methods {meth}`~lamindb.core.CanValidate.validate`, {meth}`~lamindb.core.CanValidate.inspect`, {meth}`~lamindb.core.CanValidate.standardize`, {meth}`~lamindb.core.Registry.from_values` take 2 important parameters: `values` and `field`. The parameter `values` takes an iterable of input categorical values, and the parameter `field` takes a typed field of a registry.\n",
-    "\n",
     ":::\n",
     "\n",
     "```{toctree}\n",
@@ -34,14 +29,6 @@
     "```"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "id": "553332e1",
-   "metadata": {},
-   "source": [
-    "## Set up"
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -71,20 +58,12 @@
     "ln.settings.verbosity = \"hint\""
    ]
   },
-  {
-   "cell_type": "markdown",
-   "id": "65c71f2f",
-   "metadata": {},
-   "source": [
-    "## A DataFrame with labels"
-   ]
-  },
   {
    "cell_type": "markdown",
    "id": "52bbbac2",
    "metadata": {},
    "source": [
-    "Let's start with a DataFrame object that we'd like to validate and curate:"
+    "Let's start with a DataFrame that we'd like to validate:"
    ]
   },
   {
@@ -95,6 +74,7 @@
    "outputs": [],
    "source": [
     "df = pd.DataFrame({\n",
+    "    \"temperature\": [37.2, 36.3, 38.2],\n",
     "    \"cell_type\": [\"cerebral pyramidal neuron\", \"astrocyte\", \"oligodendrocyte\"],\n",
     "    \"assay_ontology_id\": [\"EFO:0008913\", \"EFO:0008913\", \"EFO:0008913\"],\n",
     "    \"donor\": [\"D0001\", \"D0002\", \"DOOO3\"],\n",
@@ -107,15 +87,7 @@
    "id": "946a3371",
    "metadata": {},
    "source": [
-    "## Validate and curate metadata"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "abc3aafa",
-   "metadata": {},
-   "source": [
-    "Define validation criteria for the columns:"
+    "## Validate and standardize metadata"
    ]
   },
   {
@@ -125,48 +97,16 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "fields = {\n",
+    "# define validation criteria for the categoricals\n",
+    "categoricals = {\n",
     "    \"cell_type\": bt.CellType.name,\n",
     "    \"assay_ontology_id\": bt.ExperimentalFactor.ontology_id,\n",
     "    \"donor\": ln.ULabel.name,\n",
-    "}"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "acdb45d8",
-   "metadata": {},
-   "source": [
-    "Validate the Pandas DataFrame:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "15137652",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "annotate = ln.Annotate.from_df(df, fields=fields)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "24e00633",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "validated = annotate.validate()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "f75d3f15",
-   "metadata": {},
-   "outputs": [],
-   "source": [
+    "}\n",
+    "# create an object to guide validation and annotation\n",
+    "annotate = ln.Annotate.from_df(df, categoricals=categoricals)\n",
+    "# validate\n",
+    "validated = annotate.validate()\n",
     "validated"
    ]
   },
@@ -177,12 +117,9 @@
    "source": [
     "## Validate using registries in another instance\n",
     "\n",
-    "Sometimes you want to validate against existing registries others might have created.\n",
-    "\n",
-    "Here we use the [cellxgene instance](https://lamin.ai/laminlabs/cellxgene) registries to curate against. You will notice more terms are validated than above.\n",
+    "Sometimes you want to validate against other existing registries, for instance [cellxgene](https://lamin.ai/laminlabs/cellxgene).\n",
     "\n",
-    "This allows us to register values that are currently missing in our instance from the [cellxgene instance](https://lamin.ai/laminlabs/cellxgene) directly.\n",
-    "By having our own registry but also validating against the [cellxgene instance](https://lamin.ai/laminlabs/cellxgene), we enable the addition of new registry values while keeping the [cellxgene instance](https://lamin.ai/laminlabs/cellxgene) focused on the [cellxgene schema](https://github.com/chanzuckerberg/single-cell-curation/tree/main/schema)."
+    "This allows us to directly transfer values that are currently missing in our registries from the [cellxgene instance](https://lamin.ai/laminlabs/cellxgene)."
    ]
   },
   {
@@ -194,9 +131,9 @@
    "source": [
     "annotate = ln.Annotate.from_df(\n",
     "    df, \n",
-    "    fields=fields, \n",
+    "    categoricals=categoricals,\n",
     "    using=\"laminlabs/cellxgene\",  # pass the instance slug\n",
-    "    )\n",
+    ")\n",
     "annotate.validate()"
    ]
   },
@@ -207,9 +144,7 @@
    "source": [
     "## Register new metadata labels\n",
     "\n",
-    "Following the suggestions above to register labels that aren't present in the current instance:\n",
-    "\n",
-    "(Note that our current instance is empty. Once you filled up the registries, registering new labels won't be frequently needed)"
+    "Our current database instance is empty. Once you populated its registries, saving new labels will only rarely be needed. You'll mostly use your lamindb instance to validate any incoming new data and annotate it."
    ]
   },
   {
@@ -219,15 +154,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "annotate.update_registry(\"cell_type\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "2dcd5a3d",
-   "metadata": {},
-   "source": [
-    "Fix typo and register again:"
+    "annotate.add_validated_from(df.cell_type.name)"
    ]
   },
   {
@@ -239,16 +166,7 @@
    "source": [
     "# use a lookup object to get the correct spelling of categories from public reference\n",
     "# pass \"public\" to use the public reference\n",
-    "lookup = annotate.lookup()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "24a56369",
-   "metadata": {},
-   "outputs": [],
-   "source": [
+    "lookup = annotate.lookup()\n",
     "lookup"
    ]
   },
@@ -259,7 +177,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "cell_types = lookup[\"cell_type\"]"
+    "cell_types = lookup[df.cell_type.name]"
    ]
   },
   {
@@ -280,9 +198,9 @@
    "outputs": [],
    "source": [
     "# fix the typo\n",
-    "df[\"cell_type\"] = df[\"cell_type\"].replace({\"cerebral pyramidal neuron\": cell_types.cerebral_cortex_pyramidal_neuron.name})\n",
+    "df.cell_type = df.cell_type.replace({\"cerebral pyramidal neuron\": cell_types.cerebral_cortex_pyramidal_neuron.name})\n",
     "\n",
-    "annotate.update_registry(\"cell_type\")"
+    "annotate.add_validated_from(df.cell_type.name)"
    ]
   },
   {
@@ -292,33 +210,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "annotate.update_registry(\"donor\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "b182af1c",
-   "metadata": {},
-   "source": [
-    "To register non-validated terms, pass `validated_only=False`:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "b393accf",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "annotate.update_registry(\"donor\", validated_only=False)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "1f3cd3a9",
-   "metadata": {},
-   "source": [
-    "Let's validate it again:"
+    "# register non-validated terms\n",
+    "annotate.add_new_from(df.donor.name)"
    ]
   },
   {
@@ -328,16 +221,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "validated = annotate.validate()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "55b9b30e",
-   "metadata": {},
-   "outputs": [],
-   "source": [
+    "# validate again\n",
+    "validated = annotate.validate()\n",
     "validated"
    ]
   },
@@ -348,8 +233,6 @@
    "source": [
     "## Validate an AnnData object\n",
     "\n",
-    "We offer an AnnData specific annotate that is aware of the variables in addition to the observations DataFrame.\n",
-    "\n",
     "Here we specify which `var_fields` and `obs_fields` to validate against."
    ]
   },
@@ -377,10 +260,10 @@
    "source": [
     "annotate = ln.Annotate.from_anndata(\n",
     "    adata, \n",
-    "    obs_fields=fields, \n",
-    "    var_field=bt.Gene.symbol, # specify the field for the var\n",
+    "    var_index=bt.Gene.symbol,\n",
+    "    categoricals=categoricals, \n",
     "    organism=\"human\",\n",
-    "    )"
+    ")"
    ]
   },
   {
@@ -400,7 +283,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "annotate.update_registry(\"all\")"
+    "annotate.add_validated_from(\"all\")"
    ]
   },
   {
@@ -418,7 +301,7 @@
    "id": "38a30170",
    "metadata": {},
    "source": [
-    "## Register file"
+    "## Save an artifact"
    ]
   },
   {
@@ -426,19 +309,7 @@
    "id": "a814ef37",
    "metadata": {},
    "source": [
-    "The validated object can be subsequently registered as an {class}`~lamindb.Artifact` in your LaminDB instance:"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "a70fe66d",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "ln.transform.stem_uid = \"WOK3vP0bNGLx\"\n",
-    "ln.transform.version = \"0\"\n",
-    "ln.track()"
+    "The validated object can be subsequently saved as an {class}`~lamindb.Artifact`:"
    ]
   },
   {
@@ -451,14 +322,6 @@
     "artifact = annotate.save_artifact(description=\"test AnnData\")"
    ]
   },
-  {
-   "cell_type": "markdown",
-   "id": "4fc354e4",
-   "metadata": {},
-   "source": [
-    "View the registered artifact with metadata:"
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -474,7 +337,7 @@
    "id": "1aa82222",
    "metadata": {},
    "source": [
-    "## Register collection"
+    "## Save a collection"
    ]
   },
   {
@@ -498,7 +361,8 @@
     "    name=\"Experiment X in brain\",  # title of the publication\n",
     "    description=\"10.1126/science.xxxxx\",  # DOI of the publication\n",
     "    reference=\"E-MTAB-xxxxx\", # accession number (e.g. GSE#, E-MTAB#, etc.)\n",
-    "    reference_type=\"ArrayExpress\") # source type (e.g. GEO, ArrayExpress, SRA, etc.)"
+    "    reference_type=\"ArrayExpress\" # source type (e.g. GEO, ArrayExpress, SRA, etc.)\n",
+    ")"
    ]
   },
   {
@@ -508,17 +372,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "collection.artifacts"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "2a46e57a",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "artifact.collections"
+    "collection.artifacts.df()"
    ]
   }
  ],

{lamindb-0.69.4 → lamindb-0.69.6}/docs/bio-registries.ipynb

@@ -111,7 +111,7 @@
     "\n",
     "`.public()` gives you access to the `ReferenceTable`, a class for accessing public ontologies.\n",
     "\n",
-    "`ReferenceTable` provides simple access to curated public ontologies that Lamin hosts for reliable and performant access. For most `ReferenceTable` objects, you can access the underlying ontology through [Pronto](https://github.com/althonos/pronto).\n",
+    "`ReferenceTable` provides simple access to standardized public ontologies that Lamin hosts for reliable and performant access. For most `ReferenceTable` objects, you can access the underlying ontology through [Pronto](https://github.com/althonos/pronto).\n",
     "\n",
     "Let's create a `ReferenceTable` object:"
    ]
@@ -337,7 +337,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "When accessing data sources, one often encounters bulk references to entities that might be corrupted or curated using different standardization schemes.\n",
+    "When accessing data sources, one often encounters bulk references to entities that might be corrupted or standardized using different standardization schemes.\n",
     "\n",
     "Let's consider an example based on an `AnnData` object:"
    ]

{lamindb-0.69.4 → lamindb-0.69.6}/docs/changelog.md

@@ -6,6 +6,15 @@
 
 ## 0.69
 
+### 0.69.6 {small}`2024-04-02`
+
+- 🚑️ Temp fix region for non-hosted buckets [PR](https://github.com/laminlabs/lamindb/pull/1543) [@sunnyosun](https://github.com/sunnyosun)
+
+### 0.69.5 {small}`2024-03-30`
+
+- ♻️ Improve Annotate API [PR](https://github.com/laminlabs/lamindb/pull/1542) [PR](https://github.com/laminlabs/lamindb/pull/1539)  [@sunnyosun](https://github.com/sunnyosun) [@falexwolf](https://github.com/falexwolf)
+- ✨ Introduce `Registry.get()` and `lamin get` (replaces `lamin stage`) [PR](https://github.com/laminlabs/lamindb/pull/1538) [@falexwolf](https://github.com/falexwolf)
+
 ### 0.69.4 {small}`2024-03-30`
 
 - ♻️ Add Vitessce integration [PR](https://github.com/laminlabs/lamindb/pull/1532) [@falexwolf](https://github.com/falexwolf)