lamindb 0.72.0__tar.gz → 0.73.0__tar.gz

{lamindb-0.72.0 → lamindb-0.73.0}/.gitignore

@@ -59,6 +59,7 @@ test-search2
 test-search3
 test-search4
 test-search5
+default_storage
 
 # General
 .DS_Store
@@ -211,12 +212,3 @@ _build
 
 # VSCode
 .vscode
-
-# fred files
-docs/guide/insert-some-data-into-frederic-hub.ipynb
-docs/guide/insert-data-in-test-user-1-profile.ipynb
-
-# key guide
-docs/faq/complex_biological_project
-key-eval
-sample_001

{lamindb-0.72.0 → lamindb-0.73.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: lamindb
-Version: 0.72.0
+Version: 0.73.0
 Summary: A data framework for biology.
 Author-email: Lamin Labs <open-source@lamin.ai>
 Requires-Python: >=3.8
@@ -9,10 +9,10 @@ Classifier: Programming Language :: Python :: 3.8
 Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
-Requires-Dist: lnschema_core==0.67.1
-Requires-Dist: lamindb_setup==0.72.1
+Requires-Dist: lnschema_core==0.68.0
+Requires-Dist: lamindb_setup==0.73.0
 Requires-Dist: lamin_utils==0.13.2
-Requires-Dist: lamin_cli==0.13.2
+Requires-Dist: lamin_cli==0.14.0
 Requires-Dist: rapidfuzz
 Requires-Dist: pyarrow
 Requires-Dist: typing_extensions!=4.6.0
@@ -25,7 +25,7 @@ Requires-Dist: graphviz
 Requires-Dist: psycopg2-binary
 Requires-Dist: psutil
 Requires-Dist: lamindb_setup[aws] ; extra == "aws"
-Requires-Dist: bionty==0.43.0 ; extra == "bionty"
+Requires-Dist: bionty==0.43.1 ; extra == "bionty"
 Requires-Dist: pandas<2 ; extra == "dev"
 Requires-Dist: pre-commit ; extra == "dev"
 Requires-Dist: nox ; extra == "dev"
@@ -33,6 +33,7 @@ Requires-Dist: laminci>=0.3 ; extra == "dev"
 Requires-Dist: pytest>=6.0 ; extra == "dev"
 Requires-Dist: coverage ; extra == "dev"
 Requires-Dist: pytest-cov ; extra == "dev"
+Requires-Dist: mudata ; extra == "dev"
 Requires-Dist: nbproject_test>=0.5.1 ; extra == "dev"
 Requires-Dist: faker-biology ; extra == "dev"
 Requires-Dist: django-schema-graph ; extra == "erdiagram"

{lamindb-0.72.0 → lamindb-0.73.0}/docs/annotate-for-developers.ipynb

@@ -382,7 +382,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "artifact.features.add_from_anndata(var_field=bt.Gene.symbol, organism=\"human\")"
+    "artifact.features._add_set_from_anndata(var_field=bt.Gene.symbol, organism=\"human\")"
    ]
   },
   {

{lamindb-0.72.0 → lamindb-0.73.0}/docs/changelog.md

@@ -12,8 +12,57 @@ LaminDB implements "migration-based versioning". When upgrading your LaminDB ins
 .. role:: small
 ```
 
+## 0.73
+
+### 0.73.0 {small}`2024-05-29`
+
+Annotating & querying by features improved:
+- ✨ Support non-categorical feature values [PR](https://github.com/laminlabs/lamindb/pull/1674) [@falexwolf](https://github.com/falexwolf)
+- ✨ Annotate dict-style with features & values [PR](https://github.com/laminlabs/lamindb/pull/1674) [@falexwolf](https://github.com/falexwolf)
+- ✨ Query by features via `.features.filter(key=value)` [PR](https://github.com/laminlabs/lamindb/pull/1680) [@falexwolf](https://github.com/falexwolf)
+- 🏗️ Feature values decoupled from feature sets [PR](https://github.com/laminlabs/lamindb/pull/1674) [@falexwolf](https://github.com/falexwolf)
+
+Example:
+
+```python
+# annotate dict-style (feature & category names get validated)
+artifact.features.add_values({
+    "species": "setosa",
+    "scientist": ["Barabara McClintock", "Edgar Anderson"],
+    "instrument": "Leica IIIc Camera",
+    "temperature": 27.6,
+    "study": "Study 0: initial plant gathering",
+    "is_awesome": True
+})
+
+# get the dict back
+artifact.features.get_values()
+
+# query by feature
+ln.Artifact.features.filter(is_awesome=True)
+```
+
+Various improvements:
+- 🚚 Additional non-breaking constraints in the core schema [PR](https://github.com/laminlabs/lamindb/pull/1681) [@falexwolf](https://github.com/falexwolf)
+- 🚸 Make `.upload_from()`, `.download_to()`, and `.view_tree()` more user friendly [PR](https://github.com/laminlabs/lamindb/pull/1677) [@falexwolf](https://github.com/falexwolf) [PR](https://github.com/laminlabs/lamindb/pull/1678) [@Koncopd](https://github.com/Koncopd)
+- 🚸 More intuitive version updating dialogue [PR](https://github.com/laminlabs/lamindb/pull/1676) [@falexwolf](https://github.com/falexwolf)
+- 🐛 Actually add tracking run for entities beyond Artifact & Collection [PR](https://github.com/laminlabs/lamindb/pull/1673) [@falexwolf](https://github.com/falexwolf)
+- 🚸 `ln.track()` returns `run` [PR](https://github.com/laminlabs/lamindb/pull/1674) [@falexwolf](https://github.com/falexwolf)
+- 🚸 Better duplicate detection and search [PR](https://github.com/laminlabs/lamindb/pull/1674) [@falexwolf](https://github.com/falexwolf)
+- 🚸 Prettier `.describe()` [PR](https://github.com/laminlabs/lamindb/pull/1674) [@falexwolf](https://github.com/falexwolf)
+- 🚸 More interactivity in `lamin save` [PR](https://github.com/laminlabs/lamindb/pull/1674) [@falexwolf](https://github.com/falexwolf)
+- 🚸 `create` flag in `.from_values()` [PR](https://github.com/laminlabs/lamindb/pull/1674) [@falexwolf](https://github.com/falexwolf)
+- 🚸 Better ordering of fields in dataframe & record representations [PR](https://github.com/laminlabs/lamindb/pull/1674) [@falexwolf](https://github.com/falexwolf)
+- 📝 Improved API reference: docs now show relationship attributes [PR](https://github.com/laminlabs/lamindb/pull/1680) [@falexwolf](https://github.com/falexwolf)
+
+
 ## 0.72
 
+### 0.72.1 {small}`2024-05-19`
+
+- ⬆️ Update bionty [PR](https://github.com/laminlabs/lamindb/pull/1671) [@sunnyosun](https://github.com/sunnyosun)
+- 🐛 Deal with migration errors when keep-artifacts-local is true [PR](https://github.com/laminlabs/lamindb-setup/pull/767) [@falexwolf](https://github.com/falexwolf)
+
 ### 0.72.0 {small}`2024-05-19`
 
 - ✨ Extend managed access for AWS S3 to arbitrary paths [PR](https://github.com/laminlabs/lamindb/pull/1668) [@Koncopd](https://github.com/Koncopd) [@fredericenard](https://github.com/fredericenard)

{lamindb-0.72.0 → lamindb-0.73.0}/docs/introduction.ipynb

@@ -83,9 +83,9 @@
    "source": [
     "### Track\n",
     "\n",
-    "Run {meth}`~lamindb.track` to track the inputs and outputs of your code ({class}`~lamindb.Transform`).\n",
+    "Run {meth}`~lamindb.track` to track the inputs and outputs of your code.\n",
     "\n",
-    "When you first run `ln.track()`, it creates a `stem_uid` & `version` to unambiguously identify your notebook or script."
+    "When you first run `ln.track()`, it raises an exception and creates a `stem_uid` & `version` to identify a notebook or script:"
    ]
   },
   {
@@ -105,7 +105,17 @@
     "ln.settings.transform.version = \"1\"  # <-- auto-generated by running ln.track()\n",
     "\n",
     "# track the execution of your notebook or script\n",
-    "ln.track()"
+    "run = ln.track()\n",
+    "\n",
+    "# get your currently running transform\n",
+    "run.transform"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "`ln.track()` added a record to the {class}`~lamindb.Transform` registry to query & search your notebook and a {class}`~lamindb.Run` registry to store the specific notebook run."
    ]
   },
   {
@@ -147,12 +157,46 @@
     "artifact.describe()\n",
     "\n",
     "# save data & metadata in one operation\n",
-    "artifact.save()\n",
-    "\n",
-    "# view data lineage\n",
-    "artifact.view_lineage()\n",
-    "\n",
-    "# load an artifact\n",
+    "artifact.save()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "View data lineage:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "artifact.view_lineage()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Load an artifact:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
     "artifact.load()"
    ]
   },
@@ -187,17 +231,17 @@
     "\n",
     ":::\n",
     "\n",
-    "```{dropdown} Does LaminDB give me a file system?\n",
+    "```{dropdown} How does LaminDB compare to a file system or object store?\n",
     "\n",
-    "You can organize artifacts using the `key` parameter of {class}`~lamindb.Artifact` as you would in cloud storage.\n",
+    "Similar to organizing files in file systems & object stores with paths, you can organize artifacts using the `key` parameter of {class}`~lamindb.Artifact`.\n",
     "\n",
-    "However, LaminDB encourages you to **not** rely on semantic keys.\n",
+    "However, LaminDB encourages you to **not** rely on semantic keys but instead organize your data based on metadata.\n",
     "\n",
     "Rather than memorizing names of folders and files, you find data via the entities you care about: people, code, experiments, genes, proteins, cell types, etc.\n",
     "\n",
     "LaminDB embeds each artifact into rich relational metadata and indexes them in storage with a universal ID (`uid`).\n",
     "\n",
-    "This scales much better than semantic keys, which lead to deep hierarchical information structures that hard to navigate for humans & machines.\n",
+    "This scales much better than semantic keys, which lead to deep hierarchical information structures that can become hard to navigate.\n",
     "\n",
     "```\n",
     "\n",
@@ -244,7 +288,7 @@
    "source": [
     "### Labels\n",
     "\n",
-    "Label an artifact with a {class}`~lamindb.ULabel`."
+    "Label an artifact with a label managed by the {class}`~lamindb.ULabel` registry."
    ]
   },
   {
@@ -261,10 +305,50 @@
     "candidate_marker_study = ln.ULabel(name=\"Candidate marker study\").save()\n",
     "\n",
     "# label an artifact\n",
-    "artifact.labels.add(candidate_marker_study)\n",
-    "artifact.describe()\n",
+    "artifact.ulabels.add(candidate_marker_study)\n",
+    "artifact.describe()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Registries"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "LaminDB's central classes are registries that manage metadata.\n",
     "\n",
-    "# the ULabel registry\n",
+    "The easiest way to see what's in a registry is to call `.df()`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ln.Artifact.df()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ln.Transform.df()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
     "ln.ULabel.df() "
    ]
   },
@@ -274,7 +358,7 @@
    "source": [
     "### Queries\n",
     "\n",
-    "Write arbitrary relational queries (under-the-hood, LaminDB is SQL & Django)."
+    "You can write arbitrary relational queries using Django's query syntax."
    ]
   },
   {
@@ -283,17 +367,20 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# get an entity by uid\n",
+    "# get an entity by uid (here, the current notebook)\n",
     "transform = ln.Transform.get(\"FPnfDtJz8qbE\")\n",
     "\n",
     "# filter by description\n",
     "ln.Artifact.filter(description=\"my RNA-seq\").df()\n",
     "\n",
-    "# query all artifacts ingested from a notebook named \"Introduction\"\n",
-    "artifacts = ln.Artifact.filter(transform__name=\"Introduction\").all()\n",
+    "# query all artifacts ingested from the current notebook\n",
+    "artifacts = ln.Artifact.filter(transform=transform).all()\n",
     "\n",
     "# query all artifacts ingested from a notebook with \"intro\" in the name and labeled \"Candidate marker study\"\n",
-    "artifacts = ln.Artifact.filter(transform__name__icontains=\"intro\", ulabels=candidate_marker_study).all()"
+    "artifacts = ln.Artifact.filter(\n",
+    "    transform__name__icontains=\"intro\",\n",
+    "    ulabels=candidate_marker_study\n",
+    ").all()"
    ]
   },
   {
@@ -313,7 +400,7 @@
     "ln.Transform.search(\"intro\").df()\n",
     "\n",
     "# look up records with auto-complete\n",
-    "labels = ln.ULabel.lookup()"
+    "ulabels = ln.ULabel.lookup()"
    ]
   },
   {
@@ -331,11 +418,102 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Validate & annotate\n",
+    "### Features"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can annotate artifacts with features & values."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "import pytest\n",
+    "\n",
+    "with pytest.raises(ln.core.exceptions.ValidationError) as e:\n",
+    "    artifact.features.add_values({\"temperature\": 21.6})\n",
+    "\n",
+    "print(e.exconly())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "LaminDB validates all user input against its registries. As the `temperature` feature didn't exist, we got an error.\n",
+    "\n",
+    "Let's follow the hint in the error message:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# register the \"temperature\" feature\n",
+    "ln.Feature(name='temperature', dtype='float').save()\n",
+    "\n",
+    "# now we can annotate with the feature & the value\n",
+    "artifact.features.add_values({\"temperature\": 21.6})\n",
+    "artifact.describe()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We can also annotate with categorical features:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# register a categorical feature\n",
+    "ln.Feature(name='study', dtype='cat').save()\n",
+    "\n",
+    "# add a categorical value\n",
+    "artifact.features.add_values({\"study\": \"Candidate marker study\"})\n",
     "\n",
-    "In LaminDB, validation & annotation of categoricals are closely related by mapping categories on registry content.\n",
+    "# describe the artifact and add type information\n",
+    "artifact.describe(print_types=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Features provide a way to bucket labels beyond their type/registry."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Validate & annotate\n",
     "\n",
-    "Let's validate a `DataFrame` by passing validation criteria while constructing an {class}`~lamindb.Annotate` flow object."
+    "LaminDB validates & annotates categorical metadata by mapping categories on registries."
    ]
   },
   {
@@ -345,6 +523,13 @@
     "### Validate"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let's use the high-level {class}`~lamindb.Annotate`class to validate a `DataFrame`:"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -360,10 +545,10 @@
     "    df,\n",
     "    # define validation criteria\n",
     "    columns=ln.Feature.name,  # map column names\n",
-    "    categoricals={df.perturbation.name: ln.ULabel.name},  # map categories\n",
+    "    categoricals={\"perturbation\": ln.ULabel.name},  # map categories\n",
     ")\n",
     "\n",
-    "# the dataframe doesn't validate because registries don't contain the identifiers\n",
+    "# the dataframe doesn't validate because registries don't contain the categories\n",
     "annotate.validate()"
    ]
   },
@@ -384,13 +569,23 @@
    },
    "outputs": [],
    "source": [
-    "# add non-validated identifiers to their mapped registries\n",
+    "# add non-validated features based on the DataFrame columns\n",
     "annotate.add_new_from_columns()\n",
-    "annotate.add_new_from(df.perturbation.name)\n",
+    "# see the updated content of the features registry\n",
+    "ln.Feature.df()\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# add non-validated labels based on the perturbations\n",
+    "annotate.add_new_from(\"perturbation\")\n",
     "\n",
-    "# the registered labels & features that will from now on be used for validation\n",
-    "ln.ULabel.df()\n",
-    "ln.Feature.df()"
+    "# see the updated content of the ULabel registry\n",
+    "ln.ULabel.df()"
    ]
   },
   {
@@ -668,7 +863,7 @@
    "outputs": [],
    "source": [
     "collection = ln.Collection([artifact, artifact2], name=\"my RNA-seq collection\", version=\"1\")\n",
-    "collection.save(transfer_labels=True)  # transfer labels from artifacts to collection\n",
+    "collection.save()\n",
     "collection.describe()\n",
     "collection.view_lineage()"
    ]
@@ -867,7 +1062,7 @@
     "\n",
     "<img src=\"https://lamin-site-assets.s3.amazonaws.com/.lamindb/XoTQFCmmj2uU4d2xyj9t.png\" width=\"350px\" style=\"background: transparent\" align=\"right\">\n",
     "\n",
-    "LaminDB provides a SQL schema for common entities: {class}`~lamindb.Artifact`, {class}`~lamindb.Collection`, {class}`~lamindb.Transform`, {class}`~lamindb.Feature`, {class}`~lamindb.ULabel` etc. - see the [API reference](reference) or the [source code](https://github.com/laminlabs/lnschema-core/blob/main/lnschema_core/models.py).\n",
+    "LaminDB provides a SQL schema for common entities: {class}`~lamindb.Artifact`, {class}`~lamindb.Collection`, {class}`~lamindb.Transform`, {class}`~lamindb.Feature`, {class}`~lamindb.ULabel` etc. - see the [API reference](/reference) or the [source code](https://github.com/laminlabs/lnschema-core/blob/main/lnschema_core/models.py).\n",
     "\n",
     "The core schema is extendable through plugins (see blue vs. red entities in **graphic**), e.g., with basic biological ({class}`~bionty.Gene`, {class}`~bionty.Protein`, {class}`~bionty.CellLine`, etc.) & operational entities (`Biosample`, `Techsample`, `Treatment`, etc.).\n",
     "\n",
@@ -902,11 +1097,6 @@
     "\n",
     "LaminDB was influenced by many other projects, see {doc}`docs:influences`."
    ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": []
   }
  ],
  "metadata": {

{lamindb-0.72.0 → lamindb-0.73.0}/docs/storage/prepare-transfer-local-to-cloud.ipynb

@@ -65,6 +65,15 @@
     "\n",
     "artifact.describe()"
    ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "assert artifact.features[\"var\"].count() == 2"
+   ]
   }
  ],
  "metadata": {

{lamindb-0.72.0 → lamindb-0.73.0}/docs/storage/transfer-local-to-cloud.ipynb

@@ -35,7 +35,8 @@
     "    experiments = artifact.experiments.all()\n",
     "    try:\n",
     "        artifact.delete(permanent=True)\n",
-    "    except Exception:\n",
+    "    except Exception as e:\n",
+    "        print(e)\n",
     "        pass\n",
     "    features_sets.delete()\n",
     "    experiments.delete()\n",

{lamindb-0.72.0 → lamindb-0.73.0}/docs/transfer.ipynb

@@ -17,7 +17,11 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {},
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
    "outputs": [],
    "source": [
     "!lamin init --storage ./test-transfer --schema bionty"
@@ -26,7 +30,11 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {},
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
    "outputs": [],
    "source": [
     "import lamindb as ln\n",