lamindb 0.57.2__tar.gz → 0.58.1__tar.gz

{lamindb-0.57.2 → lamindb-0.58.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: lamindb
-Version: 0.57.2
+Version: 0.58.1
 Summary: A data framework for biology.
 Author-email: Lamin Labs <open-source@lamin.ai>
 Requires-Python: >=3.8
@@ -8,9 +8,9 @@ Description-Content-Type: text/markdown
 Classifier: Programming Language :: Python :: 3.8
 Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
-Requires-Dist: lnschema_core==0.52.0
-Requires-Dist: lamindb_setup==0.55.6
-Requires-Dist: lamin_utils==0.11.4
+Requires-Dist: lnschema_core==0.53.0
+Requires-Dist: lamindb_setup==0.56.3
+Requires-Dist: lamin_utils==0.11.5
 Requires-Dist: rapidfuzz
 Requires-Dist: pyarrow
 Requires-Dist: typing_extensions!=4.6.0
@@ -24,7 +24,7 @@ Requires-Dist: urllib3<2 ; extra == "aws"
 Requires-Dist: boto3==1.28.17 ; extra == "aws"
 Requires-Dist: aiobotocore==2.5.4 ; extra == "aws"
 Requires-Dist: fsspec[s3]==2023.9.0 ; extra == "aws"
-Requires-Dist: lnschema_bionty==0.33.0 ; extra == "bionty"
+Requires-Dist: lnschema_bionty==0.34.0 ; extra == "bionty"
 Requires-Dist: pandas<2 ; extra == "dev"
 Requires-Dist: pre-commit ; extra == "dev"
 Requires-Dist: nox ; extra == "dev"

{lamindb-0.57.2 → lamindb-0.58.1}/docs/bio-registries.ipynb

@@ -825,7 +825,31 @@
    "source": [
     "Sometimes, the default source doesn't contain the ontology term you are looking for.\n",
     "\n",
-    "You can then specify to create a record from a non-default source:"
+    "You can then specify to create a record from a non-default source. For instance, instead of using untyped labels for iris organisms as {doc}`/tutorial2`, we can use the `ncbitaxon` ontology:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "bionty_source = lb.BiontySource.filter(entity=\"Organism\", source=\"ncbitaxon\").one()\n",
+    "iris_setosa = lb.Organism.from_bionty(name=\"iris setosa\", bionty_source=bionty_source)\n",
+    "iris_setosa.save()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "iris_setosa"
    ]
   },
   {
@@ -834,9 +858,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "bionty_source = lb.BiontySource.filter(entity=\"Phenotype\", source=\"pato\").one()\n",
-    "age = lb.Phenotype.from_bionty(name=\"age\", bionty_source=bionty_source)\n",
-    "age"
+    "iris_setosa.bionty_source"
    ]
   },
   {
@@ -845,7 +867,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "age.bionty_source"
+    "iris_setosa.parents.df()"
    ]
   },
   {
@@ -855,16 +877,40 @@
     "Analogously, you can pass `bionty_source` to bulk-create records from a non-default source:"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "records = lb.Organism.from_values(\n",
+    "    [\"iris setosa\", \"iris versicolor\", \"iris virginica\"], bionty_source=bionty_source\n",
+    ")\n",
+    "ln.save(records)"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
-    "records = lb.Phenotype.from_values([\"age\", \"life span\"], bionty_source=bionty_source)\n",
     "records"
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "iris_setosa.parents.get(name=\"iris\").view_parents(with_children=True)"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -896,7 +942,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.13"
+   "version": "3.9.16"
   },
   "nbproject": {
    "id": "s7F7PezA1HdQ",

{lamindb-0.57.2 → lamindb-0.58.1}/docs/changelog.md

@@ -1,9 +1,26 @@
 # Changelog
 
+
 ```{eval-rst}
 .. role:: small
 ```
 
+## 0.58
+
+### 0.58.1 {small}`2023-10-26`
+
+- 🔒️ Re-enable read-only access for public instances [PR1257](https://github.com/laminlabs/lamindb/pull/1257) [@falexwolf](https://github.com/falexwolf)
+- ✨ Introduce `visibility` to `File` and `Dataset` [PR1254](https://github.com/laminlabs/lamindb/pull/1254) [@sunnyosun](https://github.com/sunnyosun)
+- 🔒️ Enable to pass DB URLs during load [PR1256](https://github.com/laminlabs/lamindb/pull/1256) [@falexwolf](https://github.com/falexwolf)
+- ✨ Enable passing filter expressions to lookup and search [PR1255](https://github.com/laminlabs/lamindb/pull/1255) [@sunnyosun](https://github.com/sunnyosun)
+- ♻️ Use `file.uid` instead of `file.id` in `dataset.load()` [PR1250](https://github.com/laminlabs/lamindb/pull/1250) [@falexwolf](https://github.com/falexwolf)
+
+### 0.58.0 {small}`2023-10-24`
+
+- ⚡️ Speed up dataset creation in presence of millions of feature links [PR1248](https://github.com/laminlabs/lamindb/pull/1248) [@falexwolf](https://github.com/falexwolf)
+- 📝 Add a faq for custom validator [PR1249](https://github.com/laminlabs/lamindb/pull/1249) [@sunnyosun](https://github.com/sunnyosun)
+- ✨ Add `parents` to `Organism` [PR1245](https://github.com/laminlabs/lamindb/pull/1245) [@sunnyosun](https://github.com/sunnyosun)
+
 ## 0.57
 
 ### 0.57.2 {small}`2023-10-23`

{lamindb-0.57.2 → lamindb-0.58.1}/docs/faq/acid.ipynb

@@ -6,7 +6,7 @@
    "id": "41b33aca-1a7d-42b4-afa6-c9a5bccc060c",
    "metadata": {},
    "source": [
-    "# Are LaminDB's operations ACID?"
+    "# Will data & metadata stay in sync?"
    ]
   },
   {
@@ -15,9 +15,15 @@
    "id": "2096971d",
    "metadata": {},
    "source": [
-    "Yes, they are [ACID](https://en.wikipedia.org/wiki/ACID).\n",
-    "\n",
-    "Here, we walk through different errors that can occur while saving files & metadata records, and show that the LaminDB instance does not get corrupted by dangling metadata or files."
+    "Here, we walk through different errors that can occur while saving files & metadata records, and show that the LaminDB instance does not get corrupted by dangling metadata or files. You could say transactions across data & metadata are [ACID](https://en.wikipedia.org/wiki/ACID)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cf453c11",
+   "metadata": {},
+   "source": [
+    "## Setup"
    ]
   },
   {
@@ -29,7 +35,16 @@
    "source": [
     "from laminci.db import setup_local_test_postgres\n",
     "\n",
-    "pgurl = setup_local_test_postgres()\n",
+    "pgurl = setup_local_test_postgres()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "71969f0c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
     "!lamin init --db {pgurl} --storage ./test-acid"
    ]
   },
@@ -44,8 +59,7 @@
    "source": [
     "import pytest\n",
     "import lamindb as ln\n",
-    "from upath import UPath\n",
-    "from django.db.utils import IntegrityError"
+    "from upath import UPath"
    ]
   },
   {
@@ -157,15 +171,6 @@
     "## Save error during bulk creation"
    ]
   },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "a19ab581",
-   "metadata": {},
-   "source": [
-    "### Error during metadata save"
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -175,7 +180,7 @@
    "source": [
     "filepath = ln.dev.datasets.file_jpg_paradisi05()\n",
     "file = ln.File(filepath, description=\"My image\")\n",
-    "files = [file, \"this is not an Registry\"]"
+    "files = [file, \"this is not a record\"]"
    ]
   },
   {
@@ -219,15 +224,6 @@
     "assert len(files) == 0"
    ]
   },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "1c6c06d5",
-   "metadata": {},
-   "source": [
-    "## Error in one of the uploads"
-   ]
-  },
   {
    "attachments": {},
    "cell_type": "markdown",
@@ -237,15 +233,6 @@
     "If a list of data objects is passed to `ln.save()` and the upload of one of these data objects fails, the successful uploads are maintained and a `RuntimeError` is raised, listing the successfully uploaded data objects up until that point."
    ]
   },
-  {
-   "attachments": {},
-   "cell_type": "markdown",
-   "id": "f1738dfa",
-   "metadata": {},
-   "source": [
-    "Need a proper test here!"
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": null,

lamindb-0.58.1/docs/faq/delete.ipynb

@@ -0,0 +1,191 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# How to delete records?\n",
+    "\n",
+    "Registry records can be deleted with `record.delete()`, which will **permanently remove** them from your database.\n",
+    "\n",
+    "When it comes to **records of `File` and `Dataset`**, they are **\"moved into trash\"** when you first call `record.delete()`.\n",
+    "- Trashed records are invisible in the UI and excluded from the query results, see [visibility faq](/faq/visibility).\n",
+    "- If a record is already in the trash or `permanent=True` is passed, calling `record.delete()` triggers permanent delete.\n",
+    "- During permanent deletion of a record, its file in storage is also deleted unless it has a semantic `key`."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Setup"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!lamin init --storage test-delete"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import lamindb as ln\n",
+    "import pandas as pd"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "file = ln.File(pd.DataFrame({\"a\": [1, 2], \"b\": [3, 4]}), description=\"mydf\")\n",
+    "file.save()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ln.File.filter().df()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Trash a file"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "file.delete()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "No longer visible:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ln.File.filter().df()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "But the file still exists in the database, you can find it by not filtering for visibility:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ln.File.filter(visibility=None).df()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can restore a file from trash:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "file.restore()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ln.File.filter().df()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Permanent delete\n",
+    "\n",
+    "Calling `file.delete` on a trashed file triggers a permanent delete dialog. You can pass `permanent=True` to auto-confirm the deletion."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "file.delete(permanent=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now its gone in the database:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ln.File.filter(visibility=None).df()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "py39",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.16"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

{lamindb-0.57.2 → lamindb-0.58.1}/docs/faq/idempotency.ipynb

@@ -6,7 +6,7 @@
    "id": "fe2d8275-0883-4ffe-9912-920b28efc9d2",
    "metadata": {},
    "source": [
-    "# Are LaminDB's operations idempotent? (What happens if I save the same files & records twice?)"
+    "# What happens if I save the same files & records twice?"
    ]
   },
   {
@@ -15,9 +15,9 @@
    "id": "dda847ba-c174-4442-acfb-752d6a875645",
    "metadata": {},
    "source": [
-    "LaminDB's operations are idempotent. This is essential for two reasons:\n",
-    "- Avoid that re-running a workflow errors.\n",
-    "- Avoid that re-running a workflow inserts duplicated data.\n",
+    "LaminDB's operations are idempotent in the sense defined in this document.\n",
+    "\n",
+    "This allows you to re-run a notebook or script without erroring or duplicating data. Similar behavior holds for human data entry.\n",
     "\n",
     "## Summary\n",
     "\n",
@@ -27,7 +27,7 @@
     "\n",
     "1. LaminDB will warn you if a record with similar `name` exists and display a table of similar existing records.\n",
     "2. You can then decide whether you'd like to save a record to the database or rather query an existing one from the table.\n",
-    "3. If a name already has an exact match in a registry, LaminDB will return it instead of creating a new record.\n",
+    "3. If a name already has an exact match in a registry, LaminDB will return it instead of creating a new record. For versioned entities, also the version must be passed.\n",
     "\n",
     "If you set {attr}`~lamindb.dev.Settings.upon_create_search_names` to `False`, you'll directly populate the DB.\n",
     "\n",
@@ -251,7 +251,7 @@
    "id": "01bfec00",
    "metadata": {},
    "source": [
-    "### Data artifacts"
+    "### Files"
    ]
   },
   {
@@ -260,7 +260,7 @@
    "id": "431fbcc4",
    "metadata": {},
    "source": [
-    "#### `upon_file_create_if_hash_exists = \"warn_return_existing\"`"
+    "#### Warn upon trying to re-ingest an existing file"
    ]
   },
   {
@@ -380,7 +380,7 @@
    "id": "721bbe4e",
    "metadata": {},
    "source": [
-    "#### `upon_file_create_if_hash_exists = \"error\"`"
+    "#### Error upon trying to re-ingest an existing file"
    ]
   },
   {
@@ -419,7 +419,15 @@
    "id": "cff06da7",
    "metadata": {},
    "source": [
-    "#### `upon_file_create_if_hash_exists = \"warn_create_new\"`"
+    "#### Warn and create a new file"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fd806cd8",
+   "metadata": {},
+   "source": [
+    "Lastly, let us discuss the following setting:"
    ]
   },
   {

{lamindb-0.57.2 → lamindb-0.58.1}/docs/faq/notebooks.ipynb

@@ -10,7 +10,11 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {},
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
    "outputs": [],
    "source": [
     "!lamin init --storage ./test-notebooks"
@@ -31,7 +35,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "The following call will create a new run, if there is none in the database, yet."
+    "The following call will create a new run if there is none in the database."
    ]
   },
   {
@@ -43,40 +47,29 @@
     "ln.track()"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "run_id = ln.dev.run_context.run.id"
-   ]
-  },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Calling `ln.track()` when there is a new run in the database will load the latest run:"
+    "Let's store this run:"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
-    "ln.track()"
+    "run = ln.dev.run_context.run"
    ]
   },
   {
-   "cell_type": "code",
-   "execution_count": null,
+   "cell_type": "markdown",
    "metadata": {},
-   "outputs": [],
    "source": [
-    "ln.Run.filter().df()"
+    "If there is a run in the database, `ln.track()` will load the latest run:"
    ]
   },
   {
@@ -85,37 +78,32 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "run_id"
+    "ln.track()"
    ]
   },
   {
-   "cell_type": "code",
-   "execution_count": null,
+   "cell_type": "markdown",
    "metadata": {},
-   "outputs": [],
    "source": [
-    "ln.run_context.run.id"
+    "Check whether this is in fact the case:"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
-    "tags": [
-     "hide-cell"
-    ]
+    "tags": []
    },
    "outputs": [],
    "source": [
-    "# this tests that we're still using the previous run\n",
-    "assert run_id == ln.run_context.run.id"
+    "assert run.id == ln.run_context.run.id"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "If you'd like to create a new run, pass `new_run=True`."
+    "If you'd like to force creating a new run, pass `new_run=True` (this is the default outside of notebooks)."
    ]
   },
   {
@@ -127,6 +115,13 @@
     "ln.track(new_run=True)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "If you'd like to save a notebook including its execution report & source file, use the CLI: `lamin save <my-notebook.ipynb>`"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -137,15 +132,9 @@
    },
    "outputs": [],
    "source": [
-    "assert run_id != ln.run_context.run.id"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
+    "# test new run id differs from previous run id.\n",
+    "assert run.id != ln.run_context.run.id\n",
+    "\n",
     "# test re-initialize notebook if it got renamed\n",
     "from lamindb.dev._run_context import reinitialize_notebook\n",
     "\n",
@@ -154,15 +143,8 @@
     "# outside of CI, will expect interactive user input\n",
     "transform, metadata = reinitialize_notebook(transform, ask_for_new_id=False)\n",
     "\n",
-    "assert not transform.uid.startswith(\"FkCpHEubpD\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
+    "assert not transform.uid.startswith(\"FkCpHEubpD\")\n",
+    "\n",
     "!lamin delete --force test-notebooks\n",
     "!rm -r test-notebooks"
    ]