lamindb 0.58.0__tar.gz → 0.58.1__tar.gz

{lamindb-0.58.0 → lamindb-0.58.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: lamindb
-Version: 0.58.0
+Version: 0.58.1
 Summary: A data framework for biology.
 Author-email: Lamin Labs <open-source@lamin.ai>
 Requires-Python: >=3.8
@@ -8,8 +8,8 @@ Description-Content-Type: text/markdown
 Classifier: Programming Language :: Python :: 3.8
 Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
-Requires-Dist: lnschema_core==0.52.1
-Requires-Dist: lamindb_setup==0.55.6
+Requires-Dist: lnschema_core==0.53.0
+Requires-Dist: lamindb_setup==0.56.3
 Requires-Dist: lamin_utils==0.11.5
 Requires-Dist: rapidfuzz
 Requires-Dist: pyarrow

{lamindb-0.58.0 → lamindb-0.58.1}/docs/changelog.md

@@ -7,6 +7,14 @@
 
 ## 0.58
 
+### 0.58.1 {small}`2023-10-26`
+
+- 🔒️ Re-enable read-only access for public instances [PR1257](https://github.com/laminlabs/lamindb/pull/1257) [@falexwolf](https://github.com/falexwolf)
+- ✨ Introduce `visibility` to `File` and `Dataset` [PR1254](https://github.com/laminlabs/lamindb/pull/1254) [@sunnyosun](https://github.com/sunnyosun)
+- 🔒️ Enable to pass DB URLs during load [PR1256](https://github.com/laminlabs/lamindb/pull/1256) [@falexwolf](https://github.com/falexwolf)
+- ✨ Enable passing filter expressions to lookup and search [PR1255](https://github.com/laminlabs/lamindb/pull/1255) [@sunnyosun](https://github.com/sunnyosun)
+- ♻️ Use `file.uid` instead of `file.id` in `dataset.load()` [PR1250](https://github.com/laminlabs/lamindb/pull/1250) [@falexwolf](https://github.com/falexwolf)
+
 ### 0.58.0 {small}`2023-10-24`
 
 - ⚡️ Speed up dataset creation in presence of millions of feature links [PR1248](https://github.com/laminlabs/lamindb/pull/1248) [@falexwolf](https://github.com/falexwolf)

lamindb-0.58.1/docs/faq/delete.ipynb

@@ -0,0 +1,191 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# How to delete records?\n",
+    "\n",
+    "Registry records can be deleted with `record.delete()`, which will **permanently remove** them from your database.\n",
+    "\n",
+    "When it comes to **records of `File` and `Dataset`**, they are **\"moved into trash\"** when you first call `record.delete()`.\n",
+    "- Trashed records are invisible in the UI and excluded from the query results, see [visibility faq](/faq/visibility).\n",
+    "- If a record is already in the trash or `permanent=True` is passed, calling `record.delete()` triggers permanent delete.\n",
+    "- During permanent deletion of a record, its file in storage is also deleted unless it has a semantic `key`."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Setup"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!lamin init --storage test-delete"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import lamindb as ln\n",
+    "import pandas as pd"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "file = ln.File(pd.DataFrame({\"a\": [1, 2], \"b\": [3, 4]}), description=\"mydf\")\n",
+    "file.save()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ln.File.filter().df()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Trash a file"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "file.delete()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "No longer visible:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ln.File.filter().df()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "But the file still exists in the database, you can find it by not filtering for visibility:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ln.File.filter(visibility=None).df()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can restore a file from trash:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "file.restore()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ln.File.filter().df()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Permanent delete\n",
+    "\n",
+    "Calling `file.delete` on a trashed file triggers a permanent delete dialog. You can pass `permanent=True` to auto-confirm the deletion."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "file.delete(permanent=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now its gone in the database:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ln.File.filter(visibility=None).df()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "py39",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.16"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

{lamindb-0.58.0 → lamindb-0.58.1}/docs/faq/validator.ipynb

@@ -4,7 +4,23 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Custom validators"
+    "# How do I create custom validators?"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Here, we will extend the basic [validation guide](/validate) to enforce more constraints with dictionary-like validators.\n",
+    "\n",
+    "This is similar to what is enforced by the CZ CELLxGENE data portal, see [here](https://github.com/chanzuckerberg/single-cell-curation)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Setup"
    ]
   },
   {
@@ -26,9 +42,17 @@
     "import lnschema_bionty as lb\n",
     "from lamin_utils import logger\n",
     "\n",
+    "ln.settings.verbosity = \"success\"\n",
     "lb.settings.organism = \"human\""
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Access data"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -46,6 +70,13 @@
     "adata"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Define validation criteria"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -70,6 +101,13 @@
     "}"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Run validation"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -87,22 +125,22 @@
     "\n",
     "for slot, slot_validators in validators.items():\n",
     "    for name, validator in slot_validators.items():\n",
-    "        # registry model\n",
-    "        model = validator.field.model\n",
+    "        # access registry (a Django model)\n",
+    "        registry = validator.field.model\n",
     "\n",
     "        # validate index\n",
     "        if name == \"index\":\n",
     "            logger.print(f\"validating {slot}.{name}:\")\n",
-    "            index = adata.__getattribute__(slot).index\n",
-    "            validated = model.validate(index, validator)\n",
+    "            index = getattr(adata, slot).index\n",
+    "            validated = registry.validate(index, validator)\n",
     "            if validated.sum() == len(index):\n",
-    "                logger.print(\"🎉 PASS\")\n",
+    "                logger.success(\"index matches\")\n",
     "\n",
     "        # validate columns\n",
     "        else:\n",
     "            logger.print(f\"\\nvalidating {slot}.{name}:\")\n",
     "            # check if the column name exist\n",
-    "            if name not in adata.__getattribute__(slot).columns:\n",
+    "            if name not in getattr(adata, slot).columns:\n",
     "                logger.warning(f\"{slot}.{name} field is missing\")\n",
     "            else:\n",
     "                # check if a feature is registered for the column\n",
@@ -110,10 +148,17 @@
     "                    logger.warning(f\"feature '{name}' is not registered\")\n",
     "                # validate categorical labels in a column\n",
     "                else:\n",
-    "                    labels = adata.__getattribute__(slot)[name]\n",
-    "                    validated = model.validate(labels, validator)\n",
+    "                    labels = getattr(adata, slot)[name]\n",
+    "                    validated = registry.validate(labels, validator)\n",
     "                    if validated.sum() == len(labels):\n",
-    "                        logger.print(\"🎉 PASS\")"
+    "                        logger.success(\"labels match\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Delete test instance:"
    ]
   },
   {

lamindb-0.58.1/docs/faq/visibility.ipynb

@@ -0,0 +1,205 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# When do visibility of files and datasets change?\n",
+    "\n",
+    "Files and datasets in LaminDB have the following 3 levels if visibility:\n",
+    "- 0: \"default\", visible in the UI by default, returned from the query results\n",
+    "- 1: \"hidden\", excluded from the query results, notebook files (generated by `lamin save`) are default to be hidden\n",
+    "- 2: \"trash\", excluded from the query results, set with `.delete()`"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Setup"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!lamin init --storage test-visibility"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import lamindb as ln\n",
+    "import pandas as pd"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "file = ln.File(pd.DataFrame({\"a\": [1, 2], \"b\": [3, 4]}), description=\"mydf\")\n",
+    "file.save()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "New file has default visibility 0:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "assert file.visibility == 0"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "When you delete a file, its visibility is set to 2 (\"trash\"):"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "file.delete()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "assert file.visibility == 2"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Files in trash won't be returned from default queries:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ln.File.filter(description=\"mydf\").all()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Unless you specify `visibility=None` to see all hidden and trashed files:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ln.File.filter(description=\"mydf\", visibility=None).all()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can restore a file from trash:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "file.restore()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "assert file.visibility == 0"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ln.File.filter(description=\"mydf\").all()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Delete test file and instance:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "file.delete(permanent=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!lamin delete --force test-visibility"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "py39",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.16"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

{lamindb-0.58.0 → lamindb-0.58.1}/docs/faq.md

@@ -11,4 +11,6 @@ faq/setup
 faq/import-schema
 faq/reference-field
 faq/validator
+faq/visibility
+faq/delete
 ```

{lamindb-0.58.0 → lamindb-0.58.1}/docs/storage/add-replace-stage.ipynb

@@ -244,7 +244,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "file.delete(storage=True)"
+    "file.delete(permanent=True)"
    ]
   },
   {
@@ -435,7 +435,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "file.delete(storage=True)"
+    "file.delete(permanent=True, storage=True)"
    ]
   },
   {
@@ -616,7 +616,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "file.delete(storage=True)"
+    "file.delete(permanent=True, storage=True)"
    ]
   },
   {

{lamindb-0.58.0 → lamindb-0.58.1}/docs/storage/upload.ipynb

@@ -140,7 +140,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "pbmc68k_h5ad.delete(storage=True)"
+    "pbmc68k_h5ad.delete(permanent=True, storage=True)"
    ]
   },
   {
@@ -162,7 +162,7 @@
     "# Runs too long, should be tested elsewhere\n",
     "# pbmc68k_zarr = ln.File(pbmc68k, key=\"test-upload/pbmc68k.zarr\", format=\"zarr\")\n",
     "# ln.save(pbmc68k_zarr)\n",
-    "# pbmc68k_zarr.delete(storage=True)"
+    "# pbmc68k_zarr.delete(permanent=True, storage=True)"
    ]
   },
   {
@@ -210,7 +210,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "pbmc68k_h5ad.delete(storage=True)"
+    "pbmc68k_h5ad.delete(permanent=True, storage=True)"
    ]
   },
   {
@@ -232,7 +232,7 @@
     "# Runs too long, should be tested elsewhere\n",
     "# pbmc68k_zarr = ln.File(pbmc68k, name=\"pbmc68k.zarr\", format=\"zarr\")\n",
     "# ln.save(pbmc68k_zarr)\n",
-    "# pbmc68k_zarr.delete(storage=True)"
+    "# pbmc68k_zarr.delete(permanent=True, storage=True)"
    ]
   },
   {

{lamindb-0.58.0 → lamindb-0.58.1}/docs/tutorial.ipynb

@@ -820,14 +820,18 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "lamin save <notebook_path>  # e.g., tutorial.ipynb"
+    "```\n",
+    "lamin save <notebook_path>  # e.g., tutorial.ipynb\n",
+    "```"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "This will enable you to query the report and source code via `transform.latest_report` and `transform.source_file` and see it in the hub, e.g., [here](https://lamin.ai/laminlabs/lamindata/record/core/Transform?id=NJvdsWWbJlZSz8)."
+    "This will enable you to query the report and source code via `transform.latest_report` and `transform.source_file` and see it in the hub, e.g., [here](https://lamin.ai/laminlabs/lamindata/record/core/Transform?id=NJvdsWWbJlZSz8).\n",
+    "\n",
+    "<img src=\"https://lamin-site-assets.s3.amazonaws.com/.lamindb/RGXj5wcAf7EAc6J8aBoM.png\" width=\"700px\">"
    ]
   },
   {