lamindb 1.7a1__tar.gz → 1.8.0__tar.gz

{lamindb-1.7a1 → lamindb-1.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: lamindb
-Version: 1.7a1
+Version: 1.8.0
 Summary: A data framework for biology.
 Author-email: Lamin Labs <open-source@lamin.ai>
 Requires-Python: >=3.10,<3.14
@@ -10,8 +10,8 @@ Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Requires-Dist: lamin_utils==0.15.0
-Requires-Dist: lamin_cli==1.5.1
-Requires-Dist: lamindb_setup[aws]==1.7.0
+Requires-Dist: lamin_cli==1.5.5
+Requires-Dist: lamindb_setup[aws]==1.8.1
 Requires-Dist: pyyaml
 Requires-Dist: pyarrow
 Requires-Dist: pandera>=0.24.0

{lamindb-1.7a1 → lamindb-1.8.0}/docs/faq/keep-artifacts-local.ipynb

@@ -11,7 +11,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "If you want to default to keeping artifacts local in a cloud instance, enable {attr}`~lamindb.setup.core.InstanceSettings.keep_artifacts_local`."
+    "If you want to default to keeping artifacts local in a cloud instance, enable {attr}`~lamindb.setup.core.InstanceSettings.keep_artifacts_local`.\n",
+    "\n",
+    "Let us first create a cloud instance that woul store artifacts exclusively on S3."
    ]
   },
   {
@@ -19,7 +21,7 @@
    "execution_count": null,
    "metadata": {
     "tags": [
-     "hide-cell"
+     "hide-output"
     ]
    },
    "outputs": [],
@@ -29,15 +31,10 @@
    ]
   },
   {
-   "cell_type": "code",
-   "execution_count": null,
+   "cell_type": "markdown",
    "metadata": {},
-   "outputs": [],
    "source": [
-    "# !pip install 'lamindb[jupyter,aws]'\n",
-    "import lamindb as ln\n",
-    "\n",
-    "ln.track(\"l9lFf83aPwRc0000\")"
+    "Let's import lamindb and track the current notebook run."
    ]
   },
   {
@@ -45,21 +42,38 @@
    "execution_count": null,
    "metadata": {
     "tags": [
-     "hide-cell"
+     "hide-output"
     ]
    },
    "outputs": [],
    "source": [
-    "# the setting should be enabled on lamin.ai\n",
-    "# we're temporarily setting it here only for testing purposes\n",
-    "ln.setup.settings.instance._keep_artifacts_local = True"
+    "# !pip install 'lamindb[jupyter]'\n",
+    "import lamindb as ln\n",
+    "\n",
+    "ln.track(\"l9lFf83aPwRc0000\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Toggling setting \"keep artifacts local\""
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "You can register a managed local storage location as follows:"
+    "You checkmark the \"Keep artifacts local\" box on the instance settings tab.\n",
+    "\n",
+    "<img src=\"https://lamin-site-assets.s3.amazonaws.com/.lamindb/6Kt20kV5sQIFyV0Q0000.png\" width=\"400px\">"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "For the current test instance, we toggle it through the following private attribute."
    ]
   },
   {
@@ -68,14 +82,21 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "ln.settings.storage_local = \"./my_storage_local\""
+    "ln.setup.settings.instance._keep_artifacts_local = True"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Now, you have two storage locations: one in the S3 bucket, and the other locally."
+    "## Create a local storage location"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Call the following for a -- potentially pre-existing -- root path and a unique host identifier."
    ]
   },
   {
@@ -88,32 +109,26 @@
    },
    "outputs": [],
    "source": [
-    "ln.Storage.df()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Update storage description"
+    "ln.Storage(root=\"./our_local_storage\", host=\"abc-institute-drive1\").save()"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "You can add a description to the storage by using the description parameter:"
+    "Now, you have two storage locations: one in the S3 bucket, and the other locally."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {},
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
    "outputs": [],
    "source": [
-    "storage_record = ln.Storage.get(root=ln.settings.storage_local.root)\n",
-    "storage_record.description = \"Files stored locally in site X on server Y for reason ABC\"\n",
-    "storage_record.save()\n",
     "ln.Storage.df()"
    ]
   },
@@ -121,14 +136,14 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Use local storage"
+    "## Use a local storage location"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "If you save an artifact, by default, it's stored in local storage."
+    "If you save an artifact in keep-artifacts-local mode, by default, it's stored in local storage."
    ]
   },
   {
@@ -142,8 +157,8 @@
    "outputs": [],
    "source": [
     "original_filepath = ln.core.datasets.file_fcs()\n",
-    "artifact = ln.Artifact(original_filepath, description=\"My fcs file\").save()\n",
-    "local_path = artifact.path\n",
+    "artifact = ln.Artifact(original_filepath, key=\"example_datasets/file1.fcs\").save()\n",
+    "local_path = artifact.path  # local storage path\n",
     "local_path"
    ]
   },
@@ -151,7 +166,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "You'll see the `.fcs` file named by the `uid` in your `.lamindb/` directory under `./my_storage_local/`:"
+    "You'll see the `.fcs` file named by the `uid` in your `.lamindb/` directory under `./our_local_storage/`:"
    ]
   },
   {
@@ -164,28 +179,23 @@
    },
    "outputs": [],
    "source": [
-    "ln.settings.storage_local.root.view_tree()"
+    "assert artifact.path.exists()\n",
+    "assert artifact.path.as_posix().startswith(ln.settings.local_storage.root.as_posix())\n",
+    "ln.settings.local_storage.root.view_tree()"
    ]
   },
   {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "tags": [
-     "hide-cell"
-    ]
-   },
-   "outputs": [],
+   "cell_type": "markdown",
+   "metadata": {},
    "source": [
-    "assert local_path.exists()\n",
-    "assert artifact.path.as_posix().startswith(ln.settings.storage_local.root.as_posix())"
+    "## Pre-existing artifacts"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "If you'd like to upload an artifact, you pass `upload=True` to the `save()` method."
+    "Assume you already have a file in your local storage location:"
    ]
   },
   {
@@ -198,14 +208,16 @@
    },
    "outputs": [],
    "source": [
-    "artifact.save(upload=True)"
+    "file_in_local_storage = ln.core.datasets.file_bam()\n",
+    "file_in_local_storage.rename(\"./our_local_storage/output.bam\")\n",
+    "ln.UPath(\"our_local_storage/\").view_tree()"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "You now see the artifact in the S3 bucket:"
+    "When registering an artifact for it, it remains where it is."
    ]
   },
   {
@@ -218,14 +230,15 @@
    },
    "outputs": [],
    "source": [
-    "ln.setup.settings.storage.root.view_tree()"
+    "my_existing_file = ln.Artifact(\"./our_local_storage/output.bam\").save()\n",
+    "ln.UPath(\"our_local_storage/\").view_tree()"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "And it's no longer present in local storage:"
+    "The storage path of the artifact matches the pre-existing file:"
    ]
   },
   {
@@ -238,38 +251,21 @@
    },
    "outputs": [],
    "source": [
-    "ln.setup.settings.instance.storage_local.root.view_tree()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {
-    "tags": [
-     "hide-cell"
-    ]
-   },
-   "outputs": [],
-   "source": [
-    "assert artifact.path.exists()\n",
-    "assert not local_path.exists()\n",
-    "assert artifact.path.as_posix().startswith(\n",
-    "    ln.setup.settings.instance.storage.root.as_posix()\n",
-    ")"
+    "my_existing_file.path"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Direct upload"
+    "## Switching between local storage locations"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "You can also directly upload a file by passing `upload=True`:"
+    "You might have several local storage locations. Here is how you can switch between them."
    ]
   },
   {
@@ -282,29 +278,32 @@
    },
    "outputs": [],
    "source": [
-    "filepath = ln.core.datasets.file_mini_csv()\n",
-    "artifact2 = ln.Artifact(filepath, description=\"My csv file\").save(upload=True)\n",
-    "artifact2.path"
+    "ln.Storage(root=\"./our_local_storage2\", host=\"abc-institute-drive1\").save()\n",
+    "ln.settings.local_storage = \"./our_local_storage2\"  # switch to the new storage location"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Now we have two files on S3:"
+    "Ingest a file into the new local storage location."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "tags": [
-     "hide-output"
-    ]
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
-    "ln.Artifact.df(include=\"storage__root\")"
+    "filepath = ln.core.datasets.file_fastq()\n",
+    "artifact3 = ln.Artifact(filepath, key=\"example_datasets/file.fastq.gz\").save()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Inspect where all the files are."
    ]
   },
   {
@@ -312,147 +311,137 @@
    "execution_count": null,
    "metadata": {
     "tags": [
-     "hide-cell"
+     "hide-output"
     ]
    },
    "outputs": [],
    "source": [
-    "assert artifact2.path.exists()"
+    "ln.Artifact.df(include=[\"storage__root\", \"storage__region\"])"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Pre-existing artifacts"
+    "## Upload a local artifact to the cloud"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Assume we already have a file in our registered local storage location:"
+    "If you'd like to upload an artifact to the cloud storage location to more easily share it or view it through web applications, you pass `upload=True` to the `save()` method."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "tags": [
-     "hide-input"
-    ]
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
-    "file_in_local_storage = ln.core.datasets.file_bam()\n",
-    "file_in_local_storage.rename(\"./my_storage_local/output.bam\")\n",
-    "ln.UPath(\"my_storage_local/\").view_tree()"
+    "artifact.save(upload=True)"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "If we create an artifact from it, it remains where it is during saving:"
+    "You now see the artifact in the S3 bucket:"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "tags": [
-     "hide-output"
-    ]
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
-    "my_existing_file = ln.Artifact(\n",
-    "    \"./my_storage_local/output.bam\", description=\"my existing file\"\n",
-    ").save()\n",
-    "ln.UPath(\"my_storage_local/\").view_tree()"
+    "ln.settings.storage.root.view_tree()"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "The storage path of the artifact is constructed using `key` because  `key_is_virtual=False`:"
+    "And it's no longer present in local storage:"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "tags": [
-     "hide-output"
-    ]
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
-    "my_existing_file"
+    "assert artifact.path.exists()\n",
+    "assert not local_path.exists()\n",
+    "assert artifact.path.as_posix().startswith(ln.settings.storage.root.as_posix())\n",
+    "ln.settings.local_storage.root.view_tree()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Upload directly to the cloud"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "However, if we decide to upload the artifact, we'll use the `uid` for constructing the storage path and switch `key_is_virtual=True`:"
+    "You can also directly upload via `upload=True`:"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "tags": [
-     "hide-output"
-    ]
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
-    "my_existing_file.save(upload=True)"
+    "filepath = ln.core.datasets.file_mini_csv()\n",
+    "artifact2 = ln.Artifact(filepath, key=\"example_datasets/mini.csv\").save(upload=True)\n",
+    "artifact2.path"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Here is the remote path of the artifact:"
+    "Now we have two files on S3:"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "tags": [
-     "hide-output"
-    ]
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
-    "my_existing_file.path"
+    "ln.Artifact.df(include=\"storage__root\")"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "And here are the contents of the storage locations:"
+    "## Update storage description"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can add a description to the storage location by using the `description` field."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "tags": [
-     "hide-output"
-    ]
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
-    "# the path on S3\n",
-    "ln.setup.settings.storage.root.view_tree()\n",
-    "# the local path\n",
-    "ln.setup.settings.instance.storage_local.root.view_tree()"
+    "storage_record = ln.Storage.get(root__endswith=\"our_local_storage\")\n",
+    "storage_record.description = \"Our shared directory for project X\"\n",
+    "storage_record.save()\n",
+    "ln.Storage.df()"
    ]
   },
   {
@@ -477,7 +466,8 @@
    "source": [
     "artifact.delete(permanent=True)\n",
     "artifact2.delete(permanent=True)\n",
-    "my_existing_file.delete(permanent=True)"
+    "artifact3.delete(permanent=True)\n",
+    "my_existing_file.delete(permanent=True, storage=False)"
    ]
   },
   {
@@ -503,7 +493,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "py310",
+   "display_name": "py312",
    "language": "python",
    "name": "python3"
   },
@@ -517,7 +507,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.13"
+   "version": "3.12.8"
   }
  },
  "nbformat": 4,

{lamindb-1.7a1 → lamindb-1.8.0}/lamindb/__init__.py

@@ -108,7 +108,7 @@ Backwards compatibility.
 
 # ruff: noqa: I001
 # denote a release candidate for 0.1.0 with 0.1rc1, 0.1a1, 0.1b1, etc.
-__version__ = "1.7a1"
+__version__ = "1.8.0"
 
 import warnings
 

{lamindb-1.7a1 → lamindb-1.8.0}/lamindb/core/_context.py

@@ -17,20 +17,18 @@ from lamin_utils import logger
 from lamindb_setup.core import deprecated
 from lamindb_setup.core.hashing import hash_file
 
-from lamindb.base import ids
-from lamindb.base.ids import base62_12
-from lamindb.models import Run, Transform, format_field_value
-
-from ..core._settings import settings
+from ..base.ids import base62_12
 from ..errors import (
     InvalidArgument,
     TrackNotCalled,
     UpdateContext,
 )
+from ..models import Run, Transform, format_field_value
 from ..models._is_versioned import bump_version as bump_version_function
 from ..models._is_versioned import (
     increment_base62,
 )
+from ._settings import is_read_only_connection, settings
 from ._sync_git import get_transform_reference_from_git_repo
 from ._track_environment import track_python_environment
 
@@ -334,8 +332,8 @@ class Context:
         Args:
             transform: A transform (stem) `uid` (or record). If `None`, auto-creates a `transform` with its `uid`.
             project: A project (or its `name` or `uid`) for labeling entities.
-            space: A restricted space (or its `name` or `uid`) in which to store artifacts, collections, transforms, and runs.
-                Default: the `"All"` space.
+            space: A restricted space (or its `name` or `uid`) in which to store entities.
+                Default: the `"all"` space. Note that bionty entities ignore this setting and always get written to the `"all"` space.
                 If you want to manually move entities to a different space, set the `.space` field (:doc:`docs:access`).
             branch: A branch (or its `name` or `uid`) on which to store records.
             params: A dictionary of parameters to track for the run.
@@ -365,10 +363,8 @@ class Context:
             save_context_core,
         )
 
-        instance_settings = ln_setup.settings.instance
         # similar logic here: https://github.com/laminlabs/lamindb/pull/2527
-        # TODO: refactor upon new access management
-        if instance_settings.dialect == "postgresql" and "read" in instance_settings.db:
+        if is_read_only_connection():
             logger.warning("skipping track(), connected in read-only mode")
             return None
         if project is None:
@@ -856,7 +852,7 @@ class Context:
                 and not transform_was_saved
             ):
                 raise UpdateContext(
-                    f'{transform.created_by.name} ({transform.created_by.handle}) already works on this draft {transform.type}.\n\nPlease create a revision via `ln.track("{uid[:-4]}{increment_base62(uid[-4:])}")` or a new transform with a *different* key and `ln.track("{ids.base62_12()}0000")`.'
+                    f'{transform.created_by.name} ({transform.created_by.handle}) already works on this draft {transform.type}.\n\nPlease create a revision via `ln.track("{uid[:-4]}{increment_base62(uid[-4:])}")` or a new transform with a *different* key and `ln.track("{base62_12()}0000")`.'
                 )
             # check whether transform source code was already saved
             if transform_was_saved: