lamindb 0.69.8__tar.gz → 0.69.9__tar.gz

{lamindb-0.69.8 → lamindb-0.69.9}/.github/workflows/build.yml

@@ -30,6 +30,11 @@ jobs:
           submodules: recursive
           fetch-depth: 0
       - uses: actions/setup-python@v5
+        if: ${{ matrix.group == 'storage' }}
+        with:
+          python-version: "3.9" # run just one job with a different python version
+      - uses: actions/setup-python@v5
+        if: ${{ matrix.group != 'storage' }}
         with:
           python-version: "3.11"
       - name: cache pre-commit
@@ -100,7 +105,7 @@ jobs:
           aws-region: us-east-1
       - uses: actions/setup-python@v5
         with:
-          python-version: "3.10"
+          python-version: "3.10" # use 3.10, 3.8 and 3.11 give issues
       - run: pip install -U laminci
       - run: nox -s "install(group='docs')"
       - uses: actions/download-artifact@v2

{lamindb-0.69.8 → lamindb-0.69.9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: lamindb
-Version: 0.69.8
+Version: 0.69.9
 Summary: A data framework for biology.
 Author-email: Lamin Labs <open-source@lamin.ai>
 Requires-Python: >=3.8
@@ -9,10 +9,10 @@ Classifier: Programming Language :: Python :: 3.8
 Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
-Requires-Dist: lnschema_core==0.64.5
-Requires-Dist: lamindb_setup==0.68.5
+Requires-Dist: lnschema_core==0.64.6
+Requires-Dist: lamindb_setup==0.69.0
 Requires-Dist: lamin_utils==0.13.1
-Requires-Dist: lamin_cli==0.12.0
+Requires-Dist: lamin_cli==0.12.1
 Requires-Dist: rapidfuzz
 Requires-Dist: pyarrow
 Requires-Dist: typing_extensions!=4.6.0

{lamindb-0.69.8 → lamindb-0.69.9}/docs/changelog.md

@@ -6,9 +6,12 @@
 
 ## 0.69
 
+### 0.69.9 {small}`2024-04-04`
+
+- 🐛 Fix clashes for multiple processes [PR](https://github.com/laminlabs/lamindb/pull/1553) [@falexwolf](https://github.com/falexwolf)
+
 ### 0.69.8 {small}`2024-04-04`
 
-- 🐛 Avoid an error due to deleting the same file by multiple processes [PR](https://github.com/laminlabs/lamindb-setup/pull/703) [@Koncopd](https://github.com/Koncopd)
 - ♻️ Use future annotations [PR](https://github.com/laminlabs/lamindb/pull/1549) [@Zethson](https://github.com/Zethson)
 
 ### 0.69.7 {small}`2024-04-03`

{lamindb-0.69.8 → lamindb-0.69.9}/docs/introduction.ipynb

@@ -77,9 +77,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Provenance\n",
+    "### Data lineage\n",
     "\n",
-    "Run {meth}`~lamindb.track` to auto-generate IDs to track data lineage."
+    "Run {meth}`~lamindb.track` to track data lineage."
    ]
   },
   {
@@ -88,7 +88,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# tag your code with auto-generated identifiers\n",
+    "# tag your notebook or script with auto-generated identifiers\n",
     "ln.settings.transform.stem_uid = \"FPnfDtJz8qbE\"\n",
     "ln.settings.transform.version = \"1\"\n",
     "\n",
@@ -157,7 +157,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Label\n",
+    "### Label artifacts\n",
     "\n",
     "Label an artifact with {class}`~lamindb.ULabel`."
    ]
@@ -181,7 +181,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Query\n",
+    "### Query anything\n",
     "\n",
     "Because, under-the-hood, LaminDB is SQL, you can write arbitrary relational queries."
    ]
@@ -255,9 +255,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Public ontologies\n",
+    "## Validate & annotate\n",
     "\n",
-    "Every {py:mod}`bionty` registry is based on public ontologies."
+    "Let's validate a `DataFrame` by passing validation criteria in the `Annotate` constructor."
    ]
   },
   {
@@ -266,10 +266,12 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import bionty as bt\n",
-    "\n",
-    "cell_types = bt.CellType.public()\n",
-    "cell_types"
+    "annotate = ln.Annotate.from_df(\n",
+    "    df,\n",
+    "    columns=ln.Feature.name,  # map column names\n",
+    "    categoricals={df.perturbation.name: ln.ULabel.name},  # map categories\n",
+    ")\n",
+    "annotate.validate()"
    ]
   },
   {
@@ -278,16 +280,22 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "cell_types.search(\"gamma delta T cell\").head(2)"
+    "# add non-validated terms to registries\n",
+    "annotate.add_new_from_columns()\n",
+    "annotate.add_new_from(df.perturbation.name)\n",
+    "\n",
+    "# view the registered features\n",
+    "ln.Feature.df()"
    ]
   },
   {
-   "cell_type": "markdown",
+   "cell_type": "code",
+   "execution_count": null,
    "metadata": {},
+   "outputs": [],
    "source": [
-    "## Validate & annotate\n",
-    "\n",
-    "Let's validate a `DataFrame` by passing validation criteria in the `Annotate` constructor."
+    "# view the registered ULabels\n",
+    "ln.ULabel.df()"
    ]
   },
   {
@@ -296,12 +304,12 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "annotate = ln.Annotate.from_df(\n",
-    "    df,\n",
-    "    columns=ln.Feature.name,  # validate column names\n",
-    "    categoricals={df.perturbation.name: ln.ULabel.name},  # validate categories\n",
-    ")\n",
-    "annotate.validate()"
+    "# now the validation passes\n",
+    "annotate.validate()\n",
+    "\n",
+    "# annotate and save the artifact\n",
+    "artifact = annotate.save_artifact(description=\"my RNA-seq\", version=\"1\")\n",
+    "artifact.describe()"
    ]
   },
   {
@@ -310,12 +318,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# add non-validated terms to registries for column names and perturbations\n",
-    "annotate.add_new_from_columns()\n",
-    "annotate.add_new_from(df.perturbation.name)\n",
-    "\n",
-    "# view the registered features\n",
-    "ln.Feature.df()"
+    "# get lookup object for the entities of interest\n",
+    "lookups = annotate.lookup()\n",
+    "lookups"
    ]
   },
   {
@@ -324,15 +329,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# view the registered ULabels\n",
-    "ln.ULabel.df()\n",
-    "\n",
-    "# now the validation passes\n",
-    "annotate.validate()\n",
-    "\n",
-    "# annotate and save the artifact\n",
-    "artifact = annotate.save_artifact(description=\"my RNA-seq\", version=\"1\")\n",
-    "artifact.describe()"
+    "# filter artifacts with specific labels\n",
+    "perturbations = lookups[\"perturbation\"]\n",
+    "ln.Artifact.filter(ulabels=candidate_marker_study).filter(ulabels=perturbations.ifng).one()"
    ]
   },
   {
@@ -347,49 +346,49 @@
    ]
   },
   {
-   "cell_type": "code",
-   "execution_count": null,
+   "cell_type": "markdown",
    "metadata": {},
-   "outputs": [],
    "source": [
-    "# get lookup object for the entities of interest\n",
-    "lookups = annotate.lookup()\n",
-    "lookups"
+    "## Biological registries"
    ]
   },
   {
-   "cell_type": "code",
-   "execution_count": null,
+   "cell_type": "markdown",
    "metadata": {},
-   "outputs": [],
-   "source": []
+   "source": [
+    "The generic {class}`~lamindb.Feature` and {class}`~lamindb.ULabel` registries will get you pretty far.\n",
+    "\n",
+    "Let's now look at what you do can with a dedicated biological registry like {class}`~bionty.Gene`."
+   ]
   },
   {
-   "cell_type": "code",
-   "execution_count": null,
+   "cell_type": "markdown",
    "metadata": {},
-   "outputs": [],
    "source": [
-    "# filter artifacts with specific labels\n",
-    "perturbations = lookups[\"perturbation\"]\n",
-    "ln.Artifact.filter(ulabels=candidate_marker_study).filter(ulabels=perturbations.ifng).one()"
+    "### Access public ontologies\n",
+    "\n",
+    "Every {py:mod}`bionty` registry is based on public ontologies."
    ]
   },
   {
-   "cell_type": "markdown",
+   "cell_type": "code",
+   "execution_count": null,
    "metadata": {},
+   "outputs": [],
    "source": [
-    "## Biological types"
+    "import bionty as bt\n",
+    "\n",
+    "cell_types = bt.CellType.public()\n",
+    "cell_types"
    ]
   },
   {
-   "cell_type": "markdown",
+   "cell_type": "code",
+   "execution_count": null,
    "metadata": {},
+   "outputs": [],
    "source": [
-    "{class}`~lamindb.Feature` and {class}`~lamindb.ULabel` will get you pretty far.\n",
-    "However, if you frequently use a specific entity, you'll want a dedicated registry.\n",
-    "\n",
-    "Let’s look at the example of {class}`~bionty.Gene` and use it to register features."
+    "cell_types.search(\"gamma delta T cell\").head(2)"
    ]
   },
   {
@@ -446,7 +445,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Manage biological registries\n",
+    "### Add new entities to biological registries\n",
     "\n",
     "Create a cell type record and add a new cell state."
    ]
@@ -487,7 +486,14 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Collections of artifacts"
+    "## Scale up learning"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "How do you learn from new data batches that extend your previous data history? Leverage {class}`~lamindb.Collection`."
    ]
   },
   {
@@ -549,16 +555,9 @@
    "outputs": [],
    "source": [
     "# if it's small enough, you can load the entire collection into memory as if it was one\n",
-    "collection.load()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# you can always iterate over its artifacts\n",
+    "collection.load()\n",
+    "\n",
+    "# typically, it's too big, hence, iterate over its artifacts\n",
     "collection.artifacts.df()"
    ]
   },
@@ -567,7 +566,7 @@
    "metadata": {},
    "source": [
     "```\n",
-    "# you can also batch iterate through collections\n",
+    "# to train models, batch iterate through the collection as if it was one array\n",
     "from torch.utils.data import DataLoader, WeightedRandomSampler\n",
     "dataset = collection.mapped(label_keys=[\"perturbation\"])\n",
     "sampler = WeightedRandomSampler(\n",
@@ -579,6 +578,13 @@
     "```"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Read this [blog post](https://lamin.ai/blog/arrayloader-benchmarks) for more on training models on sharded datasets."
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -625,7 +631,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Data lineage\n",
+    "## Data lineage across entire projects\n",
     "\n",
     "View the sequence of data transformations ({class}`~lamindb.Transform`) in a project (from [here](docs:project-flow), based on [Schmidt _et al._, 2022](https://pubmed.ncbi.nlm.nih.gov/35113687/)):\n",
     "\n",
@@ -651,13 +657,19 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Loading instances\n",
+    "## Distributed databases\n",
+    "\n",
+    "LaminDB is a distributed system like git. Similar to cloning a repository, collaborators can connect to your instance via:\n",
     "\n",
-    "LaminDB is a distributed system like git. Similar to cloning a repository, collaborators can load your instance on the command-line using:\n",
+    "```python\n",
+    "ln.connect(\"account-handle/instance-name\")\n",
+    "```\n",
+    "\n",
+    "Or you can load an instance on the command line for auto-connecting in a Python session:\n",
     "\n",
     "```shell\n",
-    "lamin load myhandle/myinstance\n",
-    "```"
+    "lamin load \"account-handle/instance-name\"\n",
+    "```\n"
    ]
   },
   {

{lamindb-0.69.8 → lamindb-0.69.9}/lamindb/__init__.py

@@ -40,7 +40,7 @@ Modules & settings:
 
 """
 
-__version__ = "0.69.8"  # denote a release candidate for 0.1.0 with 0.1rc1
+__version__ = "0.69.9"  # denote a release candidate for 0.1.0 with 0.1rc1
 
 import os as _os
 

{lamindb-0.69.8 → lamindb-0.69.9}/lamindb/_artifact.py

@@ -22,7 +22,6 @@ from lamindb_setup.core.upath import (
 from lnschema_core import Artifact, Run, Storage
 from lnschema_core.models import IsTree
 from lnschema_core.types import (
-    DataLike,
     VisibilityChoice,
 )
 
@@ -114,7 +113,7 @@ def process_pathlike(
 
 def process_data(
     provisional_uid: str,
-    data: UPathStr | DataLike,
+    data: UPathStr | pd.DataFrame | AnnData,
     format: str | None,
     key: str | None,
     default_storage: Storage,
@@ -138,7 +137,7 @@ def process_data(
         )
         suffix = extract_suffix_from_path(path)
         memory_rep = None
-    elif isinstance(data, (pd.DataFrame, AnnData)):  # DataLike, spelled out
+    elif isinstance(data, (pd.DataFrame, AnnData)):
         storage = default_storage
         memory_rep = data
         if key is not None:
@@ -427,7 +426,7 @@ def log_storage_hint(
     logger.hint(hint)
 
 
-def data_is_anndata(data: DataLike):
+def data_is_anndata(data: AnnData | UPathStr):
     if isinstance(data, AnnData):
         return True
     if isinstance(data, (str, Path, UPath)):
@@ -435,7 +434,7 @@ def data_is_anndata(data: DataLike):
     return False  # pragma: no cover
 
 
-def data_is_mudata(data: DataLike):  # pragma: no cover
+def data_is_mudata(data: Any | UPathStr):  # pragma: no cover
     try:
         from mudata import MuData
     except ModuleNotFoundError:
@@ -725,7 +724,7 @@ def from_dir(
 # docstring handled through attach_func_to_class_method
 def replace(
     self,
-    data: UPathStr | DataLike,
+    data: UPathStr,
     run: Run | None = None,
     format: str | None = None,
 ) -> None:
@@ -808,9 +807,7 @@ def backed(self, is_run_input: bool | None = None) -> AnnDataAccessor | BackedAc
 
 
 # docstring handled through attach_func_to_class_method
-def load(
-    self, is_run_input: bool | None = None, stream: bool = False, **kwargs
-) -> DataLike:
+def load(self, is_run_input: bool | None = None, stream: bool = False, **kwargs) -> Any:
     _track_run_input(self, is_run_input)
     if hasattr(self, "_memory_rep") and self._memory_rep is not None:
         return self._memory_rep

{lamindb-0.69.8 → lamindb-0.69.9}/lamindb/_collection.py

@@ -16,7 +16,7 @@ from lamin_utils import logger
 from lamindb_setup.core._docs import doc_args
 from lamindb_setup.core.hashing import hash_set
 from lnschema_core.models import Collection, CollectionArtifact, FeatureSet
-from lnschema_core.types import DataLike, VisibilityChoice
+from lnschema_core.types import VisibilityChoice
 
 from lamindb._utils import attach_func_to_class_method
 from lamindb.core._data import _track_run_input
@@ -65,7 +65,7 @@ def __init__(
     data: Artifact | Iterable[Artifact] = (
         kwargs.pop("data") if len(args) == 0 else args[0]
     )
-    meta: str | None = kwargs.pop("meta") if "meta" in kwargs else None
+    meta: Artifact | None = kwargs.pop("meta") if "meta" in kwargs else None
     name: str | None = kwargs.pop("name") if "name" in kwargs else None
     description: str | None = (
         kwargs.pop("description") if "description" in kwargs else None
@@ -273,7 +273,7 @@ def load(
     join: Literal["inner", "outer"] = "outer",
     is_run_input: bool | None = None,
     **kwargs,
-) -> DataLike:
+) -> Any:
     # cannot call _track_run_input here, see comment further down
     all_artifacts = self.artifacts.all()
     suffixes = [artifact.suffix for artifact in all_artifacts]

{lamindb-0.69.8 → lamindb-0.69.9}/lamindb/core/_feature_manager.py

@@ -23,7 +23,7 @@ from lamindb.core.storage import LocalPathClasses
 from ._settings import settings
 
 if TYPE_CHECKING:
-    from lnschema_core.types import AnnDataLike, FieldAttr
+    from lnschema_core.types import FieldAttr
 
     from lamindb._query_set import QuerySet
 
@@ -132,7 +132,7 @@ def print_features(self: Data) -> str:
 
 
 def parse_feature_sets_from_anndata(
-    adata: AnnDataLike,
+    adata: AnnData,
     var_field: FieldAttr,
     obs_field: FieldAttr = Feature.name,
     **kwargs,

{lamindb-0.69.8 → lamindb-0.69.9}/lamindb/core/storage/file.py

@@ -136,6 +136,9 @@ def delete_storage_using_key(
 
 def delete_storage(storagepath: Path):
     """Delete arbitrary artifact."""
+    # TODO is_relative_to is not available in 3.8 and deprecated since 3.12
+    # replace with check_path_is_child_of_root but this needs to first be debugged
+    # if not check_path_is_child_of_root(storagepath, settings.storage):
     if not storagepath.is_relative_to(settings.storage):  # type: ignore
         logger.warning("couldn't delete files outside of default storage")
         return "did-not-delete"

{lamindb-0.69.8 → lamindb-0.69.9}/lamindb/core/types.py

@@ -4,14 +4,12 @@
    :toctree: .
 
    UPathStr
-   DataLike
    StrField
    ListLike
    TransformType
 """
 from lamindb_setup.core.types import UPathStr
 from lnschema_core.types import (
-    DataLike,
     ListLike,
     StrField,
     TransformType,

lamindb-0.69.9/lamindb/setup/core/__init__.py

@@ -0,0 +1,4 @@
+import lamindb_setup as _lamindb_setup
+from lamindb_setup.core import *  # noqa: F403
+
+__doc__ = _lamindb_setup.core.__doc__.replace("lamindb_setup", "lamindb.setup")

{lamindb-0.69.8 → lamindb-0.69.9}/pyproject.toml

@@ -16,10 +16,10 @@ classifiers = [
 ]
 dependencies = [
     # Lamin PINNED packages
-    "lnschema_core==0.64.5",
-    "lamindb_setup==0.68.5",
+    "lnschema_core==0.64.6",
+    "lamindb_setup==0.69.0",
     "lamin_utils==0.13.1",
-    "lamin_cli==0.12.0",
+    "lamin_cli==0.12.1",
     # others
     "rapidfuzz",
     "pyarrow",

lamindb-0.69.9/sub/lamin-cli/lamin_cli/__init__.py

@@ -0,0 +1,3 @@
+"""Lamin CLI."""
+
+__version__ = "0.12.1"

{lamindb-0.69.8 → lamindb-0.69.9}/sub/lamin-cli/lamin_cli/__main__.py

@@ -1,3 +1,4 @@
+from __future__ import annotations
 import os
 import sys
 from importlib.metadata import PackageNotFoundError, version

{lamindb-0.69.8 → lamindb-0.69.9}/sub/lamin-cli/lamin_cli/_cache.py

@@ -1,3 +1,4 @@
+from __future__ import annotations
 import os
 
 if os.environ.get("NO_RICH"):

{lamindb-0.69.8 → lamindb-0.69.9}/sub/lamin-cli/lamin_cli/_get.py

@@ -1,3 +1,4 @@
+from __future__ import annotations
 from typing import Tuple
 from lamin_utils import logger
 import lamindb_setup as ln_setup

{lamindb-0.69.8 → lamindb-0.69.9}/sub/lamin-cli/lamin_cli/_migration.py

@@ -1,3 +1,4 @@
+from __future__ import annotations
 import os
 from typing import Optional
 

{lamindb-0.69.8 → lamindb-0.69.9}/sub/lamin-cli/lamin_cli/_save.py

@@ -1,3 +1,4 @@
+from __future__ import annotations
 from pathlib import Path
 from typing import Optional, Union
 import lamindb_setup as ln_setup

lamindb-0.69.9/sub/lamin-cli/tests/scripts/initialized-no-git-sync.py

@@ -0,0 +1,9 @@
+import lamindb as ln
+
+ln.connect("lamindb-unit-tests")
+
+if __name__ == "__main__":
+    # we're using new_run here to mock the notebook situation
+    # and cover the look up of an existing run in the tests
+    # new_run = True is trivial
+    print("hello!")

lamindb-0.69.9/sub/lamin-cli/tests/test_multi_process.py

@@ -0,0 +1,35 @@
+from multiprocessing import Process
+from pathlib import Path
+import subprocess
+
+scripts_dir = Path(__file__).parent.resolve() / "scripts"
+
+
+def run_script():
+    filepath = scripts_dir / "initialized-no-git-sync.py"
+    result = subprocess.run(
+        f"python {filepath}",
+        shell=True,
+        capture_output=True,
+    )
+    print(result.stdout.decode())
+    print(result.stderr.decode())
+    assert result.returncode == 0
+
+
+def test_parallel_execution():
+    n_processes = 32
+    processes = []
+
+    # Create and start new processes
+    for _ in range(n_processes):
+        p = Process(target=run_script)
+        p.start()
+        processes.append(p)
+
+    # Wait for all processes to finish
+    for p in processes:
+        p.join()
+
+    for p in processes:
+        assert p.exitcode == 0

{lamindb-0.69.8 → lamindb-0.69.9}/sub/lamindb-setup/docs/changelog.md

@@ -3,6 +3,13 @@
 <!-- prettier-ignore -->
 Name | PR | Developer | Date | Version
 --- | --- | --- | --- | ---
+🚸 Skip hub request for a purely local instance | [713](https://github.com/laminlabs/lamindb-setup/pull/713) | [falexwolf](https://github.com/falexwolf) | 2024-04-08 | 0.69.0
+🐛 Fix clashes for multi process | [712](https://github.com/laminlabs/lamindb-setup/pull/712) | [falexwolf](https://github.com/falexwolf) | 2024-04-08 |
+♻️ No longer need AWS account | [711](https://github.com/laminlabs/lamindb-setup/pull/711) | [falexwolf](https://github.com/falexwolf) | 2024-04-08 |
+📝 Fix docs | [709](https://github.com/laminlabs/lamindb-setup/pull/709) | [falexwolf](https://github.com/falexwolf) | 2024-04-05 |
+♻️ Replace TypeVar with TypeAlias | [707](https://github.com/laminlabs/lamindb-setup/pull/707) | [falexwolf](https://github.com/falexwolf) | 2024-04-04 |
+♻️ Refactor clean up | [706](https://github.com/laminlabs/lamindb-setup/pull/706) | [falexwolf](https://github.com/falexwolf) | 2024-04-04 |
+🩹 Do bucket injection trick only for s3 | [705](https://github.com/laminlabs/lamindb-setup/pull/705) | [Koncopd](https://github.com/Koncopd) | 2024-04-04 |
 🚸 Add region param | [704](https://github.com/laminlabs/lamindb-setup/pull/704) | [falexwolf](https://github.com/falexwolf) | 2024-04-04 |
 🐛 Avoid the error due to deleting the same file by multiple processes | [703](https://github.com/laminlabs/lamindb-setup/pull/703) | [Koncopd](https://github.com/Koncopd) | 2024-04-04 | 0.68.5
 🐛 Fix directory uploads for hosted instances | [702](https://github.com/laminlabs/lamindb-setup/pull/702) | [Koncopd](https://github.com/Koncopd) | 2024-04-04 |