lamindb 0.73.0__tar.gz → 0.73.2__tar.gz

{lamindb-0.73.0 → lamindb-0.73.2}/.github/workflows/build.yml

@@ -119,6 +119,12 @@ jobs:
         env:
           NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
           NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }}
+      - uses: peter-evans/repository-dispatch@v2
+        if: ${{ github.event_name == 'push' }}
+        with:
+          token: ${{ secrets.LAMIN_BUILD_DOCS }}
+          repository: "laminlabs/lamin-docs"
+          event-type: build
 
   coverage:
     needs: test

{lamindb-0.73.0 → lamindb-0.73.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: lamindb
-Version: 0.73.0
+Version: 0.73.2
 Summary: A data framework for biology.
 Author-email: Lamin Labs <open-source@lamin.ai>
 Requires-Python: >=3.8
@@ -9,8 +9,8 @@ Classifier: Programming Language :: Python :: 3.8
 Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
-Requires-Dist: lnschema_core==0.68.0
-Requires-Dist: lamindb_setup==0.73.0
+Requires-Dist: lnschema_core==0.69.0
+Requires-Dist: lamindb_setup==0.73.1
 Requires-Dist: lamin_utils==0.13.2
 Requires-Dist: lamin_cli==0.14.0
 Requires-Dist: rapidfuzz
@@ -25,7 +25,7 @@ Requires-Dist: graphviz
 Requires-Dist: psycopg2-binary
 Requires-Dist: psutil
 Requires-Dist: lamindb_setup[aws] ; extra == "aws"
-Requires-Dist: bionty==0.43.1 ; extra == "bionty"
+Requires-Dist: bionty==0.44.0 ; extra == "bionty"
 Requires-Dist: pandas<2 ; extra == "dev"
 Requires-Dist: pre-commit ; extra == "dev"
 Requires-Dist: nox ; extra == "dev"

{lamindb-0.73.0 → lamindb-0.73.2}/docs/changelog.md

@@ -2,7 +2,7 @@
 
 :::{note}
 
-If using LaminHub, please use the latest version of lamindb.
+If using LaminHub, please use the latest version of LaminDB.
 
 LaminDB implements "migration-based versioning". When upgrading your LaminDB installation to a new `minor` version in `major.minor.patch`, you also migrate your database by calling `lamin migrate deploy`.
 
@@ -14,6 +14,18 @@ LaminDB implements "migration-based versioning". When upgrading your LaminDB ins
 
 ## 0.73
 
+### 0.73.2 {small}`2024-06-13`
+
+- 🐛 Fix clashing reverse accessors for `.previous_runs` and `.run` [PR](https://github.com/laminlabs/lamindb/pull/1698) [@falexwolf](https://github.com/falexwolf)
+- 🐛 Import IPython inside view [PR](https://github.com/laminlabs/lamindb/pull/1696) [@Koncopd](https://github.com/Koncopd)
+
+### 0.73.1 {small}`2024-06-05`
+
+- 🏗️ Instantly synchronize instance schema with the hub [PR](https://github.com/laminlabs/lamindb/pull/1689) [@fredericenard](https://github.com/fredericenard)
+- ⬆️ Upgrade `universal_pathlib` to 0.2.2 [PR](https://github.com/laminlabs/lamindb/pull/1687) [@Koncopd](https://github.com/Koncopd)
+- 🐛 Fix generation of `uid` for manual Transform constructor [PR](https://github.com/laminlabs/lamindb/pull/1684) [@falexwolf](https://github.com/falexwolf)
+- 🔥 Deleting `artifact.stage()` in favor of `artifact.cache()` (was deprecated in 0.70.0)
+
 ### 0.73.0 {small}`2024-05-29`
 
 Annotating & querying by features improved:
@@ -28,7 +40,7 @@ Example:
 # annotate dict-style (feature & category names get validated)
 artifact.features.add_values({
     "species": "setosa",
-    "scientist": ["Barabara McClintock", "Edgar Anderson"],
+    "scientist": ["Barbara McClintock", "Edgar Anderson"],
     "instrument": "Leica IIIc Camera",
     "temperature": 27.6,
     "study": "Study 0: initial plant gathering",

{lamindb-0.73.0 → lamindb-0.73.2}/docs/introduction.ipynb

@@ -77,6 +77,26 @@
     "!lamin init --storage ./lamin-intro --schema bionty"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now you're ready to import lamindb in a Python session."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "import lamindb as ln"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -98,8 +118,6 @@
    },
    "outputs": [],
    "source": [
-    "import lamindb as ln\n",
-    "\n",
     "# copy-pasted identifiers for your notebook or script\n",
     "ln.settings.transform.stem_uid = \"FPnfDtJz8qbE\"  # <-- auto-generated by running ln.track()\n",
     "ln.settings.transform.version = \"1\"  # <-- auto-generated by running ln.track()\n",
@@ -107,7 +125,7 @@
     "# track the execution of your notebook or script\n",
     "run = ln.track()\n",
     "\n",
-    "# get your currently running transform\n",
+    "# see your currently running transform\n",
     "run.transform"
    ]
   },
@@ -115,7 +133,31 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "`ln.track()` added a record to the {class}`~lamindb.Transform` registry to query & search your notebook and a {class}`~lamindb.Run` registry to store the specific notebook run."
+    "Calling `ln.track()` created a run and a transform and stored them in {class}`~lamindb.Run` and {class}`~lamindb.Transform`, respectively. The transform registry allows to find your notebook, scripts and pipelines. The {class}`~lamindb.Run` registry allows to find their runs.\n",
+    "\n",
+    ":::{dropdown} Is this compliant with OpenLineage?\n",
+    "\n",
+    "Yes. What OpenLineage calls a \"job\", LaminDB calls a \"transform\". What OpenLineage calls a \"run\", LaminDB calls a \"run\".\n",
+    "\n",
+    ":::\n",
+    "\n",
+    ":::{dropdown} Are the global identifiers `stem_uid` and `version` really necessary?\n",
+    "\n",
+    "Yes, if you want to track data lineage outside of workflow managers.\n",
+    "\n",
+    "To tie a piece of code to a record in a database in a way that survives name changes (notebook file name, script name, pipeline name) and content changes, you need to attach the code to an immutable identifier.\n",
+    "\n",
+    "Content changes need to correlate with the identifier, but shouldn't give rise to an entirely new identifier. To implement this behavior we code hashes with user input for `version` and create a universal identifier as follows:\n",
+    "\n",
+    "```python\n",
+    "stem_uid = random_base62(n_char)  # a random base62 sequence of length n_char\n",
+    "version_uid = encode_base62(md5_hash(version))[:4]  # version is, e.g., \"1\" or \"2.1.0\" or \"2022-03-01\"\n",
+    "uid = f\"{stem_uid}{version_uid}\"  # concatenate the stem_uid & version_uid\n",
+    "```\n",
+    "\n",
+    "git, by comparison, identifies code by its content hash alone. Notebook platforms like Google Colab and DeepNote [do not encode versions into their notebook ids](https://lamin.ai/blog/nbproject#metadata-tracking). \n",
+    "\n",
+    ":::"
    ]
   },
   {

{lamindb-0.73.0 → lamindb-0.73.2}/docs/tutorial.ipynb

@@ -367,7 +367,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Visualize data lineage with {meth}`~lamindb.core.Data.view_lineage`:"
+    "Visualize data lineage with {meth}`~lamindb.core.HasFeatures.view_lineage`:"
    ]
   },
   {

{lamindb-0.73.0 → lamindb-0.73.2}/lamindb/__init__.py

@@ -41,7 +41,7 @@ Modules & settings:
 """
 
 # denote a release candidate for 0.1.0 with 0.1rc1, 0.1a1, 0.1b1, etc.
-__version__ = "0.73.0"
+__version__ = "0.73.2"
 
 import os as _os
 

{lamindb-0.73.0 → lamindb-0.73.2}/lamindb/_annotate.py

@@ -813,7 +813,10 @@ def standardize_and_inspect(
     values: Iterable[str], field: FieldAttr, registry: Registry, **kwargs
 ):
     """Standardize and inspect values using a registry."""
-    if hasattr(registry, "standardize"):
+    if hasattr(registry, "standardize") and hasattr(
+        registry,
+        "synonyms",  # https://github.com/laminlabs/lamindb/issues/1685
+    ):
         values = registry.standardize(values, field=field, mute=True, **kwargs)
     return registry.inspect(values, field=field, mute=True, **kwargs)
 
@@ -976,7 +979,7 @@ def save_artifact(
         except ImportError:
             pass
     if artifact is None:
-        raise ValueError("data must be a DataFrame, AnnData or MuData object")
+        raise ValueError("data must be a DataFrame, AnnData or MuData object.")
     artifact.save()
 
     feature_kwargs = check_registry_organism(

{lamindb-0.73.0 → lamindb-0.73.2}/lamindb/_artifact.py

@@ -196,6 +196,8 @@ def get_stat_or_artifact(
     if not isinstance(path, LocalPathClasses):
         size, hash, hash_type = None, None, None
         if stat is not None:
+            # convert UPathStatResult to fsspec info dict
+            stat = stat.as_info()
             if "ETag" in stat:  # is file
                 size, hash, hash_type = get_stat_file_cloud(stat)
             elif stat["type"] == "directory":
@@ -1109,6 +1111,5 @@ for name in METHOD_NAMES:
 Artifact._delete_skip_storage = _delete_skip_storage
 Artifact._save_skip_storage = _save_skip_storage
 Artifact.path = path
-Artifact.stage = cache
 # this seems a Django-generated function
 delattr(Artifact, "get_visibility_display")

{lamindb-0.73.0 → lamindb-0.73.2}/lamindb/_registry.py

@@ -54,7 +54,7 @@ def suggest_records_with_similar_names(record: Registry, kwargs) -> bool:
     """
     if kwargs.get("name") is None:
         return False
-    queryset = _search(record.__class__, kwargs["name"], truncate_words=True, limit=5)
+    queryset = _search(record.__class__, kwargs["name"], truncate_words=True, limit=20)
     if not queryset.exists():  # empty queryset
         return False
     for alternative_record in queryset:
@@ -549,24 +549,3 @@ if ln_setup._TESTING:  # type: ignore
 
 for name in METHOD_NAMES:
     attach_func_to_class_method(name, Registry, globals())
-
-
-@classmethod  # type: ignore
-def __get_schema_name__(cls) -> str:
-    schema_module_name = cls.__module__.split(".")[0]
-    schema_name = schema_module_name.replace("lnschema_", "")
-    return schema_name
-
-
-@classmethod  # type: ignore
-def __get_name_with_schema__(cls) -> str:
-    schema_name = cls.__get_schema_name__()
-    if schema_name == "core":
-        schema_prefix = ""
-    else:
-        schema_prefix = f"{schema_name}."
-    return f"{schema_prefix}{cls.__name__}"
-
-
-Registry.__get_schema_name__ = __get_schema_name__
-Registry.__get_name_with_schema__ = __get_name_with_schema__

{lamindb-0.73.0 → lamindb-0.73.2}/lamindb/_view.py

@@ -1,14 +1,16 @@
 from __future__ import annotations
 
+import builtins
 import importlib
 import inspect
 
-from IPython.display import display
 from lamin_utils import colors, logger
 from lamindb_setup import settings
 from lamindb_setup._init_instance import get_schema_module_name
 from lnschema_core import Registry
 
+is_run_from_ipython = getattr(builtins, "__IPYTHON__", False)
+
 
 def view(
     n: int = 7, schema: str | None = None, registries: list[str] | None = None
@@ -25,6 +27,11 @@ def view(
     Examples:
         >>> ln.view()
     """
+    if is_run_from_ipython:
+        from IPython.display import display as show
+    else:
+        show = logger.print
+
     if schema is not None:
         schema_names = [schema]
     else:
@@ -60,4 +67,4 @@ def view(
                 df = orm.df().iloc[-n:]
             if df.shape[0] > 0:
                 logger.print(colors.blue(colors.bold(orm.__name__)))
-                display(df)
+                show(df)

{lamindb-0.73.0 → lamindb-0.73.2}/lamindb/core/_label_manager.py

@@ -125,13 +125,13 @@ def validate_labels(labels: QuerySet | list | dict, parents: bool = True):
 
 
 class LabelManager:
-    """Label manager (:attr:`~lamindb.core.Data.labels`).
+    """Label manager (:attr:`~lamindb.core.HasFeatures.labels`).
 
     This allows to manage untyped labels :class:`~lamindb.ULabel` and arbitrary
     typed labels (e.g., :class:`~bionty.CellLine`) and associate labels
     with features.
 
-    See :class:`~lamindb.core.Data` for more information.
+    See :class:`~lamindb.core.HasFeatures` for more information.
     """
 
     def __init__(self, host: Artifact | Collection):

{lamindb-0.73.0 → lamindb-0.73.2}/lamindb/core/schema.py

@@ -1,10 +1,10 @@
-from typing import Type
+from __future__ import annotations
 
 from django.db.models import ManyToManyField
 from lnschema_core.models import Feature, FeatureSet, LinkORM, Registry
 
 
-def dict_schema_name_to_model_name(orm: Type[Registry]) -> dict[str, Registry]:
+def dict_schema_name_to_model_name(orm: type[Registry]) -> dict[str, Registry]:
     d: dict = {
         i.related_model.__get_name_with_schema__(): i.related_model
         for i in orm._meta.related_objects
@@ -21,7 +21,7 @@ def dict_schema_name_to_model_name(orm: Type[Registry]) -> dict[str, Registry]:
 
 
 def dict_related_model_to_related_name(
-    orm: Type[Registry], links: bool = False
+    orm: type[Registry], links: bool = False
 ) -> dict[str, str]:
     def include(model: Registry):
         return not links != issubclass(model, LinkORM)

{lamindb-0.73.0 → lamindb-0.73.2}/lamindb/core/versioning.py

@@ -138,7 +138,7 @@ def process_is_new_version_of(
     if is_new_version_of is not None and not isinstance(is_new_version_of, type):
         raise TypeError(f"is_new_version_of has to be of type {type}")
     if is_new_version_of is None:
-        uid = init_uid(version=version, n_full_id=type._len_stem_uid)
+        uid = init_uid(version=version, n_full_id=type._len_full_uid)
     else:
         uid, version = get_uid_from_old_version(is_new_version_of, version)
         if name is None:

{lamindb-0.73.0 → lamindb-0.73.2}/lamindb/integrations/_vitessce.py

@@ -39,9 +39,13 @@ def save_vitessce_config(vitessce_config: VitessceConfig, description: str) -> A
             if "url" not in file:
                 raise ValueError("Each file must have a 'url' key.")
             filename = file["url"].split("/")[-1]
-            assert filename.endswith((".anndata.zarr", ".spatialdata.zarr"))
-            filestem = filename.replace(".anndata.zarr", "").replace(
-                ".spatialdata.zarr", ""
+            assert filename.endswith(
+                (".anndata.zarr", ".spatialdata.zarr", ".ome.zarr")
+            )
+            filestem = (
+                filename.replace(".anndata.zarr", "")
+                .replace(".spatialdata.zarr", "")
+                .replace(".ome.zarr", "")
             )
             artifact = Artifact.filter(uid__startswith=filestem).one_or_none()
             if artifact is None:

{lamindb-0.73.0 → lamindb-0.73.2}/pyproject.toml

@@ -16,8 +16,8 @@ classifiers = [
 ]
 dependencies = [
     # Lamin PINNED packages
-    "lnschema_core==0.68.0",
-    "lamindb_setup==0.73.0",
+    "lnschema_core==0.69.0",
+    "lamindb_setup==0.73.1",
     "lamin_utils==0.13.2",
     "lamin_cli==0.14.0",
     # others
@@ -39,7 +39,7 @@ Home = "https://github.com/laminlabs/lamindb"
 
 [project.optional-dependencies]
 bionty = [
-    "bionty==0.43.1",
+    "bionty==0.44.0",
 ]
 aws = [
     "lamindb_setup[aws]",

{lamindb-0.73.0 → lamindb-0.73.2}/sub/bionty/bionty/__init__.py

@@ -122,7 +122,7 @@ Developer API:
    core
 """
 
-__version__ = "0.43.1"
+__version__ = "0.44.0"
 
 from lamindb_setup._check_setup import InstanceNotSetupError, _check_instance_setup
 from lnschema_bionty import ids

{lamindb-0.73.0 → lamindb-0.73.2}/sub/bionty/pyproject.toml

@@ -8,7 +8,7 @@ authors = [{name = "Lamin Labs", email = "open-source@lamin.ai"}]
 readme = "README.md"
 dynamic = ["version", "description"]
 dependencies = [
-    "lnschema-bionty==0.42.2",
+    "lnschema-bionty==0.43.0",
 ]
 
 [project.urls]

{lamindb-0.73.0 → lamindb-0.73.2}/sub/lamindb-setup/docs/changelog.md

@@ -3,6 +3,9 @@
 <!-- prettier-ignore -->
 Name | PR | Developer | Date | Version
 --- | --- | --- | --- | ---
+✨ Add .ome.zarr as composite suffix | [775](https://github.com/laminlabs/lamindb-setup/pull/775) | [sunnyosun](https://github.com/sunnyosun) | 2024-06-05 |
+🏗️ Update instance schema in the hub | [774](https://github.com/laminlabs/lamindb-setup/pull/774) | [fredericenard](https://github.com/fredericenard) | 2024-06-04 |
+⬆️ Migrate to upath 0.2.2 | [723](https://github.com/laminlabs/lamindb-setup/pull/723) | [Koncopd](https://github.com/Koncopd) | 2024-06-02 |
 🐛 Fix trailing slash in upload_from source | [773](https://github.com/laminlabs/lamindb-setup/pull/773) | [Koncopd](https://github.com/Koncopd) | 2024-05-23 | 0.73.0
 🚸 Make `upload_from()`, `download_to()`, and `view_tree()` more user friendly | [772](https://github.com/laminlabs/lamindb-setup/pull/772) | [falexwolf](https://github.com/falexwolf) | 2024-05-23 |
 ✨ Resolve s3 bucket region even without access rights | [771](https://github.com/laminlabs/lamindb-setup/pull/771) | [Koncopd](https://github.com/Koncopd) | 2024-05-22 |

{lamindb-0.73.0 → lamindb-0.73.2}/sub/lamindb-setup/docs/hub-cloud/06-connect-hosted-instance.ipynb

@@ -86,7 +86,7 @@
    "outputs": [],
    "source": [
     "target_dir = root / \"test-dir-upload\"\n",
-    "target_dir.upload_from(test_dir)\n",
+    "target_dir.upload_from(test_dir, create_folder=True) # default\n",
     "\n",
     "assert target_dir.is_dir()\n",
     "assert (target_dir / \"test-dir-upload\").exists()\n",
@@ -102,34 +102,11 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# test trailing slash in target_dir\n",
-    "target_dir = root / \"test-dir-upload/\"\n",
-    "assert target_dir.path.endswith(\"/\")\n",
-    "\n",
-    "dest_dir = target_dir.upload_from(test_dir, create_folder=True)\n",
-    "\n",
-    "assert \"//\" not in dest_dir.path\n",
-    "assert dest_dir.as_posix() == (root / \"test-dir-upload/test-dir-upload\").as_posix()\n",
-    "assert dest_dir.is_dir()\n",
-    "\n",
-    "dest_dir.rmdir()\n",
-    "assert not dest_dir.exists()"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "target_dir_check = root / \"test-dir-upload\"\n",
-    "assert not target_dir_check.exists()\n",
-    "\n",
     "target_dir.upload_from(test_dir, create_folder=False)\n",
     "\n",
-    "assert target_dir_check.is_dir()\n",
-    "assert (target_dir_check / \"file1\").exists()\n",
-    "assert not (target_dir_check / \"test-dir-upload\").exists()"
+    "assert target_dir.is_dir()\n",
+    "assert (target_dir / \"file1\").exists()\n",
+    "assert not (target_dir / \"test-dir-upload\").exists()"
    ]
   },
   {

{lamindb-0.73.0 → lamindb-0.73.2}/sub/lamindb-setup/docs/hub-prod/test-cloud-sync.ipynb

@@ -237,7 +237,8 @@
    "source": [
     "time.sleep(1)\n",
     "cloud_file = dir_sync / \"file1\"\n",
-    "cloud_file.touch()  # update cloud timestamp\n",
+    "# update cloud timestamp, exist_ok=False needed due to truncate=not exist_ok in upath\n",
+    "cloud_file.touch(exist_ok=False)  \n",
     "\n",
     "assert cloud_file.modified.timestamp() > local_file_new.stat().st_mtime"
    ]

{lamindb-0.73.0 → lamindb-0.73.2}/sub/lamindb-setup/lamindb_setup/__init__.py

@@ -34,7 +34,7 @@ Modules & settings:
 
 """
 
-__version__ = "0.73.0"  # denote a release candidate for 0.1.0 with 0.1rc1
+__version__ = "0.73.1"  # denote a release candidate for 0.1.0 with 0.1rc1
 
 import sys
 from os import name as _os_name

{lamindb-0.73.0 → lamindb-0.73.2}/sub/lamindb-setup/lamindb_setup/_init_instance.py

@@ -16,7 +16,7 @@ from ._silence_loggers import silence_loggers
 from .core import InstanceSettings
 from .core._settings import settings
 from .core._settings_storage import StorageSettings, init_storage
-from .core.upath import convert_pathlike
+from .core.upath import UPath
 
 if TYPE_CHECKING:
     from pydantic import PostgresDsn
@@ -351,7 +351,7 @@ def infer_instance_name(
         return str(db).split("/")[-1]
     if storage == "create-s3":
         raise ValueError("pass name to init if storage = 'create-s3'")
-    storage_path = convert_pathlike(storage)
+    storage_path = UPath(storage)
     if storage_path.name != "":
         name = storage_path.name
     else:

{lamindb-0.73.0 → lamindb-0.73.2}/sub/lamindb-setup/lamindb_setup/_migrate.py

@@ -68,6 +68,8 @@ class migrate:
     @classmethod
     def deploy(cls) -> None:
         """Deploy a migration."""
+        from ._schema_metadata import update_schema_in_hub
+
         if _check_instance_setup():
             raise RuntimeError("Restart Python session to migrate or use CLI!")
         from lamindb_setup.core._hub_client import call_with_fallback_auth
@@ -104,6 +106,9 @@ class migrate:
         # this populates the hub
         if instance_is_on_hub:
             logger.important(f"updating lamindb version in hub: {lamindb.__version__}")
+            # TODO: integrate update of instance table within update_schema_in_hub & below
+            if settings.instance.dialect != "sqlite":
+                update_schema_in_hub()
             call_with_fallback_auth(
                 update_instance,
                 instance_id=settings.instance._id.hex,