lamindb 0.63.5__tar.gz → 0.64.1__tar.gz

{lamindb-0.63.5 → lamindb-0.64.1}/.github/workflows/build.yml

@@ -4,7 +4,7 @@ on:
   push:
     branches: [release]
   pull_request:
-    branches: [release, main]
+    branches: [release, main, staging]
 
 jobs:
   test:

{lamindb-0.63.5 → lamindb-0.64.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: lamindb
-Version: 0.63.5
+Version: 0.64.1
 Summary: A data framework for biology.
 Author-email: Lamin Labs <open-source@lamin.ai>
 Requires-Python: >=3.8
@@ -8,10 +8,10 @@ Description-Content-Type: text/markdown
 Classifier: Programming Language :: Python :: 3.8
 Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
-Requires-Dist: lnschema_core==0.57.6
-Requires-Dist: lamindb_setup==0.61.2
+Requires-Dist: lnschema_core==0.58.1
+Requires-Dist: lamindb_setup==0.61.4
 Requires-Dist: lamin_utils==0.12.0
-Requires-Dist: lamin_cli==0.2.5
+Requires-Dist: lamin_cli==0.3.0
 Requires-Dist: rapidfuzz
 Requires-Dist: pyarrow
 Requires-Dist: typing_extensions!=4.6.0
@@ -27,7 +27,7 @@ Requires-Dist: boto3==1.28.17 ; extra == "aws"
 Requires-Dist: aiobotocore==2.5.4 ; extra == "aws"
 Requires-Dist: fsspec[s3]==2023.9.0 ; extra == "aws"
 Requires-Dist: s3fs>=2023.9.0 ; extra == "aws"
-Requires-Dist: lnschema_bionty==0.35.3 ; extra == "bionty"
+Requires-Dist: lnschema_bionty==0.36.1 ; extra == "bionty"
 Requires-Dist: pandas<2 ; extra == "dev"
 Requires-Dist: pre-commit ; extra == "dev"
 Requires-Dist: nox ; extra == "dev"

{lamindb-0.63.5 → lamindb-0.64.1}/docs/annotate.ipynb

@@ -668,11 +668,11 @@
    "name": "python3"
   },
   "language_info": {
+   "artifact_extension": ".py",
    "codemirror_mode": {
     "name": "ipython",
     "version": 3
    },
-   "file_extension": ".py",
    "mimetype": "text/x-python",
    "name": "python",
    "nbconvert_exporter": "python",

{lamindb-0.63.5 → lamindb-0.64.1}/docs/bio-registries.ipynb

@@ -726,7 +726,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "You can also pass organism for validating features upon registering data, e.g., in `ln.File.from_anndata(..., field=lb.Gene.ensembl_gene_id, organism=...)`."
+    "You can also pass organism for validating features upon registering data, e.g., in `ln.Artifact.from_anndata(..., field=lb.Gene.ensembl_gene_id, organism=...)`."
    ]
   },
   {
@@ -934,11 +934,11 @@
    "name": "python3"
   },
   "language_info": {
+   "artifact_extension": ".py",
    "codemirror_mode": {
     "name": "ipython",
     "version": 3
    },
-   "file_extension": ".py",
    "mimetype": "text/x-python",
    "name": "python",
    "nbconvert_exporter": "python",

{lamindb-0.63.5 → lamindb-0.64.1}/docs/changelog.md

@@ -1,13 +1,27 @@
 # Changelog
 
+```{eval-rst}
 .. role:: small
 ```
 
+## 0.64
+
+### 0.64.1 {small}`2023-11-24`
+
+- 🚸 Silence the httpx logger [PR1356](https://github.com/laminlabs/lamindb/pull/1356) [@falexwolf](https://github.com/falexwolf)
+- 🐛 Fix hash inference for cloud non-s3 objects and use stored gcs credentials [PR1355](https://github.com/laminlabs/lamindb/pull/1355) [@Koncopd](https://github.com/Koncopd)
+
+### 0.64.0 {small}`2023-11-24`
+
+🏗️ Replace `File` with `Artifact` to model directories and directory-like array stores [PR1345](https://github.com/laminlabs/lamindb/pull/1345) [@falexwolf](https://github.com/falexwolf).
+
+For artifacts that are simple files, rename `File` to `Artifact` along with the word `file` to `artifact` in registry fields. Otherwise, your code will behave the same.
+
 ## 0.63
 
 ### 0.63.5 {small}`2023-12-11`
 
-- 🐛 Fix `lamin save` bug for scripts [PR1346](https://github.com/laminlabs/lamindb/pull/1346) [@sunnyosun](https://github.com/sunnyosun)
+- 🐛 Fix `lamin save` for scripts [PR1346](https://github.com/laminlabs/lamindb/pull/1346) [@sunnyosun](https://github.com/sunnyosun)
 
 ### 0.63.4 {small}`2023-12-08`
 

{lamindb-0.63.5 → lamindb-0.64.1}/docs/data.ipynb

@@ -20,7 +20,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "We saw how LaminDB allows to query & search across files & datasets using registries: {doc}`meta`.\n",
+    "We saw how LaminDB allows to query & search across artifacts & datasets using registries: {doc}`meta`.\n",
     "\n",
     "Let us now look at the following case:\n",
     "\n",
@@ -28,12 +28,12 @@
     "# get a lookup for labels\n",
     "ulabels = ln.ULabel.lookup()\n",
     "# query a parquet file matching an \"setosa\"\n",
-    "df = ln.File.filter(ulabels=ulabels.setosa, suffix=\".suffix\").first().load()\n",
+    "df = ln.Artifact.filter(ulabels=ulabels.setosa, suffix=\".suffix\").first().load()\n",
     "# query all observations in the DataFrame matching \"setosa\"\n",
     "df_setosa = df.loc[:, df.iris_organism_name == ulabels.setosa.name]  \n",
     "```\n",
     "\n",
-    "Because the file was validated, querying the `DataFrame` is guaranteed to succeed!\n",
+    "Because the artifact was validated, querying the `DataFrame` is guaranteed to succeed!\n",
     "\n",
     "Such within-dataset queries are also possible for cloud-backed datasets using DuckDB, TileDB, zarr, HDF5, parquet, and other storage backends.\n",
     "\n",
@@ -96,8 +96,8 @@
    },
    "outputs": [],
    "source": [
-    "ln.File(\"s3://lamindb-ci/lndb-storage/pbmc68k.h5ad\").save()\n",
-    "ln.File(\"s3://lamindb-ci/lndb-storage/testfile.hdf5\").save()"
+    "ln.Artifact(\"s3://lamindb-ci/lndb-storage/pbmc68k.h5ad\").save()\n",
+    "ln.Artifact(\"s3://lamindb-ci/lndb-storage/testfile.hdf5\").save()"
    ]
   },
   {
@@ -112,7 +112,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "An `h5ad` file stored on s3:"
+    "An `h5ad` artifact stored on s3:"
    ]
   },
   {
@@ -121,7 +121,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "file = ln.File.filter(key=\"lndb-storage/pbmc68k.h5ad\").one()"
+    "artifact = ln.Artifact.filter(key=\"lndb-storage/pbmc68k.h5ad\").one()"
    ]
   },
   {
@@ -130,7 +130,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "file.path"
+    "artifact.path"
    ]
   },
   {
@@ -139,7 +139,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "adata = file.backed()"
+    "adata = artifact.backed()"
    ]
   },
   {
@@ -244,7 +244,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Let us query a generic HDF5 file:"
+    "Let us query a generic HDF5 artifact:"
    ]
   },
   {
@@ -253,7 +253,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "file = ln.File.filter(key=\"lndb-storage/testfile.hdf5\").one()"
+    "artifact = ln.Artifact.filter(key=\"lndb-storage/testfile.hdf5\").one()"
    ]
   },
   {
@@ -270,7 +270,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "backed = file.backed()"
+    "backed = artifact.backed()"
    ]
   },
   {
@@ -322,11 +322,11 @@
    "name": "python3"
   },
   "language_info": {
+   "artifact_extension": ".py",
    "codemirror_mode": {
     "name": "ipython",
     "version": 3
    },
-   "file_extension": ".py",
    "mimetype": "text/x-python",
    "name": "python",
    "nbconvert_exporter": "python",

{lamindb-0.63.5 → lamindb-0.64.1}/docs/faq/acid.ipynb

@@ -15,7 +15,7 @@
    "id": "2096971d",
    "metadata": {},
    "source": [
-    "Here, we walk through different errors that can occur while saving files & metadata records, and show that the LaminDB instance does not get corrupted by dangling metadata or files. You could say transactions across data & metadata are [ACID](https://en.wikipedia.org/wiki/ACID)."
+    "Here, we walk through different errors that can occur while saving artifacts & metadata records, and show that the LaminDB instance does not get corrupted by dangling metadata or artifacts. You could say transactions across data & metadata are [ACID](https://en.wikipedia.org/wiki/ACID)."
    ]
   },
   {
@@ -77,7 +77,7 @@
    "id": "0c582719-42de-4e9d-bdcb-51f8aabb3ca5",
    "metadata": {},
    "source": [
-    "Let's try to save a file to a storage location without permission."
+    "Let's try to save a artifact to a storage location without permission."
    ]
   },
   {
@@ -87,7 +87,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "file = ln.File(\n",
+    "artifact = ln.Artifact(\n",
     "    ln.dev.datasets.anndata_mouse_sc_lymph_node(),\n",
     "    description=\"Mouse Lymph Node scRNA-seq\",\n",
     ")"
@@ -139,7 +139,7 @@
    "outputs": [],
    "source": [
     "with pytest.raises(RuntimeError) as error:\n",
-    "    file.save()\n",
+    "    artifact.save()\n",
     "print(error.exconly())"
    ]
   },
@@ -159,7 +159,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "assert len(ln.File.filter().all()) == 0"
+    "assert len(ln.Artifact.filter().all()) == 0"
    ]
   },
   {
@@ -179,8 +179,8 @@
    "outputs": [],
    "source": [
     "filepath = ln.dev.datasets.file_jpg_paradisi05()\n",
-    "file = ln.File(filepath, description=\"My image\")\n",
-    "files = [file, \"this is not a record\"]"
+    "artifact = ln.Artifact(filepath, description=\"My image\")\n",
+    "artifacts = [artifact, \"this is not a record\"]"
    ]
   },
   {
@@ -200,7 +200,7 @@
    "outputs": [],
    "source": [
     "with pytest.raises(Exception) as error:\n",
-    "    ln.save(files)\n",
+    "    ln.save(artifacts)\n",
     "print(error.exconly())"
    ]
   },
@@ -220,8 +220,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "files = ln.File.filter().all()\n",
-    "assert len(files) == 0"
+    "artifacts = ln.Artifact.filter().all()\n",
+    "assert len(artifacts) == 0"
    ]
   },
   {
@@ -257,11 +257,11 @@
    "name": "python3"
   },
   "language_info": {
+   "artifact_extension": ".py",
    "codemirror_mode": {
     "name": "ipython",
     "version": 3
    },
-   "file_extension": ".py",
    "mimetype": "text/x-python",
    "name": "python",
    "nbconvert_exporter": "python",

{lamindb-0.63.5 → lamindb-0.64.1}/docs/faq/delete.ipynb

@@ -11,7 +11,7 @@
     "When it comes to **records of `File` and `Dataset`**, they are **\"moved into trash\"** when you first call `record.delete()`.\n",
     "- Trashed records are invisible in the UI and excluded from the query results, see [visibility faq](/faq/visibility).\n",
     "- If a record is already in the trash or `permanent=True` is passed, calling `record.delete()` triggers permanent delete.\n",
-    "- During permanent deletion of a record, its file in storage is also deleted unless it has a semantic `key`."
+    "- During permanent deletion of a record, its artifact in storage is also deleted unless it has a semantic `key`."
    ]
   },
   {
@@ -46,8 +46,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "file = ln.File(pd.DataFrame({\"a\": [1, 2], \"b\": [3, 4]}), description=\"mydf\")\n",
-    "file.save()"
+    "artifact = ln.Artifact(pd.DataFrame({\"a\": [1, 2], \"b\": [3, 4]}), description=\"mydf\")\n",
+    "artifact.save()"
    ]
   },
   {
@@ -56,14 +56,14 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "ln.File.filter().df()"
+    "ln.Artifact.filter().df()"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Trash a file"
+    "## Trash a artifact"
    ]
   },
   {
@@ -72,7 +72,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "file.delete()"
+    "artifact.delete()"
    ]
   },
   {
@@ -88,14 +88,14 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "ln.File.filter().df()"
+    "ln.Artifact.filter().df()"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "But the file still exists in the database, you can find it by not filtering for visibility:"
+    "But the artifact still exists in the database, you can find it by not filtering for visibility:"
    ]
   },
   {
@@ -104,14 +104,14 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "ln.File.filter(visibility=None).df()"
+    "ln.Artifact.filter(visibility=None).df()"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "You can restore a file from trash:"
+    "You can restore a artifact from trash:"
    ]
   },
   {
@@ -120,7 +120,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "file.restore()"
+    "artifact.restore()"
    ]
   },
   {
@@ -129,7 +129,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "ln.File.filter().df()"
+    "ln.Artifact.filter().df()"
    ]
   },
   {
@@ -138,7 +138,7 @@
    "source": [
     "## Permanent delete\n",
     "\n",
-    "Calling `file.delete` on a trashed file triggers a permanent delete dialog. You can pass `permanent=True` to auto-confirm the deletion."
+    "Calling `artifact.delete` on a trashed artifact triggers a permanent delete dialog. You can pass `permanent=True` to auto-confirm the deletion."
    ]
   },
   {
@@ -147,7 +147,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "file.delete(permanent=True)"
+    "artifact.delete(permanent=True)"
    ]
   },
   {
@@ -163,7 +163,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "ln.File.filter(visibility=None).df()"
+    "ln.Artifact.filter(visibility=None).df()"
    ]
   }
  ],
@@ -174,11 +174,11 @@
    "name": "python3"
   },
   "language_info": {
+   "artifact_extension": ".py",
    "codemirror_mode": {
     "name": "ipython",
     "version": 3
    },
-   "file_extension": ".py",
    "mimetype": "text/x-python",
    "name": "python",
    "nbconvert_exporter": "python",

{lamindb-0.63.5 → lamindb-0.64.1}/docs/faq/idempotency.ipynb

@@ -6,7 +6,7 @@
    "id": "fe2d8275-0883-4ffe-9912-920b28efc9d2",
    "metadata": {},
    "source": [
-    "# What happens if I save the same files & records twice?"
+    "# What happens if I save the same artifacts & records twice?"
    ]
   },
   {
@@ -33,11 +33,11 @@
     "\n",
     "### Files\n",
     "\n",
-    "If you try to create a {class}`~lamindb.File` object from the same content, depending on {attr}`~lamindb.dev.Settings.upon_file_create_if_hash_exists`, \n",
+    "If you try to create a {class}`~lamindb.Artifact` object from the same content, depending on {attr}`~lamindb.dev.Settings.upon_artifact_create_if_hash_exists`, \n",
     "\n",
-    "- you'll get an existing object, if `upon_file_create_if_hash_exists = \"warn_return_existing\"` (the default)\n",
-    "- you'll get an error, if `upon_file_create_if_hash_exists = \"error\"`\n",
-    "- you'll get a warning and a new object, if `upon_file_create_if_hash_exists = \"warn_create_new\"`"
+    "- you'll get an existing object, if `upon_artifact_create_if_hash_exists = \"warn_return_existing\"` (the default)\n",
+    "- you'll get an error, if `upon_artifact_create_if_hash_exists = \"error\"`\n",
+    "- you'll get a warning and a new object, if `upon_artifact_create_if_hash_exists = \"warn_create_new\"`"
    ]
   },
   {
@@ -260,7 +260,7 @@
    "id": "431fbcc4",
    "metadata": {},
    "source": [
-    "#### Warn upon trying to re-ingest an existing file"
+    "#### Warn upon trying to re-ingest an existing artifact"
    ]
   },
   {
@@ -270,7 +270,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "assert ln.settings.upon_file_create_if_hash_exists == \"warn_return_existing\""
+    "assert ln.settings.upon_artifact_create_if_hash_exists == \"warn_return_existing\""
    ]
   },
   {
@@ -299,8 +299,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "file = ln.File(filepath, description=\"My fcs file\")\n",
-    "file.save()"
+    "artifact = ln.Artifact(filepath, description=\"My fcs artifact\")\n",
+    "artifact.save()"
    ]
   },
   {
@@ -314,7 +314,7 @@
    },
    "outputs": [],
    "source": [
-    "assert file.hash == \"KCEXRahJ-Ui9Y6nksQ8z1A\""
+    "assert artifact.hash == \"KCEXRahJ-Ui9Y6nksQ8z1A\""
    ]
   },
   {
@@ -333,7 +333,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "file2 = ln.File(filepath)"
+    "artifact2 = ln.Artifact(filepath)"
    ]
   },
   {
@@ -352,7 +352,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "assert file.id == file2.id"
+    "assert artifact.id == artifact2.id"
    ]
   },
   {
@@ -371,7 +371,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "file2.save()"
+    "artifact2.save()"
    ]
   },
   {
@@ -380,7 +380,7 @@
    "id": "721bbe4e",
    "metadata": {},
    "source": [
-    "#### Error upon trying to re-ingest an existing file"
+    "#### Error upon trying to re-ingest an existing artifact"
    ]
   },
   {
@@ -390,7 +390,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "ln.settings.upon_file_create_if_hash_exists = \"error\""
+    "ln.settings.upon_artifact_create_if_hash_exists = \"error\""
    ]
   },
   {
@@ -410,7 +410,7 @@
    "outputs": [],
    "source": [
     "with pytest.raises(RuntimeError):\n",
-    "    file3 = ln.File(filepath, description=\"My new fcs file\")"
+    "    artifact3 = ln.Artifact(filepath, description=\"My new fcs artifact\")"
    ]
   },
   {
@@ -419,7 +419,7 @@
    "id": "cff06da7",
    "metadata": {},
    "source": [
-    "#### Warn and create a new file"
+    "#### Warn and create a new artifact"
    ]
   },
   {
@@ -437,7 +437,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "ln.settings.upon_file_create_if_hash_exists = \"warn_create_new\""
+    "ln.settings.upon_artifact_create_if_hash_exists = \"warn_create_new\""
    ]
   },
   {
@@ -456,8 +456,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "file4 = ln.File(filepath, description=\"My new fcs file\")\n",
-    "file4.save()"
+    "artifact4 = ln.Artifact(filepath, description=\"My new fcs artifact\")\n",
+    "artifact4.save()"
    ]
   },
   {
@@ -476,7 +476,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "assert file4.id != file.id"
+    "assert artifact4.id != artifact.id"
    ]
   },
   {
@@ -486,7 +486,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "file4.filter(hash=\"KCEXRahJ-Ui9Y6nksQ8z1A\").df()"
+    "artifact4.filter(hash=\"KCEXRahJ-Ui9Y6nksQ8z1A\").df()"
    ]
   },
   {
@@ -500,7 +500,7 @@
    },
    "outputs": [],
    "source": [
-    "assert len(file.filter(hash=\"KCEXRahJ-Ui9Y6nksQ8z1A\").list()) == 2"
+    "assert len(artifact.filter(hash=\"KCEXRahJ-Ui9Y6nksQ8z1A\").list()) == 2"
    ]
   },
   {
@@ -522,11 +522,11 @@
    "name": "python3"
   },
   "language_info": {
+   "artifact_extension": ".py",
    "codemirror_mode": {
     "name": "ipython",
     "version": 3
    },
-   "file_extension": ".py",
    "mimetype": "text/x-python",
    "name": "python",
    "nbconvert_exporter": "python",

{lamindb-0.63.5 → lamindb-0.64.1}/docs/faq/import-schema.ipynb

@@ -70,11 +70,11 @@
    "name": "python3"
   },
   "language_info": {
+   "artifact_extension": ".py",
    "codemirror_mode": {
     "name": "ipython",
     "version": 3
    },
-   "file_extension": ".py",
    "mimetype": "text/x-python",
    "name": "python",
    "nbconvert_exporter": "python",