lamindb 0.76.9__tar.gz → 0.76.11__tar.gz

{lamindb-0.76.9 → lamindb-0.76.11}/.github/workflows/build.yml

@@ -107,7 +107,7 @@ jobs:
       - uses: actions/setup-python@v5
         with:
           python-version: "3.10" # use 3.10, 3.8 and 3.11 give issues
-      - run: pip install -U laminci
+      - run: pip install "laminci@git+https://x-access-token:${{ secrets.LAMIN_BUILD_DOCS }}@github.com/laminlabs/laminci"
       - run: nox -s "install_ci(group='docs')"
       - uses: actions/download-artifact@v4
       - run: nox -s docs

{lamindb-0.76.9 → lamindb-0.76.11}/.gitmodules

@@ -16,3 +16,6 @@
 [submodule "sub/clinicore"]
 	path = sub/clinicore
 	url = https://github.com/laminlabs/clinicore
+[submodule "sub/findrefs"]
+	path = sub/findrefs
+	url = https://github.com/laminlabs/findrefs

{lamindb-0.76.9 → lamindb-0.76.11}/PKG-INFO

@@ -1,18 +1,17 @@
 Metadata-Version: 2.1
 Name: lamindb
-Version: 0.76.9
+Version: 0.76.11
 Summary: A data framework for biology.
 Author-email: Lamin Labs <open-source@lamin.ai>
-Requires-Python: >=3.8
+Requires-Python: >=3.9
 Description-Content-Type: text/markdown
-Classifier: Programming Language :: Python :: 3.8
 Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
-Requires-Dist: lnschema_core==0.74.5
-Requires-Dist: lamindb_setup==0.77.5
-Requires-Dist: lamin_utils==0.13.4
-Requires-Dist: lamin_cli==0.17.6
+Requires-Dist: lnschema_core==0.74.6
+Requires-Dist: lamindb_setup==0.77.7
+Requires-Dist: lamin_utils==0.13.6
+Requires-Dist: lamin_cli==0.17.8
 Requires-Dist: rapidfuzz
 Requires-Dist: pyarrow
 Requires-Dist: typing_extensions!=4.6.0
@@ -24,7 +23,8 @@ Requires-Dist: pandas
 Requires-Dist: graphviz
 Requires-Dist: psycopg2-binary
 Requires-Dist: lamindb_setup[aws] ; extra == "aws"
-Requires-Dist: bionty==0.50.2 ; extra == "bionty"
+Requires-Dist: bionty==0.51.1 ; extra == "bionty"
+Requires-Dist: line_profiler ; extra == "dev"
 Requires-Dist: pre-commit ; extra == "dev"
 Requires-Dist: nox ; extra == "dev"
 Requires-Dist: laminci>=0.3 ; extra == "dev"

{lamindb-0.76.9 → lamindb-0.76.11}/docs/api.md

@@ -18,6 +18,8 @@ cli
 lamindb
 bionty
 wetlab
+clinicore
+findrefs
 ```
 
 ```{toctree}

{lamindb-0.76.9 → lamindb-0.76.11}/docs/bio-registries.ipynb

@@ -275,7 +275,7 @@
    "source": [
     "We'd like to load the corresponding records in our in-house registry to annotate a dataset.\n",
     "\n",
-    "To this end, you'll typically use {class}`~lamindb.core.Record.from_values`, which will both validate & retrieve records that match the values."
+    "To this end, you'll typically use {class}`~lamindb.core.CanValidate.from_values`, which will both validate & retrieve records that match the values."
    ]
   },
   {

lamindb-0.76.11/docs/clinicore.md

@@ -0,0 +1,5 @@
+# `clinicore`
+
+```{eval-rst}
+.. automodule:: clinicore
+```

{lamindb-0.76.9 → lamindb-0.76.11}/docs/curate-any.ipynb

@@ -24,7 +24,7 @@
     "\n",
     "{class}`~lamindb.core.CanValidate` methods validate against the registries in your LaminDB instance.\n",
     "In {doc}`./bio-registries`, you'll see how to extend standard validation to validation against _public references_ using a `ReferenceTable` ontology object: `public = Record.public()`.\n",
-    "By default, {meth}`~lamindb.core.Record.from_values` considers a match in a public reference a validated value for any {mod}`bionty` entity.\n",
+    "By default, {meth}`~lamindb.core.CanValidate.from_values` considers a match in a public reference a validated value for any {mod}`bionty` entity.\n",
     "\n",
     ":::"
    ]
@@ -131,7 +131,7 @@
    "source": [
     "Following the suggestions to register new labels:\n",
     "\n",
-    "Bulk creating records using {meth}`~lamindb.core.Record.from_values` only returns validated records:\n",
+    "Bulk creating records using {meth}`~lamindb.core.CanValidate.from_values` only returns validated records:\n",
     "\n",
     "Note: Terms validated with public reference are also created with `.from_values`, see {doc}`/bio-registries` for details."
    ]

{lamindb-0.76.9 → lamindb-0.76.11}/docs/faq/idempotency.ipynb

@@ -71,8 +71,7 @@
     "import pytest\n",
     "\n",
     "ln.settings.verbosity = \"hint\"\n",
-    "ln.context.uid = \"ANW20Fr4eZgM0000\"\n",
-    "ln.context.track()"
+    "ln.track(\"ANW20Fr4eZgM0000\")"
    ]
   },
   {

{lamindb-0.76.9 → lamindb-0.76.11}/docs/faq/keep-artifacts-local.ipynb

@@ -37,8 +37,7 @@
     "# !pip install 'lamindb[jupyter,aws]'\n",
     "import lamindb as ln\n",
     "\n",
-    "ln.context.uid = \"l9lFf83aPwRc0000\"\n",
-    "ln.context.track()"
+    "ln.track(\"l9lFf83aPwRc0000\")"
    ]
   },
   {

{lamindb-0.76.9 → lamindb-0.76.11}/docs/faq/key.ipynb

@@ -119,8 +119,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "ln.context.uid = \"WIwaNDvlEkwS0000\"\n",
-    "ln.context.track()"
+    "ln.track(\"WIwaNDvlEkwS0000\")"
    ]
   },
   {

{lamindb-0.76.9 → lamindb-0.76.11}/docs/faq/setup.ipynb

@@ -19,7 +19,7 @@
    },
    "outputs": [],
    "source": [
-    "!lamin load --unload"
+    "!lamin disconnect"
    ]
   },
   {
@@ -54,7 +54,7 @@
    "outputs": [],
    "source": [
     "with pytest.raises(ln._InstanceNotSetupError):\n",
-    "    ln.context.track()"
+    "    ln.track()"
    ]
   },
   {
@@ -82,8 +82,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "ln.context.uid = \"2lhqA4uTKSFP0000\"\n",
-    "ln.context.track()"
+    "ln.track(\"2lhqA4uTKSFP0000\")"
    ]
   },
   {

{lamindb-0.76.9 → lamindb-0.76.11}/docs/faq/track-run-inputs.ipynb

@@ -63,7 +63,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Call `ln.context.track()`:"
+    "Call `ln.track()`:"
    ]
   },
   {
@@ -72,8 +72,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "ln.context.uid = \"Rx2s9aPTMQLY0000\"\n",
-    "ln.context.track()"
+    "ln.track(\"Rx2s9aPTMQLY0000\")\n"
    ]
   },
   {

lamindb-0.76.11/docs/findrefs.md

@@ -0,0 +1,5 @@
+# `findrefs`
+
+```{eval-rst}
+.. automodule:: findrefs
+```

lamindb-0.76.11/docs/scripts/run-track-with-params.py

@@ -0,0 +1,17 @@
+import argparse
+import lamindb as ln
+
+if __name__ == "__main__":
+    p = argparse.ArgumentParser()
+    p.add_argument("--input-dir", type=str)
+    p.add_argument("--downsample", action="store_true")
+    p.add_argument("--learning-rate", type=float)
+    args = p.parse_args()
+    params = {
+        "input_dir": args.input_dir,
+        "learning_rate": args.learning_rate,
+        "downsample": args.downsample,
+    }
+    ln.track("JjRF4mACd9m00000", params=params)
+    # your code
+    ln.finish()

lamindb-0.76.11/docs/scripts/synced-with-git.py

@@ -0,0 +1,6 @@
+import lamindb as ln
+
+ln.settings.sync_git_repo = "https://github.com/..."
+ln.track()
+# your code
+ln.finish()

{lamindb-0.76.9 → lamindb-0.76.11}/docs/setup.ipynb

@@ -111,16 +111,16 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Load an instance"
+    "## Connecting to an instance"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Load your own instance:\n",
+    "Connect to your own instance:\n",
     "```\n",
-    "lamin load <instance_name>\n",
+    "lamin connect <instance_name>\n",
     "````"
    ]
   },
@@ -128,9 +128,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Load somebody else's instance:\n",
+    "Connect to somebody else's instance:\n",
     "```\n",
-    "lamin load <account_handle/instance_name>\n",
+    "lamin connect <account_handle/instance_name>\n",
     "```"
    ]
   },
@@ -170,7 +170,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "!lamin info"
+    "!lamin settings"
    ]
   },
   {
@@ -225,25 +225,19 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Update default storage"
+    "## Update the default storage"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "It's easiest to see and update default storage in the Python API using {attr}`~lamindb.core.Settings.storage`:\n",
+    "It's easiest to see and update the default storage for the current Python session in the Python API using {attr}`~lamindb.core.Settings.storage`:\n",
     "\n",
     "```python\n",
     "import lamindb as ln\n",
     "ln.settings.storage  # set via ln.settings.storage = \"s3://other-bucket\"\n",
     "#> s3://default-bucket\n",
-    "```\n",
-    "\n",
-    "You can also change it using the CLI via\n",
-    "\n",
-    "```\n",
-    "lamin set --storage s3://other-bucket\n",
     "```"
    ]
   },
@@ -291,18 +285,18 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Close an instance"
+    "## Dsiconnect from an instance"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Loading an instance means loading an environment for managing your datasets.\n",
+    "Connecting to an instance means loading an environment for managing your data.\n",
     "\n",
-    "When loading a new instance, you automatically _close_ the previously loaded old instance.\n",
+    "When connecting to a new instance, you automatically _disconnect_ from the previously connected instance.\n",
     "\n",
-    "If you want to close the instance without loading a new instance, use `lamin load --unload`"
+    "If you want to disconnect from the instance without connecting to a new instance, use `lamin disconnect`"
    ]
   },
   {
@@ -330,7 +324,7 @@
     "Here is an example:\n",
     "\n",
     "```\n",
-    "% lamin load testdb\n",
+    "% lamin connect testdb\n",
     "🔶 \n",
     "\n",
     "Your database is not up to date with your installed Python library.\n",
@@ -343,7 +337,7 @@
     "\n",
     "Otherwise, downgrade your Python library to match the database!\n",
     "\n",
-    "✅ loaded instance: testuser1/testdb\n",
+    "✅ connected lamindb: testuser1/testdb\n",
     "```\n",
     ":::"
    ]

{lamindb-0.76.9 → lamindb-0.76.11}/docs/signup-login.md

@@ -32,5 +32,5 @@ lamin login <email> --key <API-key>
 Log out:
 
 ```
-lamin login --logout
+lamin logout
 ```

{lamindb-0.76.9 → lamindb-0.76.11}/docs/storage/prepare-transfer-local-to-cloud.ipynb

@@ -13,7 +13,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "!lamin set auto-connect false"
+    "!lamin settings set auto-connect false"
    ]
   },
   {

lamindb-0.76.11/docs/track.ipynb

@@ -0,0 +1,339 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Track notebooks & scripts\n",
+    "\n",
+    "This guide explains how to use {meth}`~lamindb.track` & {meth}`~lamindb.finish` to track notebook & scripts along with their inputs and outputs.\n",
+    "\n",
+    "For tracking data lineage in pipelines, see {doc}`docs:pipelines`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "# !pip install 'lamindb[jupyter]'\n",
+    "!lamin init --storage ./test-track"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Track data lineage"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Call {meth}`~lamindb.track` to register a data transformation and start tracking inputs & outputs of a run. You will find your notebook or script in the {class}`~lamindb.Transform` registry along with pipelines & functions. {class}`~lamindb.Run` stores executions."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "```python\n",
+    "import lamindb as ln\n",
+    "\n",
+    "# --> `ln.track()` generates a uid for your code\n",
+    "# --> `ln.track(uid)` initiates a tracked run\n",
+    "ln.track(\"9priar0hoE5u0000\")\n",
+    "\n",
+    "# your code\n",
+    "\n",
+    "ln.finish()  # mark run as finished, save execution report, source code & environment\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Below is how [a notebook](https://lamin.ai/laminlabs/lamindata/transform/PtTXoc0RbOIq65cN) with run report looks on the hub.\n",
+    "\n",
+    "<img src=\"https://lamin-site-assets.s3.amazonaws.com/.lamindb/RGXj5wcAf7EAc6J8dJfH.png\" width=\"900px\">"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Query & load a notebook or script"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In the API, filter {class}`~lamindb.Transform` to obtain a transform record:\n",
+    "\n",
+    "```python\n",
+    "transform = ln.Transform.get(name=\"Track notebooks & scripts\")\n",
+    "transform.source_code  # source code\n",
+    "transform.latest_run.report  # report of latest run\n",
+    "transform.latest_run.environment  # environment of latest run\n",
+    "transform.runs  # all runs\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "On the hub, search or filter the `transform` page and then load a script or notebook on the CLI. For example,\n",
+    "\n",
+    "```bash\n",
+    "lamin load https://lamin.ai/laminlabs/lamindata/transform/13VINnFk89PE0004\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Sync scripts with GitHub"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "To sync with your git commit, add the following line to your script:\n",
+    "\n",
+    "```python\n",
+    "ln.settings.sync_git_repo = <YOUR-GIT-REPO-URL>\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "```{eval-rst}\n",
+    ".. literalinclude:: scripts/synced-with-git.py\n",
+    "   :language: python\n",
+    "   :caption: synced-with-git.py\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You'll now see the GitHub emoji clickable on the hub.\n",
+    "\n",
+    "<img src=\"https://lamin-site-assets.s3.amazonaws.com/.lamindb/IpV8Kiq4xUbgXhzlUYT7.png\" width=\"900px\">\n",
+    "<br>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Track run parameters"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "LaminDB's validation dialogue auto-generates code for run parameters. Here is an example:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "import lamindb as ln\n",
+    "\n",
+    "ln.Param(name=\"input_dir\", dtype=\"str\").save()\n",
+    "ln.Param(name=\"learning_rate\", dtype=\"float\").save()\n",
+    "ln.Param(name=\"downsample\", dtype=\"bool\").save()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "```{eval-rst}\n",
+    ".. literalinclude:: scripts/run-track-with-params.py\n",
+    "   :language: python\n",
+    "   :caption: run-track-with-params.py\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Run the script."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "!python scripts/run-track-with-params.py  --input-dir ./mydataset --learning-rate 0.01 --downsample"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Query by run parameters"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Query for all runs that match a certain parameters:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "ln.Run.params.filter(learning_rate=0.01, input_dir=\"./mydataset\").df()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Look at the parameter values that were used for a given run."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "run = ln.Run.params.filter(learning_rate=0.01).order_by(\"-started_at\").first()\n",
+    "run.params.get_values()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Manage notebook templates"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "A notebook acts like a template upon using `lamin load` to load it. Consider you run:\n",
+    "\n",
+    "```bash\n",
+    "lamin load https://lamin.ai/account/instance/transform/Akd7gx7Y9oVO0000\n",
+    "```\n",
+    "\n",
+    "Upon running the returned notebook, you'll automatically create a new version and be able to browse it via the version dropdown on the UI.\n",
+    "\n",
+    "Additionally, you can:\n",
+    "\n",
+    "- label using `ULabel`, e.g., `transform.ulabels.add(template_label)`\n",
+    "- tag with an indicative `version` string, e.g., `transform.version = \"T1\"; transform.save()`"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    ":::{dropdown} Saving a notebook as an artifact\n",
+    "\n",
+    "Sometimes you might want to save a notebook as an artifact. This is how you can do it:\n",
+    "\n",
+    "```bash\n",
+    "lamin save template1.ipynb --key templates/template1.ipynb --description \"Template for analysis type 1\" --registry artifact\n",
+    "```\n",
+    "\n",
+    ":::"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "tags": [
+     "hide-cell"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "assert run.params.get_values() == {'downsample': True, 'input_dir': './mydataset', 'learning_rate': 0.01}\n",
+    "\n",
+    "# clean up test instance\n",
+    "!rm -r ./test-track\n",
+    "!lamin delete --force test-track"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.13"
+  },
+  "nbproject": {
+   "id": "9priar0hoE5u",
+   "parent": null,
+   "pypackage": null,
+   "time_init": "2023-12-04T18:09:49.226879+00:00",
+   "user_handle": null,
+   "user_id": null,
+   "user_name": null,
+   "version": "0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}