lamindb 1.1.0__tar.gz → 1.1.1__tar.gz

{lamindb-1.1.0 → lamindb-1.1.1}/.gitignore

@@ -59,6 +59,7 @@ default_storage
 default_storage_unit_core
 default_storage_unit_storage
 test.ipynb
+test2.ipynb
 run-tests
 test-django-validation/
 

{lamindb-1.1.0 → lamindb-1.1.1}/.pre-commit-config.yaml

@@ -6,8 +6,8 @@ default_stages:
   - pre-push
 minimum_pre_commit_version: 2.16.0
 repos:
-  - repo: https://github.com/pre-commit/mirrors-prettier
-    rev: v4.0.0-alpha.8
+  - repo: https://github.com/rbubley/mirrors-prettier
+    rev: v3.5.1
     hooks:
       - id: prettier
         exclude: |

{lamindb-1.1.0 → lamindb-1.1.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: lamindb
-Version: 1.1.0
+Version: 1.1.1
 Summary: A data framework for biology.
 Author-email: Lamin Labs <open-source@lamin.ai>
 Requires-Python: >=3.10,<3.13
@@ -10,7 +10,7 @@ Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Requires-Dist: lamin_utils==0.13.10
 Requires-Dist: lamin_cli==1.1.0
-Requires-Dist: lamindb_setup[aws]==1.1.0
+Requires-Dist: lamindb_setup[aws]==1.2.0
 Requires-Dist: pyyaml
 Requires-Dist: pyarrow
 Requires-Dist: pandera

{lamindb-1.1.0 → lamindb-1.1.1}/docs/arrays.ipynb

@@ -125,7 +125,11 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {},
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
    "outputs": [],
    "source": [
     "artifact_shard1 = ln.Artifact(\n",
@@ -472,7 +476,11 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {},
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
    "outputs": [],
    "source": [
     "backed.to_table().to_pandas()"

{lamindb-1.1.0 → lamindb-1.1.1}/docs/curate-df.ipynb

@@ -63,7 +63,7 @@
     "\n",
     "df = pd.DataFrame(\n",
     "    {\n",
-    "        \"cell_medium\": pd.Categorical([\"DMSO\", \"IFNG\", \"DMSO\"]),\n",
+    "        \"perturbation\": pd.Categorical([\"DMSO\", \"IFNG\", \"DMSO\"]),\n",
     "        \"temperature\": [37.2, 36.3, 38.2],\n",
     "        \"cell_type\": pd.Categorical(\n",
     "            [\n",
@@ -75,7 +75,7 @@
     "        \"assay_ontology_id\": pd.Categorical(\n",
     "            [\"EFO:0008913\", \"EFO:0008913\", \"EFO:0008913\"]\n",
     "        ),\n",
-    "        \"donor\": [\"D0001\", \"D0002\", \"D0003\"],\n",
+    "        \"donor\": [\"D0001\", \"D0002\", None],\n",
     "    },\n",
     "    index=[\"obs1\", \"obs2\", \"obs3\"],\n",
     ")\n",
@@ -104,23 +104,31 @@
     "schema = ln.Schema(\n",
     "    name=\"My example schema\",\n",
     "    features=[\n",
-    "        ln.Feature(name=\"cell_medium\", dtype=ln.ULabel).save(),\n",
+    "        ln.Feature(name=\"perturbation\", dtype=ln.ULabel).save(),\n",
     "        ln.Feature(name=\"temperature\", dtype=float).save(),\n",
     "        ln.Feature(name=\"cell_type\", dtype=bt.CellType).save(),\n",
     "        ln.Feature(\n",
     "            name=\"assay_ontology_id\", dtype=bt.ExperimentalFactor.ontology_id\n",
     "        ).save(),\n",
-    "        ln.Feature(name=\"donor\", dtype=str).save(),\n",
+    "        ln.Feature(name=\"donor\", dtype=str, nullable=True).save(),\n",
     "    ],\n",
     ").save()\n",
-    "# look at the schema\n",
+    "# display the associated features as a dataframe\n",
     "schema.features.df()"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "7",
+   "metadata": {},
+   "source": [
+    "Create a `Curator` using the dataset & the schema."
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "7",
+   "id": "8",
    "metadata": {
     "tags": [
      "hide-output"
@@ -133,16 +141,16 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8",
+   "id": "9",
    "metadata": {},
    "source": [
-    "The {meth}`~lamindb.Curator.validate` method checks our data against the defined criteria. It identifies which values are already validated (exist in our registries) and which are potentially problematic (do not yet exist in our registries)."
+    "The {meth}`~lamindb.curators.Curator.validate` method validates that your dataset adheres to the criteria defined by the `schema`. It identifies which values are already validated (exist in our registries) and which are potentially problematic (do not yet exist in our registries)."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "9",
+   "id": "10",
    "metadata": {
     "tags": [
      "hide-output"
@@ -159,8 +167,12 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "10",
-   "metadata": {},
+   "id": "11",
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
    "outputs": [],
    "source": [
     "# check the non-validated terms\n",
@@ -169,7 +181,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "11",
+   "id": "12",
    "metadata": {},
    "source": [
     "For `cell_type`, we saw that \"cerebral pyramidal neuron\", \"astrocytic glia\" are not validated.\n",
@@ -180,7 +192,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "12",
+   "id": "13",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -190,8 +202,12 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "13",
-   "metadata": {},
+   "id": "14",
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
    "outputs": [],
    "source": [
     "# now we have only one non-validated cell type left\n",
@@ -200,7 +216,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "14",
+   "id": "15",
    "metadata": {},
    "source": [
     "For \"cerebral pyramidal neuron\", let's understand which cell type in the public ontology might be the actual match."
@@ -209,7 +225,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "15",
+   "id": "16",
    "metadata": {
     "tags": [
      "hide-output"
@@ -226,7 +242,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "16",
+   "id": "17",
    "metadata": {
     "tags": [
      "hide-output"
@@ -242,7 +258,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "17",
+   "id": "18",
    "metadata": {
     "tags": [
      "hide-output"
@@ -251,23 +267,23 @@
    "outputs": [],
    "source": [
     "# fix the cell type\n",
-    "df.cell_type = df.cell_type.replace(\n",
+    "df.cell_type = df.cell_type.cat.rename_categories(\n",
     "    {\"cerebral pyramidal neuron\": cell_types.cerebral_cortex_pyramidal_neuron.name}\n",
     ")"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "18",
+   "id": "19",
    "metadata": {},
    "source": [
-    "For donor, we want to add the new donors: \"D0001\", \"D0002\", \"D0003\""
+    "For perturbation, we want to add the new values: \"DMSO\", \"IFNG\""
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "19",
+   "id": "20",
    "metadata": {
     "tags": [
      "hide-output"
@@ -275,14 +291,14 @@
    },
    "outputs": [],
    "source": [
-    "# this adds donors that were _not_ validated\n",
-    "curator.cat.add_new_from(\"cell_medium\")"
+    "# this adds perturbations that were _not_ validated\n",
+    "curator.cat.add_new_from(\"perturbation\")"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "20",
+   "id": "21",
    "metadata": {
     "tags": [
      "hide-output"
@@ -296,7 +312,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "21",
+   "id": "22",
    "metadata": {},
    "source": [
     "Save a curated artifact."
@@ -305,8 +321,12 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "22",
-   "metadata": {},
+   "id": "23",
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
    "outputs": [],
    "source": [
     "artifact = curator.save_artifact(key=\"my_datasets/my_curated_dataset.parquet\")"
@@ -315,8 +335,12 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "23",
-   "metadata": {},
+   "id": "24",
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
    "outputs": [],
    "source": [
     "artifact.describe()"
@@ -324,7 +348,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "24",
+   "id": "25",
    "metadata": {},
    "source": [
     "## Curate an AnnData\n",
@@ -335,7 +359,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "25",
+   "id": "26",
    "metadata": {
     "tags": [
      "hide-output"
@@ -363,35 +387,25 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "26",
+   "id": "27",
    "metadata": {},
    "outputs": [],
    "source": [
     "# define var schema\n",
     "var_schema = ln.Schema(\n",
     "    name=\"my_var_schema\",\n",
-    "    itype=bt.Gene.ensembl_gene_id,\n",
+    "    itype=bt.Gene.ensembl_gene_id,  # identifier type\n",
     "    dtype=int,\n",
     ").save()\n",
     "\n",
     "# define composite schema\n",
     "anndata_schema = ln.Schema(\n",
     "    name=\"small_dataset1_anndata_schema\",\n",
-    "    otype=\"AnnData\",\n",
+    "    otype=\"AnnData\",  # object type\n",
     "    components={\"obs\": schema, \"var\": var_schema},\n",
     ").save()"
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "27",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "var_schema.itype"
-   ]
-  },
   {
    "cell_type": "code",
    "execution_count": null,
@@ -448,10 +462,7 @@
    "outputs": [],
    "source": [
     "curator = ln.curators.AnnDataCurator(adata_validated, anndata_schema)\n",
-    "try:\n",
-    "    curator.validate()\n",
-    "except ln.errors.ValidationError as error:\n",
-    "    print(error)"
+    "curator.validate()"
    ]
   },
   {
@@ -459,7 +470,7 @@
    "id": "33",
    "metadata": {},
    "source": [
-    "The validated object can be subsequently saved as an {class}`~lamindb.Artifact`:"
+    "The validated `AnnData` can be subsequently saved as an {class}`~lamindb.Artifact`:"
    ]
   },
   {
@@ -481,7 +492,7 @@
    "id": "35",
    "metadata": {},
    "source": [
-    "Saved artifact has been annotated with validated features and labels:"
+    "The saved artifact has been annotated with validated features and labels:"
    ]
   },
   {
@@ -502,6 +513,95 @@
    "cell_type": "markdown",
    "id": "37",
    "metadata": {},
+   "source": [
+    "## Standardize an AnnData"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "38",
+   "metadata": {},
+   "source": [
+    "If you need more control, you can access the underlying `\"var\"` and `\"obs\"` `DataFrameCurator` objects directly."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "39",
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
+   "outputs": [],
+   "source": [
+    "curator.slots[\"var\"]\n",
+    "curator.slots[\"obs\"]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "40",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# revert the previous cell type standardization\n",
+    "df[\"cell_type\"] = df[\"cell_type\"].cat.rename_categories(\n",
+    "    {\"astrocyte\": \"astrocytic glia\"}\n",
+    ")\n",
+    "# an AnnData where a cell type matches a synonym\n",
+    "adata_with_synonym = ad.AnnData(X=adata_validated.X, var=adata_validated.var, obs=df)\n",
+    "adata_with_synonym"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "41",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "curator = ln.curators.AnnDataCurator(adata_with_synonym, anndata_schema)\n",
+    "try:\n",
+    "    curator.validate()\n",
+    "except ln.errors.ValidationError as error:\n",
+    "    print(error)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "42",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "curator.slots[\"obs\"].cat.standardize(\"cell_type\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "43",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "curator.validate()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "44",
+   "metadata": {},
+   "source": [
+    "## Summary"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "45",
+   "metadata": {},
    "source": [
     "We've walked through the process of validating, standardizing, and annotating datasets going through these key steps:\n",
     "\n",
@@ -518,8 +618,12 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "38",
-   "metadata": {},
+   "id": "46",
+   "metadata": {
+    "tags": [
+     "hide-cell"
+    ]
+   },
    "outputs": [],
    "source": [
     "!rm -rf ./test-curate\n",

{lamindb-1.1.0 → lamindb-1.1.1}/docs/faq/idempotency.ipynb

@@ -33,11 +33,7 @@
     "\n",
     "### Data: artifacts & collections\n",
     "\n",
-    "If you try to create a {class}`~lamindb.Artifact` object from the same content, depending on {attr}`~lamindb.core.subsettings.CreationSettings.artifact_if_hash_exists`, \n",
-    "\n",
-    "- you'll get an existing object, if `creation.artifact_if_hash_exists = \"warn_return_existing\"` (the default)\n",
-    "- you'll get an error, if `creation.artifact_if_hash_exists = \"error\"`\n",
-    "- you'll get a warning and a new object, if `creation.artifact_if_hash_exists = \"warn_create_new\"`"
+    "If you try to create a {class}`~lamindb.Artifact` object from the same content, you'll get an existing artifact instead."
    ]
   },
   {

{lamindb-1.1.0 → lamindb-1.1.1}/docs/registries.ipynb

@@ -56,12 +56,12 @@
     "\n",
     "# Create non-curated datasets\n",
     "ln.Artifact(datasets.file_jpg_paradisi05(), key=\"images/my_image.jpg\").save()\n",
-    "ln.Artifact(datasets.file_fastq(), key=\"raw/my_fastq.fastq\").save()\n",
+    "ln.Artifact(datasets.file_fastq(), key=\"raw/my_fastq.fastq.gz\").save()\n",
     "ln.Artifact.from_df(datasets.df_iris(), key=\"iris/iris_collection.parquet\").save()\n",
     "\n",
     "# Create a more complex case\n",
     "# observation-level metadata\n",
-    "ln.Feature(name=\"cell_medium\", dtype=\"cat[ULabel]\").save()\n",
+    "ln.Feature(name=\"perturbation\", dtype=\"cat[ULabel]\").save()\n",
     "ln.Feature(name=\"sample_note\", dtype=\"str\").save()\n",
     "ln.Feature(name=\"cell_type_by_expert\", dtype=\"cat[bionty.CellType]\").save()\n",
     "ln.Feature(name=\"cell_type_by_model\", dtype=\"cat[bionty.CellType]\").save()\n",
@@ -84,7 +84,7 @@
     "    adata,\n",
     "    var_index=bt.Gene.ensembl_gene_id,\n",
     "    categoricals={\n",
-    "        \"cell_medium\": ln.ULabel.name,\n",
+    "        \"perturbation\": ln.ULabel.name,\n",
     "        \"cell_type_by_expert\": bt.CellType.name,\n",
     "        \"cell_type_by_model\": bt.CellType.name,\n",
     "    },\n",
@@ -99,7 +99,7 @@
     "    adata2,\n",
     "    var_index=bt.Gene.ensembl_gene_id,\n",
     "    categoricals={\n",
-    "        \"cell_medium\": ln.ULabel.name,\n",
+    "        \"perturbation\": ln.ULabel.name,\n",
     "        \"cell_type_by_model\": bt.CellType.name,\n",
     "    },\n",
     "    organism=\"human\",\n",

{lamindb-1.1.0 → lamindb-1.1.1}/docs/setup.ipynb

@@ -241,6 +241,62 @@
     "```"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Use paths with s3-compatible endpoints"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This is an experimental feature."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "It is possible to init an instance with a path that uses an s3-compatible endpoint url. Such endpoints allow to access non-s3 buckets using the same API that is used for s3."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "```\n",
+    "lamin init --storage s3://<bucket_name>?endpoint_url=http://endpoint.com:port\n",
+    "```"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This assumes that the endpoint url is `http://endpoint.com` with a port specified."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "It is also possible to set a path with s3-compatible endpoint as a default storage for an existing instance for the current python session."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "```python\n",
+    "import lamindb as ln # connected an existing instance\n",
+    "ln.settings.storage = \"s3://<bucket_name>?endpoint_url=http://endpoint.com:port\"\n",
+    "# or using ln.UPath\n",
+    "ln.settings.storage = ln.UPath(\"s3://<bucket_name>\", endpoint_url=\"http://endpoint.com:port\")\n",
+    "```"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -299,7 +355,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Dsiconnect from an instance"
+    "## Disconnect from an instance"
    ]
   },
   {
@@ -408,7 +464,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.17"
+   "version": "3.10.16"
   },
   "vscode": {
    "interpreter": {

{lamindb-1.1.0 → lamindb-1.1.1}/lamindb/__init__.py

@@ -1,12 +1,13 @@
 """A data framework for biology.
 
-Tracking notebooks & scripts.
+Tracking notebooks, scripts & functions.
 
 .. autosummary::
    :toctree: .
 
    track
    finish
+   tracked
 
 Registries.
 
@@ -57,11 +58,12 @@ Backward compatibility.
    :toctree: .
 
    FeatureSet
+   Curator
 
 """
 
 # denote a release candidate for 0.1.0 with 0.1rc1, 0.1a1, 0.1b1, etc.
-__version__ = "1.1.0"
+__version__ = "1.1.1"
 
 from lamindb_setup._check_setup import InstanceNotSetupError as _InstanceNotSetupError
 from lamindb_setup._check_setup import _check_instance_setup