datachain 0.21.1__tar.gz → 0.23.0__tar.gz

{datachain-0.21.1 → datachain-0.23.0}/.github/workflows/tests-studio.yml

@@ -98,6 +98,7 @@ jobs:
       - name: Run tests
         # Generate `.test_durations` file with `pytest --store-durations --durations-path ../.github/.test_durations ...`
         run: >
+          DATACHAIN_METASTORE_ARG_PROJECT=john
           PYTHONPATH="$(pwd)/..:${PYTHONPATH}"
           pytest
           --config-file=pyproject.toml -rs

{datachain-0.21.1 → datachain-0.23.0}/.pre-commit-config.yaml

@@ -24,7 +24,7 @@ repos:
       - id: trailing-whitespace
         exclude: '^LICENSES/'
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: 'v0.11.13'
+    rev: 'v0.12.0'
     hooks:
       - id: ruff
         args: [--fix, --exit-non-zero-on-fix]

{datachain-0.21.1 → datachain-0.23.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: datachain
-Version: 0.21.1
+Version: 0.23.0
 Summary: Wrangle unstructured AI data at scale
 Author-email: Dmitry Petrov <support@dvc.org>
 License-Expression: Apache-2.0
@@ -94,7 +94,7 @@ Requires-Dist: scipy; extra == "tests"
 Requires-Dist: ultralytics; extra == "tests"
 Provides-Extra: dev
 Requires-Dist: datachain[docs,tests]; extra == "dev"
-Requires-Dist: mypy==1.16.0; extra == "dev"
+Requires-Dist: mypy==1.16.1; extra == "dev"
 Requires-Dist: types-python-dateutil; extra == "dev"
 Requires-Dist: types-pytz; extra == "dev"
 Requires-Dist: types-PyYAML; extra == "dev"

datachain-0.23.0/docs/guide/db_migrations.md

@@ -0,0 +1,114 @@
+# Handling Local Database Migrations (CLI)
+
+When using the DataChain CLI, datasets are stored in a local SQLite database located at:
+
+```
+.datachain/db
+```
+
+Unlike the SaaS version (Studio), the CLI does **not** support automatic database migrations. This means that after upgrading the DataChain CLI, the local database schema may become incompatible with the updated codebase.
+
+## Schema Mismatch Detection
+
+The CLI automatically checks for schema compatibility. If a mismatch is detected, you’ll see an error like:
+
+```
+OutdatedDatabaseSchemaError: You have an old version of the database schema. Please refer to the documentation for more information.
+```
+
+This typically happens after upgrading the CLI to a newer version.
+
+## How to Fix It
+
+The recommended fix is to **delete the local database** and let the CLI recreate it. To avoid losing datasets, you should **export them before removing the database**.
+
+Before deleting the file, we strongly recommend making a backup of your current database:
+
+```bash
+cp .datachain/db .datachain/db.backup
+```
+
+This allows you to recover data manually if needed later.
+
+---
+
+## Exporting and Re-Importing All Local Datasets
+
+**Important:** Exporting datasets must be done **before upgrading** to a new DataChain version. Export with the old version to avoid the `OutdatedDatabaseSchemaError` during export. After deleting the database file, upgrade/install the new DataChain version.
+
+### Step 1: Export All Datasets to Parquet
+
+Export all datasets into a folder named `exported_datasets` (created if it doesn't exist). Each dataset will be saved to a file in the format:
+
+```
+<dataset_name>.<dataset_version>.parquet
+```
+
+Example: `metrics.1.0.1.parquet`
+
+```python
+import os
+import datachain as dc
+
+export_dir = "exported_datasets"
+os.makedirs(export_dir, exist_ok=True)
+
+# dc.datasets() returns a chain of DatasetInfo objects
+for ds_info in dc.datasets(column="dataset").to_values("dataset"):
+    ds = dc.read_dataset(ds_info.name, version=ds_info.version)
+    filename = f"{ds_info.name}.{ds_info.version}.parquet"
+    filepath = os.path.join(export_dir, filename)
+    ds.to_parquet(filepath)
+```
+
+### Step 2: Delete Local Database
+
+Make sure you've backed it up (see above), then:
+
+```bash
+rm .datachain/db
+```
+
+### Step 3: Re-import All Datasets from Parquet (In Correct Version Order)
+
+To avoid import errors due to semantic versioning constraints, datasets must be imported in ascending order by version for each dataset name.
+
+```python
+import os
+import datachain as dc
+from packaging.version import Version
+
+import_dir = "exported_datasets"
+
+# Gather all dataset files
+datasets = []
+
+for fname in os.listdir(import_dir):
+    if not fname.endswith(".parquet"):
+        continue
+    base = fname[:-8]  # remove '.parquet'
+    name, version = base.split('.', 1)  # split on first dot
+    filepath = os.path.join(import_dir, fname)
+    datasets.append((name, Version(version), filepath))
+
+# Sort by dataset name and then by version ascending
+datasets.sort(key=lambda x: (x[0], x[1]))
+
+# Import datasets in order
+for name, version, filepath in datasets:
+    dc.read_parquet(filepath).save(name, version=str(version))
+```
+
+**Note:** While exporting and importing datasets to Parquet files preserves the datasets and their data, some metadata — such as dataset dependencies — will **not** be preserved. This information will be lost during this process.
+
+---
+
+## Notes
+
+- This limitation only applies to the **CLI**, which uses a local SQLite database.
+- The **Studio (SaaS)** version handles all schema migrations automatically — no manual steps are required.
+- The CLI only supports the default namespace/project: `local.local`.
+
+---
+
+This export/import workflow is the recommended way to preserve your datasets during local CLI upgrades that involve database schema changes.

datachain-0.23.0/docs/guide/env.md

@@ -0,0 +1,22 @@
+# Environment Variables
+
+List of environment variables used to configure DataChain behavior.
+
+### Core Configuration
+
+- `DATACHAIN_ROOT_DIR` – Specifies the root directory where DataChain will create the `.datachain` folder to store its internal data. (default: the current working directory).
+- `DATACHAIN_SYSTEM_CONFIG_DIR` – Overrides the system-wide configuration directory (default depends on the platform).
+- `DATACHAIN_GLOBAL_CONFIG_DIR` – Overrides the user's global configuration directory (default depends on the platform).
+- `DATACHAIN_NO_ANALYTICS` – Disables telemetry.
+
+### Studio Integration
+
+- `DATACHAIN_STUDIO_URL` – Custom Studio URL.
+- `DATACHAIN_STUDIO_TOKEN` – Authentication token for Studio.
+- `DATACHAIN_STUDIO_TEAM` – Studio team name.
+
+### Namespaces and projects
+- `DATACHAIN_NAMESPACE` – Namespace name to use as default.
+- `DATACHAIN_PROJECT` – Project name or combination of namespace name and project name separated by `.` to use as default, example: `DATACHAIN_PROJECT=dev.analytics`
+
+Note: Some environment variables are used internally and may not be documented here. For the most up-to-date list, refer to the source code.

{datachain-0.21.1 → datachain-0.23.0}/docs/guide/index.md

@@ -10,3 +10,6 @@ Welcome to the DataChain User Guide! This section provides comprehensive documen
 - [Data Processing Overview](./processing.md) - Discover DataChain's specialized data processing features.
     - [Delta Processing](./delta.md) - Incremental data processing to efficiently handle large datasets that change over time.
     - [Error Handling and Retries](./retry.md) - Learn how to handle processing errors and selectively reprocess problematic records.
+- [Environment Variables](./env.md) - Configure DataChain's behavior using environment variables.
+- [Namespaces](./namespaces.md) - Learn more about namespaces and projects.
+- [Local DB Migrations](./namespaces.md) - Learn how to handle local DB migrations after upgrading datachain.

datachain-0.23.0/docs/guide/namespaces.md

@@ -0,0 +1,161 @@
+# Organizing Datasets with Namespace and Project
+
+DataChain allows you to organize datasets using namespaces and projects. These provide an additional structure for managing data across different workflows, use cases, or organizational structures.
+
+A dataset in DataChain is organized as:
+
+```
+<namespace>.<project>.<dataset>
+```
+
+For example:
+
+```
+dev.analytics.metrics
+```
+
+## Default Namespace and Project
+
+If no namespace or project is specified, DataChain uses defaults depending on whether you're using **Studio** or the **CLI**.
+
+### Studio
+
+- **Namespace:** `users`
+- **Project:** your username (e.g. `jondoe`)
+- Saving without namespace/project:
+
+```python
+dc.read_values(scores=[1.2, 3.4, 2.5]).save("metrics")
+# Saved as users.jondoe.metrics
+```
+
+### CLI
+
+- **Namespace:** `local`
+- **Project:** `local`
+- Saving without namespace/project:
+
+```python
+dc.read_values(scores=[2.0, 2.2, 2.8]).save("metrics")
+# Saved as local.local.metrics
+```
+
+In the CLI, you cannot create or use any namespaces or projects other than the default `local.local`.
+
+## Creating a Project (Studio only)
+
+In Studio, you can explicitly create a project and namespace using:
+
+```python
+import datachain as dc
+
+dc.create_project("dev", "analytics")
+```
+
+This creates the `dev` namespace (if it doesn't exist) and a project called `analytics` inside it.
+
+**Note:** Creating custom namespaces and projects is only supported in **Studio**. In the **CLI**, only the default `local` namespace and `local` project are available.
+
+## Saving a Dataset Using a Fully Qualified Name
+
+You can implicitly create and use namespaces and projects by saving a dataset using a fully qualified name:
+
+```python
+dc.read_values(scores=[1.2, 3.4, 2.5]).save("dev.analytics.metrics")
+```
+
+In Studio, this automatically creates the namespace and project if they don’t already exist.
+
+In CLI, only `local.local.<dataset>` is supported. Using any other namespace or project will result in an error.
+
+## Using `.settings()` to Set Namespace and Project
+
+You can also set the namespace and project using `.settings()`:
+
+```python
+dc.read_values(scores=[1.2, 3.4, 2.5])
+  .settings(namespace="dev", project="analytics")
+  .save("metrics")
+```
+
+This is equivalent to saving to `dev.analytics.metrics`.
+
+In CLI, `.settings()` is only supported when both `namespace` and `project` are set to `"local"`.
+
+## Setting Namespace and Project via Environment Variables
+
+In addition to using `.settings()`, you can configure the namespace and project using environment variables:
+
+- `DATACHAIN_NAMESPACE` sets the namespace.
+- `DATACHAIN_PROJECT` sets the project name, or both the namespace and project using the format `namespace.project`.
+
+### Examples
+
+```
+# Set namespace only
+export DATACHAIN_NAMESPACE=dev
+
+# Set project only
+export DATACHAIN_PROJECT=analytics
+
+# Set both namespace and project
+export DATACHAIN_PROJECT=dev.analytics
+```
+
+##  How Namespace and Project Are Resolved
+
+When determining which namespace and project to use, Datachain applies the following precedence:
+
+1. **Fully qualified dataset name**
+   If the dataset name includes both the namespace and project, these values take highest precedence.
+   ```python
+   dc.read_dataset("dev.analytics.metrics")
+
+2. **Explicit settings in code**
+   Values provided via `.settings()` or passed directly to `read_dataset()` or similar methods.
+   ```python
+   dc.settings(namespace="dev", project="analytics")
+   dc.read_dataset("metrics", namespace="dev", project="analytics")
+   ```
+3. **Environment variables**
+   Namespace and project set using environment variables:
+   ```console
+   export DATACHAIN_PROJECT=dev.analytics
+   ```
+4. **Defaults**
+If none of the above are provided, Datachain falls back to the default namespace and project.
+
+## Reading a Dataset from a Project
+
+To read a dataset from a specific namespace and project:
+
+```python
+ds = dc.read_dataset("dev.analytics.metrics")
+```
+
+In CLI, this only works for datasets saved in the default `local.local` project.
+
+
+## Example (Studio)
+
+```python
+import datachain as dc
+
+dc.create_project("prod", "analytics")
+
+dc.read_csv("gs://bucket/metrics.csv") \
+  .save("prod.analytics.metrics")
+
+ds = dc.read_dataset("prod.analytics.metrics")
+ds.show()
+```
+
+## Example (CLI – default only)
+
+```python
+import datachain as dc
+
+dc.read_values(scores=[0.8, 1.5, 2.1]).save("metrics")
+
+ds = dc.read_dataset("local.local.metrics")
+ds.show()

{datachain-0.21.1 → datachain-0.23.0}/examples/get_started/json-csv-reader.py

@@ -48,7 +48,7 @@ def main():
 
     # Print JSON schema in Pydantic format from main COCO annotation
     chain = dc.read_storage(uri, anon="True").filter(dc.C("file.path").glob("*.json"))
-    file = next(chain.limit(1).collect("file"))
+    file = chain.limit(1).to_values("file")[0]
     print(gen_datamodel_code(file, jmespath="@", model_name="Coco"))
 
     # Static JSON schema test parsing 3/7 objects

{datachain-0.21.1 → datachain-0.23.0}/examples/incremental_processing/delta.py

@@ -47,7 +47,7 @@ def process_files_with_delta():
     print("\nDataset versions:")
     test_dataset = dc.datasets().filter(C("name") == "test_files")
 
-    for version in test_dataset.collect("version"):
+    for version in test_dataset.to_iter("version"):
         print(f"- Version: {version}")
 
     # Show the last 3 records to demonstrate the incremental processing

{datachain-0.21.1 → datachain-0.23.0}/examples/llm_and_nlp/hf-dataset-llm-eval.py

@@ -1,3 +1,5 @@
+import os
+
 from huggingface_hub import InferenceClient
 from requests import HTTPError
 
@@ -23,6 +25,7 @@ def eval_dialog(
 ) -> DialogEval:
     try:
         completion = client.chat_completion(
+            model="meta-llama/Llama-3.3-70B-Instruct",
             messages=[
                 {
                     "role": "user",
@@ -31,9 +34,10 @@ def eval_dialog(
             ],
             response_format={"type": "json", "value": DialogEval.model_json_schema()},
         )
-    except HTTPError:
+    except HTTPError as e:
         return DialogEval(
-            result="Error", reason="Error while interacting with the Hugging Face API."
+            result="Error",
+            reason=f"Error while interacting with the Hugging Face API. {e}",
         )
 
     message = completion.choices[0].message
@@ -48,9 +52,15 @@ def eval_dialog(
 # Save to HF as Parquet. Dataset can be previewed here:
 # https://huggingface.co/datasets/dvcorg/test-datachain-llm-eval/viewer
 (
-    dc.read_csv("hf://datasets/infinite-dataset-hub/MobilePlanAssistant/data.csv")
-    .settings(parallel=10)
-    .setup(client=lambda: InferenceClient("meta-llama/Llama-3.1-70B-Instruct"))
+    dc.read_csv(
+        "hf://datasets/infinite-dataset-hub/MobilePlanAssistant/data.csv", source=False
+    )
+    .settings(parallel=True)
+    .setup(
+        client=lambda: InferenceClient(
+            provider="hf-inference", api_key=os.environ["HF_TOKEN"]
+        )
+    )
     .map(response=eval_dialog)
     .to_parquet("hf://datasets/dvcorg/test-datachain-llm-eval/data.parquet")
 )

{datachain-0.21.1 → datachain-0.23.0}/mkdocs.yml

@@ -105,6 +105,9 @@ nav:
               - Overview: guide/processing.md
               - Delta Processing: guide/delta.md
               - Errors Handling and Retries: guide/retry.md
+          - Environment Variables: guide/env.md
+          - Namespaces: guide/namespaces.md
+          - Local DB Migrations: guide/db_migrations.md
       - 🤝 Contributing: contributing.md
 
   - DataChain Website ↗: https://datachain.ai" target="_blank"

{datachain-0.21.1 → datachain-0.23.0}/pyproject.toml

@@ -108,7 +108,7 @@ tests = [
 ]
 dev = [
   "datachain[docs,tests]",
-  "mypy==1.16.0",
+  "mypy==1.16.1",
   "types-python-dateutil",
   "types-pytz",
   "types-PyYAML",
@@ -221,7 +221,8 @@ ignore = [
   "PERF203",  # perflint - try-except-in-loop, irrelevant for Python>=3.11
   "PERF401",
   "D100",  # undocumented-public-module
-  "D205"  # one-blank-line-after-class
+  "D205",  # one-blank-line-after-class
+  "PLC0415"  # import-outside-top-level
 ]
 select = [
   "B",  # flake8-bugbear

{datachain-0.21.1 → datachain-0.23.0}/src/datachain/__init__.py

@@ -32,6 +32,7 @@ from datachain.lib.file import (
     VideoFrame,
 )
 from datachain.lib.model_store import ModelStore
+from datachain.lib.projects import create as create_project
 from datachain.lib.udf import Aggregator, Generator, Mapper
 from datachain.lib.utils import AbstractUDF, DataChainError
 from datachain.query import metrics, param
@@ -62,6 +63,7 @@ __all__ = [
     "VideoFile",
     "VideoFragment",
     "VideoFrame",
+    "create_project",
     "datasets",
     "delete_dataset",
     "is_chain_type",

{datachain-0.21.1 → datachain-0.23.0}/src/datachain/cache.py

@@ -39,7 +39,7 @@ def temporary_cache(
             cache.destroy()
 
 
-class Cache:
+class Cache:  # noqa: PLW1641
     def __init__(self, cache_dir: str, tmp_dir: str):
         self.odb = LocalHashFileDB(
             LocalFileSystem(),
@@ -76,9 +76,9 @@ class Cache:
     async def download(
         self, file: "File", client: "Client", callback: Optional[Callback] = None
     ) -> None:
-        from_path = f"{file.source}/{file.path}"
         from dvc_objects.fs.utils import tmp_fname
 
+        from_path = file.get_uri()
         odb_fs = self.odb.fs
         tmp_info = odb_fs.join(self.odb.tmp_dir, tmp_fname())  # type: ignore[arg-type]
         size = file.size