datachain 0.16.0__tar.gz → 0.16.2__tar.gz

{datachain-0.16.0 → datachain-0.16.2}/.pre-commit-config.yaml

@@ -24,7 +24,7 @@ repos:
       - id: trailing-whitespace
         exclude: '^LICENSES/'
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: 'v0.11.5'
+    rev: 'v0.11.6'
     hooks:
       - id: ruff
         args: [--fix, --exit-non-zero-on-fix]

{datachain-0.16.0/src/datachain.egg-info → datachain-0.16.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: datachain
-Version: 0.16.0
+Version: 0.16.2
 Summary: Wrangle unstructured AI data at scale
 Author-email: Dmitry Petrov <support@dvc.org>
 License-Expression: Apache-2.0

datachain-0.16.2/docs/commands/auth/login.md

@@ -0,0 +1,51 @@
+# auth login
+
+Authenticate DataChain with Studio to save a client access token to DataChain configuration.
+
+## Synopsis
+
+```usage
+usage: datachain auth login [-h] [-v] [-q] [-H HOSTNAME] [-s SCOPES] [-n NAME] [--no-open] [--local]
+```
+
+## Description
+
+By default, this command authenticates DataChain with Studio using default scopes and assigns a random name as the token name. The authentication token will be used for subsequent Studio operations.
+
+## Options
+
+* `-H HOSTNAME`, `--hostname HOSTNAME` - The hostname of the Studio instance to authenticate with.
+* `-s SCOPES`, `--scopes SCOPES` - Authentication token scopes. Allowed scopes: `EXPERIMENTS`, `DATASETS`, `MODELS`. Defaults to all available scopes.
+* `-n NAME`, `--name NAME` - The name of the authentication token. It will be used to identify the token shown in Studio profile. Defaults to a random name.
+* `--no-open` - Use code-based authentication without browser. You will be presented with a user code to enter in the browser. DataChain will also use this if it cannot launch the browser on your behalf.
+* `--local` - Save the token in the local project config instead of the global configuration.
+* `-h`, `--help` - Show the help message and exit.
+* `-v`, `--verbose` - Be verbose.
+* `-q`, `--quiet` - Be quiet.
+
+## Examples
+
+1. Basic authentication with default settings:
+```bash
+datachain auth login
+```
+
+2. Authenticate with specific scopes:
+```bash
+datachain auth login --scopes EXPERIMENTS,DATASETS
+```
+
+3. Authenticate with a custom token name:
+```bash
+datachain auth login --name my-token
+```
+
+4. Authenticate using code-based flow:
+```bash
+datachain auth login --no-open
+```
+
+5. Save token locally for the project:
+```bash
+datachain auth login --local
+```

datachain-0.16.2/docs/commands/auth/logout.md

@@ -0,0 +1,37 @@
+# auth logout
+
+Remove the Studio authentication token from DataChain configuration.
+
+## Synopsis
+
+```usage
+usage: datachain auth logout [-h] [-v] [-q] [--local]
+```
+
+## Description
+
+This command removes the Studio authentication token from the global DataChain configuration. By default, it removes the token from the global configuration, but you can also remove it from the local project configuration using the `--local` option.
+
+## Options
+
+* `--local` - Remove the token from the local project config instead of the global configuration.
+* `-h`, `--help` - Show the help message and exit.
+* `-v`, `--verbose` - Be verbose.
+* `-q`, `--quiet` - Be quiet.
+
+## Examples
+
+1. Remove token from global configuration:
+```bash
+datachain auth logout
+```
+
+2. Remove token from local project configuration:
+```bash
+datachain auth logout --local
+```
+
+3. Remove token with verbose output:
+```bash
+datachain auth logout -v
+```

datachain-0.16.2/docs/commands/auth/team.md

@@ -0,0 +1,36 @@
+# auth team
+
+Set the default team for Studio operations.
+
+## Synopsis
+
+```usage
+usage: datachain auth team [-h] [-v] [-q] [--global] team_name
+```
+
+## Description
+
+This command sets the default team for Studio operations. By default, the team setting is project-specific, but you can use the `--global` option to set it for all projects.
+
+## Arguments
+
+* `team_name` - Name of the team to set as default
+
+## Options
+
+* `--global` - Set team globally for all projects
+* `-h`, `--help` - Show the help message and exit.
+* `-v`, `--verbose` - Be verbose.
+* `-q`, `--quiet` - Be quiet.
+
+## Examples
+
+1. Set default team for current project:
+```bash
+datachain auth team my-team
+```
+
+2. Set default team globally for all projects:
+```bash
+datachain auth team --global my-team
+```

datachain-0.16.2/docs/commands/auth/token.md

@@ -0,0 +1,26 @@
+# auth token
+
+Display the current authentication token for Studio.
+
+## Synopsis
+
+```usage
+usage: datachain auth token [-h] [-v] [-q]
+```
+
+## Description
+
+This command displays the current authentication token that DataChain is using for Studio operations. The token is used for authenticating with Studio and accessing its features.
+
+## Options
+
+* `-h`, `--help` - Show the help message and exit.
+* `-v`, `--verbose` - Be verbose.
+* `-q`, `--quiet` - Be quiet.
+
+## Examples
+
+1. Display the current token:
+```bash
+datachain auth token
+```

datachain-0.16.2/docs/commands/index.md

@@ -0,0 +1,33 @@
+
+# Using DataChain Commands
+
+
+
+DataChain is a command-line tool for wrangling unstructured AI data at scale. Use `datachain -h` to list all available commands.
+
+
+
+## Typical DataChain Workflow
+
+
+
+1.  **Authentication with Studio**
+
+
+	- Use [`datachain auth login`](auth/login.md) to authenticate with Studio
+
+	- Set your default team with [`datachain auth team`](auth/team.md)
+
+	- View your token with [`datachain auth token`](auth/token.md)
+
+	- Log out from Studio with [`datachain auth logout`](auth/logout.md)
+
+
+
+2.  **Job Management**
+
+	- Run jobs in Studio with [`datachain job run`](job/run.md)
+
+	- Monitor job logs with [`datachain job logs`](job/logs.md)
+
+	- Cancel running jobs with [`datachain job cancel`](job/cancel.md)

datachain-0.16.2/docs/commands/job/cancel.md

@@ -0,0 +1,43 @@
+# job cancel
+
+Cancel a running job in Studio.
+
+## Synopsis
+
+```usage
+usage: datachain job cancel [-h] [-v] [-q] [--team TEAM] id
+```
+
+## Description
+
+This command cancels a running job in Studio. The job ID can be obtained from the Studio UI or from the output when the job was created. This is the recommended way to stop a running job, as simply closing the logs view (e.g., with Ctrl+C) will not cancel the job execution.
+
+## Arguments
+
+* `id` - Job ID to cancel. This ID is displayed when the job is created and can also be found in the Studio UI.
+
+## Options
+
+* `--team TEAM` - Team to cancel job for (default: from config)
+* `-h`, `--help` - Show the help message and exit.
+* `-v`, `--verbose` - Be verbose.
+* `-q`, `--quiet` - Be quiet.
+
+## Examples
+
+1. Cancel a specific job:
+```bash
+datachain job cancel job-123
+```
+
+2. Cancel a job in a specific team:
+```bash
+datachain job cancel --team my-team job-123
+```
+
+
+## Notes
+
+* The job ID is displayed when the job is created using `datachain job run`
+* You can also find the job ID in the Studio UI
+* This is the proper way to stop a running job, as simply closing the logs view will not cancel the job execution

datachain-0.16.2/docs/commands/job/logs.md

@@ -0,0 +1,47 @@
+# job logs
+
+Display logs and current status of jobs in Studio.
+
+## Synopsis
+
+```usage
+usage: datachain job logs [-h] [-v] [-q] [--team TEAM] id
+```
+
+## Description
+
+This command displays the logs and current status of a running job in Studio. The command will show real-time logs from the job execution. Note that closing this command (e.g., with Ctrl+C) will only stop displaying the logs but will not cancel the job execution. To cancel a job, use the `job cancel` command.
+
+## Arguments
+
+* `id` - Job ID to show logs for
+
+## Options
+
+* `--team TEAM` - Team to check logs for (default: from config)
+* `-h`, `--help` - Show the help message and exit.
+* `-v`, `--verbose` - Be verbose.
+* `-q`, `--quiet` - Be quiet.
+
+## Examples
+
+1. Display logs for a specific job:
+```bash
+datachain job logs job-123
+```
+
+2. Display logs for a job in a specific team:
+```bash
+datachain job logs --team my-team job-123
+```
+
+3. Display logs with verbose output:
+```bash
+datachain job logs -v job-123
+```
+
+## Notes
+
+* Closing the logs command (e.g., with Ctrl+C) will only stop displaying the logs but will not cancel the job execution
+* To cancel a running job, use the `datachain job cancel` command
+* The job will continue running in Studio even after you stop viewing the logs

datachain-0.16.2/docs/commands/job/run.md

@@ -0,0 +1,67 @@
+# job run
+
+Run a job in Studio.
+
+## Synopsis
+
+```usage
+usage: datachain job run [-h] [-v] [-q] [--team TEAM] [--env-file ENV_FILE] [--env ENV [ENV ...]]
+                         [--workers WORKERS] [--files FILES [FILES ...]] [--python-version PYTHON_VERSION]
+                         [--req-file REQ_FILE] [--req REQ [REQ ...]]
+                         file
+```
+
+## Description
+
+This command runs a job in Studio using the specified query file. You can configure various aspects of the job including environment variables, Python version, dependencies, and more.
+
+## Arguments
+
+* `file` - Query file to run
+
+## Options
+
+* `--team TEAM` - Team to run job for (default: from config)
+* `--env-file ENV_FILE` - File with environment variables for the job
+* `--env ENV` - Environment variables in KEY=VALUE format
+* `--workers WORKERS` - Number of workers for the job
+* `--files FILES` - Additional files to include in the job
+* `--python-version PYTHON_VERSION` - Python version for the job (e.g., 3.9, 3.10, 3.11)
+* `--req-file REQ_FILE` - Python requirements file
+* `--req REQ` - Python package requirements
+* `-h`, `--help` - Show the help message and exit.
+* `-v`, `--verbose` - Be verbose.
+* `-q`, `--quiet` - Be quiet.
+
+## Examples
+
+1. Run a basic job:
+```bash
+datachain job run query.py
+```
+
+2. Run a job with specific team and Python version:
+```bash
+datachain job run --team my-team --python-version 3.11 query.py
+```
+
+3. Run a job with environment variables and requirements:
+```bash
+datachain job run --env-file .env --req-file requirements.txt query.py
+```
+
+4. Run a job with multiple workers and additional files:
+```bash
+datachain job run --workers 4 --files utils.py config.json query.py
+```
+
+5. Run a job with inline environment variables and package requirements:
+```bash
+datachain job run --env API_KEY=123 --req pandas numpy query.py
+```
+
+## Notes
+
+* Closing the logs command (e.g., with Ctrl+C) will only stop displaying the logs but will not cancel the job execution
+* To cancel a running job, use the `datachain job cancel` command
+* The job will continue running in Studio even after you stop viewing the logs

{datachain-0.16.0 → datachain-0.16.2}/docs/examples.md

@@ -94,7 +94,7 @@ dc.DataModel.register(MistralModel)
 chain = (
     dc
     .read_storage("gs://datachain-demo/chatbot-KiT/", type="text")
-    .filter(dc.Column("file.name").glob("*.txt"))
+    .filter(dc.Column("file.path").glob("*.txt"))
     .limit(5)
     .settings(parallel=4, cache=True)
     .map(
@@ -228,7 +228,7 @@ Here is an example from MS COCO “captions” JSON which employs separate secti
 
 Note how complicated the setup is. Every image is references by the name, and the metadata for this file is keyed by the “id” field. This same field is references later in the “annotations” array, which is present in JSON files describing captions and the detected instances. The categories for the instances are stored in the “categories” array.
 
-However, Datachain can easily parse the entire COCO structure via several reading and merging operators:
+However, DataChain can easily parse the entire COCO structure via several reading and merging operators:
 
 ```python
 import datachain as dc
@@ -240,7 +240,7 @@ images = dc.read_storage(images_uri)
 meta = dc.read_json(captions_uri, jmespath="images")
 captions = dc.read_json(captions_uri, jmespath="annotations")
 
-images_meta = images.merge(meta, on="file.name", right_on="images.file_name")
+images_meta = images.merge(meta, on="file.path", right_on="images.file_name")
 captioned_images = images_meta.merge(captions, on="images.id", right_on="annotations.image_id")
 ```
 
@@ -248,12 +248,12 @@ The resulting dataset has image entries as files decorated with all the metadata
 
 ```python
 images_with_dogs = captioned_images.filter(dc.Column("annotations.caption").glob("*dog*"))
-images_with_dogs.select("annotations", "file.name").show()
+images_with_dogs.select("annotations", "file.path").show()
 ```
 
 ```
    captions captions                                           captions              file
-   image_id       id                                            caption              name
+   image_id       id                                            caption              path
 0     17029   778902         a dog jumping to catch a frisbee in a yard  000000017029.jpg
 1     17029   779838   A dog jumping to catch a red frisbee in a garden  000000017029.jpg
 2     17029   781941  The dog is catching the Frisbee in mid air in ...  000000017029.jpg

{datachain-0.16.0 → datachain-0.16.2}/docs/quick-start.md

@@ -184,7 +184,7 @@ chain = (
    .save("response")
 )
 
-chain.select("file.name", "status", "response.usage").show(5)
+chain.select("file.path", "status", "response.usage").show(5)
 
 success_rate = chain.filter(dc.Column("status") == "success").count() / chain.count()
 print(f"{100*success_rate:.1f}% dialogs were successful")
@@ -194,7 +194,7 @@ Output:
 
 ``` shell
 file   status      response     response          response
-name                  usage        usage             usage
+path                  usage        usage             usage
               prompt_tokens total_tokens completion_tokens
 0   1.txt  success           547          548                 1
 1  10.txt  failure          3576         3578                 2
@@ -277,7 +277,7 @@ processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
 
 chain = (
     dc.read_storage("gs://datachain-demo/dogs-and-cats/", type="image", anon=True)
-    .map(label=lambda name: name.split(".")[0], params=["file.name"])
+    .map(label=lambda name: name.split(".")[0], params=["file.path"])
     .select("file", "label").to_pytorch(
         transform=processor.image_processor,
         tokenizer=processor.tokenizer,

{datachain-0.16.0 → datachain-0.16.2}/mkdocs.yml

@@ -84,6 +84,18 @@ nav:
           - Torch: references/torch.md
           - Functions: references/func.md
           - Toolkit: references/toolkit.md
+      - 📖 CLI Reference:
+          - Overview: commands/index.md
+          - Commands:
+              - auth:
+                  - login: commands/auth/login.md
+                  - logout: commands/auth/logout.md
+                  - token: commands/auth/token.md
+                  - team: commands/auth/team.md
+              - job:
+                  - run: commands/job/run.md
+                  - logs: commands/job/logs.md
+                  - cancel: commands/job/cancel.md
       - 📡 Interacting with remote storage: references/remotes.md
       - 🤝 Contributing: contributing.md
 

{datachain-0.16.0 → datachain-0.16.2}/src/datachain/func/aggregate.py

@@ -165,7 +165,7 @@ def any_value(col: str) -> Func:
     Example:
         ```py
         dc.group_by(
-            file_example=func.any_value("file.name"),
+            file_example=func.any_value("file.path"),
             partition_by="signal.category",
         )
         ```
@@ -227,7 +227,7 @@ def concat(col: str, separator="") -> Func:
     Example:
         ```py
         dc.group_by(
-            files=func.concat("file.name", separator=", "),
+            files=func.concat("file.path", separator=", "),
             partition_by="signal.category",
         )
         ```
@@ -343,7 +343,7 @@ def first(col: str) -> Func:
         ```py
         window = func.window(partition_by="signal.category", order_by="created_at")
         dc.mutate(
-            first_file=func.first("file.name").over(window),
+            first_file=func.first("file.path").over(window),
         )
         ```
 

{datachain-0.16.0 → datachain-0.16.2}/src/datachain/lib/convert/values_to_tuples.py

@@ -1,6 +1,6 @@
 import itertools
 from collections.abc import Sequence
-from typing import Any, Optional, Union
+from typing import Any, Union
 
 from datachain.lib.data_model import (
     DataType,
@@ -71,14 +71,13 @@ def values_to_tuples(  # noqa: C901, PLR0912
             # If a non-None value appears early, it won't check the remaining items for
             # `None` values.
             try:
-                pos, first_not_none_element = next(
-                    itertools.dropwhile(lambda pair: pair[1] is None, enumerate(v))
+                first_not_none_element = next(
+                    itertools.dropwhile(lambda i: i is None, v)
                 )
             except StopIteration:
-                typ = str  # default to str if all values are None or has length 0
-                nullable = True
+                # set default type to `str` if column is empty or all values are `None`
+                typ = str
             else:
-                nullable = pos > 0
                 typ = type(first_not_none_element)  # type: ignore[assignment]
                 if not is_chain_type(typ):
                     raise ValuesToTupleError(
@@ -88,8 +87,7 @@ def values_to_tuples(  # noqa: C901, PLR0912
                     )
                 if isinstance(first_not_none_element, list):
                     typ = list[type(first_not_none_element[0])]  # type: ignore[assignment, misc]
-
-            types_map[k] = Optional[typ] if nullable else typ  # type: ignore[assignment]
+            types_map[k] = typ
 
         if length < 0:
             length = len_

{datachain-0.16.0 → datachain-0.16.2}/src/datachain/lib/dc/datachain.py

@@ -756,7 +756,7 @@ class DataChain:
 
         Example:
             ```py
-            dc.distinct("file.parent", "file.name")
+            dc.distinct("file.path")
             ```
         """
         return self._evolve(
@@ -882,7 +882,7 @@ class DataChain:
         ```py
          dc.mutate(
             area=Column("image.height") * Column("image.width"),
-            extension=file_ext(Column("file.name")),
+            extension=file_ext(Column("file.path")),
             dist=cosine_distance(embedding_text, embedding_image)
         )
         ```
@@ -1071,13 +1071,13 @@ class DataChain:
 
             Iterating over all rows with selected columns:
             ```py
-            for name, size in dc.collect("file.name", "file.size"):
+            for name, size in dc.collect("file.path", "file.size"):
                 print(name, size)
             ```
 
             Iterating over a single column:
             ```py
-            for file in dc.collect("file.name"):
+            for file in dc.collect("file.path"):
                 print(file)
             ```
         """
@@ -1630,7 +1630,7 @@ class DataChain:
             import datachain as dc
 
             chain = dc.read_storage("s3://mybucket")
-            chain = chain.filter(dc.C("file.name").glob("*.jsonl"))
+            chain = chain.filter(dc.C("file.path").glob("*.jsonl"))
             chain = chain.parse_tabular(format="json")
             ```
         """
@@ -2089,25 +2089,31 @@ class DataChain:
 
             Using glob to match patterns
             ```py
-            dc.filter(C("file.name").glob("*.jpg"))
+            dc.filter(C("file.path").glob("*.jpg"))
+            ```
+
+            Using in to match lists
+            ```py
+            ids = [1,2,3]
+            dc.filter(C("experiment_id").in_(ids))
             ```
 
             Using `datachain.func`
             ```py
             from datachain.func import string
-            dc.filter(string.length(C("file.name")) > 5)
+            dc.filter(string.length(C("file.path")) > 5)
             ```
 
             Combining filters with "or"
             ```py
-            dc.filter(C("file.name").glob("cat*") | C("file.name").glob("dog*))
+            dc.filter(C("file.path").glob("cat*") | C("file.path").glob("dog*))
             ```
 
             Combining filters with "and"
             ```py
             dc.filter(
-                C("file.name").glob("*.jpg) &
-                (string.length(C("file.name")) > 5)
+                C("file.path").glob("*.jpg) &
+                (string.length(C("file.path")) > 5)
             )
             ```
         """

{datachain-0.16.0 → datachain-0.16.2}/src/datachain/lib/dc/records.py

@@ -4,12 +4,9 @@ from typing import TYPE_CHECKING, Optional, Union
 import sqlalchemy
 
 from datachain.lib.data_model import DataType
-from datachain.lib.file import (
-    File,
-)
+from datachain.lib.file import File
 from datachain.lib.signal_schema import SignalSchema
 from datachain.query import Session
-from datachain.query.schema import Column
 
 if TYPE_CHECKING:
     from typing_extensions import ParamSpec
@@ -41,6 +38,10 @@ def read_records(
         single_record = dc.read_records(dc.DEFAULT_FILE_RECORD)
         ```
     """
+    from datachain.query.dataset import INSERT_BATCH_SIZE, adjust_outputs, get_col_types
+    from datachain.sql.types import SQLType
+    from datachain.utils import batched
+
     from .datasets import read_dataset
 
     session = Session.get(session, in_memory=in_memory)
@@ -52,11 +53,10 @@ def read_records(
 
     if schema:
         signal_schema = SignalSchema(schema)
-        columns = []
-        for c in signal_schema.db_signals(as_columns=True):
-            assert isinstance(c, Column)
-            kw = {"nullable": c.nullable} if c.nullable is not None else {}
-            columns.append(sqlalchemy.Column(c.name, c.type, **kw))
+        columns = [
+            sqlalchemy.Column(c.name, c.type)  # type: ignore[union-attr]
+            for c in signal_schema.db_signals(as_columns=True)
+        ]
     else:
         columns = [
             sqlalchemy.Column(name, typ)
@@ -83,6 +83,14 @@ def read_records(
     warehouse = catalog.warehouse
     dr = warehouse.dataset_rows(dsr)
     table = dr.get_table()
-    warehouse.insert_rows(table, to_insert)
+
+    # Optimization: Compute row types once, rather than for every row.
+    col_types = get_col_types(
+        warehouse,
+        {c.name: c.type for c in columns if isinstance(c.type, SQLType)},
+    )
+    records = (adjust_outputs(warehouse, record, col_types) for record in to_insert)
+    for chunk in batched(records, INSERT_BATCH_SIZE):
+        warehouse.insert_rows(table, chunk)
     warehouse.insert_rows_done(table)
     return read_dataset(name=dsr.name, session=session, settings=settings)

{datachain-0.16.0 → datachain-0.16.2}/src/datachain/lib/dc/utils.py

@@ -31,8 +31,8 @@ def resolve_columns(
 ) -> "Callable[Concatenate[D, P], D]":
     """Decorator that resolvs input column names to their actual DB names. This is
     specially important for nested columns as user works with them by using dot
-    notation e.g (file.name) but are actually defined with default delimiter
-    in DB, e.g file__name.
+    notation e.g (file.path) but are actually defined with default delimiter
+    in DB, e.g file__path.
     If there are any sql functions in arguments, they will just be transferred as is
     to a method.
     """