datachain 0.16.1__tar.gz → 0.16.3__tar.gz

{datachain-0.16.1 → datachain-0.16.3}/.pre-commit-config.yaml

@@ -24,7 +24,7 @@ repos:
       - id: trailing-whitespace
         exclude: '^LICENSES/'
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: 'v0.11.5'
+    rev: 'v0.11.6'
     hooks:
       - id: ruff
         args: [--fix, --exit-non-zero-on-fix]

{datachain-0.16.1/src/datachain.egg-info → datachain-0.16.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: datachain
-Version: 0.16.1
+Version: 0.16.3
 Summary: Wrangle unstructured AI data at scale
 Author-email: Dmitry Petrov <support@dvc.org>
 License-Expression: Apache-2.0
@@ -23,7 +23,7 @@ Requires-Dist: tqdm
 Requires-Dist: numpy<3,>=1
 Requires-Dist: pandas>=2.0.0
 Requires-Dist: packaging
-Requires-Dist: pyarrow
+Requires-Dist: pyarrow<20
 Requires-Dist: typing-extensions
 Requires-Dist: python-dateutil>=2
 Requires-Dist: attrs>=21.3.0

datachain-0.16.3/docs/commands/auth/login.md

@@ -0,0 +1,51 @@
+# auth login
+
+Authenticate DataChain with Studio to save a client access token to DataChain configuration.
+
+## Synopsis
+
+```usage
+usage: datachain auth login [-h] [-v] [-q] [-H HOSTNAME] [-s SCOPES] [-n NAME] [--no-open] [--local]
+```
+
+## Description
+
+By default, this command authenticates DataChain with Studio using default scopes and assigns a random name as the token name. The authentication token will be used for subsequent Studio operations.
+
+## Options
+
+* `-H HOSTNAME`, `--hostname HOSTNAME` - The hostname of the Studio instance to authenticate with.
+* `-s SCOPES`, `--scopes SCOPES` - Authentication token scopes. Allowed scopes: `EXPERIMENTS`, `DATASETS`, `MODELS`. Defaults to all available scopes.
+* `-n NAME`, `--name NAME` - The name of the authentication token. It will be used to identify the token shown in Studio profile. Defaults to a random name.
+* `--no-open` - Use code-based authentication without browser. You will be presented with a user code to enter in the browser. DataChain will also use this if it cannot launch the browser on your behalf.
+* `--local` - Save the token in the local project config instead of the global configuration.
+* `-h`, `--help` - Show the help message and exit.
+* `-v`, `--verbose` - Be verbose.
+* `-q`, `--quiet` - Be quiet.
+
+## Examples
+
+1. Basic authentication with default settings:
+```bash
+datachain auth login
+```
+
+2. Authenticate with specific scopes:
+```bash
+datachain auth login --scopes EXPERIMENTS,DATASETS
+```
+
+3. Authenticate with a custom token name:
+```bash
+datachain auth login --name my-token
+```
+
+4. Authenticate using code-based flow:
+```bash
+datachain auth login --no-open
+```
+
+5. Save token locally for the project:
+```bash
+datachain auth login --local
+```

datachain-0.16.3/docs/commands/auth/logout.md

@@ -0,0 +1,37 @@
+# auth logout
+
+Remove the Studio authentication token from DataChain configuration.
+
+## Synopsis
+
+```usage
+usage: datachain auth logout [-h] [-v] [-q] [--local]
+```
+
+## Description
+
+This command removes the Studio authentication token from the global DataChain configuration. By default, it removes the token from the global configuration, but you can also remove it from the local project configuration using the `--local` option.
+
+## Options
+
+* `--local` - Remove the token from the local project config instead of the global configuration.
+* `-h`, `--help` - Show the help message and exit.
+* `-v`, `--verbose` - Be verbose.
+* `-q`, `--quiet` - Be quiet.
+
+## Examples
+
+1. Remove token from global configuration:
+```bash
+datachain auth logout
+```
+
+2. Remove token from local project configuration:
+```bash
+datachain auth logout --local
+```
+
+3. Remove token with verbose output:
+```bash
+datachain auth logout -v
+```

datachain-0.16.3/docs/commands/auth/team.md

@@ -0,0 +1,36 @@
+# auth team
+
+Set the default team for Studio operations.
+
+## Synopsis
+
+```usage
+usage: datachain auth team [-h] [-v] [-q] [--global] team_name
+```
+
+## Description
+
+This command sets the default team for Studio operations. By default, the team setting is project-specific, but you can use the `--global` option to set it for all projects.
+
+## Arguments
+
+* `team_name` - Name of the team to set as default
+
+## Options
+
+* `--global` - Set team globally for all projects
+* `-h`, `--help` - Show the help message and exit.
+* `-v`, `--verbose` - Be verbose.
+* `-q`, `--quiet` - Be quiet.
+
+## Examples
+
+1. Set default team for current project:
+```bash
+datachain auth team my-team
+```
+
+2. Set default team globally for all projects:
+```bash
+datachain auth team --global my-team
+```

datachain-0.16.3/docs/commands/auth/token.md

@@ -0,0 +1,26 @@
+# auth token
+
+Display the current authentication token for Studio.
+
+## Synopsis
+
+```usage
+usage: datachain auth token [-h] [-v] [-q]
+```
+
+## Description
+
+This command displays the current authentication token that DataChain is using for Studio operations. The token is used for authenticating with Studio and accessing its features.
+
+## Options
+
+* `-h`, `--help` - Show the help message and exit.
+* `-v`, `--verbose` - Be verbose.
+* `-q`, `--quiet` - Be quiet.
+
+## Examples
+
+1. Display the current token:
+```bash
+datachain auth token
+```

datachain-0.16.3/docs/commands/index.md

@@ -0,0 +1,33 @@
+
+# Using DataChain Commands
+
+
+
+DataChain is a command-line tool for wrangling unstructured AI data at scale. Use `datachain -h` to list all available commands.
+
+
+
+## Typical DataChain Workflow
+
+
+
+1.  **Authentication with Studio**
+
+
+	- Use [`datachain auth login`](auth/login.md) to authenticate with Studio
+
+	- Set your default team with [`datachain auth team`](auth/team.md)
+
+	- View your token with [`datachain auth token`](auth/token.md)
+
+	- Log out from Studio with [`datachain auth logout`](auth/logout.md)
+
+
+
+2.  **Job Management**
+
+	- Run jobs in Studio with [`datachain job run`](job/run.md)
+
+	- Monitor job logs with [`datachain job logs`](job/logs.md)
+
+	- Cancel running jobs with [`datachain job cancel`](job/cancel.md)

datachain-0.16.3/docs/commands/job/cancel.md

@@ -0,0 +1,43 @@
+# job cancel
+
+Cancel a running job in Studio.
+
+## Synopsis
+
+```usage
+usage: datachain job cancel [-h] [-v] [-q] [--team TEAM] id
+```
+
+## Description
+
+This command cancels a running job in Studio. The job ID can be obtained from the Studio UI or from the output when the job was created. This is the recommended way to stop a running job, as simply closing the logs view (e.g., with Ctrl+C) will not cancel the job execution.
+
+## Arguments
+
+* `id` - Job ID to cancel. This ID is displayed when the job is created and can also be found in the Studio UI.
+
+## Options
+
+* `--team TEAM` - Team to cancel job for (default: from config)
+* `-h`, `--help` - Show the help message and exit.
+* `-v`, `--verbose` - Be verbose.
+* `-q`, `--quiet` - Be quiet.
+
+## Examples
+
+1. Cancel a specific job:
+```bash
+datachain job cancel job-123
+```
+
+2. Cancel a job in a specific team:
+```bash
+datachain job cancel --team my-team job-123
+```
+
+
+## Notes
+
+* The job ID is displayed when the job is created using `datachain job run`
+* You can also find the job ID in the Studio UI
+* This is the proper way to stop a running job, as simply closing the logs view will not cancel the job execution

datachain-0.16.3/docs/commands/job/logs.md

@@ -0,0 +1,47 @@
+# job logs
+
+Display logs and current status of jobs in Studio.
+
+## Synopsis
+
+```usage
+usage: datachain job logs [-h] [-v] [-q] [--team TEAM] id
+```
+
+## Description
+
+This command displays the logs and current status of a running job in Studio. The command will show real-time logs from the job execution. Note that closing this command (e.g., with Ctrl+C) will only stop displaying the logs but will not cancel the job execution. To cancel a job, use the `job cancel` command.
+
+## Arguments
+
+* `id` - Job ID to show logs for
+
+## Options
+
+* `--team TEAM` - Team to check logs for (default: from config)
+* `-h`, `--help` - Show the help message and exit.
+* `-v`, `--verbose` - Be verbose.
+* `-q`, `--quiet` - Be quiet.
+
+## Examples
+
+1. Display logs for a specific job:
+```bash
+datachain job logs job-123
+```
+
+2. Display logs for a job in a specific team:
+```bash
+datachain job logs --team my-team job-123
+```
+
+3. Display logs with verbose output:
+```bash
+datachain job logs -v job-123
+```
+
+## Notes
+
+* Closing the logs command (e.g., with Ctrl+C) will only stop displaying the logs but will not cancel the job execution
+* To cancel a running job, use the `datachain job cancel` command
+* The job will continue running in Studio even after you stop viewing the logs

datachain-0.16.3/docs/commands/job/run.md

@@ -0,0 +1,67 @@
+# job run
+
+Run a job in Studio.
+
+## Synopsis
+
+```usage
+usage: datachain job run [-h] [-v] [-q] [--team TEAM] [--env-file ENV_FILE] [--env ENV [ENV ...]]
+                         [--workers WORKERS] [--files FILES [FILES ...]] [--python-version PYTHON_VERSION]
+                         [--req-file REQ_FILE] [--req REQ [REQ ...]]
+                         file
+```
+
+## Description
+
+This command runs a job in Studio using the specified query file. You can configure various aspects of the job including environment variables, Python version, dependencies, and more.
+
+## Arguments
+
+* `file` - Query file to run
+
+## Options
+
+* `--team TEAM` - Team to run job for (default: from config)
+* `--env-file ENV_FILE` - File with environment variables for the job
+* `--env ENV` - Environment variables in KEY=VALUE format
+* `--workers WORKERS` - Number of workers for the job
+* `--files FILES` - Additional files to include in the job
+* `--python-version PYTHON_VERSION` - Python version for the job (e.g., 3.9, 3.10, 3.11)
+* `--req-file REQ_FILE` - Python requirements file
+* `--req REQ` - Python package requirements
+* `-h`, `--help` - Show the help message and exit.
+* `-v`, `--verbose` - Be verbose.
+* `-q`, `--quiet` - Be quiet.
+
+## Examples
+
+1. Run a basic job:
+```bash
+datachain job run query.py
+```
+
+2. Run a job with specific team and Python version:
+```bash
+datachain job run --team my-team --python-version 3.11 query.py
+```
+
+3. Run a job with environment variables and requirements:
+```bash
+datachain job run --env-file .env --req-file requirements.txt query.py
+```
+
+4. Run a job with multiple workers and additional files:
+```bash
+datachain job run --workers 4 --files utils.py config.json query.py
+```
+
+5. Run a job with inline environment variables and package requirements:
+```bash
+datachain job run --env API_KEY=123 --req pandas numpy query.py
+```
+
+## Notes
+
+* Closing the logs command (e.g., with Ctrl+C) will only stop displaying the logs but will not cancel the job execution
+* To cancel a running job, use the `datachain job cancel` command
+* The job will continue running in Studio even after you stop viewing the logs

{datachain-0.16.1 → datachain-0.16.3}/mkdocs.yml

@@ -84,6 +84,18 @@ nav:
           - Torch: references/torch.md
           - Functions: references/func.md
           - Toolkit: references/toolkit.md
+      - 📖 CLI Reference:
+          - Overview: commands/index.md
+          - Commands:
+              - auth:
+                  - login: commands/auth/login.md
+                  - logout: commands/auth/logout.md
+                  - token: commands/auth/token.md
+                  - team: commands/auth/team.md
+              - job:
+                  - run: commands/job/run.md
+                  - logs: commands/job/logs.md
+                  - cancel: commands/job/cancel.md
       - 📡 Interacting with remote storage: references/remotes.md
       - 🤝 Contributing: contributing.md
 

{datachain-0.16.1 → datachain-0.16.3}/pyproject.toml

@@ -27,7 +27,7 @@ dependencies = [
   "numpy>=1,<3",
   "pandas>=2.0.0",
   "packaging",
-  "pyarrow",
+  "pyarrow<20",
   "typing-extensions",
   "python-dateutil>=2",
   "attrs>=21.3.0",

{datachain-0.16.1 → datachain-0.16.3}/src/datachain/func/array.py

@@ -1,5 +1,5 @@
 from collections.abc import Sequence
-from typing import Any, Union
+from typing import Any, Optional, Union
 
 from datachain.sql.functions import array
 
@@ -178,6 +178,61 @@ def contains(arr: Union[str, Sequence, Func], elem: Any) -> Func:
     return Func("contains", inner=inner, cols=cols, args=args, result_type=int)
 
 
+def get_element(arg: Union[str, Sequence, Func], index: int) -> Func:
+    """
+    Returns the element at the given index from the array.
+    If the index is out of bounds, it returns None or columns default value.
+
+    Args:
+        arg (str | Sequence | Func): Array to get the element from.
+            If a string is provided, it is assumed to be the name of the array column.
+            If a sequence is provided, it is assumed to be an array of values.
+            If a Func is provided, it is assumed to be a function returning an array.
+        index (int): Index of the element to get from the array.
+
+    Returns:
+        Func: A Func object that represents the array get_element function.
+
+    Example:
+        ```py
+        dc.mutate(
+            first_el=func.array.get_element("signal.values", 0),
+            second_el=func.array.get_element([1, 2, 3, 4, 5], 1),
+        )
+        ```
+
+    Note:
+        - Result column will always be the same type as the elements of the array.
+    """
+
+    def type_from_args(arr, _):
+        if isinstance(arr, list):
+            try:
+                return type(arr[0])
+            except IndexError:
+                return str  # if the array is empty, return str as default type
+        return None
+
+    cols: Optional[Union[str, Sequence, Func]]
+    args: Union[str, Sequence, Func, int]
+
+    if isinstance(arg, (str, Func)):
+        cols = [arg]
+        args = [index]
+    else:
+        cols = None
+        args = [arg, index]
+
+    return Func(
+        "get_element",
+        inner=array.get_element,
+        cols=cols,
+        args=args,
+        from_array=True,
+        type_from_args=type_from_args,
+    )
+
+
 def sip_hash_64(arg: Union[str, Sequence]) -> Func:
     """
     Computes the SipHash-64 hash of the array.

{datachain-0.16.1 → datachain-0.16.3}/src/datachain/func/func.py

@@ -1,6 +1,6 @@
 import inspect
 from collections.abc import Sequence
-from typing import TYPE_CHECKING, Any, Callable, Optional, Union
+from typing import TYPE_CHECKING, Any, Callable, Optional, Union, get_args, get_origin
 
 from sqlalchemy import BindParameter, Case, ColumnElement, Integer, cast, desc
 from sqlalchemy.sql import func as sa_func
@@ -36,7 +36,9 @@ class Func(Function):
         args: Optional[Sequence[Any]] = None,
         kwargs: Optional[dict[str, Any]] = None,
         result_type: Optional["DataType"] = None,
+        type_from_args: Optional[Callable[..., "DataType"]] = None,
         is_array: bool = False,
+        from_array: bool = False,
         is_window: bool = False,
         window: Optional["Window"] = None,
         label: Optional[str] = None,
@@ -47,7 +49,9 @@ class Func(Function):
         self.args = args or []
         self.kwargs = kwargs or {}
         self.result_type = result_type
+        self.type_from_args = type_from_args
         self.is_array = is_array
+        self.from_array = from_array
         self.is_window = is_window
         self.window = window
         self.col_label = label
@@ -66,7 +70,9 @@ class Func(Function):
             self.args,
             self.kwargs,
             self.result_type,
+            self.type_from_args,
             self.is_array,
+            self.from_array,
             self.is_window,
             window,
             self.col_label,
@@ -101,6 +107,20 @@ class Func(Function):
                     "Columns must have the same type to infer result type",
                 )
 
+        if self.from_array:
+            if get_origin(col_type) is list:
+                col_args = get_args(col_type)
+                if len(col_args) != 1:
+                    raise DataChainColumnError(
+                        str(self),
+                        "Array column must have a single type argument",
+                    )
+                return col_args[0]
+            raise DataChainColumnError(
+                str(self),
+                "Array column must be of type list",
+            )
+
         return list[col_type] if self.is_array else col_type  # type: ignore[valid-type]
 
     def __add__(self, other: Union[ColT, float]) -> "Func":
@@ -339,7 +359,9 @@ class Func(Function):
             self.args,
             self.kwargs,
             self.result_type,
+            self.type_from_args,
             self.is_array,
+            self.from_array,
             self.is_window,
             self.window,
             label,
@@ -368,6 +390,15 @@ class Func(Function):
         if signals_schema and (col_type := self._db_col_type(signals_schema)):
             return col_type
 
+        if (
+            self.type_from_args
+            and (self.cols is None or self.cols == [])
+            and self.args is not None
+            and len(self.args) > 0
+            and (result_type := self.type_from_args(*self.args)) is not None
+        ):
+            return result_type
+
         raise DataChainColumnError(
             str(self),
             "Column name is required to infer result type",

{datachain-0.16.1 → datachain-0.16.3}/src/datachain/lib/dc/database.py

@@ -127,9 +127,11 @@ def read_database(
         ```
 
     Notes:
-        This function works with a variety of databases — including, but not limited to,
-        SQLite, DuckDB, PostgreSQL, and Snowflake, provided the appropriate driver is
-        installed.
+        - This function works with a variety of databases — including,
+        but not limited to, SQLite, DuckDB, PostgreSQL, and Snowflake,
+        provided the appropriate driver is installed.
+        - This call is blocking, and will execute the query and return once the
+          results are saved.
     """
     from datachain.lib.dc.records import read_records
 

{datachain-0.16.1 → datachain-0.16.3}/src/datachain/lib/dc/records.py

@@ -37,9 +37,13 @@ def read_records(
         import datachain as dc
         single_record = dc.read_records(dc.DEFAULT_FILE_RECORD)
         ```
+
+    Notes:
+        This call blocks until all records are inserted.
     """
-    from datachain.query.dataset import adjust_outputs, get_col_types
+    from datachain.query.dataset import INSERT_BATCH_SIZE, adjust_outputs, get_col_types
     from datachain.sql.types import SQLType
+    from datachain.utils import batched
 
     from .datasets import read_dataset
 
@@ -89,6 +93,7 @@ def read_records(
         {c.name: c.type for c in columns if isinstance(c.type, SQLType)},
     )
     records = (adjust_outputs(warehouse, record, col_types) for record in to_insert)
-    warehouse.insert_rows(table, records)
+    for chunk in batched(records, INSERT_BATCH_SIZE):
+        warehouse.insert_rows(table, chunk)
     warehouse.insert_rows_done(table)
     return read_dataset(name=dsr.name, session=session, settings=settings)

{datachain-0.16.1 → datachain-0.16.3}/src/datachain/sql/functions/array.py

@@ -48,6 +48,16 @@ class contains(GenericFunction):  # noqa: N801
     inherit_cache = True
 
 
+class get_element(GenericFunction):  # noqa: N801
+    """
+    Returns the element at the given index in the array.
+    """
+
+    package = "array"
+    name = "get_element"
+    inherit_cache = True
+
+
 class sip_hash_64(GenericFunction):  # noqa: N801
     """
     Computes the SipHash-64 hash of the array.
@@ -63,4 +73,5 @@ compiler_not_implemented(cosine_distance)
 compiler_not_implemented(euclidean_distance)
 compiler_not_implemented(length)
 compiler_not_implemented(contains)
+compiler_not_implemented(get_element)
 compiler_not_implemented(sip_hash_64)

{datachain-0.16.1 → datachain-0.16.3}/src/datachain/sql/sqlite/base.py

@@ -88,6 +88,7 @@ def setup():
     compiles(sql_path.file_ext, "sqlite")(compile_path_file_ext)
     compiles(array.length, "sqlite")(compile_array_length)
     compiles(array.contains, "sqlite")(compile_array_contains)
+    compiles(array.get_element, "sqlite")(compile_array_get_element)
     compiles(string.length, "sqlite")(compile_string_length)
     compiles(string.split, "sqlite")(compile_string_split)
     compiles(string.regexp_replace, "sqlite")(compile_string_regexp_replace)
@@ -270,6 +271,13 @@ def register_user_defined_sql_functions() -> None:
 
     _registered_function_creators["string_functions"] = create_string_functions
 
+    def create_array_functions(conn):
+        conn.create_function(
+            "json_array_get_element", 2, py_json_array_get_element, deterministic=True
+        )
+
+    _registered_function_creators["array_functions"] = create_array_functions
+
     has_json_extension = functions_exist(["json_array_length", "json_array_contains"])
     if not has_json_extension:
 
@@ -438,6 +446,20 @@ def py_json_array_contains(arr, value, is_json):
     return value in orjson.loads(arr)
 
 
+def py_json_array_get_element(val, idx):
+    arr = orjson.loads(val)
+    try:
+        return arr[idx]
+    except IndexError:
+        return None
+
+
+def compile_array_get_element(element, compiler, **kwargs):
+    return compiler.process(
+        func.json_array_get_element(*element.clauses.clauses), **kwargs
+    )
+
+
 def compile_array_length(element, compiler, **kwargs):
     return compiler.process(func.json_array_length(*element.clauses.clauses), **kwargs)
 

{datachain-0.16.1 → datachain-0.16.3/src/datachain.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: datachain
-Version: 0.16.1
+Version: 0.16.3
 Summary: Wrangle unstructured AI data at scale
 Author-email: Dmitry Petrov <support@dvc.org>
 License-Expression: Apache-2.0
@@ -23,7 +23,7 @@ Requires-Dist: tqdm
 Requires-Dist: numpy<3,>=1
 Requires-Dist: pandas>=2.0.0
 Requires-Dist: packaging
-Requires-Dist: pyarrow
+Requires-Dist: pyarrow<20
 Requires-Dist: typing-extensions
 Requires-Dist: python-dateutil>=2
 Requires-Dist: attrs>=21.3.0