datachain 0.16.1__tar.gz → 0.16.2__tar.gz

{datachain-0.16.1 → datachain-0.16.2}/.pre-commit-config.yaml

@@ -24,7 +24,7 @@ repos:
       - id: trailing-whitespace
         exclude: '^LICENSES/'
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: 'v0.11.5'
+    rev: 'v0.11.6'
     hooks:
       - id: ruff
         args: [--fix, --exit-non-zero-on-fix]

{datachain-0.16.1/src/datachain.egg-info → datachain-0.16.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: datachain
-Version: 0.16.1
+Version: 0.16.2
 Summary: Wrangle unstructured AI data at scale
 Author-email: Dmitry Petrov <support@dvc.org>
 License-Expression: Apache-2.0

datachain-0.16.2/docs/commands/auth/login.md

@@ -0,0 +1,51 @@
+# auth login
+
+Authenticate DataChain with Studio to save a client access token to DataChain configuration.
+
+## Synopsis
+
+```usage
+usage: datachain auth login [-h] [-v] [-q] [-H HOSTNAME] [-s SCOPES] [-n NAME] [--no-open] [--local]
+```
+
+## Description
+
+By default, this command authenticates DataChain with Studio using default scopes and assigns a random name as the token name. The authentication token will be used for subsequent Studio operations.
+
+## Options
+
+* `-H HOSTNAME`, `--hostname HOSTNAME` - The hostname of the Studio instance to authenticate with.
+* `-s SCOPES`, `--scopes SCOPES` - Authentication token scopes. Allowed scopes: `EXPERIMENTS`, `DATASETS`, `MODELS`. Defaults to all available scopes.
+* `-n NAME`, `--name NAME` - The name of the authentication token. It will be used to identify the token shown in Studio profile. Defaults to a random name.
+* `--no-open` - Use code-based authentication without browser. You will be presented with a user code to enter in the browser. DataChain will also use this if it cannot launch the browser on your behalf.
+* `--local` - Save the token in the local project config instead of the global configuration.
+* `-h`, `--help` - Show the help message and exit.
+* `-v`, `--verbose` - Be verbose.
+* `-q`, `--quiet` - Be quiet.
+
+## Examples
+
+1. Basic authentication with default settings:
+```bash
+datachain auth login
+```
+
+2. Authenticate with specific scopes:
+```bash
+datachain auth login --scopes EXPERIMENTS,DATASETS
+```
+
+3. Authenticate with a custom token name:
+```bash
+datachain auth login --name my-token
+```
+
+4. Authenticate using code-based flow:
+```bash
+datachain auth login --no-open
+```
+
+5. Save token locally for the project:
+```bash
+datachain auth login --local
+```

datachain-0.16.2/docs/commands/auth/logout.md

@@ -0,0 +1,37 @@
+# auth logout
+
+Remove the Studio authentication token from DataChain configuration.
+
+## Synopsis
+
+```usage
+usage: datachain auth logout [-h] [-v] [-q] [--local]
+```
+
+## Description
+
+This command removes the Studio authentication token from the global DataChain configuration. By default, it removes the token from the global configuration, but you can also remove it from the local project configuration using the `--local` option.
+
+## Options
+
+* `--local` - Remove the token from the local project config instead of the global configuration.
+* `-h`, `--help` - Show the help message and exit.
+* `-v`, `--verbose` - Be verbose.
+* `-q`, `--quiet` - Be quiet.
+
+## Examples
+
+1. Remove token from global configuration:
+```bash
+datachain auth logout
+```
+
+2. Remove token from local project configuration:
+```bash
+datachain auth logout --local
+```
+
+3. Remove token with verbose output:
+```bash
+datachain auth logout -v
+```

datachain-0.16.2/docs/commands/auth/team.md

@@ -0,0 +1,36 @@
+# auth team
+
+Set the default team for Studio operations.
+
+## Synopsis
+
+```usage
+usage: datachain auth team [-h] [-v] [-q] [--global] team_name
+```
+
+## Description
+
+This command sets the default team for Studio operations. By default, the team setting is project-specific, but you can use the `--global` option to set it for all projects.
+
+## Arguments
+
+* `team_name` - Name of the team to set as default
+
+## Options
+
+* `--global` - Set team globally for all projects
+* `-h`, `--help` - Show the help message and exit.
+* `-v`, `--verbose` - Be verbose.
+* `-q`, `--quiet` - Be quiet.
+
+## Examples
+
+1. Set default team for current project:
+```bash
+datachain auth team my-team
+```
+
+2. Set default team globally for all projects:
+```bash
+datachain auth team --global my-team
+```

datachain-0.16.2/docs/commands/auth/token.md

@@ -0,0 +1,26 @@
+# auth token
+
+Display the current authentication token for Studio.
+
+## Synopsis
+
+```usage
+usage: datachain auth token [-h] [-v] [-q]
+```
+
+## Description
+
+This command displays the current authentication token that DataChain is using for Studio operations. The token is used for authenticating with Studio and accessing its features.
+
+## Options
+
+* `-h`, `--help` - Show the help message and exit.
+* `-v`, `--verbose` - Be verbose.
+* `-q`, `--quiet` - Be quiet.
+
+## Examples
+
+1. Display the current token:
+```bash
+datachain auth token
+```

datachain-0.16.2/docs/commands/index.md

@@ -0,0 +1,33 @@
+
+# Using DataChain Commands
+
+
+
+DataChain is a command-line tool for wrangling unstructured AI data at scale. Use `datachain -h` to list all available commands.
+
+
+
+## Typical DataChain Workflow
+
+
+
+1.  **Authentication with Studio**
+
+
+	- Use [`datachain auth login`](auth/login.md) to authenticate with Studio
+
+	- Set your default team with [`datachain auth team`](auth/team.md)
+
+	- View your token with [`datachain auth token`](auth/token.md)
+
+	- Log out from Studio with [`datachain auth logout`](auth/logout.md)
+
+
+
+2.  **Job Management**
+
+	- Run jobs in Studio with [`datachain job run`](job/run.md)
+
+	- Monitor job logs with [`datachain job logs`](job/logs.md)
+
+	- Cancel running jobs with [`datachain job cancel`](job/cancel.md)

datachain-0.16.2/docs/commands/job/cancel.md

@@ -0,0 +1,43 @@
+# job cancel
+
+Cancel a running job in Studio.
+
+## Synopsis
+
+```usage
+usage: datachain job cancel [-h] [-v] [-q] [--team TEAM] id
+```
+
+## Description
+
+This command cancels a running job in Studio. The job ID can be obtained from the Studio UI or from the output when the job was created. This is the recommended way to stop a running job, as simply closing the logs view (e.g., with Ctrl+C) will not cancel the job execution.
+
+## Arguments
+
+* `id` - Job ID to cancel. This ID is displayed when the job is created and can also be found in the Studio UI.
+
+## Options
+
+* `--team TEAM` - Team to cancel job for (default: from config)
+* `-h`, `--help` - Show the help message and exit.
+* `-v`, `--verbose` - Be verbose.
+* `-q`, `--quiet` - Be quiet.
+
+## Examples
+
+1. Cancel a specific job:
+```bash
+datachain job cancel job-123
+```
+
+2. Cancel a job in a specific team:
+```bash
+datachain job cancel --team my-team job-123
+```
+
+
+## Notes
+
+* The job ID is displayed when the job is created using `datachain job run`
+* You can also find the job ID in the Studio UI
+* This is the proper way to stop a running job, as simply closing the logs view will not cancel the job execution

datachain-0.16.2/docs/commands/job/logs.md

@@ -0,0 +1,47 @@
+# job logs
+
+Display logs and current status of jobs in Studio.
+
+## Synopsis
+
+```usage
+usage: datachain job logs [-h] [-v] [-q] [--team TEAM] id
+```
+
+## Description
+
+This command displays the logs and current status of a running job in Studio. The command will show real-time logs from the job execution. Note that closing this command (e.g., with Ctrl+C) will only stop displaying the logs but will not cancel the job execution. To cancel a job, use the `job cancel` command.
+
+## Arguments
+
+* `id` - Job ID to show logs for
+
+## Options
+
+* `--team TEAM` - Team to check logs for (default: from config)
+* `-h`, `--help` - Show the help message and exit.
+* `-v`, `--verbose` - Be verbose.
+* `-q`, `--quiet` - Be quiet.
+
+## Examples
+
+1. Display logs for a specific job:
+```bash
+datachain job logs job-123
+```
+
+2. Display logs for a job in a specific team:
+```bash
+datachain job logs --team my-team job-123
+```
+
+3. Display logs with verbose output:
+```bash
+datachain job logs -v job-123
+```
+
+## Notes
+
+* Closing the logs command (e.g., with Ctrl+C) will only stop displaying the logs but will not cancel the job execution
+* To cancel a running job, use the `datachain job cancel` command
+* The job will continue running in Studio even after you stop viewing the logs

datachain-0.16.2/docs/commands/job/run.md

@@ -0,0 +1,67 @@
+# job run
+
+Run a job in Studio.
+
+## Synopsis
+
+```usage
+usage: datachain job run [-h] [-v] [-q] [--team TEAM] [--env-file ENV_FILE] [--env ENV [ENV ...]]
+                         [--workers WORKERS] [--files FILES [FILES ...]] [--python-version PYTHON_VERSION]
+                         [--req-file REQ_FILE] [--req REQ [REQ ...]]
+                         file
+```
+
+## Description
+
+This command runs a job in Studio using the specified query file. You can configure various aspects of the job including environment variables, Python version, dependencies, and more.
+
+## Arguments
+
+* `file` - Query file to run
+
+## Options
+
+* `--team TEAM` - Team to run job for (default: from config)
+* `--env-file ENV_FILE` - File with environment variables for the job
+* `--env ENV` - Environment variables in KEY=VALUE format
+* `--workers WORKERS` - Number of workers for the job
+* `--files FILES` - Additional files to include in the job
+* `--python-version PYTHON_VERSION` - Python version for the job (e.g., 3.9, 3.10, 3.11)
+* `--req-file REQ_FILE` - Python requirements file
+* `--req REQ` - Python package requirements
+* `-h`, `--help` - Show the help message and exit.
+* `-v`, `--verbose` - Be verbose.
+* `-q`, `--quiet` - Be quiet.
+
+## Examples
+
+1. Run a basic job:
+```bash
+datachain job run query.py
+```
+
+2. Run a job with specific team and Python version:
+```bash
+datachain job run --team my-team --python-version 3.11 query.py
+```
+
+3. Run a job with environment variables and requirements:
+```bash
+datachain job run --env-file .env --req-file requirements.txt query.py
+```
+
+4. Run a job with multiple workers and additional files:
+```bash
+datachain job run --workers 4 --files utils.py config.json query.py
+```
+
+5. Run a job with inline environment variables and package requirements:
+```bash
+datachain job run --env API_KEY=123 --req pandas numpy query.py
+```
+
+## Notes
+
+* Closing the logs command (e.g., with Ctrl+C) will only stop displaying the logs but will not cancel the job execution
+* To cancel a running job, use the `datachain job cancel` command
+* The job will continue running in Studio even after you stop viewing the logs

{datachain-0.16.1 → datachain-0.16.2}/mkdocs.yml

@@ -84,6 +84,18 @@ nav:
           - Torch: references/torch.md
           - Functions: references/func.md
           - Toolkit: references/toolkit.md
+      - 📖 CLI Reference:
+          - Overview: commands/index.md
+          - Commands:
+              - auth:
+                  - login: commands/auth/login.md
+                  - logout: commands/auth/logout.md
+                  - token: commands/auth/token.md
+                  - team: commands/auth/team.md
+              - job:
+                  - run: commands/job/run.md
+                  - logs: commands/job/logs.md
+                  - cancel: commands/job/cancel.md
       - 📡 Interacting with remote storage: references/remotes.md
       - 🤝 Contributing: contributing.md
 

{datachain-0.16.1 → datachain-0.16.2}/src/datachain/lib/dc/records.py

@@ -38,8 +38,9 @@ def read_records(
         single_record = dc.read_records(dc.DEFAULT_FILE_RECORD)
         ```
     """
-    from datachain.query.dataset import adjust_outputs, get_col_types
+    from datachain.query.dataset import INSERT_BATCH_SIZE, adjust_outputs, get_col_types
     from datachain.sql.types import SQLType
+    from datachain.utils import batched
 
     from .datasets import read_dataset
 
@@ -89,6 +90,7 @@ def read_records(
         {c.name: c.type for c in columns if isinstance(c.type, SQLType)},
     )
     records = (adjust_outputs(warehouse, record, col_types) for record in to_insert)
-    warehouse.insert_rows(table, records)
+    for chunk in batched(records, INSERT_BATCH_SIZE):
+        warehouse.insert_rows(table, chunk)
     warehouse.insert_rows_done(table)
     return read_dataset(name=dsr.name, session=session, settings=settings)

{datachain-0.16.1 → datachain-0.16.2/src/datachain.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: datachain
-Version: 0.16.1
+Version: 0.16.2
 Summary: Wrangle unstructured AI data at scale
 Author-email: Dmitry Petrov <support@dvc.org>
 License-Expression: Apache-2.0

{datachain-0.16.1 → datachain-0.16.2}/src/datachain.egg-info/SOURCES.txt

@@ -26,6 +26,14 @@ docs/tutorials.md
 docs/assets/captioned_cartoons.png
 docs/assets/datachain-white.svg
 docs/assets/datachain.svg
+docs/commands/index.md
+docs/commands/auth/login.md
+docs/commands/auth/logout.md
+docs/commands/auth/team.md
+docs/commands/auth/token.md
+docs/commands/job/cancel.md
+docs/commands/job/logs.md
+docs/commands/job/run.md
 docs/css/github-permalink-style.css
 docs/overrides/main.html
 docs/references/datachain.md