datachain 0.19.1__tar.gz → 0.20.0__tar.gz

{datachain-0.19.1 → datachain-0.20.0}/.github/workflows/tests-studio.yml

@@ -98,6 +98,7 @@ jobs:
       - name: Run tests
         # Generate `.test_durations` file with `pytest --store-durations --durations-path ../.github/.test_durations ...`
         run: >
+          DATACHAIN_METASTORE_ARG_PROJECT=john
           PYTHONPATH="$(pwd)/..:${PYTHONPATH}"
           pytest
           --config-file=pyproject.toml -rs

{datachain-0.19.1 → datachain-0.20.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: datachain
-Version: 0.19.1
+Version: 0.20.0
 Summary: Wrangle unstructured AI data at scale
 Author-email: Dmitry Petrov <support@dvc.org>
 License-Expression: Apache-2.0
@@ -94,7 +94,7 @@ Requires-Dist: scipy; extra == "tests"
 Requires-Dist: ultralytics; extra == "tests"
 Provides-Extra: dev
 Requires-Dist: datachain[docs,tests]; extra == "dev"
-Requires-Dist: mypy==1.16.0; extra == "dev"
+Requires-Dist: mypy==1.16.1; extra == "dev"
 Requires-Dist: types-python-dateutil; extra == "dev"
 Requires-Dist: types-pytz; extra == "dev"
 Requires-Dist: types-PyYAML; extra == "dev"

datachain-0.20.0/docs/commands/auth/team.md

@@ -0,0 +1,42 @@
+# auth team
+
+Set the default team for Studio operations.
+
+## Synopsis
+
+```usage
+usage: datachain auth team [-h] [-v] [-q] [--local] [team_name]
+```
+
+## Description
+
+This command sets or shows the default team for Studio operations. By default, the team setting is saved globally, but you can use the `--local` option to set it for the current project.
+If team_name is not passed, the current team in use is shown to the user.
+
+## Arguments
+
+* `team_name` - Optional,  Name of the team to set as default
+
+## Options
+
+* `--local` - Set team locally for the current project
+* `-h`, `--help` - Show the help message and exit.
+* `-v`, `--verbose` - Be verbose.
+* `-q`, `--quiet` - Be quiet.
+
+## Examples
+
+1. Set default team for all projects:
+```bash
+datachain auth team my-team
+```
+
+2. Set default team locally for the current project:
+```bash
+datachain auth team --local my-team
+```
+
+3. Print the current default in use
+```bash
+datachain auth team
+```

{datachain-0.19.1 → datachain-0.20.0}/docs/commands/index.md

@@ -31,3 +31,5 @@ DataChain is a command-line tool for wrangling unstructured AI data at scale. Us
 	- Monitor job logs with [`datachain job logs`](job/logs.md)
 
 	- Cancel running jobs with [`datachain job cancel`](job/cancel.md)
+
+	- Check for the clusters available for jobs [`datachain job clusters`](job/clusters.md)

datachain-0.20.0/docs/commands/job/clusters.md

@@ -0,0 +1,39 @@
+# job clusters
+
+List compute clusters in Studio.
+
+## Synopsis
+
+```usage
+usage: datachain job clusters [-h] [-v] [-q] [--team TEAM]
+```
+
+## Description
+
+This command lists compute clusters available in Studio. You can specify a team to list clusters for. The command provides information about the compute resources available for running jobs.
+
+## Options
+
+* `--team TEAM` - Team to list clusters for (default: from config)
+* `-h`, `--help` - Show the help message and exit
+* `-v`, `--verbose` - Be verbose
+* `-q`, `--quiet` - Be quiet
+
+## Examples
+
+1. List all clusters for the default team:
+```bash
+datachain job clusters
+```
+
+2. List clusters for a specific team:
+```bash
+datachain job clusters --team my-team
+```
+
+
+## Notes
+
+* The command shows all compute clusters available to your team
+* Clusters represent the compute resources where your jobs can run
+* Use the `--team` option to view clusters for a different team

{datachain-0.19.1 → datachain-0.20.0}/docs/commands/job/run.md

@@ -24,6 +24,7 @@ This command runs a job in Studio using the specified query file. You can config
 * `--team TEAM` - Team to run job for (default: from config)
 * `--env-file ENV_FILE` - File with environment variables for the job
 * `--env ENV` - Environment variables in KEY=VALUE format
+* `--cluster-id CLUSTER_ID` - Compute cluster ID to run the job on
 * `--workers WORKERS` - Number of workers for the job
 * `--files FILES` - Additional files to include in the job
 * `--python-version PYTHON_VERSION` - Python version for the job (e.g., 3.9, 3.10, 3.11)
@@ -71,8 +72,17 @@ datachain job run --repository https://github.com/iterative/datachain query.py
 datachain job run --priority 2 query.py
 ```
 
+8. Run a job in a specific cluster
+```bash
+# Get the cluster id using following command
+datachain job clusters
+# Use the id  of an active clusters from above
+datachain job run --cluster-id 1 query.py
+```
+
 ## Notes
 
 * Closing the logs command (e.g., with Ctrl+C) will only stop displaying the logs but will not cancel the job execution
 * To cancel a running job, use the `datachain job cancel` command
 * The job will continue running in Studio even after you stop viewing the logs
+* You can get the list of compute clusters using `datachain job clusters` command.

datachain-0.20.0/docs/guide/env.md

@@ -0,0 +1,18 @@
+# Environment Variables
+
+List of environment variables used to configure DataChain behavior.
+
+### Core Configuration
+
+- `DATACHAIN_ROOT_DIR` – Specifies the root directory where DataChain will create the `.datachain` folder to store its internal data. (default: the current working directory).
+- `DATACHAIN_SYSTEM_CONFIG_DIR` – Overrides the system-wide configuration directory (default depends on the platform).
+- `DATACHAIN_GLOBAL_CONFIG_DIR` – Overrides the user's global configuration directory (default depends on the platform).
+- `DATACHAIN_NO_ANALYTICS` – Disables telemetry.
+
+### Studio Integration
+
+- `DATACHAIN_STUDIO_URL` – Custom Studio URL.
+- `DATACHAIN_STUDIO_TOKEN` – Authentication token for Studio.
+- `DATACHAIN_STUDIO_TEAM` – Studio team name.
+
+Note: Some environment variables are used internally and may not be documented here. For the most up-to-date list, refer to the source code.

{datachain-0.19.1 → datachain-0.20.0}/docs/guide/index.md

@@ -10,3 +10,4 @@ Welcome to the DataChain User Guide! This section provides comprehensive documen
 - [Data Processing Overview](./processing.md) - Discover DataChain's specialized data processing features.
     - [Delta Processing](./delta.md) - Incremental data processing to efficiently handle large datasets that change over time.
     - [Error Handling and Retries](./retry.md) - Learn how to handle processing errors and selectively reprocess problematic records.
+- [Environment Variables](./env.md) - Configure DataChain's behavior using environment variables.

{datachain-0.19.1 → datachain-0.20.0}/mkdocs.yml

@@ -97,6 +97,7 @@ nav:
                   - logs: commands/job/logs.md
                   - cancel: commands/job/cancel.md
                   - ls: commands/job/ls.md
+                  - clusters: commands/job/clusters.md
       - 📚 User Guide:
           - Overview: guide/index.md
           - 📡 Interacting with remote storage: guide/remotes.md
@@ -104,6 +105,7 @@ nav:
               - Overview: guide/processing.md
               - Delta Processing: guide/delta.md
               - Errors Handling and Retries: guide/retry.md
+          - Environment Variables: guide/env.md
       - 🤝 Contributing: contributing.md
 
   - DataChain Website ↗: https://datachain.ai" target="_blank"

{datachain-0.19.1 → datachain-0.20.0}/pyproject.toml

@@ -108,7 +108,7 @@ tests = [
 ]
 dev = [
   "datachain[docs,tests]",
-  "mypy==1.16.0",
+  "mypy==1.16.1",
   "types-python-dateutil",
   "types-pytz",
   "types-PyYAML",

{datachain-0.19.1 → datachain-0.20.0}/src/datachain/__init__.py

@@ -1,3 +1,4 @@
+from datachain.lib import namespaces, projects
 from datachain.lib.data_model import DataModel, DataType, is_chain_type
 from datachain.lib.dc import (
     C,
@@ -67,7 +68,9 @@ __all__ = [
     "is_chain_type",
     "listings",
     "metrics",
+    "namespaces",
     "param",
+    "projects",
     "read_csv",
     "read_database",
     "read_dataset",