datachain 0.37.3__tar.gz → 0.37.5__tar.gz

{datachain-0.37.3 → datachain-0.37.5}/.pre-commit-config.yaml

@@ -24,7 +24,7 @@ repos:
       - id: trailing-whitespace
         exclude: '^LICENSES/'
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: 'v0.14.1'
+    rev: 'v0.14.2'
     hooks:
       - id: ruff
         args: [--fix, --exit-non-zero-on-fix]

{datachain-0.37.3 → datachain-0.37.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: datachain
-Version: 0.37.3
+Version: 0.37.5
 Summary: Wrangle unstructured AI data at scale
 Author-email: Dmitry Petrov <support@dvc.org>
 License-Expression: Apache-2.0

{datachain-0.37.3 → datachain-0.37.5}/docs/guide/checkpoints.md

@@ -2,10 +2,12 @@
 
 Checkpoints allow DataChain to automatically skip re-creating datasets that were successfully saved in previous script runs. When a script fails or is interrupted, you can re-run it and DataChain will resume from where it left off, reusing datasets that were already created.
 
-**Note:** Checkpoints are currently available only for local script runs. Support for Studio is planned for future releases.
+Checkpoints are available for both local script runs and Studio executions.
 
 ## How Checkpoints Work
 
+### Local Script Runs
+
 When you run a Python script locally (e.g., `python my_script.py`), DataChain automatically:
 
 1. **Creates a job** for the script execution, using the script's absolute path as the job name
@@ -16,6 +18,18 @@ When you run a Python script locally (e.g., `python my_script.py`), DataChain au
 
 This means that if your script creates multiple datasets and fails partway through, the next run will skip recreating the datasets that were already successfully saved.
 
+### Studio Runs
+
+When running jobs on Studio, the checkpoint workflow is managed through the UI:
+
+1. **Job execution** is triggered using the Run button in the Studio interface
+2. **Checkpoint control** is explicit - you choose between:
+   - **Run from scratch**: Ignores any existing checkpoints and recreates all datasets
+   - **Continue from last checkpoint**: Resumes from the last successful checkpoint, skipping already-completed stages
+3. **Parent-child job linking** is handled automatically by the system - no need for script path matching or job name conventions
+4. **Checkpoint behavior** during execution is the same as local runs: datasets are saved at each `.save()` call and can be reused on retry
+
+
 ## Example
 
 Consider this script that processes data in multiple stages:
@@ -66,7 +80,6 @@ Checkpoints are **not** used when:
 - Running code interactively (Python REPL, Jupyter notebooks)
 - Running code as a module (e.g., `python -m mymodule`)
 - The `DATACHAIN_CHECKPOINTS_RESET` environment variable is set (see below)
-- Running on Studio (checkpoints support planned for future releases)
 
 ## Resetting Checkpoints
 
@@ -176,17 +189,12 @@ for ds in dc.datasets():
 
 ## Limitations
 
-- **Local only:** Checkpoints currently work only for local script runs. Studio support is planned.
 - **Script-based:** Code must be run as a script (not interactively or as a module).
 - **Hash-based matching:** Any change to the chain will create a different hash, preventing checkpoint reuse.
 - **Same script path:** The script must be run from the same absolute path for parent job linking to work.
 
 ## Future Plans
 
-### Studio Support
-
-Support for checkpoints on Studio is planned for future releases, which will enable checkpoint functionality for collaborative workflows and cloud-based data processing.
-
 ### UDF-Level Checkpoints
 
 Currently, checkpoints are created only when datasets are saved using `.save()`. This means that if a script fails during a long-running UDF operation (like `.map()`, `.gen()`, or `.agg()`), the entire UDF computation must be rerun on the next execution.

{datachain-0.37.3 → datachain-0.37.5}/src/datachain/query/dataset.py

@@ -426,8 +426,10 @@ class UDFStep(Step, ABC):
         """Method that creates a table where temp udf results will be saved"""
 
     def process_input_query(self, query: Select) -> tuple[Select, list["Table"]]:
-        """Apply any necessary processing to the input query"""
-        return query, []
+        """Materialize inputs, ensure sys columns are available, needed for checkpoints,
+        needed for map to work (merge results)"""
+        table = self.catalog.warehouse.create_pre_udf_table(query)
+        return sqlalchemy.select(*table.c), [table]
 
     @abstractmethod
     def create_result_query(
@@ -675,13 +677,6 @@ class UDFSignal(UDFStep):
 
         return self.catalog.warehouse.create_udf_table(udf_output_columns)
 
-    def process_input_query(self, query: Select) -> tuple[Select, list["Table"]]:
-        if os.getenv("DATACHAIN_DISABLE_QUERY_CACHE", "") not in ("", "0"):
-            return query, []
-        table = self.catalog.warehouse.create_pre_udf_table(query)
-        q: Select = sqlalchemy.select(*table.c)
-        return q, [table]
-
     def create_result_query(
         self, udf_table, query
     ) -> tuple[QueryGeneratorFunc, list["sqlalchemy.Column"]]:

{datachain-0.37.3 → datachain-0.37.5}/src/datachain.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: datachain
-Version: 0.37.3
+Version: 0.37.5
 Summary: Wrangle unstructured AI data at scale
 Author-email: Dmitry Petrov <support@dvc.org>
 License-Expression: Apache-2.0

{datachain-0.37.3 → datachain-0.37.5}/tests/func/test_datachain.py

@@ -1699,10 +1699,16 @@ def test_agg_offset_limit(catalog_tmpfile, parallel, offset, limit, files):
         value=list(range(100)),
         session=catalog_tmpfile.session,
     )
+    # Read values in general doesn't guarantee order, so we need to order first
+    ds = ds.order_by("filename")
     if offset is not None:
         ds = ds.offset(offset)
     if limit is not None:
         ds = ds.limit(limit)
+
+    limited_filenames = ds.to_values("filename")
+    assert set(limited_filenames) == set(files)
+
     ds = (
         ds.settings(parallel=parallel)
         .agg(

{datachain-0.37.3 → datachain-0.37.5}/tests/func/test_union.py

@@ -57,3 +57,30 @@ def test_union_parallel_udf_ids_only_no_dup(test_session_tmpfile, monkeypatch):
     assert total == 2 * n
     assert len(distinct_idx) == 2 * n
     assert total == len(distinct_idx)
+
+
+def test_union_parallel_gen_ids_only_no_dup(test_session_tmpfile, monkeypatch):
+    monkeypatch.setattr("datachain.query.dispatch.DEFAULT_BATCH_SIZE", 5, raising=False)
+    n = 30
+
+    x_ids = list(range(n))
+    y_ids = list(range(n, 2 * n))
+
+    x = dc.read_values(idx=x_ids, session=test_session_tmpfile)
+    y = dc.read_values(idx=y_ids, session=test_session_tmpfile)
+
+    xy = x.union(y)
+
+    def expand(idx):
+        yield f"val-{idx}"
+
+    generated = xy.settings(parallel=2).gen(
+        gen=expand,
+        params=("idx",),
+        output={"val": str},
+    )
+
+    values = generated.to_values("val")
+
+    assert len(values) == 2 * n
+    assert set(values) == {f"val-{i}" for i in range(2 * n)}