datachain 0.37.2__tar.gz → 0.37.4__tar.gz

{datachain-0.37.2 → datachain-0.37.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: datachain
-Version: 0.37.2
+Version: 0.37.4
 Summary: Wrangle unstructured AI data at scale
 Author-email: Dmitry Petrov <support@dvc.org>
 License-Expression: Apache-2.0

{datachain-0.37.2 → datachain-0.37.4}/docs/guide/checkpoints.md

@@ -2,10 +2,12 @@
 
 Checkpoints allow DataChain to automatically skip re-creating datasets that were successfully saved in previous script runs. When a script fails or is interrupted, you can re-run it and DataChain will resume from where it left off, reusing datasets that were already created.
 
-**Note:** Checkpoints are currently available only for local script runs. Support for Studio is planned for future releases.
+Checkpoints are available for both local script runs and Studio executions.
 
 ## How Checkpoints Work
 
+### Local Script Runs
+
 When you run a Python script locally (e.g., `python my_script.py`), DataChain automatically:
 
 1. **Creates a job** for the script execution, using the script's absolute path as the job name
@@ -16,6 +18,18 @@ When you run a Python script locally (e.g., `python my_script.py`), DataChain au
 
 This means that if your script creates multiple datasets and fails partway through, the next run will skip recreating the datasets that were already successfully saved.
 
+### Studio Runs
+
+When running jobs on Studio, the checkpoint workflow is managed through the UI:
+
+1. **Job execution** is triggered using the Run button in the Studio interface
+2. **Checkpoint control** is explicit - you choose between:
+   - **Run from scratch**: Ignores any existing checkpoints and recreates all datasets
+   - **Continue from last checkpoint**: Resumes from the last successful checkpoint, skipping already-completed stages
+3. **Parent-child job linking** is handled automatically by the system - no need for script path matching or job name conventions
+4. **Checkpoint behavior** during execution is the same as local runs: datasets are saved at each `.save()` call and can be reused on retry
+
+
 ## Example
 
 Consider this script that processes data in multiple stages:
@@ -66,7 +80,6 @@ Checkpoints are **not** used when:
 - Running code interactively (Python REPL, Jupyter notebooks)
 - Running code as a module (e.g., `python -m mymodule`)
 - The `DATACHAIN_CHECKPOINTS_RESET` environment variable is set (see below)
-- Running on Studio (checkpoints support planned for future releases)
 
 ## Resetting Checkpoints
 
@@ -176,17 +189,12 @@ for ds in dc.datasets():
 
 ## Limitations
 
-- **Local only:** Checkpoints currently work only for local script runs. Studio support is planned.
 - **Script-based:** Code must be run as a script (not interactively or as a module).
 - **Hash-based matching:** Any change to the chain will create a different hash, preventing checkpoint reuse.
 - **Same script path:** The script must be run from the same absolute path for parent job linking to work.
 
 ## Future Plans
 
-### Studio Support
-
-Support for checkpoints on Studio is planned for future releases, which will enable checkpoint functionality for collaborative workflows and cloud-based data processing.
-
 ### UDF-Level Checkpoints
 
 Currently, checkpoints are created only when datasets are saved using `.save()`. This means that if a script fails during a long-running UDF operation (like `.map()`, `.gen()`, or `.agg()`), the entire UDF computation must be rerun on the next execution.

{datachain-0.37.2 → datachain-0.37.4}/src/datachain/data_storage/sqlite.py

@@ -29,7 +29,6 @@ from sqlalchemy.sql.selectable import Select
 from tqdm.auto import tqdm
 
 import datachain.sql.sqlite
-from datachain import semver
 from datachain.data_storage import AbstractDBMetastore, AbstractWarehouse
 from datachain.data_storage.db_engine import DatabaseEngine
 from datachain.data_storage.schema import DefaultSchema
@@ -692,61 +691,6 @@ class SQLiteWarehouse(AbstractWarehouse):
             for row in self.db.execute(query, cursor=cur)
         ]
 
-    def merge_dataset_rows(
-        self,
-        src: DatasetRecord,
-        dst: DatasetRecord,
-        src_version: str,
-        dst_version: str,
-    ) -> None:
-        dst_empty = False
-
-        if not self.db.has_table(self.dataset_table_name(src, src_version)):
-            # source table doesn't exist, nothing to do
-            return
-
-        src_dr = self.dataset_rows(src, src_version).table
-
-        if not self.db.has_table(self.dataset_table_name(dst, dst_version)):
-            # destination table doesn't exist, create it
-            self.create_dataset_rows_table(
-                self.dataset_table_name(dst, dst_version),
-                columns=src_dr.columns,
-            )
-            dst_empty = True
-
-        dst_dr = self.dataset_rows(dst, dst_version).table
-        merge_fields = [c.name for c in src_dr.columns if c.name != "sys__id"]
-        select_src = select(*(getattr(src_dr.columns, f) for f in merge_fields))
-
-        if dst_empty:
-            # we don't need union, but just select from source to destination
-            insert_query = sqlite.insert(dst_dr).from_select(merge_fields, select_src)
-        else:
-            dst_version_latest = None
-            # find the previous version of the destination dataset
-            dst_previous_versions = [
-                v.version
-                for v in dst.versions  # type: ignore [union-attr]
-                if semver.compare(v.version, dst_version) == -1
-            ]
-            if dst_previous_versions:
-                dst_version_latest = max(dst_previous_versions)
-
-            dst_dr_latest = self.dataset_rows(dst, dst_version_latest).table
-
-            select_dst_latest = select(
-                *(getattr(dst_dr_latest.c, f) for f in merge_fields)
-            )
-            union_query = sqlalchemy.union(select_src, select_dst_latest)
-            insert_query = (
-                sqlite.insert(dst_dr)
-                .from_select(merge_fields, union_query)
-                .prefix_with("OR IGNORE")
-            )
-
-        self.db.execute(insert_query)
-
     def prepare_entries(self, entries: "Iterable[File]") -> Iterable[dict[str, Any]]:
         return (e.model_dump() for e in entries)
 

{datachain-0.37.2 → datachain-0.37.4}/src/datachain/data_storage/warehouse.py

@@ -371,21 +371,6 @@ class AbstractWarehouse(ABC, Serializable):
         table = sa.Table(table_name, self.db.metadata)
         self.db.drop_table(table, if_exists=if_exists)
 
-    @abstractmethod
-    def merge_dataset_rows(
-        self,
-        src: "DatasetRecord",
-        dst: "DatasetRecord",
-        src_version: str,
-        dst_version: str,
-    ) -> None:
-        """
-        Merges source dataset rows and current latest destination dataset rows
-        into a new rows table created for new destination dataset version.
-        Note that table for new destination version must be created upfront.
-        Merge results should not contain duplicates.
-        """
-
     def dataset_rows_select(
         self,
         query: sa.Select,

{datachain-0.37.2 → datachain-0.37.4}/src/datachain.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: datachain
-Version: 0.37.2
+Version: 0.37.4
 Summary: Wrangle unstructured AI data at scale
 Author-email: Dmitry Petrov <support@dvc.org>
 License-Expression: Apache-2.0