datachain 0.26.4__tar.gz → 0.28.0__tar.gz

{datachain-0.26.4 → datachain-0.28.0}/.pre-commit-config.yaml

@@ -24,7 +24,7 @@ repos:
       - id: trailing-whitespace
         exclude: '^LICENSES/'
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: 'v0.12.3'
+    rev: 'v0.12.4'
     hooks:
       - id: ruff
         args: [--fix, --exit-non-zero-on-fix]

{datachain-0.26.4 → datachain-0.28.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: datachain
-Version: 0.26.4
+Version: 0.28.0
 Summary: Wrangle unstructured AI data at scale
 Author-email: Dmitry Petrov <support@dvc.org>
 License-Expression: Apache-2.0
@@ -45,7 +45,7 @@ Requires-Dist: datamodel-code-generator>=0.25
 Requires-Dist: Pillow<12,>=10.0.0
 Requires-Dist: msgpack<2,>=1.0.4
 Requires-Dist: psutil
-Requires-Dist: huggingface_hub
+Requires-Dist: huggingface_hub<0.34.0
 Requires-Dist: iterative-telemetry>=0.0.10
 Requires-Dist: platformdirs
 Requires-Dist: dvc-studio-client<1,>=0.21
@@ -120,7 +120,7 @@ Dynamic: license-file
 |logo| DataChain
 ================
 
-|PyPI| |Python Version| |Codecov| |Tests|
+|PyPI| |Python Version| |Codecov| |Tests| |DeepWiki|
 
 .. |logo| image:: docs/assets/datachain.svg
    :height: 24
@@ -136,6 +136,9 @@ Dynamic: license-file
 .. |Tests| image:: https://github.com/iterative/datachain/actions/workflows/tests.yml/badge.svg
    :target: https://github.com/iterative/datachain/actions/workflows/tests.yml
    :alt: Tests
+.. |DeepWiki| image:: https://deepwiki.com/badge.svg
+   :target: https://deepwiki.com/iterative/datachain
+   :alt: DeepWiki
 
 DataChain is a Python-based AI-data warehouse for transforming and analyzing unstructured
 data like images, audio, videos, text and PDFs. It integrates with external storage

{datachain-0.26.4 → datachain-0.28.0}/README.rst

@@ -2,7 +2,7 @@
 |logo| DataChain
 ================
 
-|PyPI| |Python Version| |Codecov| |Tests|
+|PyPI| |Python Version| |Codecov| |Tests| |DeepWiki|
 
 .. |logo| image:: docs/assets/datachain.svg
    :height: 24
@@ -18,6 +18,9 @@
 .. |Tests| image:: https://github.com/iterative/datachain/actions/workflows/tests.yml/badge.svg
    :target: https://github.com/iterative/datachain/actions/workflows/tests.yml
    :alt: Tests
+.. |DeepWiki| image:: https://deepwiki.com/badge.svg
+   :target: https://deepwiki.com/iterative/datachain
+   :alt: DeepWiki
 
 DataChain is a Python-based AI-data warehouse for transforming and analyzing unstructured
 data like images, audio, videos, text and PDFs. It integrates with external storage

{datachain-0.26.4 → datachain-0.28.0}/docs/commands/job/run.md

@@ -5,15 +5,22 @@ Run a job in Studio.
 ## Synopsis
 
 ```usage
-usage: datachain job run [-h] [-v] [-q] [--team TEAM] [--env-file ENV_FILE] [--env ENV [ENV ...]] [--cluster CLUSTER] [--workers WORKERS]
-                         [--files FILES [FILES ...]] [--python-version PYTHON_VERSION] [--repository REPOSITORY] [--req-file REQ_FILE] [--req REQ [REQ ...]]
-                         [--priority PRIORITY] [--start-time START_TIME] [--cron CRON]
+usage: datachain job run [-h] [-v] [-q] [--team TEAM] [--env-file ENV_FILE]
+                         [--env ENV [ENV ...]]
+                         [--cluster CLUSTER] [--workers WORKERS]
+                         [--files FILES [FILES ...]]
+                         [--python-version PYTHON_VERSION]
+                         [--repository REPOSITORY]
+                         [--req-file REQ_FILE] [--req REQ [REQ ...]]
+                         [--priority PRIORITY]
+                         [--start-time START_TIME] [--cron CRON]
+                         [--no-wait]
                          file
 ```
 
 ## Description
 
-This command runs a job in Studio using the specified query file. You can configure various aspects of the job including environment variables, Python version, dependencies, and more. When using --start-time or --cron, the job is scheduled as a task and will not show logs immediately. The job will be executed according to the schedule.
+This command runs a job in Studio using the specified query file. You can configure various aspects of the job including environment variables, Python version, dependencies, and more. When using --start-time or --cron, the job is scheduled to run but won't start immediately. (can be seen in the Tasks tab in UI)
 
 ## Arguments
 
@@ -32,8 +39,9 @@ This command runs a job in Studio using the specified query file. You can config
 * `--req-file REQ_FILE` - Python requirements file
 * `--req REQ` - Python package requirements
 * `--priority PRIORITY` - Priority for the job in range 0-5. Lower value is higher priority (default: 5)
-* `--start-time START_TIME` - Start time in ISO format or natural language for the cron task.
+* `--start-time START_TIME` - Time to schedule the task in YYYY-MM-DDTHH:mm format or natural language.
 * `--cron CRON` - Cron expression for the cron task.
+* `--no-wait` - Do not wait for the job to finish.
 * `-h`, `--help` - Show the help message and exit.
 * `-v`, `--verbose` - Be verbose.
 * `-q`, `--quiet` - Be quiet.
@@ -125,6 +133,12 @@ datachain job run --cron "@monthly" query.py
 datachain job run --start-time "tomorrow 3pm" --cron "0 0 * * *" query.py
 ```
 
+12. Start the job and do not wait for the job to complete
+```bash
+# Do not follow or tail the logs from Studio.
+datachain job run query.py --no-wait
+```
+
 ## Notes
 
 * Closing the logs command (e.g., with Ctrl+C) will only stop displaying the logs but will not cancel the job execution
@@ -132,7 +146,7 @@ datachain job run --start-time "tomorrow 3pm" --cron "0 0 * * *" query.py
 * The job will continue running in Studio even after you stop viewing the logs
 * You can get the list of compute clusters using `datachain job clusters` command.
 * When using `--start-time` or `--cron` options, the job is scheduled as a task and will not show logs immediately. The job will be executed according to the schedule.
-* The `--start-time` option supports natural language parsing using the dateparser library, allowing flexible time expressions like "tomorrow 3pm", "in 2 hours", "monday 9am", etc.
+* The `--start-time` option supports natural language parsing using the [dateparser](https://dateparser.readthedocs.io/en/latest/) library, allowing flexible time expressions like "tomorrow 3pm", "in 2 hours", "monday 9am", etc.
 * Cron expressions follow the standard format: minute hour day-of-month month day-of-week (e.g., "0 0 * * *" for daily at midnight) or Vixie cron-style “@” keyword expressions.
 * Following options for Vixie cron-style expressions are supported:
     * @midnight

{datachain-0.26.4 → datachain-0.28.0}/docs/examples.md

@@ -10,55 +10,45 @@ title: Examples
 
     Datachain is built by composing wrangling operations.
 
-    For example, let us consider the New Yorker Cartoon caption contest dataset, where cartoons are matched against the potential titles. Let us imagine we want to augment this dataset with synthetic scene descriptions coming from an AI model. The below code takes images from the cloud, and applies PaliGemma model to caption the first five of them and put the results in the column “scene”:
+    For example, let us consider the New Yorker Cartoon caption contest dataset, where cartoons are matched against the potential titles. Let us imagine we want to augment this dataset with synthetic scene descriptions coming from an AI model. The below code takes images from the cloud, and applies BLIP Large model to caption the first five of them and put the results in the column "scene":
 
     ```python
     import datachain as dc # (1)!
-    from transformers import AutoProcessor, PaliGemmaForConditionalGeneration # (2)!
+    from transformers import Pipeline, pipeline
+    from datachain import File
 
-    images = dc.read_storage("gs://datachain-demo/newyorker_caption_contest/images", type="image")
-
-    model = PaliGemmaForConditionalGeneration.from_pretrained("google/paligemma-3b-mix-224")
-    processor = AutoProcessor.from_pretrained("google/paligemma-3b-mix-224")
-
-    def process(file: File) -> str:
-      image=file.read().convert("RGB")
-      inputs = processor(text="caption", images=image, return_tensors="pt")
-      generate_ids = model.generate(**inputs, max_new_tokens=100)
-      return processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
+    def process(file: File, pipeline: Pipeline) -> str:
+        image = file.read().convert("RGB")
+        return pipeline(image)[0]["generated_text"]
 
     chain = (
-          images.limit(5)
-          .settings(cache=True)
-          .map(scene=lambda file: process(file), output = str)
-          .save()
+        dc.read_storage("gs://datachain-demo/newyorker_caption_contest/images", type="image", anon=True)
+        .limit(5)
+        .settings(cache=True)
+        .setup(pipeline=lambda: pipeline("image-to-text", model="Salesforce/blip-image-captioning-large"))
+        .map(scene=process)
+        .persist()
     )
     ```
 
-    1. `pip install datachain`
-    2. `pip install transformers`
+    1. `pip install datachain[hf]`
 
     Here is how we can view the results in a plot:
 
     ```python
     import matplotlib.pyplot as plt
-    import re
     from textwrap import wrap
 
-    def trim_text(text):
-        match = re.search(r'[A-Z][^.]*\.', text)
-        return match.group(0) if match else ''
-
-    images = chain.collect("file")
-    captions = chain.collect("scene")
-    _ , axes = plt.subplots(1, len(captions), figsize=(15, 5))
+    count = chain.count()
+    _, axes = plt.subplots(1, count, figsize=(15, 5))
 
-    for ax, img, caption in zip(axes, images, captions):
-        ax.imshow(img.read(),cmap='gray')
-        ax.axis('off')
-        wrapped_caption = "\n".join(wrap(trim_text(caption), 30))
-        ax.set_title(wrapped_caption, fontsize=6)
+    for ax, (img_file, caption) in zip(axes, chain.to_iter("file", "scene")):
+        ax.imshow(img_file.read(), cmap="gray")
+        ax.axis("off")
+        wrapped_caption = "\n".join(wrap(caption.strip(), 40))
+        ax.set_title(wrapped_caption, fontsize=10, pad=20)
 
+    plt.tight_layout()
     plt.show()
     ```
 

{datachain-0.26.4 → datachain-0.28.0}/mkdocs.yml

@@ -177,7 +177,7 @@ plugins:
             - https://numpy.org/doc/stable/objects.inv
             - https://pandas.pydata.org/docs/objects.inv
             - https://arrow.apache.org/docs/objects.inv
-            - https://docs.sqlalchemy.org/objects.inv
+            # - https://docs.sqlalchemy.org/objects.inv  # SSL certificate issue
             - https://docs.pydantic.dev/latest/objects.inv
 
 watch:

{datachain-0.26.4 → datachain-0.28.0}/pyproject.toml

@@ -49,7 +49,7 @@ dependencies = [
   "Pillow>=10.0.0,<12",
   "msgpack>=1.0.4,<2",
   "psutil",
-  "huggingface_hub",
+  "huggingface_hub<0.34.0",
   "iterative-telemetry>=0.0.10",
   "platformdirs",
   "dvc-studio-client>=0.21,<1",

{datachain-0.26.4 → datachain-0.28.0}/src/datachain/cli/parser/job.py

@@ -20,8 +20,8 @@ def add_jobs_parser(subparsers, parent_parser) -> None:
     studio_run_description = "Run a job in Studio. \n"
     studio_run_description += (
         "When using --start-time or --cron,"
-        " the job is scheduled as a task and will not show logs immediately."
-        " The job will be executed according to the schedule."
+        " the job is scheduled to run but won't start immediately"
+        " (can be seen in the Tasks tab in UI)"
     )
 
     studio_run_parser = jobs_subparser.add_parser(
@@ -104,11 +104,16 @@ def add_jobs_parser(subparsers, parent_parser) -> None:
     studio_run_parser.add_argument(
         "--start-time",
         action="store",
-        help="Start time in ISO format or natural language for the cron task.",
+        help="Time to schedule a task in YYYY-MM-DDTHH:mm format or natural language.",
     )
     studio_run_parser.add_argument(
         "--cron", action="store", help="Cron expression for the cron task."
     )
+    studio_run_parser.add_argument(
+        "--no-wait",
+        action="store_true",
+        help="Do not wait for the job to finish",
+    )
 
     studio_ls_help = "List jobs in Studio"
     studio_ls_description = "List jobs in Studio."

{datachain-0.26.4 → datachain-0.28.0}/src/datachain/data_storage/job.py

@@ -12,10 +12,11 @@ class JobStatus(int, Enum):
     CANCELING = 7
     CANCELED = 8
     CANCELING_SCHEDULED = 9
+    TASK = 11
 
     @classmethod
     def finished(cls) -> tuple[int, ...]:
-        return cls.COMPLETE, cls.FAILED, cls.CANCELED
+        return cls.COMPLETE, cls.FAILED, cls.CANCELED, cls.TASK
 
 
 class JobQueryType(int, Enum):

{datachain-0.26.4 → datachain-0.28.0}/src/datachain/lib/arrow.py

@@ -245,7 +245,7 @@ def arrow_type_mapper(col_type: pa.DataType, column: str = "") -> type:  # noqa:
             if field.nullable and not ModelStore.is_pydantic(dtype):
                 dtype = Optional[dtype]  # type: ignore[assignment]
             type_dict[field.name] = dtype
-        return dict_to_data_model(column, type_dict)
+        return dict_to_data_model(f"ArrowDataModel_{column}", type_dict)
     if pa.types.is_map(col_type):
         return dict
     if isinstance(col_type, pa.lib.DictionaryType):

datachain-0.28.0/src/datachain/lib/audio.py

@@ -0,0 +1,244 @@
+import posixpath
+from typing import TYPE_CHECKING, Optional, Union
+
+from datachain.lib.file import FileError
+
+if TYPE_CHECKING:
+    from numpy import ndarray
+
+    from datachain.lib.file import Audio, AudioFile, File
+
+try:
+    import torchaudio
+except ImportError as exc:
+    raise ImportError(
+        "Missing dependencies for processing audio.\n"
+        "To install run:\n\n"
+        "  pip install 'datachain[audio]'\n"
+    ) from exc
+
+
+def audio_info(file: "Union[File, AudioFile]") -> "Audio":
+    """Extract metadata like sample rate, channels, duration, and format."""
+    from datachain.lib.file import Audio
+
+    file = file.as_audio_file()
+
+    try:
+        with file.open() as f:
+            info = torchaudio.info(f)
+
+            sample_rate = int(info.sample_rate)
+            channels = int(info.num_channels)
+            frames = int(info.num_frames)
+            duration = float(frames / sample_rate) if sample_rate > 0 else 0.0
+
+            codec_name = getattr(info, "encoding", "")
+            file_ext = file.get_file_ext().lower()
+            format_name = _encoding_to_format(codec_name, file_ext)
+
+            bits_per_sample = getattr(info, "bits_per_sample", 0)
+            bit_rate = (
+                bits_per_sample * sample_rate * channels if bits_per_sample > 0 else -1
+            )
+
+    except Exception as exc:
+        raise FileError(
+            "unable to extract metadata from audio file", file.source, file.path
+        ) from exc
+
+    return Audio(
+        sample_rate=sample_rate,
+        channels=channels,
+        duration=duration,
+        samples=frames,
+        format=format_name,
+        codec=codec_name,
+        bit_rate=bit_rate,
+    )
+
+
+def _encoding_to_format(encoding: str, file_ext: str) -> str:
+    """
+    Map torchaudio encoding to a format name.
+
+    Args:
+        encoding: The encoding string from torchaudio.info()
+        file_ext: The file extension as a fallback
+
+    Returns:
+        Format name as a string
+    """
+    # Direct mapping for formats that match exactly
+    encoding_map = {
+        "FLAC": "flac",
+        "MP3": "mp3",
+        "VORBIS": "ogg",
+        "AMR_WB": "amr",
+        "AMR_NB": "amr",
+        "OPUS": "opus",
+        "GSM": "gsm",
+    }
+
+    if encoding in encoding_map:
+        return encoding_map[encoding]
+
+    # For PCM variants, use file extension to determine format
+    if encoding.startswith("PCM_"):
+        # Common PCM formats by extension
+        pcm_formats = {
+            "wav": "wav",
+            "aiff": "aiff",
+            "au": "au",
+            "raw": "raw",
+        }
+        return pcm_formats.get(file_ext, "wav")  # Default to wav for PCM
+
+    # Fallback to file extension if encoding is unknown
+    return file_ext if file_ext else "unknown"
+
+
+def audio_to_np(
+    audio: "AudioFile", start: float = 0, duration: Optional[float] = None
+) -> "tuple[ndarray, int]":
+    """Load audio fragment as numpy array.
+    Multi-channel audio is transposed to (samples, channels)."""
+    if start < 0:
+        raise ValueError("start must be a non-negative float")
+
+    if duration is not None and duration <= 0:
+        raise ValueError("duration must be a positive float")
+
+    if hasattr(audio, "as_audio_file"):
+        audio = audio.as_audio_file()
+
+    try:
+        with audio.open() as f:
+            info = torchaudio.info(f)
+            sample_rate = info.sample_rate
+
+            frame_offset = int(start * sample_rate)
+            num_frames = int(duration * sample_rate) if duration is not None else -1
+
+            # Reset file pointer to the beginning
+            # This is important to ensure we read from the correct position later
+            f.seek(0)
+
+            waveform, sr = torchaudio.load(
+                f, frame_offset=frame_offset, num_frames=num_frames
+            )
+
+            audio_np = waveform.numpy()
+
+            if audio_np.shape[0] > 1:
+                audio_np = audio_np.T
+            else:
+                audio_np = audio_np.squeeze()
+
+            return audio_np, int(sr)
+    except Exception as exc:
+        raise FileError(
+            "unable to read audio fragment", audio.source, audio.path
+        ) from exc
+
+
+def audio_to_bytes(
+    audio: "AudioFile",
+    format: str = "wav",
+    start: float = 0,
+    duration: Optional[float] = None,
+) -> bytes:
+    """Convert audio to bytes using soundfile.
+
+    If duration is None, converts from start to end of file.
+    If start is 0 and duration is None, converts entire file."""
+    y, sr = audio_to_np(audio, start, duration)
+
+    import io
+
+    import soundfile as sf
+
+    buffer = io.BytesIO()
+    sf.write(buffer, y, sr, format=format)
+    return buffer.getvalue()
+
+
+def save_audio(
+    audio: "AudioFile",
+    output: str,
+    format: Optional[str] = None,
+    start: float = 0,
+    end: Optional[float] = None,
+) -> "AudioFile":
+    """Save audio file or extract fragment to specified format.
+
+    Args:
+        audio: Source AudioFile object
+        output: Output directory path
+        format: Output format ('wav', 'mp3', etc). Defaults to source format
+        start: Start time in seconds (>= 0). Defaults to 0
+        end: End time in seconds. If None, extracts to end of file
+
+    Returns:
+        AudioFile: New audio file with format conversion/extraction applied
+
+    Examples:
+        save_audio(audio, "/path", "mp3")                       # Entire file to MP3
+        save_audio(audio, "s3://bucket/path", "wav", start=2.5) # From 2.5s to end
+        save_audio(audio, "/path", "flac", start=1, end=3)      # Extract 1-3s fragment
+    """
+    if format is None:
+        format = audio.get_file_ext()
+
+    # Validate start time
+    if start < 0:
+        raise ValueError(
+            f"Can't save audio for '{audio.path}', "
+            f"start time must be non-negative: {start:.3f}"
+        )
+
+    # Handle full file conversion when end is None and start is 0
+    if end is None and start == 0:
+        output_file = posixpath.join(output, f"{audio.get_file_stem()}.{format}")
+        try:
+            audio_bytes = audio_to_bytes(audio, format, start=0, duration=None)
+        except Exception as exc:
+            raise FileError(
+                "unable to convert audio file", audio.source, audio.path
+            ) from exc
+    elif end is None:
+        # Extract from start to end of file
+        output_file = posixpath.join(
+            output, f"{audio.get_file_stem()}_{int(start * 1000):06d}_end.{format}"
+        )
+        try:
+            audio_bytes = audio_to_bytes(audio, format, start=start, duration=None)
+        except Exception as exc:
+            raise FileError(
+                "unable to save audio fragment", audio.source, audio.path
+            ) from exc
+    else:
+        # Fragment extraction mode with specific end time
+        if end < 0 or start >= end:
+            raise ValueError(
+                f"Can't save audio for '{audio.path}', "
+                f"invalid time range: ({start:.3f}, {end:.3f})"
+            )
+
+        duration = end - start
+        start_ms = int(start * 1000)
+        end_ms = int(end * 1000)
+        output_file = posixpath.join(
+            output, f"{audio.get_file_stem()}_{start_ms:06d}_{end_ms:06d}.{format}"
+        )
+
+        try:
+            audio_bytes = audio_to_bytes(audio, format, start, duration)
+        except Exception as exc:
+            raise FileError(
+                "unable to save audio fragment", audio.source, audio.path
+            ) from exc
+
+    from datachain.lib.file import AudioFile
+
+    return AudioFile.upload(audio_bytes, output_file, catalog=audio._catalog)

{datachain-0.26.4 → datachain-0.28.0}/src/datachain/lib/data_model.py

@@ -1,3 +1,5 @@
+import inspect
+import uuid
 from collections.abc import Sequence
 from datetime import datetime
 from typing import ClassVar, Optional, Union, get_args, get_origin
@@ -80,7 +82,9 @@ def dict_to_data_model(
 
     fields = {
         name: (
-            anno,
+            anno
+            if inspect.isclass(anno) and issubclass(anno, BaseModel)
+            else Optional[anno],
             Field(
                 validation_alias=AliasChoices(name, original_names[idx] or name),
                 default=None,
@@ -101,6 +105,10 @@ def dict_to_data_model(
                     field_info[str(alias)] = (_name, field)
             return field_info
 
+    # Generate random unique name if not provided
+    if not name:
+        name = f"DataModel_{uuid.uuid4().hex[:8]}"
+
     return create_model(
         name,
         __base__=_DataModelStrict,

{datachain-0.26.4 → datachain-0.28.0}/src/datachain/lib/dc/hf.py

@@ -25,19 +25,23 @@ def read_hf(
     settings: Optional[dict] = None,
     column: str = "",
     model_name: str = "",
+    limit: int = 0,
     **kwargs,
 ) -> "DataChain":
-    """Generate chain from huggingface hub dataset.
+    """Generate chain from Hugging Face Hub dataset.
 
     Parameters:
         dataset : Path or name of the dataset to read from Hugging Face Hub,
             or an instance of `datasets.Dataset`-like object.
-        args : Additional positional arguments to pass to datasets.load_dataset.
+        args : Additional positional arguments to pass to `datasets.load_dataset`.
         session : Session to use for the chain.
         settings : Settings to use for the chain.
         column : Generated object column name.
         model_name : Generated model name.
-        kwargs : Parameters to pass to datasets.load_dataset.
+        limit : Limit the number of items to read from the HF dataset.
+                Adds `take(limit)` to the `datasets.load_dataset`.
+                Defaults to 0 (no limit).
+        kwargs : Parameters to pass to `datasets.load_dataset`.
 
     Example:
         Load from Hugging Face Hub:
@@ -53,6 +57,18 @@ def read_hf(
         import datachain as dc
         chain = dc.read_hf(ds)
         ```
+
+        Streaming with limit, for large datasets:
+        ```py
+        import datachain as dc
+        ds = dc.read_hf("beans", split="train", streaming=True, limit=10)
+        ```
+
+        or use HF split syntax (not supported if streaming is enabled):
+        ```py
+        import datachain as dc
+        ds = dc.read_hf("beans", split="train[%10]")
+        ```
     """
     from datachain.lib.hf import HFGenerator, get_output_schema, stream_splits
 
@@ -72,4 +88,4 @@ def read_hf(
         output = {column: model}
 
     chain = read_values(split=list(ds_dict.keys()), session=session, settings=settings)
-    return chain.gen(HFGenerator(dataset, model, *args, **kwargs), output=output)
+    return chain.gen(HFGenerator(dataset, model, limit, *args, **kwargs), output=output)