datachain 0.26.3__tar.gz → 0.27.0__tar.gz

{datachain-0.26.3 → datachain-0.27.0}/.pre-commit-config.yaml

@@ -24,7 +24,7 @@ repos:
       - id: trailing-whitespace
         exclude: '^LICENSES/'
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: 'v0.12.3'
+    rev: 'v0.12.4'
     hooks:
       - id: ruff
         args: [--fix, --exit-non-zero-on-fix]

{datachain-0.26.3 → datachain-0.27.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: datachain
-Version: 0.26.3
+Version: 0.27.0
 Summary: Wrangle unstructured AI data at scale
 Author-email: Dmitry Petrov <support@dvc.org>
 License-Expression: Apache-2.0
@@ -26,6 +26,7 @@ Requires-Dist: packaging
 Requires-Dist: pyarrow
 Requires-Dist: typing-extensions
 Requires-Dist: python-dateutil>=2
+Requires-Dist: dateparser>=1.0.0
 Requires-Dist: attrs>=21.3.0
 Requires-Dist: fsspec>=2024.2.0
 Requires-Dist: s3fs>=2024.2.0
@@ -100,6 +101,7 @@ Provides-Extra: dev
 Requires-Dist: datachain[docs,tests]; extra == "dev"
 Requires-Dist: mypy==1.17.0; extra == "dev"
 Requires-Dist: types-python-dateutil; extra == "dev"
+Requires-Dist: types-dateparser; extra == "dev"
 Requires-Dist: types-pytz; extra == "dev"
 Requires-Dist: types-PyYAML; extra == "dev"
 Requires-Dist: types-requests; extra == "dev"
@@ -118,7 +120,7 @@ Dynamic: license-file
 |logo| DataChain
 ================
 
-|PyPI| |Python Version| |Codecov| |Tests|
+|PyPI| |Python Version| |Codecov| |Tests| |DeepWiki|
 
 .. |logo| image:: docs/assets/datachain.svg
    :height: 24
@@ -134,6 +136,9 @@ Dynamic: license-file
 .. |Tests| image:: https://github.com/iterative/datachain/actions/workflows/tests.yml/badge.svg
    :target: https://github.com/iterative/datachain/actions/workflows/tests.yml
    :alt: Tests
+.. |DeepWiki| image:: https://deepwiki.com/badge.svg
+   :target: https://deepwiki.com/iterative/datachain
+   :alt: DeepWiki
 
 DataChain is a Python-based AI-data warehouse for transforming and analyzing unstructured
 data like images, audio, videos, text and PDFs. It integrates with external storage

{datachain-0.26.3 → datachain-0.27.0}/README.rst

@@ -2,7 +2,7 @@
 |logo| DataChain
 ================
 
-|PyPI| |Python Version| |Codecov| |Tests|
+|PyPI| |Python Version| |Codecov| |Tests| |DeepWiki|
 
 .. |logo| image:: docs/assets/datachain.svg
    :height: 24
@@ -18,6 +18,9 @@
 .. |Tests| image:: https://github.com/iterative/datachain/actions/workflows/tests.yml/badge.svg
    :target: https://github.com/iterative/datachain/actions/workflows/tests.yml
    :alt: Tests
+.. |DeepWiki| image:: https://deepwiki.com/badge.svg
+   :target: https://deepwiki.com/iterative/datachain
+   :alt: DeepWiki
 
 DataChain is a Python-based AI-data warehouse for transforming and analyzing unstructured
 data like images, audio, videos, text and PDFs. It integrates with external storage

{datachain-0.26.3 → datachain-0.27.0}/docs/commands/job/run.md

@@ -5,15 +5,21 @@ Run a job in Studio.
 ## Synopsis
 
 ```usage
-usage: datachain job run [-h] [-v] [-q] [--team TEAM] [--env-file ENV_FILE] [--env ENV [ENV ...]]
-                         [--workers WORKERS] [--files FILES [FILES ...]] [--python-version PYTHON_VERSION]
+usage: datachain job run [-h] [-v] [-q] [--team TEAM] [--env-file ENV_FILE]
+                         [--env ENV [ENV ...]]
+                         [--cluster CLUSTER] [--workers WORKERS]
+                         [--files FILES [FILES ...]]
+                         [--python-version PYTHON_VERSION]
+                         [--repository REPOSITORY]
                          [--req-file REQ_FILE] [--req REQ [REQ ...]]
+                         [--priority PRIORITY]
+                         [--start-time START_TIME] [--cron CRON]
                          file
 ```
 
 ## Description
 
-This command runs a job in Studio using the specified query file. You can configure various aspects of the job including environment variables, Python version, dependencies, and more.
+This command runs a job in Studio using the specified query file. You can configure various aspects of the job including environment variables, Python version, dependencies, and more. When using --start-time or --cron, the job is scheduled to run but won't start immediately. (can be seen in the Tasks tab in UI)
 
 ## Arguments
 
@@ -28,10 +34,12 @@ This command runs a job in Studio using the specified query file. You can config
 * `--workers WORKERS` - Number of workers for the job
 * `--files FILES` - Additional files to include in the job
 * `--python-version PYTHON_VERSION` - Python version for the job (e.g., 3.9, 3.10, 3.11)
+* `--repository REPOSITORY` - Repository URL to clone before running the job
 * `--req-file REQ_FILE` - Python requirements file
 * `--req REQ` - Python package requirements
 * `--priority PRIORITY` - Priority for the job in range 0-5. Lower value is higher priority (default: 5)
-* `--repository URL` - Repository URL to clone before running the job.
+* `--start-time START_TIME` - Time to schedule the task in YYYY-MM-DDTHH:mm format or natural language.
+* `--cron CRON` - Cron expression for the cron task.
 * `-h`, `--help` - Show the help message and exit.
 * `-v`, `--verbose` - Be verbose.
 * `-q`, `--quiet` - Be quiet.
@@ -66,17 +74,11 @@ datachain job run --env API_KEY=123 --req pandas numpy query.py
 6. Run a job with a repository (will be cloned in the job working directory):
 ```bash
 datachain job run --repository https://github.com/iterative/datachain query.py
-```
-
-To specify a branch / revision:
 
-```bash
+# To specify a branch / revision:
 datachain job run --repository https://github.com/iterative/datachain@main query.py
-```
-
-Git URLs are also supported:
 
-```bash
+# Git URLs are also supported:
 datachain job run --repository git@github.com:iterative/datachain.git@main query.py
 ```
 
@@ -90,7 +92,43 @@ datachain job run --priority 2 query.py
 # Get the cluster id using following command
 datachain job clusters
 # Use the id  of an active clusters from above
-datachain job run --cluster-id 1 query.py
+datachain job run --cluster 1 query.py
+```
+
+9. Schedule a job to run once at a specific time
+```bash
+# Run job tomorrow at 3pm
+datachain job run --start-time "tomorrow 3pm" query.py
+
+# Run job in 2 hours
+datachain job run --start-time "in 2 hours" query.py
+
+# Run job on Monday at 9am
+datachain job run --start-time "monday 9am" query.py
+
+# Run job at a specific date and time
+datachain job run --start-time "2024-01-15 14:30:00" query.py
+```
+
+10. Schedule a recurring job using cron expression
+```bash
+# Run job daily at midnight
+datachain job run --cron "0 0 * * *" query.py
+
+# Run job every Monday at 9am
+datachain job run --cron "0 9 * * 1" query.py
+
+# Run job every hour
+datachain job run --cron "0 * * * *" query.py
+
+# Run job every month
+datachain job run --cron "@monthly" query.py
+```
+
+11. Schedule a recurring job with a start time
+```bash
+# Start the cron job after tomorrow 3pm
+datachain job run --start-time "tomorrow 3pm" --cron "0 0 * * *" query.py
 ```
 
 ## Notes
@@ -99,3 +137,14 @@ datachain job run --cluster-id 1 query.py
 * To cancel a running job, use the `datachain job cancel` command
 * The job will continue running in Studio even after you stop viewing the logs
 * You can get the list of compute clusters using `datachain job clusters` command.
+* When using `--start-time` or `--cron` options, the job is scheduled as a task and will not show logs immediately. The job will be executed according to the schedule.
+* The `--start-time` option supports natural language parsing using the [dateparser](https://dateparser.readthedocs.io/en/latest/) library, allowing flexible time expressions like "tomorrow 3pm", "in 2 hours", "monday 9am", etc.
+* Cron expressions follow the standard format: minute hour day-of-month month day-of-week (e.g., "0 0 * * *" for daily at midnight) or Vixie cron-style “@” keyword expressions.
+* Following options for Vixie cron-style expressions are supported:
+    * @midnight
+    * @hourly
+    * @daily
+    * @weekly
+    * @monthly
+    * @yearly
+    * @annually

{datachain-0.26.3 → datachain-0.27.0}/docs/examples.md

@@ -10,55 +10,45 @@ title: Examples
 
     Datachain is built by composing wrangling operations.
 
-    For example, let us consider the New Yorker Cartoon caption contest dataset, where cartoons are matched against the potential titles. Let us imagine we want to augment this dataset with synthetic scene descriptions coming from an AI model. The below code takes images from the cloud, and applies PaliGemma model to caption the first five of them and put the results in the column “scene”:
+    For example, let us consider the New Yorker Cartoon caption contest dataset, where cartoons are matched against the potential titles. Let us imagine we want to augment this dataset with synthetic scene descriptions coming from an AI model. The below code takes images from the cloud, and applies BLIP Large model to caption the first five of them and put the results in the column "scene":
 
     ```python
     import datachain as dc # (1)!
-    from transformers import AutoProcessor, PaliGemmaForConditionalGeneration # (2)!
+    from transformers import Pipeline, pipeline
+    from datachain import File
 
-    images = dc.read_storage("gs://datachain-demo/newyorker_caption_contest/images", type="image")
-
-    model = PaliGemmaForConditionalGeneration.from_pretrained("google/paligemma-3b-mix-224")
-    processor = AutoProcessor.from_pretrained("google/paligemma-3b-mix-224")
-
-    def process(file: File) -> str:
-      image=file.read().convert("RGB")
-      inputs = processor(text="caption", images=image, return_tensors="pt")
-      generate_ids = model.generate(**inputs, max_new_tokens=100)
-      return processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
+    def process(file: File, pipeline: Pipeline) -> str:
+        image = file.read().convert("RGB")
+        return pipeline(image)[0]["generated_text"]
 
     chain = (
-          images.limit(5)
-          .settings(cache=True)
-          .map(scene=lambda file: process(file), output = str)
-          .save()
+        dc.read_storage("gs://datachain-demo/newyorker_caption_contest/images", type="image", anon=True)
+        .limit(5)
+        .settings(cache=True)
+        .setup(pipeline=lambda: pipeline("image-to-text", model="Salesforce/blip-image-captioning-large"))
+        .map(scene=process)
+        .persist()
     )
     ```
 
-    1. `pip install datachain`
-    2. `pip install transformers`
+    1. `pip install datachain[hf]`
 
     Here is how we can view the results in a plot:
 
     ```python
     import matplotlib.pyplot as plt
-    import re
     from textwrap import wrap
 
-    def trim_text(text):
-        match = re.search(r'[A-Z][^.]*\.', text)
-        return match.group(0) if match else ''
-
-    images = chain.collect("file")
-    captions = chain.collect("scene")
-    _ , axes = plt.subplots(1, len(captions), figsize=(15, 5))
+    count = chain.count()
+    _, axes = plt.subplots(1, count, figsize=(15, 5))
 
-    for ax, img, caption in zip(axes, images, captions):
-        ax.imshow(img.read(),cmap='gray')
-        ax.axis('off')
-        wrapped_caption = "\n".join(wrap(trim_text(caption), 30))
-        ax.set_title(wrapped_caption, fontsize=6)
+    for ax, (img_file, caption) in zip(axes, chain.to_iter("file", "scene")):
+        ax.imshow(img_file.read(), cmap="gray")
+        ax.axis("off")
+        wrapped_caption = "\n".join(wrap(caption.strip(), 40))
+        ax.set_title(wrapped_caption, fontsize=10, pad=20)
 
+    plt.tight_layout()
     plt.show()
     ```
 

{datachain-0.26.3 → datachain-0.27.0}/pyproject.toml

@@ -30,6 +30,7 @@ dependencies = [
   "pyarrow",
   "typing-extensions",
   "python-dateutil>=2",
+  "dateparser>=1.0.0",
   "attrs>=21.3.0",
   "fsspec>=2024.2.0",
   "s3fs>=2024.2.0",
@@ -116,6 +117,7 @@ dev = [
   "datachain[docs,tests]",
   "mypy==1.17.0",
   "types-python-dateutil",
+  "types-dateparser",
   "types-pytz",
   "types-PyYAML",
   "types-requests",

{datachain-0.26.3 → datachain-0.27.0}/src/datachain/cli/parser/job.py

@@ -17,7 +17,12 @@ def add_jobs_parser(subparsers, parent_parser) -> None:
     )
 
     studio_run_help = "Run a job in Studio"
-    studio_run_description = "Run a job in Studio."
+    studio_run_description = "Run a job in Studio. \n"
+    studio_run_description += (
+        "When using --start-time or --cron,"
+        " the job is scheduled to run but won't start immediately"
+        " (can be seen in the Tasks tab in UI)"
+    )
 
     studio_run_parser = jobs_subparser.add_parser(
         "run",
@@ -96,6 +101,14 @@ def add_jobs_parser(subparsers, parent_parser) -> None:
         help="Priority for the job in range 0-5. "
         "Lower value is higher priority (default: 5)",
     )
+    studio_run_parser.add_argument(
+        "--start-time",
+        action="store",
+        help="Time to schedule a task in YYYY-MM-DDTHH:mm format or natural language.",
+    )
+    studio_run_parser.add_argument(
+        "--cron", action="store", help="Cron expression for the cron task."
+    )
 
     studio_ls_help = "List jobs in Studio"
     studio_ls_description = "List jobs in Studio."

{datachain-0.26.3 → datachain-0.27.0}/src/datachain/lib/arrow.py

@@ -245,7 +245,7 @@ def arrow_type_mapper(col_type: pa.DataType, column: str = "") -> type:  # noqa:
             if field.nullable and not ModelStore.is_pydantic(dtype):
                 dtype = Optional[dtype]  # type: ignore[assignment]
             type_dict[field.name] = dtype
-        return dict_to_data_model(column, type_dict)
+        return dict_to_data_model(f"ArrowDataModel_{column}", type_dict)
     if pa.types.is_map(col_type):
         return dict
     if isinstance(col_type, pa.lib.DictionaryType):

datachain-0.27.0/src/datachain/lib/audio.py

@@ -0,0 +1,244 @@
+import posixpath
+from typing import TYPE_CHECKING, Optional, Union
+
+from datachain.lib.file import FileError
+
+if TYPE_CHECKING:
+    from numpy import ndarray
+
+    from datachain.lib.file import Audio, AudioFile, File
+
+try:
+    import torchaudio
+except ImportError as exc:
+    raise ImportError(
+        "Missing dependencies for processing audio.\n"
+        "To install run:\n\n"
+        "  pip install 'datachain[audio]'\n"
+    ) from exc
+
+
+def audio_info(file: "Union[File, AudioFile]") -> "Audio":
+    """Extract metadata like sample rate, channels, duration, and format."""
+    from datachain.lib.file import Audio
+
+    file = file.as_audio_file()
+
+    try:
+        with file.open() as f:
+            info = torchaudio.info(f)
+
+            sample_rate = int(info.sample_rate)
+            channels = int(info.num_channels)
+            frames = int(info.num_frames)
+            duration = float(frames / sample_rate) if sample_rate > 0 else 0.0
+
+            codec_name = getattr(info, "encoding", "")
+            file_ext = file.get_file_ext().lower()
+            format_name = _encoding_to_format(codec_name, file_ext)
+
+            bits_per_sample = getattr(info, "bits_per_sample", 0)
+            bit_rate = (
+                bits_per_sample * sample_rate * channels if bits_per_sample > 0 else -1
+            )
+
+    except Exception as exc:
+        raise FileError(
+            "unable to extract metadata from audio file", file.source, file.path
+        ) from exc
+
+    return Audio(
+        sample_rate=sample_rate,
+        channels=channels,
+        duration=duration,
+        samples=frames,
+        format=format_name,
+        codec=codec_name,
+        bit_rate=bit_rate,
+    )
+
+
+def _encoding_to_format(encoding: str, file_ext: str) -> str:
+    """
+    Map torchaudio encoding to a format name.
+
+    Args:
+        encoding: The encoding string from torchaudio.info()
+        file_ext: The file extension as a fallback
+
+    Returns:
+        Format name as a string
+    """
+    # Direct mapping for formats that match exactly
+    encoding_map = {
+        "FLAC": "flac",
+        "MP3": "mp3",
+        "VORBIS": "ogg",
+        "AMR_WB": "amr",
+        "AMR_NB": "amr",
+        "OPUS": "opus",
+        "GSM": "gsm",
+    }
+
+    if encoding in encoding_map:
+        return encoding_map[encoding]
+
+    # For PCM variants, use file extension to determine format
+    if encoding.startswith("PCM_"):
+        # Common PCM formats by extension
+        pcm_formats = {
+            "wav": "wav",
+            "aiff": "aiff",
+            "au": "au",
+            "raw": "raw",
+        }
+        return pcm_formats.get(file_ext, "wav")  # Default to wav for PCM
+
+    # Fallback to file extension if encoding is unknown
+    return file_ext if file_ext else "unknown"
+
+
+def audio_to_np(
+    audio: "AudioFile", start: float = 0, duration: Optional[float] = None
+) -> "tuple[ndarray, int]":
+    """Load audio fragment as numpy array.
+    Multi-channel audio is transposed to (samples, channels)."""
+    if start < 0:
+        raise ValueError("start must be a non-negative float")
+
+    if duration is not None and duration <= 0:
+        raise ValueError("duration must be a positive float")
+
+    if hasattr(audio, "as_audio_file"):
+        audio = audio.as_audio_file()
+
+    try:
+        with audio.open() as f:
+            info = torchaudio.info(f)
+            sample_rate = info.sample_rate
+
+            frame_offset = int(start * sample_rate)
+            num_frames = int(duration * sample_rate) if duration is not None else -1
+
+            # Reset file pointer to the beginning
+            # This is important to ensure we read from the correct position later
+            f.seek(0)
+
+            waveform, sr = torchaudio.load(
+                f, frame_offset=frame_offset, num_frames=num_frames
+            )
+
+            audio_np = waveform.numpy()
+
+            if audio_np.shape[0] > 1:
+                audio_np = audio_np.T
+            else:
+                audio_np = audio_np.squeeze()
+
+            return audio_np, int(sr)
+    except Exception as exc:
+        raise FileError(
+            "unable to read audio fragment", audio.source, audio.path
+        ) from exc
+
+
+def audio_to_bytes(
+    audio: "AudioFile",
+    format: str = "wav",
+    start: float = 0,
+    duration: Optional[float] = None,
+) -> bytes:
+    """Convert audio to bytes using soundfile.
+
+    If duration is None, converts from start to end of file.
+    If start is 0 and duration is None, converts entire file."""
+    y, sr = audio_to_np(audio, start, duration)
+
+    import io
+
+    import soundfile as sf
+
+    buffer = io.BytesIO()
+    sf.write(buffer, y, sr, format=format)
+    return buffer.getvalue()
+
+
+def save_audio(
+    audio: "AudioFile",
+    output: str,
+    format: Optional[str] = None,
+    start: float = 0,
+    end: Optional[float] = None,
+) -> "AudioFile":
+    """Save audio file or extract fragment to specified format.
+
+    Args:
+        audio: Source AudioFile object
+        output: Output directory path
+        format: Output format ('wav', 'mp3', etc). Defaults to source format
+        start: Start time in seconds (>= 0). Defaults to 0
+        end: End time in seconds. If None, extracts to end of file
+
+    Returns:
+        AudioFile: New audio file with format conversion/extraction applied
+
+    Examples:
+        save_audio(audio, "/path", "mp3")                       # Entire file to MP3
+        save_audio(audio, "s3://bucket/path", "wav", start=2.5) # From 2.5s to end
+        save_audio(audio, "/path", "flac", start=1, end=3)      # Extract 1-3s fragment
+    """
+    if format is None:
+        format = audio.get_file_ext()
+
+    # Validate start time
+    if start < 0:
+        raise ValueError(
+            f"Can't save audio for '{audio.path}', "
+            f"start time must be non-negative: {start:.3f}"
+        )
+
+    # Handle full file conversion when end is None and start is 0
+    if end is None and start == 0:
+        output_file = posixpath.join(output, f"{audio.get_file_stem()}.{format}")
+        try:
+            audio_bytes = audio_to_bytes(audio, format, start=0, duration=None)
+        except Exception as exc:
+            raise FileError(
+                "unable to convert audio file", audio.source, audio.path
+            ) from exc
+    elif end is None:
+        # Extract from start to end of file
+        output_file = posixpath.join(
+            output, f"{audio.get_file_stem()}_{int(start * 1000):06d}_end.{format}"
+        )
+        try:
+            audio_bytes = audio_to_bytes(audio, format, start=start, duration=None)
+        except Exception as exc:
+            raise FileError(
+                "unable to save audio fragment", audio.source, audio.path
+            ) from exc
+    else:
+        # Fragment extraction mode with specific end time
+        if end < 0 or start >= end:
+            raise ValueError(
+                f"Can't save audio for '{audio.path}', "
+                f"invalid time range: ({start:.3f}, {end:.3f})"
+            )
+
+        duration = end - start
+        start_ms = int(start * 1000)
+        end_ms = int(end * 1000)
+        output_file = posixpath.join(
+            output, f"{audio.get_file_stem()}_{start_ms:06d}_{end_ms:06d}.{format}"
+        )
+
+        try:
+            audio_bytes = audio_to_bytes(audio, format, start, duration)
+        except Exception as exc:
+            raise FileError(
+                "unable to save audio fragment", audio.source, audio.path
+            ) from exc
+
+    from datachain.lib.file import AudioFile
+
+    return AudioFile.upload(audio_bytes, output_file, catalog=audio._catalog)

{datachain-0.26.3 → datachain-0.27.0}/src/datachain/lib/data_model.py

@@ -1,3 +1,5 @@
+import inspect
+import uuid
 from collections.abc import Sequence
 from datetime import datetime
 from typing import ClassVar, Optional, Union, get_args, get_origin
@@ -80,7 +82,9 @@ def dict_to_data_model(
 
     fields = {
         name: (
-            anno,
+            anno
+            if inspect.isclass(anno) and issubclass(anno, BaseModel)
+            else Optional[anno],
             Field(
                 validation_alias=AliasChoices(name, original_names[idx] or name),
                 default=None,
@@ -101,6 +105,10 @@ def dict_to_data_model(
                     field_info[str(alias)] = (_name, field)
             return field_info
 
+    # Generate random unique name if not provided
+    if not name:
+        name = f"DataModel_{uuid.uuid4().hex[:8]}"
+
     return create_model(
         name,
         __base__=_DataModelStrict,

{datachain-0.26.3 → datachain-0.27.0}/src/datachain/lib/dc/datachain.py

@@ -2388,7 +2388,7 @@ class DataChain:
         placement: FileExportPlacement = "fullpath",
         link_type: Literal["copy", "symlink"] = "copy",
         num_threads: Optional[int] = EXPORT_FILES_MAX_THREADS,
-        anon: bool = False,
+        anon: Optional[bool] = None,
         client_config: Optional[dict] = None,
     ) -> None:
         """Export files from a specified signal to a directory. Files can be
@@ -2403,7 +2403,11 @@ class DataChain:
                 Falls back to `'copy'` if symlinking fails.
             num_threads : number of threads to use for exporting files.
                 By default it uses 5 threads.
-            anon: If true, we will treat cloud bucket as public one
+            anon: If True, we will treat cloud bucket as public one. Default behavior
+                depends on the previous session configuration (e.g. happens in the
+                initial `read_storage`) and particular cloud storage client
+                implementation (e.g. S3 fallbacks to anonymous access if no credentials
+                were found).
             client_config: Optional configuration for the destination storage client
 
         Example:
@@ -2421,8 +2425,8 @@ class DataChain:
         ):
             raise ValueError("Files with the same name found")
 
-        if anon:
-            client_config = (client_config or {}) | {"anon": True}
+        if anon is not None:
+            client_config = (client_config or {}) | {"anon": anon}
 
         progress_bar = tqdm(
             desc=f"Exporting files to {output}: ",

{datachain-0.26.3 → datachain-0.27.0}/src/datachain/lib/dc/hf.py

@@ -25,19 +25,23 @@ def read_hf(
     settings: Optional[dict] = None,
     column: str = "",
     model_name: str = "",
+    limit: int = 0,
     **kwargs,
 ) -> "DataChain":
-    """Generate chain from huggingface hub dataset.
+    """Generate chain from Hugging Face Hub dataset.
 
     Parameters:
         dataset : Path or name of the dataset to read from Hugging Face Hub,
             or an instance of `datasets.Dataset`-like object.
-        args : Additional positional arguments to pass to datasets.load_dataset.
+        args : Additional positional arguments to pass to `datasets.load_dataset`.
         session : Session to use for the chain.
         settings : Settings to use for the chain.
         column : Generated object column name.
         model_name : Generated model name.
-        kwargs : Parameters to pass to datasets.load_dataset.
+        limit : Limit the number of items to read from the HF dataset.
+                Adds `take(limit)` to the `datasets.load_dataset`.
+                Defaults to 0 (no limit).
+        kwargs : Parameters to pass to `datasets.load_dataset`.
 
     Example:
         Load from Hugging Face Hub:
@@ -53,6 +57,18 @@ def read_hf(
         import datachain as dc
         chain = dc.read_hf(ds)
         ```
+
+        Streaming with limit, for large datasets:
+        ```py
+        import datachain as dc
+        ds = dc.read_hf("beans", split="train", streaming=True, limit=10)
+        ```
+
+        or use HF split syntax (not supported if streaming is enabled):
+        ```py
+        import datachain as dc
+        ds = dc.read_hf("beans", split="train[%10]")
+        ```
     """
     from datachain.lib.hf import HFGenerator, get_output_schema, stream_splits
 
@@ -72,4 +88,4 @@ def read_hf(
         output = {column: model}
 
     chain = read_values(split=list(ds_dict.keys()), session=session, settings=settings)
-    return chain.gen(HFGenerator(dataset, model, *args, **kwargs), output=output)
+    return chain.gen(HFGenerator(dataset, model, limit, *args, **kwargs), output=output)