local-data-studio 1.1.0__tar.gz → 2.0.0__tar.gz

local_data_studio-2.0.0/CHANGELOG.md

@@ -0,0 +1,142 @@
+# Changelog
+
+Notable changes to Local Data Studio are recorded here. Release Please updates
+this file from Conventional Commits when it prepares a release pull request.
+
+## [2.0.0](https://github.com/Onely7/local_data_studio/compare/v1.1.1...v2.0.0) (2026-11-02)
+
+
+### ⚠ BREAKING CHANGES
+
+* **index:** Jumps past the row index return HTTP 409 with the detail {"code": "row_index_required", "row": ..., "message": ...} instead of HTTP 400, and build_line_index_with_progress reports indexed_rows and complete alongside row_count, which is None until the index is complete.
+* **preview:** resolve_raw_image_file is renamed to resolve_raw_media_file, and preview responses carry audio markers instead of hex strings for audio bytes.
+* **deps:** Sentence Transformers 6.2+, Transformers 5.19+, Pillow 13+, and LiteLLM 1.100.4+ are now required.
+* **sql:** guard_select_sql_for_dataset was removed; synchronous callers of execute_query_guarded still receive HTTP 400 for full scans of large datasets.
+* **preview:** image bytes in preview responses are thumbnails rather than the original bytes, and raw rows return image bytes as data URLs instead of hex strings.
+* **readers:** Parquet and search page tokens use a new format, search responses include next_page_token, and line rows more than 1M rows past a checkpoint require the row index.
+
+### Features
+
+* **atlas:** embed audio columns ([56c17ba](https://github.com/Onely7/local_data_studio/commit/56c17ba96bb07d1fcceaf36ed781673b408af1e4))
+* **atlas:** support EmbeddingGemma 2 and multimodal ST 6 models ([6d1a5ef](https://github.com/Onely7/local_data_studio/commit/6d1a5ef2400f66c98d6b7d2645c95d343857fe0d))
+* **index:** index rows in the background and resume interrupted scans ([858033f](https://github.com/Onely7/local_data_studio/commit/858033f7ba2fd24420f1af472dea8def643f9f59))
+* **json:** stream JSON arrays and read JSON Lines saved as .json ([ca7376a](https://github.com/Onely7/local_data_studio/commit/ca7376a66b0ff19d5827e35ae07822c9a53d4678))
+* **preview:** play audio in dataset cells ([6010bc8](https://github.com/Onely7/local_data_studio/commit/6010bc8d5061354e0a9ed4ed82fff83c39cc0d77))
+* **sql:** run full-scan queries on large datasets in the background ([be78154](https://github.com/Onely7/local_data_studio/commit/be78154f50f953d8de4d232a1e77b0d1e5e476da))
+* **transcription:** transcribe audio cells with a configurable speech model ([0f0b332](https://github.com/Onely7/local_data_studio/commit/0f0b33221dc52e1bcc05ce0641c5bcf0220379fd))
+* **ui:** jump to any row and page through search results ([3dd9ff9](https://github.com/Onely7/local_data_studio/commit/3dd9ff9cdbdec0bcc8b2386767982c3a00a3f739))
+* **ui:** show row totals and index far jumps with progress ([7b5389d](https://github.com/Onely7/local_data_studio/commit/7b5389d1a33a56f4f9dfe6c00a57d15577bd19c7))
+* support Python 3.14 ([7678336](https://github.com/Onely7/local_data_studio/commit/7678336d2df53ba67b331a7a69e61f585b132ded))
+
+
+### Bug Fixes
+
+* **atlas:** sample large datasets instead of loading every row ([6ba0907](https://github.com/Onely7/local_data_studio/commit/6ba0907499ac1967249ea95c89aefd09c53b8e67))
+* **csv:** keep quoted line breaks inside one row ([fbc8f48](https://github.com/Onely7/local_data_studio/commit/fbc8f483a9bfb18b2f64af65e735ffd318767df7))
+* **readers:** scan small line files to the end instead of rejecting jumps ([ad346ef](https://github.com/Onely7/local_data_studio/commit/ad346efa6a64d5cf450e7fb310bcf5a967cda07f))
+* **ui:** keep pagination and panel buttons readable on narrow screens ([23f4ca2](https://github.com/Onely7/local_data_studio/commit/23f4ca28fa34acba254478eff6fc7397b99b8285))
+
+
+### Performance Improvements
+
+* **preview:** convert text-only pages without the thread pool ([0a2bb3c](https://github.com/Onely7/local_data_studio/commit/0a2bb3c983b02063ec64754ec1d71eeb063d1ae7))
+* **preview:** send image thumbnails instead of full image bytes ([05eabcd](https://github.com/Onely7/local_data_studio/commit/05eabcdac895c87fd8a7dbaee9f0547183c70ab6))
+* **readers:** read pages, jumps, and visible rows without rereading files ([be8e3ec](https://github.com/Onely7/local_data_studio/commit/be8e3ec7d4e99958218f444fdae46a7cdecb89ed))
+* **readers:** scan, jump, and search large datasets in bounded time ([fef4c6f](https://github.com/Onely7/local_data_studio/commit/fef4c6fcc6c195c3ada2bcfaa23926b3973bcd70))
+
+
+### Dependencies
+
+* **deps:** bump astral-sh/setup-uv in the github-actions group ([#53](https://github.com/Onely7/local_data_studio/issues/53)) ([96d74c5](https://github.com/Onely7/local_data_studio/commit/96d74c55310d95aed8b054027cf4065659b5f146))
+* remove the unused polars dependency ([d163e9e](https://github.com/Onely7/local_data_studio/commit/d163e9ec890197aa8ef7df09fba55f4f13704257))
+* replace ydata-profiling with fg-data-profiling ([ab39eca](https://github.com/Onely7/local_data_studio/commit/ab39eca11bcbb77d3023629037d38e9f0924dd68))
+* upgrade numpy, scipy, matplotlib and numba ([c4c4411](https://github.com/Onely7/local_data_studio/commit/c4c44118a9136208c15608ac6631f2bb9cbdbd83))
+* upgrade pandas to 2.3.3 ([a0d645e](https://github.com/Onely7/local_data_studio/commit/a0d645e05d1622c5764441b61ee9971aab573f87))
+
+
+### Documentation
+
+* describe audio playback, transcription, and audio embeddings ([9bcb80d](https://github.com/Onely7/local_data_studio/commit/9bcb80da8d8ee6339f3a1a0cf7963a199eafe683))
+* describe background indexing and large-file reading ([96d9924](https://github.com/Onely7/local_data_studio/commit/96d9924b68f1e549f65e97761ea840c195a2b266))
+* describe large-dataset navigation, background SQL, and EmbeddingGemma 2 ([1cde4c2](https://github.com/Onely7/local_data_studio/commit/1cde4c26bb83d2f6a262b74c708787767cdca3ac))
+
+
+### Build System
+
+* **deps:** upgrade dependencies and resolve security advisories ([1b40f87](https://github.com/Onely7/local_data_studio/commit/1b40f87cabf8837102e20375c5d134ffb9f59537))
+
+## [1.1.1](https://github.com/Onely7/local_data_studio/compare/v1.1.0...v1.1.1) (2026-07-23)
+
+
+### Bug Fixes
+
+* preserve compatibility in dependency updates ([caf5beb](https://github.com/Onely7/local_data_studio/commit/caf5bebafbb3da47432b27bef5caf78304328f76))
+
+
+### Dependencies
+
+* **deps:** bump https://github.com/astral-sh/ty-pre-commit ([9fa50ca](https://github.com/Onely7/local_data_studio/commit/9fa50caefd27154addf2cfc9d71afd7851242ba4))
+* **deps:** bump the github-actions group with 2 updates ([6faf4ab](https://github.com/Onely7/local_data_studio/commit/6faf4ab65668734ae0ce5c247a50aeaef19e29cf))
+* **deps:** bump the python-dependencies group with 4 updates ([2e5a599](https://github.com/Onely7/local_data_studio/commit/2e5a59918bb4c8666b3caf11107007298174f66c))
+* **deps:** bump the python-dependencies group with 9 updates ([c7a47e5](https://github.com/Onely7/local_data_studio/commit/c7a47e50df31b5285fc669a4fe8a2a2bfceb4904))
+
+## [1.1.0](https://github.com/Onely7/local_data_studio/compare/v1.0.0...v1.1.0) (2026-07-15)
+
+### Translation
+
+- Added manual translation for a cell or the currently visible values in a
+  Preview column. Translation-enabled LiteLLM profiles are selected separately
+  from SQL-generation profiles, and source datasets are never modified.
+  ([3b5d102](https://github.com/Onely7/local_data_studio/commit/3b5d102500f8a250c0c5ca60525f403cca0e7b59),
+  [5537318](https://github.com/Onely7/local_data_studio/commit/5537318fae385a527230250ed073031b8a2614fa))
+- Added 68 target languages and independent defaults for the translation model
+  and target language. The target language can be set in
+  `local_data_studio.toml`, with browser preferences used as fallbacks.
+  ([943033d](https://github.com/Onely7/local_data_studio/commit/943033ddd0e93f21162db6ebbb03de13ef7c704f))
+- Added bounded, cancellable background translation jobs. Server-side limits
+  cover rows, strings, characters, chunk size, and provider concurrency;
+  malformed provider responses are validated and retried once.
+  ([3b5d102](https://github.com/Onely7/local_data_studio/commit/3b5d102500f8a250c0c5ca60525f403cca0e7b59))
+- Preserved nested list and object structure while translating natural-language
+  strings. Numeric, Boolean, binary, identifier, URL, image, and audio values
+  are excluded, and completed translations remain only in browser memory.
+  ([5537318](https://github.com/Onely7/local_data_studio/commit/5537318fae385a527230250ed073031b8a2614fa))
+
+### Data Analysis
+
+- Applied `EDA_ROW_LIMIT` to the dataset source before pandas DataFrame
+  materialization when no rows are hidden in the current session. This avoids
+  loading an entire large JSONL dataset before selecting the requested EDA
+  rows. Unlimited mode (`-1`) continues to load all selected rows.
+  ([7d799ed](https://github.com/Onely7/local_data_studio/commit/7d799ed1e99b9d969f2c6677ab7388d3889646f1))
+
+### Interface
+
+- Added translation controls and copy actions to expanded values and JSON code
+  views, with shared JSON syntax coloring for structured values.
+- Refined the dataset selector, search and pagination controls, responsive
+  one-column ordering, mobile scrolling, and compact desktop toolbar so the
+  Preview keeps as much space as possible.
+- Updated the visual theme, bundled action icons, application branding, and
+  screenshots while retaining the existing dataset, SQL, EDA, and Atlas
+  workflows.
+  ([0b4f5fd](https://github.com/Onely7/local_data_studio/commit/0b4f5fd1565d65105e6c147e365d7f05224e27ad),
+  [40944f7](https://github.com/Onely7/local_data_studio/commit/40944f739aef7ee86b546efcada9441b573b0092),
+  [cfa2b54](https://github.com/Onely7/local_data_studio/commit/cfa2b54b8d5edc46efe4a1eaea353a07f6fb972b),
+  [f9690c5](https://github.com/Onely7/local_data_studio/commit/f9690c51a918f31d6cbb07eeaeae4a7aeb08bcae))
+
+### Release And Documentation
+
+- Added Release Please automation for version updates, changelog maintenance,
+  release pull requests, tags, and GitHub Releases.
+  ([4a70f10](https://github.com/Onely7/local_data_studio/commit/4a70f105308545e1cd9789b5972f1ff085cd86b0))
+- Updated the English and Japanese user guides, implementation notes,
+  configuration example, and screenshots for translation, EDA loading, and the
+  refined workspace.
+  ([f23e800](https://github.com/Onely7/local_data_studio/commit/f23e800722f16cf00ff758761afa6f20467092d7),
+  [0a6e32c](https://github.com/Onely7/local_data_studio/commit/0a6e32c942cc7ee19e08bb27ae96def1b44a0ea),
+  [b53a213](https://github.com/Onely7/local_data_studio/commit/b53a213fa35ce8981c85e8278532c2d400878fe8))
+
+## [1.0.0](https://github.com/Onely7/local_data_studio/releases/tag/v1.0.0) (2026-07-13)
+
+- Published the first stable Local Data Studio package to PyPI.

local_data_studio-1.1.0/README.md → local_data_studio-2.0.0/PKG-INFO

@@ -1,3 +1,49 @@
+Metadata-Version: 2.5
+Name: local-data-studio
+Version: 2.0.0
+Summary: Local data studio for JSONL/JSON/CSV/TSV/Parquet with scalable previews, image inspection, DuckDB SQL, EDA, and Embedding Atlas
+Project-URL: Homepage, https://github.com/Onely7/local_data_studio
+Project-URL: Repository, https://github.com/Onely7/local_data_studio
+Project-URL: Issues, https://github.com/Onely7/local_data_studio/issues
+Author: Daiki Shiono
+License-Expression: MIT
+License-File: LICENSE
+Keywords: data-exploration,data-viewer,dataset-viewer,duckdb,eda,embedding-atlas,image-datasets,jsonl,parquet
+Classifier: Development Status :: 3 - Alpha
+Classifier: Framework :: FastAPI
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Requires-Python: <3.15,>=3.11
+Requires-Dist: av>=18.1.0
+Requires-Dist: duckdb<2.0.0,>=1.4.0
+Requires-Dist: embedding-atlas<0.21.0,>=0.20.0
+Requires-Dist: fastapi<1,>=0.139
+Requires-Dist: fg-data-profiling<5,>=4.20.0
+Requires-Dist: httpx<1,>=0.28
+Requires-Dist: litellm<2,>=1.100.4
+Requires-Dist: pandas<3,>=2.3.3
+Requires-Dist: pillow>=13.0.0
+Requires-Dist: pyarrow>=24.0.0
+Requires-Dist: pydantic-settings<3,>=2.6
+Requires-Dist: pydantic<3,>=2.10
+Requires-Dist: python-dotenv
+Requires-Dist: python-multipart
+Requires-Dist: qwen-vl-utils>=0.0.14
+Requires-Dist: requests<3,>=2.33
+Requires-Dist: scikit-learn<2,>=1.7.0
+Requires-Dist: sentence-transformers>=6.2.0
+Requires-Dist: starlette<2,>=1.3.1
+Requires-Dist: torchvision>=0.27.1
+Requires-Dist: transformers>=5.19.0
+Requires-Dist: uvicorn>=0.50.0
+Description-Content-Type: text/markdown
+
 <div align="center">
 
 # Local Data Studio <img src="images/local_data_studio_icon.png" alt="Local Data Studio Icon" align="right" width="150" style="margin-left:50px;"/>
@@ -25,21 +71,24 @@ Click the Local Data Studio logo in the top bar to open this GitHub repository i
 
 ## Key Features
 
-* Preview large datasets while limiting how much data is loaded
+* Preview terabyte-scale datasets quickly: each page reads only the rows it shows
+* Index rows in the background while you browse, so totals, far jumps, and later pages are ready sooner
 * Run read-only searches and aggregations with DuckDB SQL
 * Apply SQL timeouts, memory limits, and warnings when a query may read a large amount of data
 * Generate EDA reports for an entire dataset or SQL results
-* Visualize text columns, image columns, or SQL results with Embedding Atlas
+* Visualize text, image, or audio columns, or SQL results, with Embedding Atlas
 * Inspect complete row contents with the **Row Inspector**
 * Display images from URLs, local file paths, and values stored in `{bytes, path}` format
 * Open enlarged image previews and switch between multiple images in the same row
+* Play audio stored as bytes, Hugging Face `Audio` values, data URLs, or file paths
+* Transcribe audio with a configured speech-recognition model such as Qwen3-ASR or Whisper
 * Manually translate visible cells or columns with configured LiteLLM-compatible models
 * Upload files by dragging and dropping them into the application
 * Hide rows while the application page is open and, when needed, delete them from the source file
 
 ## Supported Environments and Data Formats
 
-* Python 3.11, 3.12, or 3.13
+* Python 3.11, 3.12, 3.13, or 3.14
 * Supported formats: `.jsonl`, `.json`, `.csv`, `.tsv`, and `.parquet`
 
 ## Quick Start
@@ -102,7 +151,17 @@ Use the controls at the top of the page to change what is displayed:
 
 * **Search**: Searches the data.
 * **Rows**: Changes the number of rows shown per page.
-* **Prev** / **Next**: Moves to the previous or next page.
+* **Go to row**: Enter a row number and press `Enter` to jump directly to that row.
+  When the row lies past the indexed part of a large JSONL, CSV, TSV, or JSON file, the rows up to it are indexed first.
+  The progress appears above the table, with a **Cancel** button.
+* **Prev** / **Next**: Moves to the previous or next page of the data or of the search results.
+
+The number at the left of each row is its position in the dataset, so it stays the same after jumps, searches, and hidden rows.
+
+The line above the table shows the number of columns and the rows on screen, followed by the total number of rows.
+Parquet totals are known immediately.
+For JSONL, CSV, TSV, and JSON files, the total is an estimate, such as `of about 2.4B`, until the background row index reaches the end of the file.
+Image cells show small thumbnails; select an image to view it at its original resolution.
 
 <img src="images/local_data_studio_03.png" alt="Searching data and moving between pages" width="45%">
 
@@ -116,6 +175,10 @@ If the server has a LiteLLM model connection setting, called a model profile, yo
 You can choose from configured OpenAI, Anthropic, Gemini, hosted vLLM, and other LiteLLM-compatible models.
 
 SQL execution is subject to timeouts, memory limits, and checks for queries that may read a large amount of data.
+
+On datasets larger than 256 MiB, queries that aggregate, sort, deduplicate, join, or use window functions scan the whole dataset in the background.
+While such a query runs, its progress is shown and **Run Query** changes to **Cancel Query**.
+Up to 1,000,000 result rows are cached, so later pages, **Show raw row**, and EDA or Atlas on the query results reuse them without scanning again.
 For restrictions on generated SQL, see [LLM Model Profiles for SQL Generation and Translation](#llm-model-profiles-for-sql-generation-and-translation).
 
 <img src="images/local_data_studio_04.png" alt="SQL console" width="45%">
@@ -141,7 +204,7 @@ The default value is `minimal`.
 
 ### 5. Visualize Embeddings
 
-An embedding represents the features of text or an image as a sequence of numbers called a vector.
+An embedding represents the features of text, an image, or audio as a sequence of numbers called a vector.
 Embeddings usually contain many numbers, which makes them difficult to compare directly on a screen.
 Local Data Studio uses methods such as UMAP, t-SNE, and PCA to convert embeddings into two-dimensional coordinates and displays them with Embedding Atlas.
 This conversion is called **dimensionality reduction**.
@@ -149,13 +212,14 @@ Some configuration keys and internal code use the term `projection`, but this RE
 
 To use this feature, you must provide a local encoder model in Hugging Face format.
 Hugging Face format means that the model's configuration files and related data are stored in the expected directory structure.
-An encoder model converts text or images into embeddings.
+An encoder model converts text, images, or audio into embeddings.
 
 Place the model under `models/embedder`, or under the directory specified by `--models-dir` or `EMBEDDER_MODELS_DIR`.
 
 Example locations:
 
 ```text
+models/embedder/google/embeddinggemma-2
 models/embedder/google/siglip2-base-patch16-224
 models/embedder/Qwen/Qwen3-Embedding-0.6B
 models/embedder/Qwen/Qwen3-VL-Embedding-2B
@@ -165,7 +229,7 @@ A directory appears in the **Model** selector when it contains model-identifying
 
 Under **Visualize Embedding**, select the following:
 
-1. A text or image column
+1. A text, image, or audio column
 2. The model to use
 3. The library that loads the model and computes embeddings, referred to as the backend
 4. The dimensionality-reduction method used to convert embeddings into two dimensions
@@ -197,6 +261,26 @@ If the field is left empty, the default prompt stored with the model is used.
 
 </details>
 
+<details>
+<summary><strong>Using EmbeddingGemma 2</strong></summary>
+
+[EmbeddingGemma 2](https://huggingface.co/google/embeddinggemma-2) embeds text, images, and audio into the same 768-dimensional space.
+The model is not gated, so you can download it without a Hugging Face token:
+
+```bash
+hf download google/embeddinggemma-2 --local-dir models/embedder/google/embeddinggemma-2
+```
+
+Select `google/embeddinggemma-2` as the **Model**.
+Sentence Transformers embeds text, image, and audio columns, and Transformers embeds text columns.
+The model expects a task prefix on text, so for visualization enter `task: clustering | query: ` in **Prompt**.
+Leave **Prompt** empty for image columns, because images are embedded without a prefix.
+Audio is also embedded without a prefix, so a prompt is ignored for audio columns.
+Audio is resampled to 16 kHz, and clips longer than 300 seconds are embedded from their start.
+In Atlas, the audio column shows each clip's file name and duration.
+
+</details>
+
 <img src="images/local_data_studio_07.png" alt="Embedding Atlas settings" width="45%"> <img src="images/local_data_studio_08.png" alt="Embedding Atlas visualization" width="45%">
 
 ### 6. Use the Row Inspector and Enlarged Image View
@@ -219,7 +303,56 @@ If that fails, it uses `path` as a fallback.
 
 <img src="images/local_data_studio_09.png" alt="Row Inspector" width="45%"> <img src="images/local_data_studio_10.png" alt="Enlarged image preview" width="45%">
 
-### 7. Translate Visible Values
+### 7. Play and Transcribe Audio
+
+Local Data Studio recognizes WAV, FLAC, MP3, OGG (including Opus), AAC, M4A, and AIFF audio stored in these forms:
+
+* Bytes in a binary column
+* Dictionaries (objects) in `{ "bytes": ..., "path": ... }` format, such as Hugging Face `Audio` values
+* `data:audio/...` URLs
+* Audio URLs and relative or absolute file paths
+
+In the table, an audio value appears as a play button with its duration, format, and size.
+WAV and FLAC show their duration before playback; other formats show it once they have been played.
+An expanded field shows a full player with a seek bar and volume control.
+Only one clip plays at a time, and playback continues while the table updates.
+
+Preview pages do not carry audio bytes.
+Audio is loaded only when you play it or expand its row, so pages of large clips stay light.
+Relative paths are resolved next to the dataset file first and then below the data directory and `file_serve_roots`; files outside those directories are not served.
+The browser loads audio URLs directly.
+
+To transcribe audio, add a speech-recognition model to the `[transcription]` section of `local_data_studio.toml` and select it in **Transcriber** in the toolbar.
+For example, download Qwen3-ASR 1.7B and configure it as follows:
+
+```bash
+hf download Qwen/Qwen3-ASR-1.7B-hf --local-dir models/asr/Qwen/Qwen3-ASR-1.7B-hf
+```
+
+```toml
+[transcription]
+default_model = "qwen3-asr-1.7b"
+
+[[transcription.models]]
+id = "qwen3-asr-1.7b"
+label = "Qwen3-ASR 1.7B"
+model = "models/asr/Qwen/Qwen3-ASR-1.7B-hf"
+```
+
+Use the `-hf` repository, because Transformers cannot load the original `Qwen/Qwen3-ASR-1.7B` checkpoint.
+`model` can also be a Hugging Face model ID such as `openai/whisper-large-v3-turbo`; any model that the Transformers speech-recognition pipeline loads works.
+Add `language = "ja"` to a model to fix the language instead of detecting it, or `device = "cpu"` or `device = "cuda:1"` to choose where it runs.
+
+The transcription icon in a column header transcribes the audio that is visible in that column; select it again to cancel.
+The **Transcribe** button in an expanded field transcribes that field's audio.
+Each transcript appears below its audio with the detected language and a copy icon.
+
+The model loads on first use, which can take a minute, and is released after `idle_unload_seconds` (600 seconds by default).
+Qwen3-ASR 1.7B uses about 4 GB of GPU memory.
+Audio longer than `chunk_seconds` is split at quiet points, and clips longer than `max_audio_seconds` are rejected.
+Transcripts are cached under the cache directory by audio content and model, so the same clip is not transcribed twice.
+
+### 8. Translate Visible Values
 
 To use translation, set `translation = true` in an LLM model profile.
 Then select a model and target language from the toolbar at the top of the page.
@@ -296,6 +429,7 @@ The main sections are as follows:
 * `[server]`: Server settings
 * `[settings]`: EDA, Embedding Atlas, source-file deletion, and related settings
 * `[llm]`: LLM model profiles used for SQL generation and manual translation
+* `[transcription]`: Speech-recognition models and limits for audio transcription
 * `[translation]`: Initial target language and translation request limits
 
 For example, configure the data location and the maximum number of rows loaded for EDA as follows:
@@ -388,7 +522,7 @@ The environment variables below can also be specified in the TOML `[settings]` s
 * `EMBEDDER_MODELS_DIR`: The parent directory containing local Hugging Face encoder models. By default, Local Data Studio uses `models/embedder` under the workspace or current working directory.
 * `ATLAS_HOST`, `ATLAS_PORT`: The IPv4 loopback host used by an Embedding Atlas child process, meaning a separate process started by Local Data Studio, and the first port number to check when selecting an internal port. A loopback host refers only to the same computer. `localhost` is normalized to `127.0.0.1`. IPv6 and externally reachable hosts are not allowed. Browsers do not connect to this port directly.
 * `ATLAS_MAX_INSTANCES`: The maximum number of pending and running Atlas child processes combined. The default is `4`, and the value must be at least `1`.
-* `ATLAS_SAMPLE`: The maximum number of rows used for embedding computation and dimensionality reduction and stored in the cached Atlas Parquet file. Sampling is applied after SQL, with a fixed random seed of 42 so that the same rows are selected for the same input. When unset or set to `0`, all selected rows are used. Negative values are rejected.
+* `ATLAS_SAMPLE`: The maximum number of rows used for embedding computation and dimensionality reduction and stored in the cached Atlas Parquet file. Sampling is applied after SQL, with a fixed random seed of 42 so that the same rows are selected for the same input. When unset or set to `0`, all selected rows are used, except that datasets larger than 256 MiB use a sample of 100,000 rows. Negative values are rejected.
 * `ATLAS_BATCH_SIZE`: The number of rows processed at once during embedding computation. When unset or set to `0`, the Embedding Atlas default is used.
 * `ATLAS_CACHE_MAX_BYTES`: The maximum total size of Embedding Atlas-related files stored in `./cache/atlas`. When the limit is exceeded, the oldest cache files are removed first.
 * `ATLAS_TEXT_MAX_CHARS`: The maximum number of characters retained from a text cell for both embedding input and the cached Atlas Parquet file. Set this to `0` to disable truncation.
@@ -399,6 +533,14 @@ The environment variables below can also be specified in the TOML `[settings]` s
 
 </details>
 
+<details>
+<summary><strong>Large Dataset Settings</strong></summary>
+
+* `BACKGROUND_INDEX`: When `true` (the default), opening a JSONL, CSV, TSV, or JSON file indexes its rows in the background, so the total row count, far jumps, and later pages are available without waiting for a full scan. The scan pauses while pages and rows are being loaded, and it stops when you open another file; its progress is kept and resumed later. Set it to `false` to index rows only when you select **Count Rows** or jump past the indexed rows. The `--background-index` and `--no-background-index` command-line options override this setting.
+* `BACKGROUND_INDEX_MAX_BYTES`: How many bytes of each file the background index reads. The default is 16 GiB (`17179869184`). Files up to this size are indexed completely, which also gives their exact row count. Set it to `0` to index whole files of any size.
+
+</details>
+
 <details>
 <summary><strong>Data Deletion Settings</strong></summary>
 
@@ -413,7 +555,7 @@ The environment variables below can also be specified in the TOML `[settings]` s
 For development or source-code changes, you can run Local Data Studio from the repository.
 The following software is required:
 
-* Python 3.11, 3.12, or 3.13
+* Python 3.11, 3.12, 3.13, or 3.14
 * Git, which is used to download the source code from GitHub
 * uv, which creates the Python environment and installs the required libraries
 
@@ -512,7 +654,8 @@ Atlas pages are served through Local Data Studio, so you do not need to forward
 * Searching and generating EDA reports may take time for large datasets.
 * Row counting, full-dataset search, sample statistics, and EDA run as background jobs that support progress reporting and cancellation.
 * Setting `EDA_ROW_LIMIT=-1` loads all selected rows into memory for analysis. Use this only when the complete dataset fits comfortably in memory.
-* JSON array files at terabyte (TB) scale are not recommended. Use JSONL or Parquet to browse very large datasets more efficiently.
+* JSON array files are previewed page by page, but counting, indexing, and searching them is several times slower than for JSONL. Use JSONL or Parquet for very large datasets.
+* **Count Rows** indexes a JSONL, CSV, TSV, or JSON file to its end. Jumps past the indexed rows index the rows up to the target automatically.
 * t-SNE can become dramatically slower and consume much more memory as the amount of data increases. Set a practical `ATLAS_SAMPLE` limit for large datasets.
 
 > [!WARNING]
@@ -530,9 +673,26 @@ You can skip it if you only need the standard browsing and analysis features.
 <details>
 <summary><strong>Large-Dataset Preview and Background Processing</strong></summary>
 
-For supported formats, previews of very large datasets use cursor-based `page_token` values instead of large `OFFSET` values.
-`OFFSET` skips a specified number of rows from the beginning, and larger values may require more work to pass over earlier rows.
-A `page_token` records the current position and is used to move efficiently between the previous and next pages.
+Previews never read a dataset from its beginning to reach a page.
+
+* JSONL, CSV, TSV, and JSON pages continue from the byte position recorded in a `page_token`.
+  A sparse row index, with one checkpoint every 10,000 rows, lets **Go to row** start reading near the requested row.
+  The background index, **Count Rows**, and jumps past the index extend one shared scan from where the last scan stopped.
+  Searches and jumps also record the checkpoints they pass.
+  Without nearby checkpoints, one jump may skip at most 1,000,000 rows unless less than 256 MiB remains to be read; farther jumps index the rows up to the target first.
+* `.json` files that hold one JSON value per line are read like JSONL. JSON arrays are read element by element, so the whole document is never loaded.
+* CSV and TSV files whose quoted fields contain line breaks are read record by record, following RFC 4180 quoting.
+  If quotes in a file are unbalanced, rows are split at every line break and a warning is shown.
+* Parquet pages are read by physical row number using the row-group metadata, so any row is available immediately.
+  The file's metadata is read once and kept in memory, and only the data pages a page needs are read.
+  The next page continues from where the previous page stopped, without decoding its row group again.
+* Each page reads at most 32 MiB of source data, so pages of very large rows contain fewer rows.
+* Rows shown on the current page are kept in memory, up to 64 MiB, so the **Row Inspector**, enlarged images, and audio playback open them without reading the file again.
+* Searches resume after the previous page's last match instead of starting over.
+
+Line scans read 16 MiB blocks and locate rows with vectorized byte searches on several threads.
+Counting rows and searching therefore run close to disk speed.
+Jumps and single-row reads start with a 256 KiB read and grow it only when needed.
 
 Row counting, full-dataset search, sample statistics, and EDA run as background jobs that support progress reporting and cancellation.
 
@@ -622,7 +782,7 @@ Values converted into encoder input format are stored only in an internal embedd
 
 When `ATLAS_SAMPLE=N` is specified, DuckDB selects rows deterministically after SQL filtering and before creating a pandas DataFrame.
 As a result, at most `N` rows are materialized for embedding computation, dimensionality reduction, and the cached Atlas Parquet file.
-When `ATLAS_SAMPLE=0`, all selected rows are processed.
+When `ATLAS_SAMPLE=0`, all selected rows are processed, except that datasets larger than 256 MiB use a sample of 100,000 rows.
 
 Text and prompt templates are expanded only for the current embedding batch.
 Image `bytes` values are written incrementally to a temporary disk-backed work area, and only the data required for the current batch is loaded into memory.
@@ -674,6 +834,8 @@ Image and multimodal models require a native `modules.json` to run through Sente
 
 As a result, image-only DINOv3 checkpoints can use only the Transformers backend and rely on the model-declared `pooler_output`.
 Qwen3-VL-Embedding models with a native Sentence Transformers pipeline can use both backends.
+EmbeddingGemma 2 declares text, image, audio, and video inputs.
+Sentence Transformers embeds its text and image columns, while the tokenizer-based Transformers adapter is offered for text only.
 
 Only backends for which an executable adapter can be confirmed are selectable.
 `remote_code` is available only when `ATLAS_TRUST_REMOTE_CODE=true` explicitly permits execution of code from the model repository.
@@ -746,7 +908,7 @@ Resolve all errors before committing.
 ## Acknowledgements
 
 * [Dataset Viewer by Hugging Face](https://github.com/huggingface/dataset-viewer): Used as a reference for the UI and feature design.
-* [YData Profiling](https://github.com/ydataai/ydata-profiling): Used to generate EDA reports.
+* [fg-data-profiling](https://github.com/Data-Centric-AI-Community/fg-data-profiling) (formerly YData Profiling): Used to generate EDA reports.
 * [Embedding Atlas](https://github.com/apple/embedding-atlas): Used for interactive embedding visualization.
 
 ## License