cocoindex 0.1.76__tar.gz → 0.1.78__tar.gz

{cocoindex-0.1.76 → cocoindex-0.1.78}/Cargo.lock

@@ -1297,7 +1297,7 @@ dependencies = [
 
 [[package]]
 name = "cocoindex"
-version = "0.1.76"
+version = "0.1.78"
 dependencies = [
  "anyhow",
  "async-openai",

{cocoindex-0.1.76 → cocoindex-0.1.78}/Cargo.toml

@@ -2,7 +2,7 @@
 name = "cocoindex"
 # Version used for local development is always higher than others to take precedence.
 # Will be overridden for specific release versions.
-version = "0.1.76"
+version = "0.1.78"
 edition = "2024"
 rust-version = "1.88"
 

{cocoindex-0.1.76 → cocoindex-0.1.78}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cocoindex
-Version: 0.1.76
+Version: 0.1.78
 Requires-Dist: click>=8.1.8
 Requires-Dist: rich>=14.0.0
 Requires-Dist: python-dotenv>=1.1.0
@@ -214,6 +214,7 @@ It defines an index flow like this:
 | [Image Search with Vision API](examples/image_search) | Generates detailed captions for images using a vision model, embeds them, enables live-updating semantic search via FastAPI and served on a React frontend|
 | [Face Recognition](examples/face_recognition) | Recognize faces in images and build embedding index |
 | [Paper Metadata](examples/paper_metadata) | Index papers in PDF files, and build metadata tables for each paper |
+| [Multi Format Indexing](examples/multi_format_indexing) | Build visual document index from PDFs and images with ColPali for semantic search |
 | [Custom Output Files](examples/custom_output_files) | Convert markdown files to HTML files and save them to a local directory, using *CocoIndex Custom Targets* |
 
 More coming and stay tuned 👀!

{cocoindex-0.1.76 → cocoindex-0.1.78}/README.md

@@ -185,6 +185,7 @@ It defines an index flow like this:
 | [Image Search with Vision API](examples/image_search) | Generates detailed captions for images using a vision model, embeds them, enables live-updating semantic search via FastAPI and served on a React frontend|
 | [Face Recognition](examples/face_recognition) | Recognize faces in images and build embedding index |
 | [Paper Metadata](examples/paper_metadata) | Index papers in PDF files, and build metadata tables for each paper |
+| [Multi Format Indexing](examples/multi_format_indexing) | Build visual document index from PDFs and images with ColPali for semantic search |
 | [Custom Output Files](examples/custom_output_files) | Convert markdown files to HTML files and save them to a local directory, using *CocoIndex Custom Targets* |
 
 More coming and stay tuned 👀!

{cocoindex-0.1.76 → cocoindex-0.1.78}/docs/docs/core/data_types.mdx

@@ -199,3 +199,23 @@ Currently, the following types are key types
 - *Uuid*
 - *Date*
 - *Struct* with all fields being key types (using `@dataclass(frozen=True)` or `NamedTuple`)
+
+## *Null* Values
+
+CocoIndex supports *Null* values. A *Null* value represents the absence of data or an unknown value, distinct from empty strings, zero numbers, or false boolean values.
+
+### Nullable Type
+
+For any data (e.g. a field of a *Struct*, an argument or return value of a CocoIndex function), if it is nullable, it means its value can be *Null*.
+We use a `?` suffix to indicate a nullable type, e.g. *Str?*, *Person?*.
+
+In Python, *Null* is represented as `None`, so a nullable type can be represented by `T | None` or `typing.Optional[T]`.
+
+### *Null* propagating on CocoIndex functions
+
+A function may specify whether each input argument is nullable or not.
+Non-nullable argument means the function needs a known value for the argument to work.
+However, it doesn't forbid the argument to be *Null* at runtime.
+When a non-nullable argument receives a *Null* value, the function execution is skipped and the result is *Null*.
+
+For example, for [`SplitRecursively` function](/docs/ops/functions#splitrecursively), the `text` and `chunk_size` arguments are not nullable. If the input value of either of them is *Null*, the function will return *Null*.

{cocoindex-0.1.76 → cocoindex-0.1.78}/docs/docs/core/flow_def.mdx

@@ -360,21 +360,40 @@ It will use `Staging__doc_embeddings` as the collection name if the current app
 
 ### Control Processing Concurrency
 
-You can control the concurrency of the processing by setting the following options:
+CocoIndex processes data in parallel to maximize throughput, but unconstrained parallelism can overwhelm your system.
+Processing too many items simultaneously can lead to:
 
-*   `max_inflight_rows`: the maximum number of concurrent inflight requests for the processing.
-*   `max_inflight_bytes`: the maximum number of concurrent inflight bytes for the processing.
+- **Memory exhaustion**: Large datasets loaded concurrently can consume excessive RAM
+- **Resource contention**: Too many parallel operations competing for CPU, disk I/O, or network bandwidth
+- **System instability**: High concurrency can cause timeouts, crashes, or degraded performance
 
-These options can be passed in to the following APIs:
+To prevent these issues, CocoIndex provides concurrency controls that limit how many data items are processed simultaneously.
 
-*   [`FlowBuilder.add_source()`](#import-from-source): The options above control the processing concurrency of multiple rows from a source. New rows will not be loaded in memory if it'll be over the limit.
+#### Concurrency Options
 
-    Besides, global limits on overall processing concurrency of all sources from all flows can be specified by [`GlobalExecutionOptions`](/docs/core/settings#globalexecutionoptions) or corresponding [environment variables](/docs/core/settings#list-of-environment-variables).
-    If both global and per-source limits are specified, both need to be satisfied to admit additional source rows.
+You can control processing concurrency using these options:
 
-*   [`DataSlice.row()`](#for-each-row):  The options above provides a finer-grained control, to limit the processing concurrency of multiple rows within a table at any level.
+*   `max_inflight_rows`: Limits the maximum number of data rows being processed concurrently
+*   `max_inflight_bytes`: Limits the total memory footprint of data being processed concurrently (measured in bytes)
 
-`max_inflight_bytes` only counts the number of bytes already existing in the current row before any further processing.
+When these limits are reached, CocoIndex will pause loading new data until some of the current processing completes, ensuring your system remains stable.
+
+#### Where to Apply Concurrency Controls
+
+These concurrency options can be configured at different levels:
+
+*   **Source level** via [`FlowBuilder.add_source()`](#import-from-source): Controls how many rows from a data source are processed simultaneously. This prevents overwhelming your system when ingesting large datasets.
+
+    You can also set global limits across all sources and flows using [`GlobalExecutionOptions`](/docs/core/settings#globalexecutionoptions) or environment variables [`COCOINDEX_SOURCE_MAX_INFLIGHT_ROWS`](/docs/core/settings#list-of-environment-variables)/[`COCOINDEX_SOURCE_MAX_INFLIGHT_BYTES`](/docs/core/settings#list-of-environment-variables).
+    When both global and per-source limits are specified, both limits are enforced independently - a new row can only be processed if there's available capacity in both the global budget (shared across all sources) and the per-source budget (specific to that source).
+
+*   **Row iteration level** via [`DataSlice.row()`](#for-each-row): Provides fine-grained control over parallel processing within nested data structures, allowing you to tune concurrency at any level of your data hierarchy.
+
+:::note
+
+The `max_inflight_bytes` limit only counts the size of data that already exists in memory before any transformations are applied. It doesn't include the memory used by intermediate processing results.
+
+:::
 
 For example:
 

{cocoindex-0.1.76 → cocoindex-0.1.78}/docs/docs/core/settings.mdx

@@ -77,24 +77,33 @@ If not set, all flows are in a default unnamed namespace.
 
 `DatabaseConnectionSpec` configures the connection to a database. Only Postgres is supported for now. It has the following fields:
 
-*   `url` (type: `str`, required): The URL of the Postgres database to use as the internal storage, e.g. `postgres://cocoindex:cocoindex@localhost/cocoindex`.
+*   `url` (type: `str`): The URL of the Postgres database to use as the internal storage, e.g. `postgres://cocoindex:cocoindex@localhost/cocoindex`.
 
     *Environment variable* for `Settings.database.url`: `COCOINDEX_DATABASE_URL`
 
-*   `user` (type: `str`, optional): The username for the Postgres database. If not provided, username will come from `url`.
+*   `user` (type: `str | None`, default: `None`): The username for the Postgres database. If not provided, username will come from `url`.
 
     *Environment variable* for `Settings.database.user`: `COCOINDEX_DATABASE_USER`
 
-*   `password` (type: `str`, optional): The password for the Postgres database. If not provided, password will come from `url`.
+*   `password` (type: `str | None`, default: `None`): The password for the Postgres database. If not provided, password will come from `url`.
 
     *Environment variable* for `Settings.database.password`: `COCOINDEX_DATABASE_PASSWORD`
 
-:::tip
+    :::tip
 
-Please be careful that all values in `url` needs to be url-encoded if they contain special characters.
-For this reason, prefer to use the separated `user` and `password` fields for username and password.
+    Please be careful that all values in `url` needs to be url-encoded if they contain special characters.
+    For this reason, prefer to use the separated `user` and `password` fields for username and password.
+
+    :::
+
+*   `max_connections` (type: `int`, default: `64`): The maximum number of connections to keep in the pool.
+
+    *Environment variable* for `Settings.database.max_connections`: `COCOINDEX_DATABASE_MAX_CONNECTIONS`
+
+*   `min_connections` (type: `int`, default: `16`): The minimum number of connections to keep in the pool.
+
+    *Environment variable* for `Settings.database.min_connections`: `COCOINDEX_DATABASE_MIN_CONNECTIONS`
 
-:::
 
 :::info
 
@@ -109,10 +118,10 @@ If you use the Postgres database hosted by [Supabase](https://supabase.com/), pl
 
 `GlobalExecutionOptions` is used to configure the global execution options shared by all flows. It has the following fields:
 
-*   `source_max_inflight_rows` (type: `int`, optional): The maximum number of concurrent inflight requests for all source operations.
-*   `source_max_inflight_bytes` (type: `int`, optional): The maximum number of concurrent inflight bytes for all source operations.
+*   `source_max_inflight_rows` (type: `int | None`, default: `256`): The maximum number of concurrent inflight rows for all source operations.
+*   `source_max_inflight_bytes` (type: `int | None`, default: `None`): The maximum number of concurrent inflight bytes for all source operations.
 
-See also [flow definition docs](/docs/core/flow_def#control-processing-concurrency) to control processing concurrency on per-source basis.
+See also [flow definition docs](/docs/core/flow_def#control-processing-concurrency) about why it's necessary to control processing concurrency, and how to configure it on per-source basis.
 If both global and per-source limits are specified, both need to be satisfied to admit additional source rows.
 
 ## List of Environment Variables
@@ -125,5 +134,7 @@ This is the list of environment variables, each of which has a corresponding fie
 | `COCOINDEX_DATABASE_URL` | `database.url` | Yes |
 | `COCOINDEX_DATABASE_USER` | `database.user` | No |
 | `COCOINDEX_DATABASE_PASSWORD` | `database.password` | No |
-| `COCOINDEX_SOURCE_MAX_INFLIGHT_ROWS` | `global_execution_options.source_max_inflight_rows` | No |
+| `COCOINDEX_DATABASE_MAX_CONNECTIONS` | `database.max_connections` | No (default: `64`) |
+| `COCOINDEX_DATABASE_MIN_CONNECTIONS` | `database.min_connections` | No (default: `16`) |
+| `COCOINDEX_SOURCE_MAX_INFLIGHT_ROWS` | `global_execution_options.source_max_inflight_rows` | No (default: `256`) |
 | `COCOINDEX_SOURCE_MAX_INFLIGHT_BYTES` | `global_execution_options.source_max_inflight_bytes` | No |

{cocoindex-0.1.76 → cocoindex-0.1.78}/docs/docs/ops/functions.md

@@ -9,12 +9,12 @@ description: CocoIndex Built-in Functions
 
 `ParseJson` parses a given text to JSON.
 
-The spec takes the following fields:
+Input data:
 
-*   `text` (`str`): The source text to parse.
-*   `language` (`str`, optional): The language of the source text.  Only `json` is supported now.  Default to `json`.
+*   `text` (*Str*): The source text to parse.
+*   `language` (*Str?*, default: `"json"`): The language of the source text.  Only `json` is supported now.
 
-Return: *Json*
+Return: *Json*, the parsed JSON object.
 
 ## SplitRecursively
 
@@ -37,7 +37,7 @@ Input data:
 
 *   `text` (*Str*): The text to split.
 *   `chunk_size` (*Int64*): The maximum size of each chunk, in bytes.
-*   `min_chunk_size` (*Int64*, optional): The minimum size of each chunk, in bytes. If not provided, default to `chunk_size / 2`.
+*   `min_chunk_size` (*Int64*, default: `chunk_size / 2`): The minimum size of each chunk, in bytes.
 
     :::note
 
@@ -48,8 +48,8 @@ Input data:
 
     :::
 
-*   `chunk_overlap` (*Int64*, optional): The maximum overlap size between adjacent chunks, in bytes.
-*   `language` (*Str*, optional): The language of the document.
+*   `chunk_overlap` (*Int64?*, default: *Null*): The maximum overlap size between adjacent chunks, in bytes.
+*   `language` (*Str*, default: `""`): The language of the document.
     Can be a language name (e.g. `Python`, `Javascript`, `Markdown`) or a file extension (e.g. `.py`, `.js`, `.md`).
 
 
@@ -61,7 +61,7 @@ Input data:
         *   `custom_languages` in the spec, against the `language_name` or `aliases` field of each entry.
         *   Builtin languages (see [Supported Languages](#supported-languages) section below), against the language, aliases or file extensions of each entry.
 
-        All matches are in a case-insensitive manner. If the value of `language` is null, it'll be treated as empty string.
+        All matches are in a case-insensitive manner.
 
     *   If no match is found, the input will be treated as plain text.
 
@@ -185,7 +185,7 @@ Not all LLM APIs support text embedding. See the [LLM API Types table](/docs/ai/
 
 Input data:
 
-*   `text` (*Str*, required): The text to embed.
+*   `text` (*Str*): The text to embed.
 
 Return: *Vector[Float32, N]*, where *N* is the dimension of the embedding vector determined by the model.
 

{cocoindex-0.1.76 → cocoindex-0.1.78}/examples/image_search/pyproject.toml

@@ -4,7 +4,7 @@ version = "0.1.0"
 description = "Image search examples for cocoindex: CLIP and ColPali-based embedding."
 requires-python = ">=3.11"
 dependencies = [
-    "cocoindex[colpali]>=0.1.75",
+    "cocoindex[colpali]>=0.1.76",
     "python-dotenv>=1.0.1",
     "fastapi>=0.100.0",
     "torch>=2.0.0",

cocoindex-0.1.78/examples/multi_format_indexing/README.md

@@ -0,0 +1,71 @@
+# Build visual document index from PDFs and images with ColPali
+[![GitHub](https://img.shields.io/github/stars/cocoindex-io/cocoindex?color=5B5BD6)](https://github.com/cocoindex-io/cocoindex)
+
+
+In this example, we build a visual document indexing flow using ColPali for embedding PDFs and images. and query the index with natural language.
+
+We appreciate a star ⭐ at [CocoIndex Github](https://github.com/cocoindex-io/cocoindex) if this is helpful.
+
+## Steps
+### Indexing Flow
+
+1. We ingest a list of PDF files and image files from the `source_files` directory.
+2. For each file:
+   - **PDF files**: convert each page to a high-resolution image (300 DPI)
+   - **Image files**: use the image directly
+   - Generate visual embeddings for each page/image using ColPali model
+3. We will save the embeddings and metadata in Qdrant vector database.
+
+### Query
+We will match against user-provided natural language text using ColPali's text-to-visual embedding capability, enabling semantic search across visual document content.
+
+
+
+## Prerequisite
+[Install Qdrant](https://qdrant.tech/documentation/guides/installation/) if you don't have one running locally.
+
+You can start Qdrant with Docker:
+```bash
+docker run -p 6333:6333 -p 6334:6334 qdrant/qdrant
+```
+
+## Run
+
+Install dependencies:
+
+```bash
+pip install -e .
+```
+
+Setup:
+
+```bash
+cocoindex setup main.py
+```
+
+Update index:
+
+```bash
+cocoindex update main.py
+```
+
+Run:
+
+```bash
+python main.py
+```
+
+## About ColPali
+This example uses [ColPali](https://github.com/illuin-tech/colpali), a state-of-the-art vision-language model that enables:
+- Direct visual understanding of document layouts, tables, and figures
+- Natural language queries against visual document content
+- No need for OCR or text extraction - works directly with document images
+
+## CocoInsight
+I used CocoInsight (Free beta now) to troubleshoot the index generation and understand the data lineage of the pipeline. It just connects to your local CocoIndex server, with Zero pipeline data retention. Run following command to start CocoInsight:
+
+```
+cocoindex server -ci main.py
+```
+
+Then open the CocoInsight UI at [https://cocoindex.io/cocoinsight](https://cocoindex.io/cocoinsight).

cocoindex-0.1.78/examples/multi_format_indexing/main.py

@@ -0,0 +1,135 @@
+import cocoindex
+import os
+import mimetypes
+
+from dotenv import load_dotenv
+from dataclasses import dataclass
+from pdf2image import convert_from_bytes
+from io import BytesIO
+
+from qdrant_client import QdrantClient
+
+QDRANT_GRPC_URL = "http://localhost:6334"
+QDRANT_COLLECTION = "MultiFormatIndexings"
+COLPALI_MODEL_NAME = os.getenv("COLPALI_MODEL", "vidore/colpali-v1.2")
+
+
+@dataclass
+class Page:
+    page_number: int | None
+    image: bytes
+
+
+@cocoindex.op.function()
+def file_to_pages(filename: str, content: bytes) -> list[Page]:
+    """
+    Classify file content based on MIME type detection.
+    Returns ClassifiedFileContent with appropriate field populated based on file type.
+    """
+    # Guess the MIME type based on the filename
+    mime_type, _ = mimetypes.guess_type(filename)
+
+    if mime_type == "application/pdf":
+        images = convert_from_bytes(content, dpi=300)
+        pages = []
+        for i, image in enumerate(images):
+            with BytesIO() as buffer:
+                image.save(buffer, format="PNG")
+                pages.append(Page(page_number=i + 1, image=buffer.getvalue()))
+        return pages
+    elif mime_type and mime_type.startswith("image/"):
+        return [Page(page_number=None, image=content)]
+    else:
+        return []
+
+
+qdrant_connection = cocoindex.add_auth_entry(
+    "qdrant_connection",
+    cocoindex.targets.QdrantConnection(grpc_url=QDRANT_GRPC_URL),
+)
+
+
+@cocoindex.flow_def(name="MultiFormatIndexing")
+def multi_format_indexing_flow(
+    flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope
+) -> None:
+    """
+    Define an example flow that embeds files into a vector database.
+    """
+    data_scope["documents"] = flow_builder.add_source(
+        cocoindex.sources.LocalFile(path="source_files", binary=True)
+    )
+
+    output_embeddings = data_scope.add_collector()
+
+    with data_scope["documents"].row() as doc:
+        doc["pages"] = flow_builder.transform(
+            file_to_pages, filename=doc["filename"], content=doc["content"]
+        )
+        with doc["pages"].row() as page:
+            page["embedding"] = page["image"].transform(
+                cocoindex.functions.ColPaliEmbedImage(model=COLPALI_MODEL_NAME)
+            )
+            output_embeddings.collect(
+                id=cocoindex.GeneratedField.UUID,
+                filename=doc["filename"],
+                page=page["page_number"],
+                embedding=page["embedding"],
+            )
+
+    output_embeddings.export(
+        "multi_format_indexings",
+        cocoindex.targets.Qdrant(
+            connection=qdrant_connection,
+            collection_name=QDRANT_COLLECTION,
+        ),
+        primary_key_fields=["id"],
+    )
+
+
+@cocoindex.transform_flow()
+def query_to_colpali_embedding(
+    text: cocoindex.DataSlice[str],
+) -> cocoindex.DataSlice[list[list[float]]]:
+    return text.transform(
+        cocoindex.functions.ColPaliEmbedQuery(model=COLPALI_MODEL_NAME)
+    )
+
+
+def _main() -> None:
+    # Initialize Qdrant client
+    client = QdrantClient(url=QDRANT_GRPC_URL, prefer_grpc=True)
+
+    # Run queries in a loop to demonstrate the query capabilities.
+    while True:
+        query = input("Enter search query (or Enter to quit): ")
+        if query == "":
+            break
+
+        # Get the embedding for the query
+        query_embedding = query_to_colpali_embedding.eval(query)
+
+        search_results = client.query_points(
+            collection_name=QDRANT_COLLECTION,
+            query=query_embedding,  # Multi-vector format: list[list[float]]
+            using="embedding",  # Specify the vector field name
+            limit=5,
+            with_payload=True,
+        )
+        print("\nSearch results:")
+        for result in search_results.points:
+            score = result.score
+            payload = result.payload
+            if payload is None:
+                continue
+            page_number = payload["page"]
+            page_number_str = f"Page:{page_number}" if page_number is not None else ""
+            print(f"[{score:.3f}] {payload['filename']} {page_number_str}")
+            print("---")
+        print()
+
+
+if __name__ == "__main__":
+    load_dotenv()
+    cocoindex.init()
+    _main()

cocoindex-0.1.78/examples/multi_format_indexing/pyproject.toml

@@ -0,0 +1,14 @@
+[project]
+name = "pdf-embedding"
+version = "0.1.0"
+description = "Simple example for cocoindex: build embedding index based on local PDF files."
+requires-python = ">=3.11"
+dependencies = [
+    "cocoindex[colpali]>=0.1.76",
+    "python-dotenv>=1.0.1",
+    "pdf2image>=1.17.0",
+    "qdrant-client>=1.15.0",
+]
+
+[tool.setuptools]
+packages = []

cocoindex-0.1.78/examples/text_embedding_qdrant/.env

@@ -0,0 +1,2 @@
+# Postgres database address for cocoindex
+COCOINDEX_DATABASE_URL=postgres://cocoindex:cocoindex@localhost/cocoindex