cocoindex 0.2.10__tar.gz → 0.2.12__tar.gz

{cocoindex-0.2.10 → cocoindex-0.2.12}/.github/workflows/_docs_release.yml

@@ -3,6 +3,10 @@ name: Release Docs (INTERNAL)
 on:
   workflow_call:
 
+permissions:
+  contents: read
+  pages: write
+
 jobs:
   deploy:
     runs-on: ubuntu-latest

{cocoindex-0.2.10 → cocoindex-0.2.12}/.github/workflows/_test.yml

@@ -3,6 +3,9 @@ name: Run Tests (INTERNAL)
 on:
     workflow_call:
 
+permissions:
+  contents: read
+
 env:
   CARGO_TERM_COLOR: always
 

{cocoindex-0.2.10 → cocoindex-0.2.12}/.github/workflows/docs_release.yml

@@ -5,6 +5,7 @@ on:
 
 permissions:
   contents: write
+  pages: write
 
 jobs:
   deploy:

{cocoindex-0.2.10 → cocoindex-0.2.12}/.github/workflows/docs_test.yml

@@ -7,6 +7,9 @@ on:
       - docs/**
       - ".github/workflows/docs_test.yml"
 
+permissions:
+  contents: read
+
 jobs:
   test-deploy:
     runs-on: ubuntu-latest

{cocoindex-0.2.10 → cocoindex-0.2.12}/.github/workflows/release.yml

@@ -12,6 +12,7 @@ on:
 
 permissions:
   contents: read
+  pages: write
 
 jobs:
   create-versioned-toml:

{cocoindex-0.2.10 → cocoindex-0.2.12}/.gitignore

@@ -21,3 +21,6 @@ examples/**/eval_*
 
 /.vscode
 /*.session.sql
+
+# mypy daemon environment
+.dmypy.json

{cocoindex-0.2.10 → cocoindex-0.2.12}/.pre-commit-config.yaml

@@ -49,7 +49,7 @@ repos:
 
         - id: mypy-check
           name: mypy type check
-          entry: mypy --cache-fine-grain
+          entry: dmypy run python
           language: system
           types: [python]
           pass_filenames: false

{cocoindex-0.2.10 → cocoindex-0.2.12}/Cargo.lock

@@ -1283,7 +1283,7 @@ dependencies = [
 
 [[package]]
 name = "cocoindex"
-version = "0.2.10"
+version = "0.2.12"
 dependencies = [
  "anyhow",
  "async-openai",

{cocoindex-0.2.10 → cocoindex-0.2.12}/Cargo.toml

@@ -2,9 +2,9 @@
 name = "cocoindex"
 # Version used for local development is always higher than others to take precedence.
 # Will be overridden for specific release versions.
-version = "0.2.10"
+version = "0.2.12"
 edition = "2024"
-rust-version = "1.88"
+rust-version = "1.89"
 license = "Apache-2.0"
 readme = "README.md"
 

{cocoindex-0.2.10 → cocoindex-0.2.12}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cocoindex
-Version: 0.2.10
+Version: 0.2.12
 Classifier: Development Status :: 3 - Alpha
 Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Operating System :: OS Independent
@@ -29,11 +29,14 @@ Requires-Dist: mypy ; extra == 'dev'
 Requires-Dist: pre-commit ; extra == 'dev'
 Requires-Dist: sentence-transformers>=3.3.1 ; extra == 'embeddings'
 Requires-Dist: colpali-engine ; extra == 'colpali'
+Requires-Dist: lancedb>=0.25.0 ; extra == 'lancedb'
 Requires-Dist: sentence-transformers>=3.3.1 ; extra == 'all'
 Requires-Dist: colpali-engine ; extra == 'all'
+Requires-Dist: lancedb>=0.25.0 ; extra == 'all'
 Provides-Extra: dev
 Provides-Extra: embeddings
 Provides-Extra: colpali
+Provides-Extra: lancedb
 Provides-Extra: all
 License-File: THIRD_PARTY_NOTICES.html
 Summary: With CocoIndex, users declare the transformation, CocoIndex creates & maintains an index, and keeps the derived index up to date based on source update, with minimal computation and changes.

{cocoindex-0.2.10 → cocoindex-0.2.12}/THIRD_PARTY_NOTICES.html

@@ -2428,7 +2428,7 @@ Software.
                 <h3 id="Apache-2.0">Apache License 2.0</h3>
                 <h4>Used by:</h4>
                 <ul class="license-used-by">
-                    <li><a href=" https://crates.io/crates/cocoindex ">cocoindex 0.2.10</a></li>
+                    <li><a href=" https://crates.io/crates/cocoindex ">cocoindex 0.2.12</a></li>
                     <li><a href=" https://github.com/awesomized/crc-fast-rust ">crc-fast 1.3.0</a></li>
                     <li><a href=" https://github.com/qdrant/rust-client ">qdrant-client 1.15.0</a></li>
                 </ul>

{cocoindex-0.2.10 → cocoindex-0.2.12}/docs/docs/contributing/guide.md

@@ -5,7 +5,7 @@ description: How to contribute to CocoIndex
 
 [CocoIndex](https://github.com/cocoindex-io/cocoindex) is an open source project. We are respectful, open and friendly. This guide explains how to get involved and contribute to [CocoIndex](https://github.com/cocoindex-io/cocoindex).
 
-Our [Discord server](https://discord.com/invite/zpA9S2DR7s) is constantly open. 
+Our [Discord server](https://discord.com/invite/zpA9S2DR7s) is constantly open.
 If you are unsure about anything, it is a good place to discuss! We'd love to collaborate and will always be friendly.
 
 ## Good First Issues

{cocoindex-0.2.10 → cocoindex-0.2.12}/docs/docs/contributing/setup_dev_environment.md

@@ -44,4 +44,4 @@ Follow the steps below to get CocoIndex built on the latest codebase locally - i
 -   Before running a specific example, set extra environment variables, for exposing extra traces, allowing dev UI, etc.
     ```sh
     . ./.env.lib_debug
-    ```
+    ```

{cocoindex-0.2.10 → cocoindex-0.2.12}/docs/docs/examples/examples/academic_papers_index.md

@@ -7,6 +7,7 @@ canonicalUrl: '/examples/academic_papers_index'
 sidebar_custom_props:
   image: /img/examples/academic_papers_index/cover.png
   tags: [vector-index, metadata]
+image: /img/examples/academic_papers_index/cover.png
 tags: [vector-index, metadata]
 ---
 
@@ -20,10 +21,10 @@ import { GitHubButton, YouTubeButton, DocumentationButton } from '../../../src/c
 
 1. Extract the paper metadata, including file name, title, author information, abstract, and number of pages.
 
-2. Build vector embeddings for the metadata, such as the title and abstract, for semantic search. 
+2. Build vector embeddings for the metadata, such as the title and abstract, for semantic search.
 This enables better metadata-driven semantic search results. For example, you can match text queries against titles and abstracts.
 
-3. Build an index of authors and all the file names associated with each author 
+3. Build an index of authors and all the file names associated with each author
 to answer questions like "Give me all the papers by Jeff Dean."
 
 4. If you want to perform full PDF embedding for the paper, you can extend the flow.
@@ -107,7 +108,7 @@ After this step, we should have the basic info of each paper.
 
 We will convert the first page to Markdown using Marker. Alternatively, you can easily plug in any PDF parser, such as Docling using CocoIndex's [custom function](https://cocoindex.io/docs/custom_ops/custom_functions).
 
-Define a marker converter function and cache it, since its initialization is resource-intensive. 
+Define a marker converter function and cache it, since its initialization is resource-intensive.
 This ensures that the same converter instance is reused for different input files.
 
 ```python
@@ -136,7 +137,7 @@ def pdf_to_markdown(content: bytes) -> str:
 Pass it to your transform
 
 ```python
-with data_scope["documents"].row() as doc:    
+with data_scope["documents"].row() as doc:
     # ... process
     doc["first_page_md"] = doc["basic_info"]["first_page"].transform(
             pdf_to_markdown
@@ -199,7 +200,7 @@ paper_metadata.collect(
 Just collect anything you need :)
 
 ### Collect `author` to `filename` information
-We’ve already extracted author list. Here we want to collect Author → Papers in a separate table to build a look up functionality. 
+We’ve already extracted author list. Here we want to collect Author → Papers in a separate table to build a look up functionality.
 Simply collect by author.
 
 ```python
@@ -228,8 +229,8 @@ doc["title_embedding"] = doc["metadata"]["title"].transform(
 
 ### Abstract
 
-Split abstract into chunks, embed each chunk and collect their embeddings. 
-Sometimes the abstract could be very long. 
+Split abstract into chunks, embed each chunk and collect their embeddings.
+Sometimes the abstract could be very long.
 
 ```python
 doc["abstract_chunks"] = doc["metadata"]["abstract"].transform(
@@ -307,7 +308,7 @@ author_papers.export(
     "author_papers",
     cocoindex.targets.Postgres(),
     primary_key_fields=["author_name", "filename"],
-)    
+)
 metadata_embeddings.export(
     "metadata_embeddings",
     cocoindex.targets.Postgres(),
@@ -327,9 +328,9 @@ In this example we use PGVector as embedding store. With CocoIndex, you can do o
 
 ## Query the index
 
-You can refer to this section of [Text Embeddings](https://cocoindex.io/blogs/text-embeddings-101#3-query-the-index) about 
-how to build query against embeddings. 
-For now CocoIndex doesn't provide additional query interface. We can write SQL or rely on the query engine by the target storage. 
+You can refer to this section of [Text Embeddings](https://cocoindex.io/blogs/text-embeddings-101#3-query-the-index) about
+how to build query against embeddings.
+For now CocoIndex doesn't provide additional query interface. We can write SQL or rely on the query engine by the target storage.
 
 - Many databases already have optimized query implementations with their own best practices
 - The query space has excellent solutions for querying, reranking, and other search-related functionality.

{cocoindex-0.2.10 → cocoindex-0.2.12}/docs/docs/examples/examples/codebase_index.md

@@ -7,6 +7,7 @@ canonicalUrl: '/examples/code_index'
 sidebar_custom_props:
   image: /img/examples/codebase_index/cover.png
   tags: [vector-index, codebase]
+image: /img/examples/codebase_index/cover.png
 tags: [vector-index, codebase]
 ---
 
@@ -18,7 +19,7 @@ import { GitHubButton, YouTubeButton, DocumentationButton } from '../../../src/c
 ![Codebase Index](/img/examples/codebase_index/cover.png)
 
 ## Overview
-In this tutorial, we will build codebase index. [CocoIndex](https://github.com/cocoindex-io/cocoindex) provides built-in support for codebase chunking, with native Tree-sitter support. It works with large codebases, and can be updated in near real-time with incremental processing - only reprocess what's changed. 
+In this tutorial, we will build codebase index. [CocoIndex](https://github.com/cocoindex-io/cocoindex) provides built-in support for codebase chunking, with native Tree-sitter support. It works with large codebases, and can be updated in near real-time with incremental processing - only reprocess what's changed.
 
 ## Use Cases
 A wide range of applications can be built with an effective codebase index that is always up-to-date.
@@ -43,14 +44,14 @@ The flow is composed of the following steps:
 - Generate embeddings for each chunk
 - Store in a vector database for retrieval
 
-## Setup 
+## Setup
 - Install Postgres, follow [installation guide](https://cocoindex.io/docs/getting_started/installation#-install-postgres).
 - Install CocoIndex
   ```bash
   pip install -U cocoindex
   ```
 
-## Add the codebase as a source. 
+## Add the codebase as a source.
 We will index the CocoIndex codebase. Here we use the `LocalFile` source to ingest files from the CocoIndex codebase root directory.
 
 ```python
@@ -66,7 +67,7 @@ def code_embedding_flow(flow_builder: cocoindex.FlowBuilder, data_scope: cocoind
 - Include files with the extensions of `.py`, `.rs`, `.toml`, `.md`, `.mdx`
 - Exclude files and directories starting `.`,  `target` in the root and `node_modules` under any directory.
 
-`flow_builder.add_source` will create a table with sub fields (`filename`, `content`). 
+`flow_builder.add_source` will create a table with sub fields (`filename`, `content`).
 <DocumentationButton url="https://cocoindex.io/docs/ops/sources" text="Sources" />
 
 
@@ -95,14 +96,14 @@ with data_scope["files"].row() as file:
     file["extension"] = file["filename"].transform(extract_extension)
     file["chunks"] = file["content"].transform(
           cocoindex.functions.SplitRecursively(),
-          language=file["extension"], chunk_size=1000, chunk_overlap=300) 
+          language=file["extension"], chunk_size=1000, chunk_overlap=300)
 ```
 <DocumentationButton url="https://cocoindex.io/docs/ops/functions#splitrecursively" text="SplitRecursively" margin="0 0 16px 0" />
 
 ![SplitRecursively](/img/examples/codebase_index/chunk.png)
 
 ### Embed the chunks
-We use `SentenceTransformerEmbed` to embed the chunks. 
+We use `SentenceTransformerEmbed` to embed the chunks.
 
 ```python
 @cocoindex.transform_flow()
@@ -140,7 +141,7 @@ code_embeddings.export(
     vector_indexes=[cocoindex.VectorIndex("embedding", cocoindex.VectorSimilarityMetric.COSINE_SIMILARITY)])
 ```
 
-We use [Cosine Similarity](https://en.wikipedia.org/wiki/Cosine_similarity) to measure the similarity between the query and the indexed data. 
+We use [Cosine Similarity](https://en.wikipedia.org/wiki/Cosine_similarity) to measure the similarity between the query and the indexed data.
 
 ## Query the index
 We match against user-provided text by a SQL query, reusing the embedding operation in the indexing flow.
@@ -229,4 +230,4 @@ Follow the url from the terminal - `https://cocoindex.io/cocoinsight` to access
 
 SplitRecursively has native support for all major programming languages.
 
-<DocumentationButton url="https://cocoindex.io/docs/ops/functions#supported-languages" text="Supported Languages" margin="0 0 16px 0" />
+<DocumentationButton url="https://cocoindex.io/docs/ops/functions#supported-languages" text="Supported Languages" margin="0 0 16px 0" />

{cocoindex-0.2.10 → cocoindex-0.2.12}/docs/docs/examples/examples/custom_targets.md

@@ -7,6 +7,7 @@ canonicalUrl: '/examples/custom_targets'
 sidebar_custom_props:
   image: /img/examples/custom_targets/cover.png
   tags: [custom-building-blocks]
+image: /img/examples/custom_targets/cover.png
 tags: [custom-building-blocks]
 ---
 import { GitHubButton, YouTubeButton, DocumentationButton } from '../../../src/components/GitHubButton';
@@ -34,7 +35,7 @@ flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope
 		refresh_interval=timedelta(seconds=5),
 	)
 ```
-This ingestion creates a table with `filename` and `content` fields. 
+This ingestion creates a table with `filename` and `content` fields.
 <DocumentationButton url="https://cocoindex.io/docs/ops/sources" text="Sources" />
 
 ## Process each file and collect
@@ -91,7 +92,7 @@ class LocalFileTargetConnector:
 
 ```
 
-The `describe()` method returns a human-readable string that describes the target, which is displayed in the CLI logs. 
+The `describe()` method returns a human-readable string that describes the target, which is displayed in the CLI logs.
 For example, it prints:
 
 `Target: Local directory ./data/output`
@@ -103,10 +104,10 @@ def describe(key: str) -> str:
     return f"Local directory {key}"
 ```
 
-`apply_setup_change()` applies setup changes to the backend. The previous and current specs are passed as arguments, 
+`apply_setup_change()` applies setup changes to the backend. The previous and current specs are passed as arguments,
 and the method is expected to update the backend setup to match the current state.
 
-A `None` spec indicates non-existence, so when `previous` is `None`, we need to create it, 
+A `None` spec indicates non-existence, so when `previous` is `None`, we need to create it,
 and when `current` is `None`, we need to delete it.
 
 
@@ -134,8 +135,8 @@ def apply_setup_change(
             os.rmdir(previous.directory)
 ```
 
-The `mutate()` method is called by CocoIndex to apply data changes to the target, 
-batching mutations to potentially multiple targets of the same type. 
+The `mutate()` method is called by CocoIndex to apply data changes to the target,
+batching mutations to potentially multiple targets of the same type.
 This allows the target connector flexibility in implementation (e.g., atomic commits, or processing items with dependencies in a specific order).
 
 Each element in the batch corresponds to a specific target and is represented by a tuple containing:
@@ -150,8 +151,8 @@ class LocalFileTargetValues:
     html: str
 ```
 
-The value type of the `dict` is `LocalFileTargetValues | None`, 
-where a non-`None` value means an upsert and `None` value means a delete. Similar to `apply_setup_changes()`, 
+The value type of the `dict` is `LocalFileTargetValues | None`,
+where a non-`None` value means an upsert and `None` value means a delete. Similar to `apply_setup_changes()`,
 idempotency is expected here.
 
 ```python
@@ -216,7 +217,5 @@ This keeps your knowledge graph continuously synchronized with your document sou
 Sometimes there may be an internal/homegrown tool or API (e.g. within a company) that's not publicly available.
 These can only be connected through custom targets.
 
-### Faster adoption of new export logic 
+### Faster adoption of new export logic
 When a new tool, database, or API joins your stack, simply define a Target Spec and Target Connector — start exporting right away, with no pipeline refactoring required.
-
-

{cocoindex-0.2.10 → cocoindex-0.2.12}/docs/docs/examples/examples/docs_to_knowledge_graph.md

@@ -7,6 +7,7 @@ canonicalUrl: '/examples/knowledge-graph-for-docs'
 sidebar_custom_props:
   image: /img/examples/docs_to_knowledge_graph/cover.png
   tags: [knowledge-graph, structured-data-extraction]
+image: /img/examples/docs_to_knowledge_graph/cover.png
 tags: [knowledge-graph, structured-data-extraction]
 ---
 
@@ -35,7 +36,7 @@ and then build a knowledge graph.
 - CocoIndex can direct map the collected data to Neo4j nodes and relationships.
 
 ## Setup
-*   [Install PostgreSQL](https://cocoindex.io/docs/getting_started/installation#-install-postgres). CocoIndex uses PostgreSQL internally for incremental processing. 
+*   [Install PostgreSQL](https://cocoindex.io/docs/getting_started/installation#-install-postgres). CocoIndex uses PostgreSQL internally for incremental processing.
 *   [Install Neo4j](https://cocoindex.io/docs/ops/targets#neo4j-dev-instance), a graph database.
 *   [Configure your OpenAI API key](https://cocoindex.io/docs/ai/llm#openai).  Alternatively, we have native support for Gemini, Ollama, LiteLLM. You can choose your favorite LLM provider and work completely on-premises.
 
@@ -50,7 +51,7 @@ and then build a knowledge graph.
 
 ### Add documents as source
 
-We will process CocoIndex documentation markdown files (`.md`, `.mdx`) from the `docs/core` directory ([markdown files](https://github.com/cocoindex-io/cocoindex/tree/main/docs/docs/core), [deployed docs](https://cocoindex.io/docs/core/basics)). 
+We will process CocoIndex documentation markdown files (`.md`, `.mdx`) from the `docs/core` directory ([markdown files](https://github.com/cocoindex-io/cocoindex/tree/main/docs/docs/core), [deployed docs](https://cocoindex.io/docs/core/basics)).
 
 ```python
 @cocoindex.flow_def(name="DocsToKG")
@@ -140,7 +141,7 @@ Next, we will use `cocoindex.functions.ExtractByLlm` to extract the relationship
 doc["relationships"] = doc["content"].transform(
     cocoindex.functions.ExtractByLlm(
         llm_spec=cocoindex.LlmSpec(
-            api_type=cocoindex.LlmApiType.OPENAI, 
+            api_type=cocoindex.LlmApiType.OPENAI,
             model="gpt-4o"
         ),
         output_type=list[Relationship],
@@ -186,7 +187,7 @@ with doc["relationships"].row() as relationship:
 
 
 ### Build knowledge graph
- 
+
 #### Basic concepts
 All nodes for Neo4j need two things:
 1. Label: The type of the node. E.g., `Document`, `Entity`.
@@ -235,10 +236,10 @@ This exports Neo4j nodes with label `Document` from the `document_node` collecto
 
 #### Export `RELATIONSHIP` and `Entity` nodes to Neo4j
 
-We don't have explicit collector for `Entity` nodes. 
+We don't have explicit collector for `Entity` nodes.
 They are part of the `entity_relationship` collector and fields are collected during the relationship extraction.
 
-To export them as Neo4j nodes, we need to first declare `Entity` nodes. 
+To export them as Neo4j nodes, we need to first declare `Entity` nodes.
 
 ```python
 flow_builder.declare(
@@ -288,7 +289,7 @@ In a relationship, there's:
 2.  A relationship connecting the source and target.
 Note that different relationships may share the same source and target nodes.
 
-`NodeFromFields` takes the fields from the `entity_relationship` collector and creates `Entity` nodes. 
+`NodeFromFields` takes the fields from the `entity_relationship` collector and creates `Entity` nodes.
 
 #### Export the `entity_mention` to Neo4j.
 
@@ -333,7 +334,7 @@ It creates relationships by:
     ```sh
     cocoindex update --setup main.py
     ```
-    
+
     You'll see the index updates state in the terminal. For example,
 
     ```
@@ -342,7 +343,7 @@ It creates relationships by:
 
 ## CocoInsight
 
-I used CocoInsight to troubleshoot the index generation and understand the data lineage of the pipeline.  It is in free beta now, you can give it a try. 
+I used CocoInsight to troubleshoot the index generation and understand the data lineage of the pipeline.  It is in free beta now, you can give it a try.
 
 ```sh
 cocoindex server -ci main
@@ -368,7 +369,7 @@ MATCH p=()-->() RETURN p
 ## Kuzu
 Cocoindex natively supports Kuzu - a high performant, embedded open source graph database.
 
-<DocumentationButton url="https://cocoindex.io/docs/ops/targets#kuzu" text="Kuzu" margin="0 0 16px 0" /> 
+<DocumentationButton url="https://cocoindex.io/docs/ops/targets#kuzu" text="Kuzu" margin="0 0 16px 0" />
 
 The GraphDB interface in CocoIndex is standardized, you just need to **switch the configuration** without any additional code changes. CocoIndex supports exporting to Kuzu through its API server. You can bring up a Kuzu API server locally by running:
 
@@ -390,4 +391,3 @@ kuzu_conn_spec = cocoindex.add_auth_entry(
 ```
 
 <GitHubButton url="https://github.com/cocoindex-io/cocoindex/blob/30761f8ab674903d742c8ab2e18d4c588df6d46f/examples/docs_to_knowledge_graph/main.py#L33-L37"  margin="0 0 16px 0" />
-

{cocoindex-0.2.10 → cocoindex-0.2.12}/docs/docs/examples/examples/document_ai.md

@@ -7,6 +7,7 @@ canonicalUrl: '/examples/document_ai'
 sidebar_custom_props:
   image: /img/examples/document_ai/cover.png
   tags: [vector-index, custom-building-block]
+image: /img/examples/document_ai/cover.png
 tags: [vector-index, custom-building-block]
 ---
 import { GitHubButton, DocumentationButton, ExampleButton } from '../../../src/components/GitHubButton';
@@ -20,7 +21,7 @@ CocoIndex is a flexible ETL framework with incremental processing.  We don’t b
 
 ## Set up
 - [Install Postgres](https://cocoindex.io/docs/getting_started/installation#-install-postgres) if you don't have one.
-- Configure Project and Processor ID for Document AI API 
+- Configure Project and Processor ID for Document AI API
     - [Official Google document AI API](https://cloud.google.com/document-ai/docs/try-docai) with free live demo.
     - Sign in to [Google Cloud Console](https://console.cloud.google.com/), create or open a project, and enable Document AI API.
       - ![image.png](/img/examples/document_ai/document_ai.png)

{cocoindex-0.2.10 → cocoindex-0.2.12}/docs/docs/examples/examples/image_search.md

@@ -7,6 +7,7 @@ canonicalUrl: '/examples/image_search'
 sidebar_custom_props:
   image: /img/examples/image_search/cover.png
   tags: [vector-index, multi-modal]
+image: /img/examples/image_search/cover.png
 tags: [vector-index, multi-modal]
 ---
 
@@ -20,7 +21,7 @@ import { GitHubButton, YouTubeButton, DocumentationButton } from '../../../src/c
 CocoIndex supports native integration with ColPali - with just a few lines of code, you embed and index images with ColPali’s late-interaction architecture. We also build a light weight image search application with FastAPI.
 
 
-## ColPali 
+## ColPali
 
 **ColPali (Contextual Late-interaction over Patches)** is a powerful model for multimodal retrieval.
 

{cocoindex-0.2.10 → cocoindex-0.2.12}/docs/docs/examples/examples/manual_extraction.md

@@ -7,6 +7,7 @@ canonicalUrl: '/examples/manual_extraction'
 sidebar_custom_props:
   image: /img/examples/manual_extraction/cover.png
   tags: [structured-data-extraction, data-mapping]
+image: /img/examples/manual_extraction/cover.png
 tags: [structured-data-extraction, data-mapping]
 ---
 
@@ -187,7 +188,7 @@ def summarize_module(module_info: ModuleInfo) -> ModuleSummary:
         num_classes=len(module_info.classes),
         num_methods=len(module_info.methods),
     )
-``` 
+```
 
 ### Plug in the function into the flow
 ```python
@@ -248,4 +249,3 @@ SELECT filename, module_info->'title' AS title, module_summary FROM modules_info
 cocoindex server -ci main
 ```
 CocoInsight dashboard is here `https://cocoindex.io/cocoinsight`.  It connects to your local CocoIndex server with zero data retention.
-

{cocoindex-0.2.10 → cocoindex-0.2.12}/docs/docs/examples/examples/multi_format_index.md

@@ -1,5 +1,5 @@
 ---
-title: Index PDFs, Images, Slides without OCR 
+title: Index PDFs, Images, Slides without OCR
 description: Build a visual document indexing pipeline using ColPali to index scanned documents, PDFs, academic papers, presentation slides, and standalone images — all mixed together with charts, tables, and figures - into the same vector space.
 sidebar_class_name: hidden
 slug: /examples/multi_format_index
@@ -7,6 +7,7 @@ canonicalUrl: '/examples/multi_format_index'
 sidebar_custom_props:
   image: /img/examples/multi_format_index/cover.png
   tags: [vector-index, multi-modal]
+image: /img/examples/multi_format_index/cover.png
 tags: [vector-index, multi-modal]
 ---
 
@@ -19,7 +20,7 @@ import { GitHubButton, YouTubeButton, DocumentationButton } from '../../../src/c
 ## Overview
 Do you have a messy collection of scanned documents, PDFs, academic papers, presentation slides, and standalone images — all mixed together with charts, tables, and figures — that you want to process into the same vector space for semantic search or to power an AI agent?
 
-In this example, we’ll walk through how to build a visual document indexing pipeline using ColPali for embedding both PDFs and images — and then query the index using natural language.  
+In this example, we’ll walk through how to build a visual document indexing pipeline using ColPali for embedding both PDFs and images — and then query the index using natural language.
 
 We’ll skip OCR entirely — ColPali can directly understand document layouts, tables, and figures from images, making it perfect for semantic search across visual-heavy content.
 
@@ -56,7 +57,7 @@ data_scope["documents"] = flow_builder.add_source(
 
 ## Convert Files to Pages
 
-We classify files by MIME type and process accordingly. 
+We classify files by MIME type and process accordingly.
 
 Define a dataclass:
 
@@ -111,7 +112,7 @@ In the flow we convert all the files to pages. this makes each pages and all ima
 
 ## Generate Visual Embeddings
 
-We use ColPali to generate embeddings for images on each page. 
+We use ColPali to generate embeddings for images on each page.
 
 ```python
 with doc["pages"].row() as page:
@@ -131,7 +132,7 @@ with doc["pages"].row() as page:
 
 ![Embedding](/img/examples/multi_format_index/embed.png)
 
-ColPali Architecture fundamentally rethinks how documents, especially visually complex or image-rich ones, are represented and searched. Instead of reducing each image or page to a single dense vector (as in traditional bi-encoders), ColPali breaks an image into many smaller patches, preserving local spatial and semantic structure. 
+ColPali Architecture fundamentally rethinks how documents, especially visually complex or image-rich ones, are represented and searched. Instead of reducing each image or page to a single dense vector (as in traditional bi-encoders), ColPali breaks an image into many smaller patches, preserving local spatial and semantic structure.
 
 Each patch receives its own embedding, which together form a multi-vector representation of the complete document.
 
@@ -142,7 +143,7 @@ Each patch receives its own embedding, which together form a multi-vector repres
 
 ## Export to Qdrant
 
-Note the way to embed image and query are different, as they’re two different types of data. 
+Note the way to embed image and query are different, as they’re two different types of data.
 
 Create a function to embed query:
 
@@ -199,9 +200,7 @@ cocoindex server -ci main
 Follow the url `https://cocoindex.io/cocoinsight`.  It connects to your local CocoIndex server, with zero pipeline data retention. You can use it to view extracted pages, see embedding vectors and metadata.
 
 
-## Connect to other sources 
+## Connect to other sources
 CocoIndex natively supports Google Drive, Amazon S3, Azure Blob Storage, and more.
 
 <DocumentationButton url="https://cocoindex.io/docs/ops/sources" text="Sources" margin="0 0 16px 0" />
-
-

{cocoindex-0.2.10 → cocoindex-0.2.12}/docs/docs/examples/examples/patient_form_extraction.md

@@ -7,6 +7,7 @@ canonicalUrl: '/examples/patient_form_extraction'
 sidebar_custom_props:
   image: /img/examples/patient_form_extraction/cover.png
   tags: [structured-data-extraction, data-mapping ]
+image: /img/examples/patient_form_extraction/cover.png
 tags: [structured-data-extraction, data-mapping]
 ---
 
@@ -21,7 +22,7 @@ import { GitHubButton, YouTubeButton, DocumentationButton } from '../../../src/c
 With CocoIndex, you can easily define nested schema in Python dataclass and use LLM to extract structured data from unstructured data. This example shows how to extract structured data from patient intake forms.
 
 :::info
-The extraction quality is highly dependent on the OCR quality. You can use CocoIndex with any commercial parser or open source ones that is tailored for your domain for better results. For example, Document AI from Google Cloud and more. 
+The extraction quality is highly dependent on the OCR quality. You can use CocoIndex with any commercial parser or open source ones that is tailored for your domain for better results. For example, Document AI from Google Cloud and more.
 :::
 
 ## Flow Overview
@@ -68,7 +69,7 @@ def patient_intake_extraction_flow(
 
 ##  Parse documents with different formats to Markdown
 
-Define a custom function to parse documents in any format to Markdown. Here we use [MarkItDown](https://github.com/microsoft/markitdown) to convert the file to Markdown. It also provides options to parse by LLM, like `gpt-4o`. At present, MarkItDown supports: PDF, Word, Excel, Images (EXIF metadata and OCR), etc. 
+Define a custom function to parse documents in any format to Markdown. Here we use [MarkItDown](https://github.com/microsoft/markitdown) to convert the file to Markdown. It also provides options to parse by LLM, like `gpt-4o`. At present, MarkItDown supports: PDF, Word, Excel, Images (EXIF metadata and OCR), etc.
 
 ```python
 class ToMarkdown(cocoindex.op.FunctionSpec):
@@ -103,7 +104,7 @@ with data_scope["documents"].row() as doc:
 
 ![Markdown](/img/examples/patient_form_extraction/tomarkdown.png)
 
-## Define output schema 
+## Define output schema
 
 We are going to define the patient info schema for structured extraction. One of the best examples to define a patient info schema is probably following the [FHIR standard - Patient Resource](https://build.fhir.org/patient.html#resource).
 
@@ -274,7 +275,7 @@ For mission-critical use cases, it is important to evaluate the quality of the e
 
 
 ## Troubleshooting
-If extraction is not ideal, this is how I troubleshoot. My original golden file for this record is [this one](https://github.com/cocoindex-io/patient-intake-extraction/blob/main/data/example_forms/Patient_Intake_Form_Joe_Artificial.pdf). 
+If extraction is not ideal, this is how I troubleshoot. My original golden file for this record is [this one](https://github.com/cocoindex-io/patient-intake-extraction/blob/main/data/example_forms/Patient_Intake_Form_Joe_Artificial.pdf).
 
 We could troubleshoot in two steps:
 1. Convert to Markdown
@@ -292,7 +293,7 @@ Go to `https://cocoindex.io/cocoinsight`. You could see an interactive UI to exp
 Click on the `markdown` column for `Patient_Intake_Form_Joe.pdf`, you could see the Markdown content. We could try a few different models with the Markdown converter/LLM to iterate and see if we can get better results, or needs manual correction.
 
 
-## Connect to other sources 
+## Connect to other sources
 CocoIndex natively supports Google Drive, Amazon S3, Azure Blob Storage, and more.
 
-<DocumentationButton url="https://cocoindex.io/docs/ops/sources" text="Sources" margin="0 0 16px 0" />
+<DocumentationButton url="https://cocoindex.io/docs/ops/sources" text="Sources" margin="0 0 16px 0" />