cocoindex 0.1.82__tar.gz → 0.2.0__tar.gz

{cocoindex-0.1.82 → cocoindex-0.2.0}/Cargo.lock

@@ -1283,7 +1283,7 @@ dependencies = [
 
 [[package]]
 name = "cocoindex"
-version = "0.1.82"
+version = "0.2.0"
 dependencies = [
  "anyhow",
  "async-openai",

{cocoindex-0.1.82 → cocoindex-0.2.0}/Cargo.toml

@@ -2,7 +2,7 @@
 name = "cocoindex"
 # Version used for local development is always higher than others to take precedence.
 # Will be overridden for specific release versions.
-version = "0.1.82"
+version = "0.2.0"
 edition = "2024"
 rust-version = "1.88"
 readme = "README.md"

{cocoindex-0.1.82 → cocoindex-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cocoindex
-Version: 0.1.82
+Version: 0.2.0
 Requires-Dist: click>=8.1.8
 Requires-Dist: rich>=14.0.0
 Requires-Dist: python-dotenv>=1.1.0

{cocoindex-0.1.82 → cocoindex-0.2.0}/docs/docs/core/basics.md

@@ -23,7 +23,7 @@ Each piece of data has a **data type**, falling into one of the following catego
 
 *   *Basic type*.
 *   *Struct type*: a collection of **fields**, each with a name and a type.
-*   *Table type*: a collection of **rows**, each of which is a struct with specified schema. A table type can be a *KTable* (which has a key field) or a *LTable* (ordered but without key field).
+*   *Table type*: a collection of **rows**, each of which is a struct with specified schema. A table type can be a *KTable* (with key columns that uniquely identify each row) or a *LTable* (rows are ordered but without keys).
 
 An indexing flow always has a top-level struct, containing all data within and managed by the flow.
 

{cocoindex-0.1.82 → cocoindex-0.2.0}/docs/docs/core/data_types.mdx

@@ -148,21 +148,27 @@ We have two specific types of *Table* types: *KTable* and *LTable*.
 
 #### KTable
 
-*KTable* is a *Table* type whose first column serves as the key.
+*KTable* is a *Table* type whose one or more columns together serve as the key.
 The row order of a *KTable* is not preserved.
-Type of the first column (key column) must be a [key type](#key-types).
+Each key column must be a [key type](#key-types). When multiple key columns are present, they form a composite key.
 
-In Python, a *KTable* type is represented by `dict[K, V]`. 
-The `K` should be the type binding to a key type,
-and the `V` should be the type binding to a *Struct* type representing the value fields of each row.
-When the specific type annotation is not provided,
-the key type is bound to a tuple with its key parts when it's a *Struct* type, the value type is bound to `dict[str, Any]`.
+In Python, a *KTable* type is represented by `dict[K, V]`.
+`K` represents the key and `V` represents the value for each row:
+
+- `K` can be a Struct type (either a frozen dataclass or a `NamedTuple`) that contains all key parts as fields. This is the general way to model multi-part keys.
+- When there is only a single key part and it is a basic type (e.g. `str`, `int`), you may use that basic type directly as the dictionary key instead of wrapping it in a Struct.
+- `V` should be the type bound to a *Struct* representing the non-key value fields of each row.
+
+When a specific type annotation is not provided:
+- For composite keys (multiple key parts), the key binds to a Python tuple of the key parts, e.g. `tuple[str, str]`.
+- For a single basic key part, the key binds to that basic Python type.
+- The value binds to `dict[str, Any]`.
 
 
 For example, you can use `dict[str, Person]` or `dict[str, PersonTuple]` to represent a *KTable*, with 4 columns: key (*Str*), `first_name` (*Str*), `last_name` (*Str*), `dob` (*Date*).
 It's bound to `dict[str, dict[str, Any]]` if you don't annotate the function argument with a specific type.
 
-Note that if you want to use a *Struct* as the key, you need to ensure its value in Python is immutable. For `dataclass`, annotate it with `@dataclass(frozen=True)`. For `NamedTuple`, immutability is built-in. For example:
+Note that when using a Struct as the key, it must be immutable in Python. For a dataclass, annotate it with `@dataclass(frozen=True)`. For `NamedTuple`, immutability is built-in. For example:
 
 ```python
 @dataclass(frozen=True)
@@ -175,8 +181,8 @@ class PersonKeyTuple(NamedTuple):
     id: str
 ```
 
-Then you can use `dict[PersonKey, Person]` or `dict[PersonKeyTuple, PersonTuple]` to represent a KTable keyed by `PersonKey` or `PersonKeyTuple`.
-It's bound to `dict[(str, str), dict[str, Any]]` if you don't annotate the function argument with a specific type.
+Then you can use `dict[PersonKey, Person]` or `dict[PersonKeyTuple, PersonTuple]` to represent a KTable keyed by both `id_kind` and `id`.
+If you don't annotate the function argument with a specific type, it's bound to `dict[tuple[str, str], dict[str, Any]]`.
 
 
 #### LTable

{cocoindex-0.1.82 → cocoindex-0.2.0}/docs/docs/examples/examples/academic_papers_index.md

@@ -5,16 +5,15 @@ sidebar_class_name: hidden
 slug: /examples/academic_papers_index
 canonicalUrl: '/examples/academic_papers_index'
 sidebar_custom_props:
-  image: /img/examples/academic_papers_index.png
+  image: /img/examples/academic_papers_index/cover.png
   tags: [vector-index, metadata]
 tags: [vector-index, metadata]
 ---
 
-import { GitHubButton, YouTubeButton } from '../../../src/components/GitHubButton';
+import { GitHubButton, YouTubeButton, DocumentationButton } from '../../../src/components/GitHubButton';
 
 <GitHubButton url="https://github.com/cocoindex-io/cocoindex/tree/main/examples/paper_metadata"/>
 
-
 ## What we will achieve
 
 1. Extract the paper metadata, including file name, title, author information, abstract, and number of pages.
@@ -27,18 +26,8 @@ to answer questions like "Give me all the papers by Jeff Dean."
 
 4. If you want to perform full PDF embedding for the paper, you can extend the flow.
 
-## Setup
-
-- [Install PostgreSQL](https://cocoindex.io/docs/getting_started/installation#-install-postgres).
-  CocoIndex uses PostgreSQL internally for incremental processing.
-- [Configure your OpenAI API key](https://cocoindex.io/docs/ai/llm#openai).  
-  Alternatively, we have native support for Gemini, Ollama, LiteLLM. Check out the [guide](https://cocoindex.io/docs/ai/llm#ollama).
-  You can choose your favorite LLM provider and work completely on-premises.
-
-## Define Indexing Flow
-
-To better help you navigate what we will walk through, here is a flow diagram: 
-
+## Flow Overview
+![Flow Overview](/img/examples/academic_papers_index/flow.png)
 1. Import a list of papers in PDF.
 2. For each file:
     - Extract the first page of the paper.
@@ -50,9 +39,15 @@ To better help you navigate what we will walk through, here is a flow diagram:
     - Author-to-paper mapping, for author-based query.
     - Embeddings for titles and abstract chunks, for semantic search.
 
-Let’s zoom in on the steps.
+## Setup
+
+- [Install PostgreSQL](https://cocoindex.io/docs/getting_started/installation#-install-postgres).
+  CocoIndex uses PostgreSQL internally for incremental processing.
+- [Configure your OpenAI API key](https://cocoindex.io/docs/ai/llm#openai). Alternatively, we have native support for Gemini, Ollama, LiteLLM. You can choose your favorite LLM provider and work completely on-premises.
 
-### Import the Papers
+  <DocumentationButton href="https://cocoindex.io/docs/ai/llm" text="LLM" margin="0 0 16px 0" />
+
+## Import the Papers
 
 ```python
 @cocoindex.flow_def(name="PaperMetadata")
@@ -65,12 +60,12 @@ def paper_metadata_flow(
     )
 ```
 
-`flow_builder.add_source` will create a table with sub fields (`filename`, `content`), 
-we can refer to the [documentation](https://cocoindex.io/docs/ops/sources) for more details.
+`flow_builder.add_source` will create a table with sub fields (`filename`, `content`).
+<DocumentationButton href="https://cocoindex.io/docs/ops/sources" text="Sources" margin="0 0 16px 0" />
 
-### Extract and collect metadata
+## Extract and collect metadata
 
-#### Extract first page for basic info
+### Extract first page for basic info
 
 Define a custom function to extract the first page and number of pages of the PDF.
 
@@ -96,20 +91,19 @@ def extract_basic_info(content: bytes) -> PaperBasicInfo:
 
 ```
 
-Now, plug this into your flow.
-We extract metadata from the first page to minimize processing cost, since the entire PDF can be very large.
+Now plug this into the flow. We extract metadata from the first page to minimize processing cost, since the entire PDF can be very large.
 
 ```python
 with data_scope["documents"].row() as doc:
     doc["basic_info"] = doc["content"].transform(extract_basic_info)
 ```
+![Extract basic info](/img/examples/academic_papers_index/basic_info.png)
 
-After this step, you should have the basic info of each paper.
+After this step, we should have the basic info of each paper.
 
 ### Parse basic info
 
-We will convert the first page to Markdown using Marker. 
-Alternatively, you can easily plug in your favorite PDF parser, such as Docling.
+We will convert the first page to Markdown using Marker. Alternatively, you can easily plug in any PDF parser, such as Docling using CocoIndex's [custom function](https://cocoindex.io/docs/custom_ops/custom_functions).
 
 Define a marker converter function and cache it, since its initialization is resource-intensive. 
 This ensures that the same converter instance is reused for different input files.
@@ -140,18 +134,20 @@ def pdf_to_markdown(content: bytes) -> str:
 Pass it to your transform
 
 ```python
-with data_scope["documents"].row() as doc:      
+with data_scope["documents"].row() as doc:    
+    # ... process
     doc["first_page_md"] = doc["basic_info"]["first_page"].transform(
             pdf_to_markdown
         )
 ```
+![First page in Markdown](/img/examples/academic_papers_index/first_page.png)
 
 After this step, you should have the first page of each paper in Markdown format.
 
-#### Extract basic info with LLM
+### Extract basic info with LLM
 
 Define a schema for LLM extraction. CocoIndex natively supports LLM-structured extraction with complex and nested schemas.
-If you are interested in learning more about nested schemas, refer to [this article](https://cocoindex.io/blogs/patient-intake-form-extraction-with-llm).
+If you are interested in learning more about nested schemas, refer to [this example](https://cocoindex.io/docs/examples/patient_form_extraction).
 
 ```python
 @dataclasses.dataclass
@@ -163,7 +159,6 @@ class PaperMetadata:
     title: str
     authors: list[Author]
     abstract: str
-
 ```
 
 Plug it into the `ExtractByLlm` function. With a dataclass defined, CocoIndex will automatically parse the LLM response into the dataclass.
@@ -181,26 +176,27 @@ doc["metadata"] = doc["first_page_md"].transform(
 ```
 
 After this step, you should have the metadata of each paper.
+![Metadata](/img/examples/academic_papers_index/metadata.png)
 
-#### Collect paper metadata
+### Collect paper metadata
 
 ```python
-  paper_metadata = data_scope.add_collector()
-  with data_scope["documents"].row() as doc:
-    # ... process
-    # Collect metadata
-    paper_metadata.collect(
-        filename=doc["filename"],
-        title=doc["metadata"]["title"],
-        authors=doc["metadata"]["authors"],
-        abstract=doc["metadata"]["abstract"],
-        num_pages=doc["basic_info"]["num_pages"],
-    )
+paper_metadata = data_scope.add_collector()
+with data_scope["documents"].row() as doc:
+# ... process
+# Collect metadata
+paper_metadata.collect(
+    filename=doc["filename"],
+    title=doc["metadata"]["title"],
+    authors=doc["metadata"]["authors"],
+    abstract=doc["metadata"]["abstract"],
+    num_pages=doc["basic_info"]["num_pages"],
+)
 ```
 
 Just collect anything you need :)
 
-#### Collect `author` to `filename` information
+### Collect `author` to `filename` information
 We’ve already extracted author list. Here we want to collect Author → Papers in a separate table to build a look up functionality. 
 Simply collect by author.
 
@@ -216,9 +212,9 @@ with data_scope["documents"].row() as doc:
 ```
 
 
-### Compute and collect embeddings
+## Compute and collect embeddings
 
-#### Title
+### Title
 
 ```python
 doc["title_embedding"] = doc["metadata"]["title"].transform(
@@ -228,7 +224,7 @@ doc["title_embedding"] = doc["metadata"]["title"].transform(
 )
 ```
 
-#### Abstract
+### Abstract
 
 Split abstract into chunks, embed each chunk and collect their embeddings. 
 Sometimes the abstract could be very long. 
@@ -252,6 +248,8 @@ doc["abstract_chunks"] = doc["metadata"]["abstract"].transform(
 
 After this step, you should have the abstract chunks of each paper.
 
+![Abstract chunks](/img/examples/academic_papers_index/abstract_chunks.png)
+
 Embed each chunk and collect their embeddings.
 
 ```python
@@ -265,7 +263,9 @@ with doc["abstract_chunks"].row() as chunk:
 
 After this step, you should have the embeddings of the abstract chunks of each paper.
 
-#### Collect embeddings
+![Abstract chunks embeddings](/img/examples/academic_papers_index/chunk_embedding.png)
+
+### Collect embeddings
 
 ```python
 metadata_embeddings = data_scope.add_collector()
@@ -292,7 +292,7 @@ with data_scope["documents"].row() as doc:
         )
 ```
 
-### Export
+## Export
 Finally, we export the data to Postgres.
 
 ```python
@@ -319,14 +319,9 @@ metadata_embeddings.export(
 )
 ```
 
-In this example we use PGVector as embedding stores/
-With CocoIndex, you can do one line switch on other supported Vector databases like Qdrant, see this [guide](https://cocoindex.io/docs/ops/targets#entry-oriented-targets) for more details.
-We aim to standardize interfaces and make it like assembling building blocks.
+In this example we use PGVector as embedding store. With CocoIndex, you can do one line switch on other supported Vector databases.
 
-## View in CocoInsight step by step
-
-You can walk through the project step by step in [CocoInsight](https://www.youtube.com/watch?v=MMrpUfUcZPk) to see 
-exactly how each field is constructed and what happens behind the scenes.
+<DocumentationButton href="https://cocoindex.io/docs/ops/targets#entry-oriented-targets" text="Entry Oriented Targets" margin="0 0 16px 0" />
 
 ## Query the index
 
@@ -338,3 +333,14 @@ For now CocoIndex doesn't provide additional query interface. We can write SQL o
 - The query space has excellent solutions for querying, reranking, and other search-related functionality.
 
 If you need assist with writing the query, please feel free to reach out to us at [Discord](https://discord.com/invite/zpA9S2DR7s).
+
+## CocoInsight
+
+You can walk through the project step by step in [CocoInsight](https://www.youtube.com/watch?v=MMrpUfUcZPk) to see exactly how each field is constructed and what happens behind the scenes.
+
+
+```sh
+cocoindex server -ci main.py
+```
+
+Follow the url `https://cocoindex.io/cocoinsight`.  It connects to your local CocoIndex server, with zero pipeline data retention.

{cocoindex-0.1.82 → cocoindex-0.2.0}/docs/docs/examples/examples/codebase_index.md

@@ -5,30 +5,57 @@ sidebar_class_name: hidden
 slug: /examples/code_index
 canonicalUrl: '/examples/code_index'
 sidebar_custom_props:
-  image: /img/examples/codebase_index.png
+  image: /img/examples/codebase_index/cover.png
   tags: [vector-index, codebase]
 tags: [vector-index, codebase]
 ---
 
-import { GitHubButton, YouTubeButton } from '../../../src/components/GitHubButton';
+import { GitHubButton, YouTubeButton, DocumentationButton } from '../../../src/components/GitHubButton';
 
 <GitHubButton url="https://github.com/cocoindex-io/cocoindex/tree/main/examples/code_embedding"/>
 <YouTubeButton url="https://youtu.be/G3WstvhHO24?si=ndYfM0XRs03_hVPR" />
 
-## Setup 
+## Overview
+In this tutorial, we will build codebase index. [CocoIndex](https://github.com/cocoindex-io/cocoindex) provides built-in support for codebase chunking, with native Tree-sitter support. It works with large codebases, and can be updated in near real-time with incremental processing - only reprocess what's changed. 
 
-If you don't have Postgres installed, please follow [installation guide](https://cocoindex.io/docs/getting_started/installation).
+## Use Cases
+A wide range of applications can be built with an effective codebase index that is always up-to-date. Some examples include:
 
-## Add the codebase as a source. 
+![Use case illustration](/img/examples/codebase_index/usecase.png)
+
+- Semantic code context for AI coding agents like Claude, Codex, Gemini CLI.
+- MCP for code editors such as Cursor, Windsurf, and VSCode.
+- Context-aware code search applications—semantic code search, natural language code retrieval.
+- Context for code review agents—AI code review, automated code analysis, code quality checks, pull request summarization.
+- Automated code refactoring, large-scale code migration.
+- Enhance SRE workflows: enable rapid root cause analysis, incident response, and change impact assessment by indexing infrastructure-as-code, deployment scripts, and config files for semantic search and lineage tracking.
+- Automatically generate design documentation from code—keep design docs up-to-date.
+
+## Flow Overview
+
+![Flow Overview](/img/examples/codebase_index/flow.png)
+
+The flow is composed of the following steps:
+
+- Read code files from the local filesystem
+- Extract file extensions, to get the language of the code for Tree-sitter to parse
+- Split code into semantic chunks using Tree-sitter
+- Generate embeddings for each chunk
+- Store in a vector database for retrieval
 
-Ingest files from the CocoIndex codebase root directory.
+## Setup 
+- Install Postgres, follow [installation guide](https://cocoindex.io/docs/getting_started/installation#-install-postgres).
+- Install CocoIndex
+  ```bash
+  pip install -U cocoindex
+  ```
+
+## Add the codebase as a source. 
+We will index the CocoIndex codebase. Here we use the `LocalFile` source to ingest files from the CocoIndex codebase root directory.
 
 ```python
 @cocoindex.flow_def(name="CodeEmbedding")
 def code_embedding_flow(flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope):
-    """
-    Define an example flow that embeds files into a vector database.
-    """
     data_scope["files"] = flow_builder.add_source(
         cocoindex.sources.LocalFile(path="../..",
                                     included_patterns=["*.py", "*.rs", "*.toml", "*.md", "*.mdx"],
@@ -40,16 +67,15 @@ def code_embedding_flow(flow_builder: cocoindex.FlowBuilder, data_scope: cocoind
 - Exclude files and directories starting `.`,  `target` in the root and `node_modules` under any directory.
 
 `flow_builder.add_source` will create a table with sub fields (`filename`, `content`). 
-See [documentation](https://cocoindex.io/docs/ops/sources) for more details.
+<DocumentationButton href="https://cocoindex.io/docs/ops/sources" text="Sources" />
 
 
-## Process each file and collect the information.
+## Process each file and collect the information
 
-###  Extract the extension of a filename
+### Extract the extension of a filename
 
 We need to pass the language (or extension) to Tree-sitter to parse the code.
 Let's define a function to extract the extension of a filename while processing each file.
-You can find the documentation for custom function [here](https://cocoindex.io/docs/core/custom_function).
 
 ```python
 @cocoindex.op.function()
@@ -58,52 +84,43 @@ def extract_extension(filename: str) -> str:
     return os.path.splitext(filename)[1]
 ```
 
-Then we are going to process each file and collect the information.
-
-```python
-with data_scope["files"].row() as file:
-    file["extension"] = file["filename"].transform(extract_extension)
-```
-
-Here we extract the extension of the filename and store it in the `extension` field.
-
+<DocumentationButton href="https://cocoindex.io/docs/custom_ops/custom_functions" text="Custom Function" margin="0 0 16px 0" />
 
 ### Split the file into chunks
-
-We will chunk the code with Tree-sitter. 
-We use the `SplitRecursively` function to split the file into chunks. 
-It is integrated with Tree-sitter, so you can pass in the language to the `language` parameter.
-To see all supported language names and extensions, see the documentation [here](https://cocoindex.io/docs/ops/functions#splitrecursively). All the major languages are supported, e.g., Python, Rust, JavaScript, TypeScript, Java, C++, etc. If it's unspecified or the specified language is not supported, it will be treated as plain text.
+We use the `SplitRecursively` function to split the file into chunks.  `SplitRecursively` is CocoIndex building block, with native integration with Tree-sitter. You need to pass in the language to the `language` parameter if you are processing code.
 
 ```python
 with data_scope["files"].row() as file:
+    # Extract the extension of the filename.
+    file["extension"] = file["filename"].transform(extract_extension)
     file["chunks"] = file["content"].transform(
           cocoindex.functions.SplitRecursively(),
           language=file["extension"], chunk_size=1000, chunk_overlap=300) 
 ```
+<DocumentationButton href="https://cocoindex.io/docs/ops/functions#splitrecursively" text="SplitRecursively" margin="0 0 16px 0" />
 
+![SplitRecursively](/img/examples/codebase_index/chunk.png)
 
 ### Embed the chunks
-
 We use `SentenceTransformerEmbed` to embed the chunks. 
-You can refer to the documentation [here](https://cocoindex.io/docs/ops/functions#sentencetransformerembed). 
 
 ```python
 @cocoindex.transform_flow()
 def code_to_embedding(text: cocoindex.DataSlice[str]) -> cocoindex.DataSlice[list[float]]:
-    """
-    Embed the text using a SentenceTransformer model.
-    """
     return text.transform(
         cocoindex.functions.SentenceTransformerEmbed(
             model="sentence-transformers/all-MiniLM-L6-v2"))
 ```
 
-Then for each chunk, we will embed it using the `code_to_embedding` function. and collect the embeddings to the `code_embeddings` collector.
+<DocumentationButton href="https://cocoindex.io/docs/ops/functions#sentencetransformerembed" text="SentenceTransformerEmbed" margin="0 0 16px 0" />
+
+:::tip
+`@cocoindex.transform_flow()` is needed to share the transformation across indexing and query. When building a vector index and querying against it, the embedding computation must remain consistent between indexing and querying.
+:::
 
-`@cocoindex.transform_flow()` is needed to share the transformation across indexing and query. We build a vector index and query against it, 
-the embedding computation needs to be consistent between indexing and querying. See [documentation](https://cocoindex.io/docs/query#transform-flow) for more details.
+<DocumentationButton href="https://cocoindex.io/docs/query#transform-flow" text="Transform Flow" margin="0 0 16px 0" />
 
+Then for each chunk, we will embed it using the `code_to_embedding` function, and collect the embeddings to the `code_embeddings` collector.
 
 ```python
 with data_scope["files"].row() as file:
@@ -113,10 +130,7 @@ with data_scope["files"].row() as file:
                                 code=chunk["text"], embedding=chunk["embedding"])
 ```
 
-
-### 2.4 Collect the embeddings
-
-Export the embeddings to a table.
+### Export the embeddings
 
 ```python
 code_embeddings.export(
@@ -126,8 +140,7 @@ code_embeddings.export(
     vector_indexes=[cocoindex.VectorIndex("embedding", cocoindex.VectorSimilarityMetric.COSINE_SIMILARITY)])
 ```
 
-We use Consine Similarity to measure the similarity between the query and the indexed data. 
-To learn more about Consine Similarity, see [Wiki](https://en.wikipedia.org/wiki/Cosine_similarity).
+We use [Cosine Similarity](https://en.wikipedia.org/wiki/Cosine_similarity) to measure the similarity between the query and the indexed data. 
 
 ## Query the index
 We match against user-provided text by a SQL query, reusing the embedding operation in the indexing flow.
@@ -180,13 +193,16 @@ if __name__ == "__main__":
 
 ## Run the index setup & update
 
-🎉 Now you are all set!
+- Install dependencies
+    ```bash
+    pip install -e .
+    ```
 
-Run following command to setup and update the index.
-```sh
-cocoindex update --setup main.py
-```
-You'll see the index updates state in the terminal
+- Setup and update the index
+    ```sh
+    cocoindex update --setup main.py
+    ```
+    You'll see the index updates state in the terminal
 
 
 ## Test the query
@@ -197,7 +213,13 @@ python main.py
 ```
 
 When you see the prompt, you can enter your search query. for example: spec.
+The returned results - each entry contains score (Cosine Similarity), filename, and the code snippet that get matched.
 
-You can find the search results in the terminal
+## CocoInsight
+To get a better understanding of the indexing flow, you can use CocoInsight to help the development step by step.
+To spin up, it is super easy.
 
-The returned results - each entry contains score (Cosine Similarity), filename, and the code snippet that get matched.
+```
+cocoindex server main.py -ci
+```
+Follow the url from the terminal - `https://cocoindex.io/cocoinsight` to access the CocoInsight.

{cocoindex-0.1.82 → cocoindex-0.2.0}/docs/docs/examples/examples/custom_targets.md

@@ -5,23 +5,17 @@ sidebar_class_name: hidden
 slug: /examples/custom_targets
 canonicalUrl: '/examples/custom_targets'
 sidebar_custom_props:
-  image: /img/examples/custom_targets.png
+  image: /img/examples/custom_targets/cover.png
   tags: [custom-building-blocks]
 tags: [custom-building-blocks]
 ---
-import { GitHubButton, YouTubeButton } from '../../../src/components/GitHubButton';
+import { GitHubButton, YouTubeButton, DocumentationButton } from '../../../src/components/GitHubButton';
 
 <GitHubButton url="https://github.com/cocoindex-io/cocoindex/tree/main/examples/custom_output_files"/>
 
 ## Overview
 
-Let’s walk through a simple example—exporting `.md` files as `.html` using a custom file-based target. This project monitors folder changes and continuously converts markdown to HTML incrementally.
-Check out the full [source code](https://github.com/cocoindex-io/cocoindex/tree/main/examples/custom_output_files).
-
-The overall flow is simple:
-This example focuses on 
-- how to configure your custom target
-- the flow effortless picks up the changes in the source, recomputes only what's changed and export to the target
+Let’s walk through a simple example—exporting `.md` files as `.html` using a custom file-based target. This project monitors folder changes and continuously converts markdown to HTML incrementally. The overall flow is simple and primarily focuses on how to configure your custom target.
 
 
 ## Ingest files
@@ -33,16 +27,13 @@ Ingest a list of markdown files:
 def custom_output_files(
 flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope
 ) -> None:
-	"""
-	Define an example flow that exports markdown files to HTML files.
-	"""
 	data_scope["documents"] = flow_builder.add_source(
 		cocoindex.sources.LocalFile(path="data", included_patterns=["*.md"]),
 		refresh_interval=timedelta(seconds=5),
 	)
 ```
 This ingestion creates a table with `filename` and `content` fields. 
-
+<DocumentationButton href="https://cocoindex.io/docs/ops/sources" text="Sources" />
 
 ## Process each file and collect
 
@@ -50,11 +41,12 @@ Define custom function that converts markdown to HTML
 
 ```python
 @cocoindex.op.function()
-
 def markdown_to_html(text: str) -> str:
     return _markdown_it.render(text)
 ```
 
+<DocumentationButton href="https://cocoindex.io/docs/custom_ops/custom_functions" text="Custom Function" margin="0 0 16px 0" />
+
 Define data collector and transform each document to html.
 
 ```python
@@ -63,12 +55,15 @@ with data_scope["documents"].row() as doc:
     doc["html"] = doc["content"].transform(markdown_to_html)
     output_html.collect(filename=doc["filename"], html=doc["html"])
 ```
+![Convert markdown to html](/img/examples/custom_targets/convert.png)
 
 
 ##  Define the custom target
 
 ### Define the target spec
 
+<DocumentationButton href="https://cocoindex.io/docs/custom_ops/custom_targets#target-spec" text="Target Spec" margin="0 0 16px 0" />
+
 The target spec contains a directory for output files:
 
 ```python
@@ -76,8 +71,11 @@ class LocalFileTarget(cocoindex.op.TargetSpec):
     directory: str
 ```
 
+
 ### Implement the connector
 
+<DocumentationButton href="https://cocoindex.io/docs/custom_ops/custom_targets#target-connector" text="Target Connector" margin="0 0 16px 0" />
+
 `get_persistent_key()` defines the persistent key,
 which uniquely identifies the target for change tracking and incremental updates. Here, we simply use the target directory as the key (e.g., `./data/output`).
 
@@ -180,17 +178,15 @@ def mutate(
 ### Use it in the Flow
 
 ```python
-    output_html.export(
-        "OutputHtml",
-        LocalFileTarget(directory="output_html"),
-        primary_key_fields=["filename"],
-    )
+output_html.export(
+    "OutputHtml",
+    LocalFileTarget(directory="output_html"),
+    primary_key_fields=["filename"],
+)
 ```
 
 ## Run the example
 
-Once your pipeline is set up, keeping your knowledge graph updated is simple:
-
 ```bash
 pip install -e .
 cocoindex update --setup main.py