cocoindex 0.1.83__tar.gz → 0.2.1__tar.gz

{cocoindex-0.1.83 → cocoindex-0.2.1}/Cargo.lock

@@ -1283,7 +1283,7 @@ dependencies = [
 
 [[package]]
 name = "cocoindex"
-version = "0.1.83"
+version = "0.2.1"
 dependencies = [
  "anyhow",
  "async-openai",

{cocoindex-0.1.83 → cocoindex-0.2.1}/Cargo.toml

@@ -2,7 +2,7 @@
 name = "cocoindex"
 # Version used for local development is always higher than others to take precedence.
 # Will be overridden for specific release versions.
-version = "0.1.83"
+version = "0.2.1"
 edition = "2024"
 rust-version = "1.88"
 readme = "README.md"

{cocoindex-0.1.83 → cocoindex-0.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cocoindex
-Version: 0.1.83
+Version: 0.2.1
 Requires-Dist: click>=8.1.8
 Requires-Dist: rich>=14.0.0
 Requires-Dist: python-dotenv>=1.1.0

{cocoindex-0.1.83 → cocoindex-0.2.1}/docs/docs/core/basics.md

@@ -23,7 +23,7 @@ Each piece of data has a **data type**, falling into one of the following catego
 
 *   *Basic type*.
 *   *Struct type*: a collection of **fields**, each with a name and a type.
-*   *Table type*: a collection of **rows**, each of which is a struct with specified schema. A table type can be a *KTable* (which has a key field) or a *LTable* (ordered but without key field).
+*   *Table type*: a collection of **rows**, each of which is a struct with specified schema. A table type can be a *KTable* (with key columns that uniquely identify each row) or a *LTable* (rows are ordered but without keys).
 
 An indexing flow always has a top-level struct, containing all data within and managed by the flow.
 

{cocoindex-0.1.83 → cocoindex-0.2.1}/docs/docs/core/data_types.mdx

@@ -148,21 +148,27 @@ We have two specific types of *Table* types: *KTable* and *LTable*.
 
 #### KTable
 
-*KTable* is a *Table* type whose first column serves as the key.
+*KTable* is a *Table* type whose one or more columns together serve as the key.
 The row order of a *KTable* is not preserved.
-Type of the first column (key column) must be a [key type](#key-types).
+Each key column must be a [key type](#key-types). When multiple key columns are present, they form a composite key.
 
-In Python, a *KTable* type is represented by `dict[K, V]`. 
-The `K` should be the type binding to a key type,
-and the `V` should be the type binding to a *Struct* type representing the value fields of each row.
-When the specific type annotation is not provided,
-the key type is bound to a tuple with its key parts when it's a *Struct* type, the value type is bound to `dict[str, Any]`.
+In Python, a *KTable* type is represented by `dict[K, V]`.
+`K` represents the key and `V` represents the value for each row:
+
+- `K` can be a Struct type (either a frozen dataclass or a `NamedTuple`) that contains all key parts as fields. This is the general way to model multi-part keys.
+- When there is only a single key part and it is a basic type (e.g. `str`, `int`), you may use that basic type directly as the dictionary key instead of wrapping it in a Struct.
+- `V` should be the type bound to a *Struct* representing the non-key value fields of each row.
+
+When a specific type annotation is not provided:
+- For composite keys (multiple key parts), the key binds to a Python tuple of the key parts, e.g. `tuple[str, str]`.
+- For a single basic key part, the key binds to that basic Python type.
+- The value binds to `dict[str, Any]`.
 
 
 For example, you can use `dict[str, Person]` or `dict[str, PersonTuple]` to represent a *KTable*, with 4 columns: key (*Str*), `first_name` (*Str*), `last_name` (*Str*), `dob` (*Date*).
 It's bound to `dict[str, dict[str, Any]]` if you don't annotate the function argument with a specific type.
 
-Note that if you want to use a *Struct* as the key, you need to ensure its value in Python is immutable. For `dataclass`, annotate it with `@dataclass(frozen=True)`. For `NamedTuple`, immutability is built-in. For example:
+Note that when using a Struct as the key, it must be immutable in Python. For a dataclass, annotate it with `@dataclass(frozen=True)`. For `NamedTuple`, immutability is built-in. For example:
 
 ```python
 @dataclass(frozen=True)
@@ -175,8 +181,8 @@ class PersonKeyTuple(NamedTuple):
     id: str
 ```
 
-Then you can use `dict[PersonKey, Person]` or `dict[PersonKeyTuple, PersonTuple]` to represent a KTable keyed by `PersonKey` or `PersonKeyTuple`.
-It's bound to `dict[(str, str), dict[str, Any]]` if you don't annotate the function argument with a specific type.
+Then you can use `dict[PersonKey, Person]` or `dict[PersonKeyTuple, PersonTuple]` to represent a KTable keyed by both `id_kind` and `id`.
+If you don't annotate the function argument with a specific type, it's bound to `dict[tuple[str, str], dict[str, Any]]`.
 
 
 #### LTable

{cocoindex-0.1.83 → cocoindex-0.2.1}/docs/docs/core/settings.mdx

@@ -96,11 +96,11 @@ If not set, all flows are in a default unnamed namespace.
 
     :::
 
-*   `max_connections` (type: `int`, default: `64`): The maximum number of connections to keep in the pool.
+*   `max_connections` (type: `int`, default: `25`): The maximum number of connections to keep in the pool.
 
     *Environment variable* for `Settings.database.max_connections`: `COCOINDEX_DATABASE_MAX_CONNECTIONS`
 
-*   `min_connections` (type: `int`, default: `16`): The minimum number of connections to keep in the pool.
+*   `min_connections` (type: `int`, default: `5`): The minimum number of connections to keep in the pool.
 
     *Environment variable* for `Settings.database.min_connections`: `COCOINDEX_DATABASE_MIN_CONNECTIONS`
 
@@ -134,7 +134,7 @@ This is the list of environment variables, each of which has a corresponding fie
 | `COCOINDEX_DATABASE_URL` | `database.url` | Yes |
 | `COCOINDEX_DATABASE_USER` | `database.user` | No |
 | `COCOINDEX_DATABASE_PASSWORD` | `database.password` | No |
-| `COCOINDEX_DATABASE_MAX_CONNECTIONS` | `database.max_connections` | No (default: `64`) |
-| `COCOINDEX_DATABASE_MIN_CONNECTIONS` | `database.min_connections` | No (default: `16`) |
+| `COCOINDEX_DATABASE_MAX_CONNECTIONS` | `database.max_connections` | No (default: `25`) |
+| `COCOINDEX_DATABASE_MIN_CONNECTIONS` | `database.min_connections` | No (default: `5`) |
 | `COCOINDEX_SOURCE_MAX_INFLIGHT_ROWS` | `global_execution_options.source_max_inflight_rows` | No (default: `1024`) |
 | `COCOINDEX_SOURCE_MAX_INFLIGHT_BYTES` | `global_execution_options.source_max_inflight_bytes` | No |

{cocoindex-0.1.83 → cocoindex-0.2.1}/docs/docs/examples/examples/academic_papers_index.md

@@ -10,11 +10,10 @@ sidebar_custom_props:
 tags: [vector-index, metadata]
 ---
 
-import { GitHubButton, YouTubeButton } from '../../../src/components/GitHubButton';
+import { GitHubButton, YouTubeButton, DocumentationButton } from '../../../src/components/GitHubButton';
 
 <GitHubButton url="https://github.com/cocoindex-io/cocoindex/tree/main/examples/paper_metadata"/>
 
-
 ## What we will achieve
 
 1. Extract the paper metadata, including file name, title, author information, abstract, and number of pages.
@@ -27,18 +26,8 @@ to answer questions like "Give me all the papers by Jeff Dean."
 
 4. If you want to perform full PDF embedding for the paper, you can extend the flow.
 
-## Setup
-
-- [Install PostgreSQL](https://cocoindex.io/docs/getting_started/installation#-install-postgres).
-  CocoIndex uses PostgreSQL internally for incremental processing.
-- [Configure your OpenAI API key](https://cocoindex.io/docs/ai/llm#openai).  
-  Alternatively, we have native support for Gemini, Ollama, LiteLLM. Check out the [guide](https://cocoindex.io/docs/ai/llm#ollama).
-  You can choose your favorite LLM provider and work completely on-premises.
-
-## Define Indexing Flow
-
-To better help you navigate what we will walk through, here is a flow diagram: 
-
+## Flow Overview
+![Flow Overview](/img/examples/academic_papers_index/flow.png)
 1. Import a list of papers in PDF.
 2. For each file:
     - Extract the first page of the paper.
@@ -50,9 +39,15 @@ To better help you navigate what we will walk through, here is a flow diagram:
     - Author-to-paper mapping, for author-based query.
     - Embeddings for titles and abstract chunks, for semantic search.
 
-Let’s zoom in on the steps.
+## Setup
+
+- [Install PostgreSQL](https://cocoindex.io/docs/getting_started/installation#-install-postgres).
+  CocoIndex uses PostgreSQL internally for incremental processing.
+- [Configure your OpenAI API key](https://cocoindex.io/docs/ai/llm#openai). Alternatively, we have native support for Gemini, Ollama, LiteLLM. You can choose your favorite LLM provider and work completely on-premises.
 
-### Import the Papers
+  <DocumentationButton href="https://cocoindex.io/docs/ai/llm" text="LLM" margin="0 0 16px 0" />
+
+## Import the Papers
 
 ```python
 @cocoindex.flow_def(name="PaperMetadata")
@@ -65,12 +60,12 @@ def paper_metadata_flow(
     )
 ```
 
-`flow_builder.add_source` will create a table with sub fields (`filename`, `content`), 
-we can refer to the [documentation](https://cocoindex.io/docs/ops/sources) for more details.
+`flow_builder.add_source` will create a table with sub fields (`filename`, `content`).
+<DocumentationButton href="https://cocoindex.io/docs/ops/sources" text="Sources" margin="0 0 16px 0" />
 
-### Extract and collect metadata
+## Extract and collect metadata
 
-#### Extract first page for basic info
+### Extract first page for basic info
 
 Define a custom function to extract the first page and number of pages of the PDF.
 
@@ -96,20 +91,19 @@ def extract_basic_info(content: bytes) -> PaperBasicInfo:
 
 ```
 
-Now, plug this into your flow.
-We extract metadata from the first page to minimize processing cost, since the entire PDF can be very large.
+Now plug this into the flow. We extract metadata from the first page to minimize processing cost, since the entire PDF can be very large.
 
 ```python
 with data_scope["documents"].row() as doc:
     doc["basic_info"] = doc["content"].transform(extract_basic_info)
 ```
+![Extract basic info](/img/examples/academic_papers_index/basic_info.png)
 
-After this step, you should have the basic info of each paper.
+After this step, we should have the basic info of each paper.
 
 ### Parse basic info
 
-We will convert the first page to Markdown using Marker. 
-Alternatively, you can easily plug in your favorite PDF parser, such as Docling.
+We will convert the first page to Markdown using Marker. Alternatively, you can easily plug in any PDF parser, such as Docling using CocoIndex's [custom function](https://cocoindex.io/docs/custom_ops/custom_functions).
 
 Define a marker converter function and cache it, since its initialization is resource-intensive. 
 This ensures that the same converter instance is reused for different input files.
@@ -140,18 +134,20 @@ def pdf_to_markdown(content: bytes) -> str:
 Pass it to your transform
 
 ```python
-with data_scope["documents"].row() as doc:      
+with data_scope["documents"].row() as doc:    
+    # ... process
     doc["first_page_md"] = doc["basic_info"]["first_page"].transform(
             pdf_to_markdown
         )
 ```
+![First page in Markdown](/img/examples/academic_papers_index/first_page.png)
 
 After this step, you should have the first page of each paper in Markdown format.
 
-#### Extract basic info with LLM
+### Extract basic info with LLM
 
 Define a schema for LLM extraction. CocoIndex natively supports LLM-structured extraction with complex and nested schemas.
-If you are interested in learning more about nested schemas, refer to [this article](https://cocoindex.io/blogs/patient-intake-form-extraction-with-llm).
+If you are interested in learning more about nested schemas, refer to [this example](https://cocoindex.io/docs/examples/patient_form_extraction).
 
 ```python
 @dataclasses.dataclass
@@ -163,7 +159,6 @@ class PaperMetadata:
     title: str
     authors: list[Author]
     abstract: str
-
 ```
 
 Plug it into the `ExtractByLlm` function. With a dataclass defined, CocoIndex will automatically parse the LLM response into the dataclass.
@@ -181,26 +176,27 @@ doc["metadata"] = doc["first_page_md"].transform(
 ```
 
 After this step, you should have the metadata of each paper.
+![Metadata](/img/examples/academic_papers_index/metadata.png)
 
-#### Collect paper metadata
+### Collect paper metadata
 
 ```python
-  paper_metadata = data_scope.add_collector()
-  with data_scope["documents"].row() as doc:
-    # ... process
-    # Collect metadata
-    paper_metadata.collect(
-        filename=doc["filename"],
-        title=doc["metadata"]["title"],
-        authors=doc["metadata"]["authors"],
-        abstract=doc["metadata"]["abstract"],
-        num_pages=doc["basic_info"]["num_pages"],
-    )
+paper_metadata = data_scope.add_collector()
+with data_scope["documents"].row() as doc:
+# ... process
+# Collect metadata
+paper_metadata.collect(
+    filename=doc["filename"],
+    title=doc["metadata"]["title"],
+    authors=doc["metadata"]["authors"],
+    abstract=doc["metadata"]["abstract"],
+    num_pages=doc["basic_info"]["num_pages"],
+)
 ```
 
 Just collect anything you need :)
 
-#### Collect `author` to `filename` information
+### Collect `author` to `filename` information
 We’ve already extracted author list. Here we want to collect Author → Papers in a separate table to build a look up functionality. 
 Simply collect by author.
 
@@ -216,9 +212,9 @@ with data_scope["documents"].row() as doc:
 ```
 
 
-### Compute and collect embeddings
+## Compute and collect embeddings
 
-#### Title
+### Title
 
 ```python
 doc["title_embedding"] = doc["metadata"]["title"].transform(
@@ -228,7 +224,7 @@ doc["title_embedding"] = doc["metadata"]["title"].transform(
 )
 ```
 
-#### Abstract
+### Abstract
 
 Split abstract into chunks, embed each chunk and collect their embeddings. 
 Sometimes the abstract could be very long. 
@@ -252,6 +248,8 @@ doc["abstract_chunks"] = doc["metadata"]["abstract"].transform(
 
 After this step, you should have the abstract chunks of each paper.
 
+![Abstract chunks](/img/examples/academic_papers_index/abstract_chunks.png)
+
 Embed each chunk and collect their embeddings.
 
 ```python
@@ -265,7 +263,9 @@ with doc["abstract_chunks"].row() as chunk:
 
 After this step, you should have the embeddings of the abstract chunks of each paper.
 
-#### Collect embeddings
+![Abstract chunks embeddings](/img/examples/academic_papers_index/chunk_embedding.png)
+
+### Collect embeddings
 
 ```python
 metadata_embeddings = data_scope.add_collector()
@@ -292,7 +292,7 @@ with data_scope["documents"].row() as doc:
         )
 ```
 
-### Export
+## Export
 Finally, we export the data to Postgres.
 
 ```python
@@ -319,14 +319,9 @@ metadata_embeddings.export(
 )
 ```
 
-In this example we use PGVector as embedding stores/
-With CocoIndex, you can do one line switch on other supported Vector databases like Qdrant, see this [guide](https://cocoindex.io/docs/ops/targets#entry-oriented-targets) for more details.
-We aim to standardize interfaces and make it like assembling building blocks.
+In this example we use PGVector as embedding store. With CocoIndex, you can do one line switch on other supported Vector databases.
 
-## View in CocoInsight step by step
-
-You can walk through the project step by step in [CocoInsight](https://www.youtube.com/watch?v=MMrpUfUcZPk) to see 
-exactly how each field is constructed and what happens behind the scenes.
+<DocumentationButton href="https://cocoindex.io/docs/ops/targets#entry-oriented-targets" text="Entry Oriented Targets" margin="0 0 16px 0" />
 
 ## Query the index
 
@@ -338,3 +333,14 @@ For now CocoIndex doesn't provide additional query interface. We can write SQL o
 - The query space has excellent solutions for querying, reranking, and other search-related functionality.
 
 If you need assist with writing the query, please feel free to reach out to us at [Discord](https://discord.com/invite/zpA9S2DR7s).
+
+## CocoInsight
+
+You can walk through the project step by step in [CocoInsight](https://www.youtube.com/watch?v=MMrpUfUcZPk) to see exactly how each field is constructed and what happens behind the scenes.
+
+
+```sh
+cocoindex server -ci main.py
+```
+
+Follow the url `https://cocoindex.io/cocoinsight`.  It connects to your local CocoIndex server, with zero pipeline data retention.

{cocoindex-0.1.83 → cocoindex-0.2.1}/docs/docs/examples/examples/docs_to_knowledge_graph.md

@@ -35,8 +35,10 @@ and then build a knowledge graph.
 ## Setup
 *   [Install PostgreSQL](https://cocoindex.io/docs/getting_started/installation#-install-postgres). CocoIndex uses PostgreSQL internally for incremental processing. 
 *   [Install Neo4j](https://cocoindex.io/docs/ops/targets#neo4j-dev-instance), a graph database.
-*   [Configure your OpenAI API key](https://cocoindex.io/docs/ai/llm#openai). Alternatively, you can switch to Ollama, which runs LLM models locally.
-    <DocumentationButton href="https://cocoindex.io/docs/ai/llm#ollama" text="Ollama" margin="0 0 16px 0" />
+*   [Configure your OpenAI API key](https://cocoindex.io/docs/ai/llm#openai).  Alternatively, we have native support for Gemini, Ollama, LiteLLM. You can choose your favorite LLM provider and work completely on-premises.
+
+    <DocumentationButton href="https://cocoindex.io/docs/ai/llm" text="LLM" margin="0 0 16px 0" />
+
 
 ## Documentation
 <DocumentationButton href="https://cocoindex.io/docs/ops/targets#property-graph-targets" text="Property Graph Targets" margin="0 0 16px 0" />

cocoindex-0.2.1/docs/docs/examples/examples/manual_extraction.md

@@ -0,0 +1,249 @@
+---
+title: Extract Structured Data from Python Manual markdowns with Ollama
+description: Extract structured data from markdowns (Python Manual)
+sidebar_class_name: hidden
+slug: /examples/manual_extraction
+canonicalUrl: '/examples/manual_extraction'
+sidebar_custom_props:
+  image: /img/examples/manual_extraction/cover.png
+  tags: [structured-data-extraction, data-mapping]
+tags: [structured-data-extraction, data-mapping]
+---
+
+import { GitHubButton, YouTubeButton, DocumentationButton } from '../../../src/components/GitHubButton';
+
+<GitHubButton url="https://github.com/cocoindex-io/cocoindex/tree/main/examples/manuals_llm_extraction"/>
+
+## Overview
+This example shows how to extract structured data from Python Manuals using Ollama.
+
+## Flow Overview
+![Flow Overview](/img/examples/manual_extraction/flow.png)
+
+- For each PDF file:
+    - Parse to markdown.
+    - Extract structured data from the markdown using LLM.
+    - Add summary to the module info.
+    - Collect the data.
+- Export the data to a table.
+
+
+## Prerequisites
+- If you don't have Postgres installed, please refer to the [installation guide](https://cocoindex.io/docs/getting_started/installation).
+
+- [Download](https://ollama.com/download) and install Ollama. Pull your favorite LLM models by:
+    ```sh
+    ollama pull llama3.2
+    ```
+
+    <DocumentationButton href="https://cocoindex.io/docs/ai/llm#ollama" text="Ollama" margin="0 0 16px 0" />
+
+    Alternatively, CocoIndex have native support for Gemini, Ollama, LiteLLM. You can choose your favorite LLM provider and work completely on-premises.
+
+    <DocumentationButton href="https://cocoindex.io/docs/ai/llm" text="LLM" margin="0 0 16px 0" />
+
+## Add Source
+Let's add Python docs as a source.
+
+```python
+@cocoindex.flow_def(name="ManualExtraction")
+def manual_extraction_flow(
+    flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope
+):
+    """
+    Define an example flow that extracts manual information from a Markdown.
+    """
+    data_scope["documents"] = flow_builder.add_source(
+        cocoindex.sources.LocalFile(path="manuals", binary=True)
+    )
+
+    modules_index = data_scope.add_collector()
+```
+
+`flow_builder.add_source` will create a table with the following sub fields:
+- `filename` (key, type: `str`): the filename of the file, e.g. `dir1/file1.md`
+- `content` (type: `str` if `binary` is `False`, otherwise `bytes`): the content of the file
+
+<DocumentationButton href="https://cocoindex.io/docs/ops/sources" text="LocalFile" margin="0 0 16px 0" />
+
+## Parse Markdown
+
+To do this, we can plugin a custom function to convert PDF to markdown. There are so many different parsers commercially and open source available, you can bring your own parser here.
+
+```python
+class PdfToMarkdown(cocoindex.op.FunctionSpec):
+    """Convert a PDF to markdown."""
+
+
+@cocoindex.op.executor_class(gpu=True, cache=True, behavior_version=1)
+class PdfToMarkdownExecutor:
+    """Executor for PdfToMarkdown."""
+
+    spec: PdfToMarkdown
+    _converter: PdfConverter
+
+    def prepare(self):
+        config_parser = ConfigParser({})
+        self._converter = PdfConverter(
+            create_model_dict(), config=config_parser.generate_config_dict()
+        )
+
+    def __call__(self, content: bytes) -> str:
+        with tempfile.NamedTemporaryFile(delete=True, suffix=".pdf") as temp_file:
+            temp_file.write(content)
+            temp_file.flush()
+            text, _, _ = text_from_rendered(self._converter(temp_file.name))
+            return text
+```
+You may wonder why we want to define a spec + executor (instead of using a standalone function) here. The main reason is there're some heavy preparation work (initialize the parser) needs to be done before being ready to process real data.
+
+<DocumentationButton href="https://cocoindex.io/docs/custom_ops/custom_functions" text="Custom Function" margin="0 0 16px 0" />
+
+Plug in the function to the flow.
+
+```python
+with data_scope["documents"].row() as doc:
+    doc["markdown"] = doc["content"].transform(PdfToMarkdown())
+```
+
+It transforms each document to markdown.
+
+
+## Extract Structured Data from Markdown files
+### Define schema
+Let's define the schema `ModuleInfo` using Python dataclasses, and we can pass it to the LLM to extract the structured data. It's easy to do this with CocoIndex.
+
+``` python
+@dataclasses.dataclass
+class ArgInfo:
+    """Information about an argument of a method."""
+    name: str
+    description: str
+
+@dataclasses.dataclass
+class MethodInfo:
+    """Information about a method."""
+    name: str
+    args: cocoindex.typing.List[ArgInfo]
+    description: str
+
+@dataclasses.dataclass
+class ClassInfo:
+    """Information about a class."""
+    name: str
+    description: str
+    methods: cocoindex.typing.List[MethodInfo]
+
+@dataclasses.dataclass
+class ModuleInfo:
+    """Information about a Python module."""
+    title: str
+    description: str
+    classes: cocoindex.typing.List[ClassInfo]
+    methods: cocoindex.typing.List[MethodInfo]
+```
+
+### Extract structured data
+
+CocoIndex provides builtin functions (e.g. ExtractByLlm) that process data using LLM.  This example uses Ollama.
+
+```python
+with data_scope["documents"].row() as doc:
+    doc["module_info"] = doc["content"].transform(
+        cocoindex.functions.ExtractByLlm(
+            llm_spec=cocoindex.LlmSpec(
+                    api_type=cocoindex.LlmApiType.OLLAMA,
+                    # See the full list of models: https://ollama.com/library
+                    model="llama3.2"
+            ),
+            output_type=ModuleInfo,
+            instruction="Please extract Python module information from the manual."))
+```
+
+<DocumentationButton href="https://cocoindex.io/docs/core/functions#extractbyllm" text="ExtractByLlm" margin="0 0 16px 0" />
+
+![ExtractByLlm](/img/examples/manual_extraction/extraction.png)
+
+## Add summarization to module info
+Using CocoIndex as framework, you can easily add any transformation on the data, and collect it as part of the data index. Let's add some simple summary to each module - like number of classes and methods, using simple Python function.
+
+### Define Schema
+``` python
+@dataclasses.dataclass
+class ModuleSummary:
+    """Summary info about a Python module."""
+    num_classes: int
+    num_methods: int
+```
+
+### A simple custom function to summarize the data
+```python
+@cocoindex.op.function()
+def summarize_module(module_info: ModuleInfo) -> ModuleSummary:
+    """Summarize a Python module."""
+    return ModuleSummary(
+        num_classes=len(module_info.classes),
+        num_methods=len(module_info.methods),
+    )
+``` 
+
+### Plug in the function into the flow
+```python
+with data_scope["documents"].row() as doc:
+    # ... after the extraction
+    doc["module_summary"] = doc["module_info"].transform(summarize_module)
+```
+
+<DocumentationButton href="https://cocoindex.io/docs/custom_ops/custom_functions" text="Custom Function" margin="0 0 16px 0" />
+
+![Summarize Module](/img/examples/manual_extraction/summary.png)
+
+## Collect the data
+
+
+After the extraction, we need to cherrypick anything we like from the output using the `collect` function from the collector of a data scope defined above.
+
+```python
+modules_index.collect(
+    filename=doc["filename"],
+    module_info=doc["module_info"],
+)
+```
+
+Finally, let's export the extracted data to a table.
+
+```python
+modules_index.export(
+    "modules",
+    cocoindex.storages.Postgres(table_name="modules_info"),
+    primary_key_fields=["filename"],
+)
+```
+
+## Query and test your index
+Run the following command to setup and update the index.
+```sh
+cocoindex update -L main.py
+```
+You'll see the index updates state in the terminal
+
+After the index is built, you have a table with the name `modules_info`. You can query it at any time, e.g., start a Postgres shell:
+
+```bash
+psql postgres://cocoindex:cocoindex@localhost/cocoindex
+```
+
+And run the SQL query:
+
+```sql
+SELECT filename, module_info->'title' AS title, module_summary FROM modules_info;
+```
+
+## CocoInsight
+[CocoInsight](https://www.youtube.com/watch?v=ZnmyoHslBSc) is a really cool tool to help you understand your data pipeline and data index. It is in Early Access now (Free).
+
+```sh
+cocoindex server -ci main.py
+```
+CocoInsight dashboard is here `https://cocoindex.io/cocoinsight`.  It connects to your local CocoIndex server with zero data retention.
+