cocoindex 0.1.81__tar.gz → 0.1.82__tar.gz

cocoindex-0.1.82/CONTRIBUTING.md

@@ -0,0 +1 @@
+We love contributions from our community ❤️. Please check out our [contributing guide](https://cocoindex.io/docs/contributing/guide).

{cocoindex-0.1.81 → cocoindex-0.1.82}/Cargo.lock

@@ -1283,7 +1283,7 @@ dependencies = [
 
 [[package]]
 name = "cocoindex"
-version = "0.1.81"
+version = "0.1.82"
 dependencies = [
  "anyhow",
  "async-openai",

{cocoindex-0.1.81 → cocoindex-0.1.82}/Cargo.toml

@@ -2,7 +2,7 @@
 name = "cocoindex"
 # Version used for local development is always higher than others to take precedence.
 # Will be overridden for specific release versions.
-version = "0.1.81"
+version = "0.1.82"
 edition = "2024"
 rust-version = "1.88"
 readme = "README.md"

{cocoindex-0.1.81 → cocoindex-0.1.82}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cocoindex
-Version: 0.1.81
+Version: 0.1.82
 Requires-Dist: click>=8.1.8
 Requires-Dist: rich>=14.0.0
 Requires-Dist: python-dotenv>=1.1.0
@@ -79,17 +79,17 @@ Ultra performant data transformation framework for AI, with core engine written
 
 </br>
 
-CocoIndex makes it super easy to transform data with AI workloads, and keep source data and target in sync effortlessly.
+CocoIndex makes it effortless to transform data with AI, and keep source data and target in sync. Whether you’re building a vector index for RAG, creating knowledge graphs, or performing any custom data transformations — goes beyond SQL.
 
 </br>
 
 <p align="center">
-    <img src="https://cocoindex.io/images/venn-features.png" alt="CocoIndex Features" width='400'>
+<img width="4187" height="1883" alt="CocoIndex Features" src="https://github.com/user-attachments/assets/6147673f-0daf-4313-a0c7-7c4205ba2e31" />
 </p>
 
 </br>
 
-Either creating embedding, building knowledge graphs, or any data transformations - beyond traditional SQL.
+
 
 ## Exceptional velocity
 Just declare transformation in dataflow with ~100 lines of python
@@ -114,8 +114,8 @@ CocoIndex follows the idea of [Dataflow](https://en.wikipedia.org/wiki/Dataflow_
 
 **Particularly**, developers don't explicitly mutate data by creating, updating and deleting. They just need to define transformation/formula for a set of source data.
 
-## Build like LEGO
-Native builtins for different source, targets and transformations. Standardize interface, make it 1-line code switch between different components.
+## Plug-and-Play Building Blocks
+Native builtins for different source, targets and transformations. Standardize interface, make it 1-line code switch between different components - as easy as assembling building blocks.
 
 <p align="center">
     <img src="https://cocoindex.io/images/components.svg" alt="CocoIndex Features">

{cocoindex-0.1.81 → cocoindex-0.1.82}/README.md

@@ -50,17 +50,17 @@ Ultra performant data transformation framework for AI, with core engine written
 
 </br>
 
-CocoIndex makes it super easy to transform data with AI workloads, and keep source data and target in sync effortlessly.
+CocoIndex makes it effortless to transform data with AI, and keep source data and target in sync. Whether you’re building a vector index for RAG, creating knowledge graphs, or performing any custom data transformations — goes beyond SQL.
 
 </br>
 
 <p align="center">
-    <img src="https://cocoindex.io/images/venn-features.png" alt="CocoIndex Features" width='400'>
+<img width="4187" height="1883" alt="CocoIndex Features" src="https://github.com/user-attachments/assets/6147673f-0daf-4313-a0c7-7c4205ba2e31" />
 </p>
 
 </br>
 
-Either creating embedding, building knowledge graphs, or any data transformations - beyond traditional SQL.
+
 
 ## Exceptional velocity
 Just declare transformation in dataflow with ~100 lines of python
@@ -85,8 +85,8 @@ CocoIndex follows the idea of [Dataflow](https://en.wikipedia.org/wiki/Dataflow_
 
 **Particularly**, developers don't explicitly mutate data by creating, updating and deleting. They just need to define transformation/formula for a set of source data.
 
-## Build like LEGO
-Native builtins for different source, targets and transformations. Standardize interface, make it 1-line code switch between different components.
+## Plug-and-Play Building Blocks
+Native builtins for different source, targets and transformations. Standardize interface, make it 1-line code switch between different components - as easy as assembling building blocks.
 
 <p align="center">
     <img src="https://cocoindex.io/images/components.svg" alt="CocoIndex Features">

{cocoindex-0.1.81 → cocoindex-0.1.82}/docs/docs/about/community.md

@@ -12,7 +12,7 @@ We are super excited for community contributions of all kinds - whether it's cod
 We would love to foster an inclusive, welcoming, and supportive environment. Contributing to CocoIndex should feel collaborative, friendly and enjoyable for everyone. Together, we can build better AI applications through robust data infrastructure.
 
 :::tip Start hacking CocoIndex
-Check out our [Contributing guide](./contributing) to get started!
+Check out our [Contributing guide](/docs/contributing/guide) to get started!
 :::
 
 

cocoindex-0.1.81/docs/docs/about/contributing.md → cocoindex-0.1.82/docs/docs/contributing/guide.md

@@ -1,68 +1,20 @@
 ---
-title: Contributing
-description: Learn how to contribute to CocoIndex
+title: Contributing Guide
+description: How to contribute to CocoIndex
 ---
 
-# Contributing
-
 [CocoIndex](https://github.com/cocoindex-io/cocoindex) is an open source project. We are respectful, open and friendly. This guide explains how to get involved and contribute to [CocoIndex](https://github.com/cocoindex-io/cocoindex).
 
-## Issues:
-
-We use [GitHub Issues](https://github.com/cocoindex-io/cocoindex/issues) to track bugs and feature requests.
+Our [Discord server](https://discord.com/invite/zpA9S2DR7s) is constantly open. 
+If you are unsure about anything, it is a good place to discuss! We'd love to collaborate and will always be friendly.
 
 ## Good First Issues
 
 We tag issues with the ["good first issue"](https://github.com/cocoindex-io/cocoindex/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) label for beginner contributors.
 
 ## How to Contribute
-- If you decide to work on an issue, unless the PR can be sent immediately (e.g. just a few lines of code), we recommend you to leave a comment on the issue like **`I'm working on it`**  or **`Can I work on this issue?`** to avoid duplicating work.
+- If you decide to take an issue, we recommend you to leave a comment on the issue like  **`Can I work on this issue?`** so we could assign it to you. This helps you and others avoid duplicating work.
 - For larger features, we recommend you to discuss with us first in our [Discord server](https://discord.com/invite/zpA9S2DR7s) to coordinate the design and work.
-- Our [Discord server](https://discord.com/invite/zpA9S2DR7s) is constantly open. If you are unsure about anything, it is a good place to discuss! We'd love to collaborate and will always be friendly.
-
-## Start hacking! Setting Up Development Environment
-Follow the steps below to get cocoindex built on the latest codebase locally - if you are making changes to cocoindex functionality and want to test it out.
-
--   🦀 [Install Rust](https://rust-lang.org/tools/install)
-
-    If you don't have Rust installed, run
-    ```sh
-    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
-    ```
-    Already have Rust? Make sure it's up to date
-    ```sh
-    rustup update
-    ```
-
--   Setup Python virtual environment:
-    ```sh
-    python3 -m venv .venv
-    ```
-    Activate the virtual environment, before any installing / building / running:
-
-    ```sh
-    . .venv/bin/activate
-    ```
-
--   Install required tools:
-    ```sh
-    pip install maturin
-    ```
-
--   Build the library. Run at the root of cocoindex directory:
-    ```sh
-    maturin develop -E all,dev
-    ```
-
--   Install and enable pre-commit hooks. This ensures all checks run automatically before each commit:
-    ```sh
-    pre-commit install
-    ```
-
--   Before running a specific example, set extra environment variables, for exposing extra traces, allowing dev UI, etc.
-    ```sh
-    . ./.env.lib_debug
-    ```
 
 ## Submit Your Code
 CocoIndex is committed to the highest standards of code quality. Please ensure your code is thoroughly tested before submitting a PR.
@@ -89,6 +41,7 @@ In your PR description, please include:
 - Note if it's a breaking change
 - Reference any related GitHub issues
 
+
 A core team member will review your PR within one business day and provide feedback on any required changes. Once approved and all tests pass, the reviewer will squash and merge your PR into the main branch.
 
 Your contribution will then be part of CocoIndex! We'll highlight your contribution in our release notes 🌴.

cocoindex-0.1.82/docs/docs/contributing/new_built_in_target.mdx

@@ -0,0 +1,136 @@
+---
+title: New Built-in Target
+description: Learn how to create a new built-in target in CocoIndex.
+toc_max_heading_level: 4
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+A target connects CocoIndex flow and external systems, and needs to synchronize changes of data from the CocoIndex flow to outside.
+
+## Built-in Targets and Custom Targets
+
+CocoIndex allows you to create [custom targets](/docs/custom_ops/custom_targets) by Python SDK.
+Custom targets share the gist as built-in targets, but the interface is simplified, so it only supports a subset of capabilities.
+The most notable difference is that custom target API exposed by Python SDK doesn't provide schema of the data to be exported, so it only fits most of the use cases where the target schema is fixed.
+But the built-in targets get the schema, so it can be used to define general-purpose targets.
+
+You're recommended to read the [custom target documentation](/docs/custom_ops/custom_targets) first as a starting point.
+
+## Life of a Built-in Target
+
+The core logic of a built-in target is the *Target Connector*, which is an implementation of the trait [`TargetFactoryBase`](https://github.com/search?q=repo%3Acocoindex-io%2Fcocoindex%20%22trait%20TargetFactoryBase%22&type=code).
+
+*Users* provide a *Target Spec* when they export specific data (in the form of [Data Collector](/docs/core/flow_def#data-collector)) to the target.
+
+```mermaid
+flowchart TD
+    FlowAnalyzer[CocoIndex Flow Analyzer] --> DataSchema[Data Schema]
+    ExecutionEngine[CocoIndex Execution Engine] --> Mutations[mutations]
+
+    TargetSpec[Target Spec] --> BuildMethod(build#40;#41;)
+    DataSchema --> BuildMethod
+
+    BuildMethod --> SetupKeyState[Setup Key & State]
+    BuildMethod --> ExportContext[Export Context]
+
+    SetupKeyState --> InternalStorage[(Internal Storage)]
+    SetupKeyState --> DiffMethod(diff_setup_states#40;#41;)
+    InternalStorage --> |previous versions| DiffMethod
+
+    DiffMethod --> SetupChange[Setup Change]
+
+    SetupChange --> ApplySetupMethod(apply_setup_changes#40;#41;)
+    ApplySetupMethod --> |setup changes #40;DDL#41;| TargetInfra[Target Infrastructure]
+
+    ExportContext --> ApplyMutationMethod(apply_mutation#40;#41;)
+    Mutations --> ApplyMutationMethod
+    ApplyMutationMethod --> |data changes #40;DML#41;| TargetInfra
+
+    %% Styling
+    classDef cocoClass fill:#aaa,color:black
+    classDef storageClass fill:#aaa,color:black
+    classDef methodClass fill:#ffc,color:black,font-weight:bold
+
+    class BuildMethod,DiffMethod,ApplySetupMethod,ApplyMutationMethod methodClass
+    class InternalStorage,TargetInfra storageClass
+    class TargetSpec,SetupKeyState,ExportContext,SetupChange,DataSchema,Mutations dataClass
+    class FlowAnalyzer,ExecutionEngine cocoClass
+```
+
+
+### States / Contexts Building
+
+*CocoIndex* analyzes the flow before execution. For each target, it will have the information of *Data Schema* exported to the target, and the *Target Spec*.
+The *Target Connector* (an implementation of `TargetFactoryBase`) takes them by the `build()` method, to digest this information and return necessary information for setup changes and data changes, including a *Setup Key*, a *Setup State* and a *Export Context* for each target.
+
+*Setup Key* and *Setup State* provides information to decide how to setup the target.
+Both *Setup Key* and *Setup State* are persisted in CocoIndex's [internal storage](/docs/core/basics#internal-storage), to keep track of targets' states.
+
+-   *Setup Key* is a key used to uniquely identify the target. It should remain stable across different calls.
+    If it changes, CocoIndex considers it as a different target.
+    For example, for Postgres, we use the table name as the setup key.
+
+-   *Setup State* is a state of the target.
+    *Setup Key* and *Setup State* should contain sufficient information to decide how to setup the target.
+    For example, for Postgres, we use the table name and its schema as the setup state.
+
+    :::note
+
+    Note that different CocoIndex data types may be projected to the same column type in Postgres, e.g. all _Struct_ and _Table_ types are projected to `jsonb`.
+    It's sufficient to keep `jsonb` instead of the specific CocoIndex type in *Setup State*, so the *Setup State* doesn't need to be changed too often.
+
+    For historical reasons, we already put the specific CocoIndex type in *Setup State* for Postgres targets, and it's not easy to update it with a smooth migration, so we keep it for now.
+    But for new targets, it's recommended to keep just sufficient information to define the target.
+
+    :::
+
+*Export Context* is an in-memory object that contains necessary information and objects to apply data changes.
+For example, it may hold a database connection pool, or a prepared statement for upserting/deleting data, etc.
+
+### Apply Setup Changes
+
+*CocoIndex* calls the *Target Connector* (`diff_setup_states()`) to compare the current *Setup State* with previous ones, which returns *Setup Change* object to describe the change (or no change).
+For example, for Postgres, the *Setup Change* object contains information such as which table needs to be created/dropped, which column needs to be added/dropped, etc.
+
+*CocoIndex* further passes the *Setup Change* to the *Target Connector* (`apply_setup_changes()`), which is responsible for applying the *Setup Change* to the target infrastructure.
+Here's the place for the target connector to issue the actual commands to setup, drop or update the target infrastructure.
+
+:::note
+
+Here CocoIndex may pass multiple versions of possible existing states to the target connector via `diff_setup_states()`. This is because there may be failures or interrupts in the previous setup attempts, so CocoIndex doesn't always clearly know the exact state of the target. CocoIndex needs to keep multiple versions of states on track (the existing one and the pending ones) until the next `apply_setup_changes()` is successful.
+
+For example, consider the following scenarios:
+
+-   A Postgres target's initial state describes it has a column `col_1`.
+-   Later, after the flow changes, a new column `col_2` is collected, so a new version of target state has columns `col_1` and `col_2`. But the attempt to apply this setup change doesn't go through (e.g. connection to the database is broken, in general CocoIndex doesn't know the exact state the backend ends up with).
+-   Now the user updates the column name from `col_2` to `col_3` in their flow. In the next attempt to apply the setup change, we get a setup state with `col_1` and `col_3`.
+
+    -   To apply setup changes for the latest, CocoIndex will pass two versions existing setup states to the target connector via `diff_setup_states()`, one with `col_1`, the other with `col_1` and `col_2`. The new setup state has `col_1` and `col_3`. And the target connector needs to create a setup change that take both of the possible existing states into account. The change should be "drop `col_2` and add `col_3`".
+    -   When applying the setup change in `apply_setup_changes()`, the target connector should do this idempotently, i.e. it should be a no-op if the column `col_2` already doesn't exist (by `DROP IF EXISTS`), so it works well on either possible existing state.
+    -   After this setup change is applied successfully, CocoIndex clearly knows the target's current state (i.e. with columns `col_1` and `col_3`), and safely forgets all previous states.
+
+:::
+
+### Apply Data Changes
+
+*CocoIndex* calls the *Target Connector* (`apply_mutation()`) to apply data changes, by list of mutations (upserts and deletes) together with their *Export Context*.
+The method should apply these mutations in an idempotent way, i.e. it should be a no-op if the mutation is already applied.
+For example, for Postgres, upserts are done by `INSERT ... ON CONFLICT ... DO UPDATE SET ...`, and deletes are done by `DELETE ... WHERE ...`.
+
+### Notes on Multiple Targets
+
+The interface of multiple methods in `TargetFactoryBase` (`build()`, `apply_setup_changes()`, `apply_mutation()`) takes a batch of inputs coming from multiple targets of the same type. This gives a chance for the connector to handle dependencies between them (if any) and apply changes in the correct order.
+For example, to support property graph databases, relationships depend on nodes, so nodes need to be added before relationships, and on deletion it happens in reverse order.
+This is not a consideration for target types that different targets are independent of each other.
+
+## Examples
+
+The following target implementations provide good examples:
+
+-   [Postgres](/docs/ops/targets#postgres), see [related code](https://github.com/search?q=repo%3Acocoindex-io%2Fcocoindex+path%3A%2Ftarget%2F+Postgres&type=code).
+    It provides a good example for targets with specific column types in the schema.
+
+-   [Qdrant](/docs/ops/targets#qdrant), see [related code](https://github.com/search?q=repo%3Acocoindex-io%2Fcocoindex+path%3A%2Ftarget%2F+Qdrant&type=code).
+    It provides a good example for targets without specific column types in the schema, as Qdrant's payloads are JSON objects.

cocoindex-0.1.82/docs/docs/contributing/setup_dev_environment.md

@@ -0,0 +1,47 @@
+---
+title: Setup Development Environment
+description: Learn how to setup your development environment to develop CocoIndex
+---
+
+Follow the steps below to get CocoIndex built on the latest codebase locally - if you are making changes to CocoIndex functionality and want to test it out.
+
+-   🦀 [Install Rust](https://rust-lang.org/tools/install)
+
+    If you don't have Rust installed, run
+    ```sh
+    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
+    ```
+    Already have Rust? Make sure it's up to date
+    ```sh
+    rustup update
+    ```
+
+-   Setup Python virtual environment:
+    ```sh
+    python3 -m venv .venv
+    ```
+    Activate the virtual environment, before any installing / building / running:
+
+    ```sh
+    . .venv/bin/activate
+    ```
+
+-   Install required tools:
+    ```sh
+    pip install maturin
+    ```
+
+-   Build the library. Run at the root of cocoindex directory:
+    ```sh
+    maturin develop -E all,dev
+    ```
+
+-   Install and enable pre-commit hooks. This ensures all checks run automatically before each commit:
+    ```sh
+    pre-commit install
+    ```
+
+-   Before running a specific example, set extra environment variables, for exposing extra traces, allowing dev UI, etc.
+    ```sh
+    . ./.env.lib_debug
+    ```

{cocoindex-0.1.81 → cocoindex-0.1.82}/docs/docs/custom_ops/custom_targets.mdx

@@ -8,12 +8,29 @@ import Tabs from '@theme/Tabs';
 import TabItem from '@theme/TabItem';
 
 A custom target allows you to export data to any destination you want, such as databases, cloud storage, file systems, APIs, or other external systems.
+You can either continuously update the destination to keep it in sync with the latest exported data, or simply publish the changes as a changelog to somewhere.
+
+## Overview
 
 Custom targets are defined by two components:
 
 *   A **target spec** that configures the behavior and connection parameters for the target.
 *   A **target connector** that handles the actual data export operations.
 
+When you define a flow within CocoIndex, you define how data are transformed, collected and exported, without worrying about how to handle data change (insert, update, delete). CocoIndex handles it for you.
+However, a target connects CocoIndex flow and external systems, and needs to synchronize changes of data from the CocoIndex flow to outside.
+The implementation of a target connector needs to deal with changes, in two aspects:
+
+-   **Setup changes**.
+    They're for basic setup of a target's corresponding infrastructure (e.g. a table, a directory) without specific data.
+    When users add a new target, delete an existing target, or make changes to the target spec,
+    the framework will trigger the connector to apply these setup changes by calling `apply_setup_change()`.
+    The connector needs apply corresponding setup changes to the external system, e.g. create/delete a table, update/delete a directory, etc.
+
+-   **Data changes**.
+    They're changes of specific data exported to the target.
+    During the flow is running, when new rows-to-export appear, or existing ones are updated or deleted in the CocoIndex flow, the framework will trigger the connector to apply these data changes by calling `mutate()`, e.g. insert/update/delete a row in a table, write/delete a file, etc.
+
 ## Target Spec
 
 The target spec defines the configuration parameters for your custom target. When you use this target in a flow (typically by calling [`export()`](/docs/core/flow_def#export)), you instantiate this target spec with specific parameter values.
@@ -44,7 +61,7 @@ Notes:
 
 A target connector handles the actual data export operations for your custom target. It defines how data should be written to your target destination.
 
-Target connectors implement two categories of methods: **setup methods** for managing target infrastructure (similar to DDL operations in databases), and **data methods** for handling specific data operations (similar to DML operations).
+Target connectors implement two categories of methods: setup methods to deal with setup changes, and data methods to deal with data changes.
 
 <Tabs>
 <TabItem value="python" label="Python" default>
@@ -158,7 +175,21 @@ def prepare(spec: CustomTarget) -> PreparedCustomTarget:
 
 If not provided, the original spec will be passed directly to `mutate`.
 
-### Complete Example
+## Best Practices
+
+### Idempotency of Methods with Side Effects
+
+`apply_setup_change()` and `mutate()` are the two methods that are expected to produce side effects.
+We expect them to be idempotent, i.e. when calling them with the same arguments multiple times, the effect should remain the same.
+
+For example,
+- For `apply_setup_change()`, if the target is a directory, it should be a no-op if we try to create it (`previous` is `None`) when the directory already exists, and also a no-op if we try to delete it (`current` is `None`) when the directory does not exist.
+- For `mutate()`, if a mutation is a deletion, it should be a no-op if the row does not exist.
+
+This is to make sure when the system if left in an intermediate state, e.g. interrupted in the middle between a change is made and CocoIndex notes down the change is completed, the targets can still be gracefully rolled forward to the desired states after the system is resumed.
+
+
+## Example
 
 In this example, we define a custom target that accepts data with the following fields:
 - `filename` (key field)
@@ -247,21 +278,4 @@ For simplicity, the type hints can be omitted and a `dict` will be created inste
 </TabItem>
 </Tabs>
 
-## Best Practices
-
-### Idempotency of Methods with Side Effects
-
-`apply_setup_change()` and `mutate()` are the two methods that are expected to produce side effects.
-We expect them to be idempotent, i.e. when calling them with the same arguments multiple times, the effect should remain the same.
-
-For example,
-- For `apply_setup_change()`, if the target is a directory, it should be a no-op if we try to create it (`previous` is `None`) when the directory already exists, and also a no-op if we try to delete it (`current` is `None`) when the directory does not exist.
-- For `mutate()`, if a mutation is a deletion, it should be a no-op if the row does not exist.
-
-This is to make sure when the system if left in an intermediate state, e.g. interrupted in the middle between a change is made and CocoIndex notes down the change is completed, the targets can still be gracefully rolled forward to the desired states after the system is resumed.
-
-## Examples
-
-The cocoindex repository contains the following examples of custom targets:
-
-*   In the [custom_output_files](https://github.com/cocoindex-io/cocoindex/blob/main/examples/custom_output_files/main.py) example, `LocalFileTarget` exports data to local HTML files.
+See the [custom_output_files](https://github.com/cocoindex-io/cocoindex/blob/main/examples/custom_output_files/main.py) for the an end-to-end example.