cocoindex 0.1.48__tar.gz → 0.1.49__tar.gz

{cocoindex-0.1.48 → cocoindex-0.1.49}/Cargo.lock

@@ -993,7 +993,7 @@ dependencies = [
 
 [[package]]
 name = "cocoindex"
-version = "0.1.48"
+version = "0.1.49"
 dependencies = [
  "anyhow",
  "async-openai",

{cocoindex-0.1.48 → cocoindex-0.1.49}/Cargo.toml

@@ -2,7 +2,7 @@
 name = "cocoindex"
 # Version used for local development is always higher than others to take precedence.
 # Will be overridden for specific release versions.
-version = "0.1.48"
+version = "0.1.49"
 edition = "2024"
 
 [profile.release]

{cocoindex-0.1.48 → cocoindex-0.1.49}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cocoindex
-Version: 0.1.48
+Version: 0.1.49
 Requires-Dist: sentence-transformers>=3.3.1
 Requires-Dist: click>=8.1.8
 Requires-Dist: rich>=14.0.0

cocoindex-0.1.49/docs/docs/getting_started/overview.md

@@ -0,0 +1,34 @@
+---
+title: Overview
+slug: /
+---
+
+# Welcome to CocoIndex
+
+CocoIndex is an ultra-performant real-time data transformation framework for AI, with incremental processing. 
+
+As a data framework, CocoIndex takes it to the next level on data freshness. **Incremental processing** is one of the core values provided by CocoIndex.
+
+## Programming Model
+CocoIndex follows the idea of [Dataflow programming](https://en.wikipedia.org/wiki/Dataflow_programming) model. Each transformation creates a new field solely based on input fields, without hidden states and value mutation. All data before/after each transformation is observable, with lineage out of the box.
+
+The gist of an example data transformation:
+```python
+# import
+data['content'] = flow_builder.add_source(...) 
+
+# transform
+data['out'] = data['content'] 
+    .transform(...)
+    .transform(...)
+
+# collect data
+collector.collect(...)
+
+# export to db, vector db, graph db ...
+collector.export(...)
+```
+
+Get Started:
+- [Quick Start](https://cocoindex.io/docs/getting_started/quickstart)
+

{cocoindex-0.1.48 → cocoindex-0.1.49}/docs/docs/ops/functions.md

@@ -39,9 +39,27 @@ Input data:
 
 *   `chunk_overlap` (type: `int`, optional): The maximum overlap size between adjacent chunks, in bytes.
 *   `language` (type: `str`, optional): The language of the document.
-    Can be a langauge name (e.g. `Python`, `Javascript`, `Markdown`) or a file extension (e.g. `.py`, `.js`, `.md`).
-    To see all supported language names and extensions, see [the code](https://github.com/search?q=org%3Acocoindex-io+lang%3Arust++%22static+TREE_SITTER_LANGUAGE_BY_LANG%22&type=code).
-    If it's unspecified or the specified language is not supported, it will be treated as plain text.
+    Can be a language name (e.g. `Python`, `Javascript`, `Markdown`) or a file extension (e.g. `.py`, `.js`, `.md`).
+
+*   `custom_languages` (type: `list[CustomLanguageSpec]`, optional): This allows you to customize the way to chunking specific languages using regular expressions. Each `CustomLanguageSpec` is a dict with the following fields:
+    *   `language_name` (type: `str`, required): Name of the language.
+    *   `aliases` (type: `list[str]`, optional): A list of aliases for the language.
+        It's an error if any language name or alias is duplicated.
+
+    *   `separators_regex` (type: `list[str]`, required): A list of regex patterns to split the text.
+        Higher-level boundaries should come first, and lower-level should be listed later. e.g. `[r"\n# ", r"\n## ", r"\n\n", r"\. "]`.
+        See [regex Syntax](https://docs.rs/regex/latest/regex/#syntax) for supported regular expression syntax.
+
+    :::note
+
+    We use the `language` field to determine how to split the input text, following these rules:
+    
+    *   We'll match the input `language` field against the `language_name` or `aliases` of each custom language specification, and use the matched one. If value of `language` is null, it'll be treated as empty string when matching `language_name` or `aliases`.
+    *   If no match is found, we'll match the `language` field against the builtin language configurations.
+        For all supported builtin language names and aliases (extensions), see [the code](https://github.com/search?q=org%3Acocoindex-io+lang%3Arust++%22static+TREE_SITTER_LANGUAGE_BY_LANG%22&type=code).
+    *   If no match is found, the input will be treated as plain text.
+
+    :::
 
 Return type: [KTable](/docs/core/data_types#ktable), each row represents a chunk, with the following sub fields:
 

{cocoindex-0.1.48 → cocoindex-0.1.49}/docs/docs/ops/storages.md

@@ -54,34 +54,21 @@ Here's how CocoIndex data elements map to Qdrant elements during export:
 |-------------------|------------------|
 | an export target  | a unique collection |  
 | a collected row   | a point |
-| a field           | a named vector (for fields with vector type); a field within payload (otherwise) |
+| a field           | a named vector, if fits into Qdrant vector; or a field within payload otherwise |
+
+A vector with `Float32`, `Float64` or `Int64` type, and with fixed dimension, fits into Qdrant vector.
 
 #### Spec
 
 The spec takes the following fields:
 
-*   `collection_name` (type: `str`, required): The name of the collection to export the data to.
-
-*   `grpc_url` (type: `str`, optional): The [gRPC URL](https://qdrant.tech/documentation/interfaces/#grpc-interface) of the Qdrant instance. Defaults to `http://localhost:6334/`.
-
-*   `api_key` (type: `str`, optional). API key to authenticate requests with.
+*   `connection` (type: [auth reference](../core/flow_def#auth-registry) to `QdrantConnection`, optional): The connection to the Qdrant instance. `QdrantConnection` has the following fields:
+    *   `grpc_url` (type: `str`): The [gRPC URL](https://qdrant.tech/documentation/interfaces/#grpc-interface) of the Qdrant instance, e.g. `http://localhost:6334/`.
+    *   `api_key` (type: `str`, optional). API key to authenticate requests with.
 
-Before exporting, you must create a collection with a [vector name](https://qdrant.tech/documentation/concepts/vectors/#named-vectors) that matches the vector field name in CocoIndex, and set `setup_by_user=True` during export.
+    If `connection` is not provided, will use local Qdrant instance at `http://localhost:6334/` by default.
 
-Example:
-
-```python
-doc_embeddings.export(
-    "doc_embeddings",
-    cocoindex.storages.Qdrant(
-        collection_name="cocoindex",
-        grpc_url="https://xyz-example.cloud-region.cloud-provider.cloud.qdrant.io:6334/",
-        api_key="<your-api-key-here>",
-    ),
-    primary_key_fields=["id_field"],
-    setup_by_user=True,
-)
-```
+*   `collection_name` (type: `str`, required): The name of the collection to export the data to.
 
 You can find an end-to-end example [here](https://github.com/cocoindex-io/cocoindex/tree/main/examples/text_embedding_qdrant).
 
@@ -399,19 +386,7 @@ You can find end-to-end examples fitting into any of supported property graphs i
 
 ### Neo4j
 
-If you don't have a Neo4j database, you can start a Neo4j database using our docker compose config:
-
-```bash
-docker compose -f <(curl -L https://raw.githubusercontent.com/cocoindex-io/cocoindex/refs/heads/main/dev/neo4j.yaml) up -d
-```
-
-:::warning
-
-The docker compose config above will start a Neo4j Enterprise instance under the [Evaluation License](https://neo4j.com/terms/enterprise_us/),
-with 30 days trial period.
-Please read and agree the license before starting the instance.
-
-:::
+#### Spec
 
 The `Neo4j` target spec takes the following fields:
 
@@ -430,17 +405,32 @@ Neo4j also provides a declaration spec `Neo4jDeclaration`, to configure indexing
     *   `primary_key_fields` (required)
     *   `vector_indexes` (optional)
 
-### Kuzu
+#### Neo4j dev instance
 
-CocoIndex supports talking to Kuzu through its [API server](https://github.com/kuzudb/api-server).
-You can bring up a Kuzu API server locally by running:
+If you don't have a Neo4j database, you can start a Neo4j database using our docker compose config:
 
 ```bash
-KUZU_DB_DIR=$HOME/.kuzudb
-KUZU_PORT=8123
-docker run -d --name kuzu -p ${KUZU_PORT}:8000 -v ${KUZU_DB_DIR}:/database kuzudb/api-server:latest
+docker compose -f <(curl -L https://raw.githubusercontent.com/cocoindex-io/cocoindex/refs/heads/main/dev/neo4j.yaml) up -d
 ```
 
+If will bring up a Neo4j instance, which can be accessed by username `neo4j` and password `cocoindex`.
+You can access the Neo4j browser at [http://localhost:7474](http://localhost:7474).
+
+:::warning
+
+The docker compose config above will start a Neo4j Enterprise instance under the [Evaluation License](https://neo4j.com/terms/enterprise_us/),
+with 30 days trial period.
+Please read and agree the license before starting the instance.
+
+:::
+
+
+### Kuzu
+
+#### Spec
+
+CocoIndex supports talking to Kuzu through its [API server](https://github.com/kuzudb/api-server).
+
 The `Kuzu` target spec takes the following fields:
 
 *   `connection` (type: [auth reference](../core/flow_def#auth-registry) to `KuzuConnectionSpec`): The connection to the Kuzu database. `KuzuConnectionSpec` has the following fields:
@@ -453,3 +443,25 @@ Kuzu also provides a declaration spec `KuzuDeclaration`, to configure indexing o
 *   Fields for [nodes to declare](#declare-extra-node-labels), including
     *   `nodes_label` (required)
     *   `primary_key_fields` (required)
+
+#### Kuzu dev instance
+
+If you don't have a Kuzu instance yet, you can bring up a Kuzu API server locally by running:
+
+```bash
+KUZU_DB_DIR=$HOME/.kuzudb
+KUZU_PORT=8123
+docker run -d --name kuzu -p ${KUZU_PORT}:8000 -v ${KUZU_DB_DIR}:/database kuzudb/api-server:latest
+```
+
+To explore the graph you built with Kuzu, you can use the [Kuzu Explorer](https://github.com/kuzudb/explorer).
+Currently Kuzu API server and the explorer cannot be up at the same time. So you need to stop the API server before running the explorer.
+
+To start the instance of the explorer, run:
+
+```bash
+KUZU_EXPLORER_PORT=8124
+docker run -d --name kuzu-explorer -p ${KUZU_EXPLORER_PORT}:8000  -v ${KUZU_DB_DIR}:/database -e MODE=READ_ONLY  kuzudb/explorer:latest
+```
+
+You can then access the explorer at [http://localhost:8124](http://localhost:8124).

{cocoindex-0.1.48 → cocoindex-0.1.49}/examples/docs_to_knowledge_graph/README.md

@@ -12,10 +12,10 @@ Please drop [Cocoindex on Github](https://github.com/cocoindex-io/cocoindex) a s
 
 ![example-explanation](https://github.com/user-attachments/assets/07ddbd60-106f-427f-b7cc-16b73b142d27)
 
-
 ## Prerequisite
 *   [Install Postgres](https://cocoindex.io/docs/getting_started/installation#-install-postgres) if you don't have one.
-*   [Install Neo4j](https://cocoindex.io/docs/ops/storages#neo4j) if you don't have one.
+*   Install [Neo4j](https://cocoindex.io/docs/ops/storages#neo4j-dev-instance) or [Kuzu](https://cocoindex.io/docs/ops/storages#kuzu-dev-instance) if you don't have one.
+    *   The example uses Neo4j by default for now. If you want to use Kuzu, find out the "SELECT ONE GRAPH DATABASE TO USE" section and switch the active branch.
 *   [Configure your OpenAI API key](https://cocoindex.io/docs/ai/llm#openai).
 
 ## Documentation
@@ -45,21 +45,18 @@ cocoindex update main.py
 
 ### Browse the knowledge graph
 
-After the knowledge graph is build, you can explore the knowledge graph you built in Neo4j Browser.
+After the knowledge graph is built, you can explore the knowledge graph.
 
-For the dev enviroment, you can connect neo4j browser using credentials:
-- username: `neo4j`
-- password: `cocoindex`
-which is pre-configured in the our docker compose [config.yaml](https://raw.githubusercontent.com/cocoindex-io/cocoindex/refs/heads/main/dev/neo4j.yaml).
+* If you're using Neo4j, you can open the explorer at [http://localhost:7474](http://localhost:7474), with username `neo4j` and password `cocoindex`.
+* If you're using Kuzu, you can start a Kuzu explorer locally. See [Kuzu dev instance](https://cocoindex.io/docs/ops/storages#kuzu-dev-instance) for more details.
 
-You can open it at [http://localhost:7474](http://localhost:7474), and run the following Cypher query to get all relationships:
+You can run the following Cypher query to get all relationships:
 
 ```cypher
 MATCH p=()-->() RETURN p
 ```
-<img width="1366" alt="neo4j-for-coco-docs" src="https://github.com/user-attachments/assets/3c8b6329-6fee-4533-9480-571399b57e57" />
-
 
+<img width="1366" alt="neo4j-for-coco-docs" src="https://github.com/user-attachments/assets/3c8b6329-6fee-4533-9480-571399b57e57" />
 
 ## CocoInsight 
 I used CocoInsight (Free beta now) to troubleshoot the index generation and understand the data lineage of the pipeline. 

{cocoindex-0.1.48 → cocoindex-0.1.49}/examples/docs_to_knowledge_graph/main.py

@@ -5,27 +5,6 @@ This example shows how to extract relationships from documents and build a knowl
 import dataclasses
 import cocoindex
 
-
-@dataclasses.dataclass
-class DocumentSummary:
-    """Describe a summary of a document."""
-
-    title: str
-    summary: str
-
-
-@dataclasses.dataclass
-class Relationship:
-    """
-    Describe a relationship between two entities.
-    Subject and object should be Core CocoIndex concepts only, should be nouns. For example, `CocoIndex`, `Incremental Processing`, `ETL`,  `Data` etc.
-    """
-
-    subject: str
-    predicate: str
-    object: str
-
-
 neo4j_conn_spec = cocoindex.add_auth_entry(
     "Neo4jConnection",
     cocoindex.storages.Neo4jConnection(
@@ -41,19 +20,43 @@ kuzu_conn_spec = cocoindex.add_auth_entry(
     ),
 )
 
-# Use Neo4j as the graph database
+# SELECT ONE GRAPH DATABASE TO USE
+# This example can use either Neo4j or Kuzu as the graph database.
+# Please make sure only one branch is live and others are commented out.
+
+# Use Neo4j
 GraphDbSpec = cocoindex.storages.Neo4j
 GraphDbConnection = cocoindex.storages.Neo4jConnection
 GraphDbDeclaration = cocoindex.storages.Neo4jDeclaration
 conn_spec = neo4j_conn_spec
 
-# Use Kuzu as the graph database
+# Use Kuzu
 #  GraphDbSpec = cocoindex.storages.Kuzu
 #  GraphDbConnection = cocoindex.storages.KuzuConnection
 #  GraphDbDeclaration = cocoindex.storages.KuzuDeclaration
 #  conn_spec = kuzu_conn_spec
 
 
+@dataclasses.dataclass
+class DocumentSummary:
+    """Describe a summary of a document."""
+
+    title: str
+    summary: str
+
+
+@dataclasses.dataclass
+class Relationship:
+    """
+    Describe a relationship between two entities.
+    Subject and object should be Core CocoIndex concepts only, should be nouns. For example, `CocoIndex`, `Incremental Processing`, `ETL`,  `Data` etc.
+    """
+
+    subject: str
+    predicate: str
+    object: str
+
+
 @cocoindex.flow_def(name="DocsToKG")
 def docs_to_kg_flow(
     flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope

{cocoindex-0.1.48 → cocoindex-0.1.49}/examples/image_search/README.md

@@ -14,7 +14,6 @@ We appreciate a star ⭐ at [CocoIndex Github](https://github.com/cocoindex-io/c
 - Qdrant for Vector Storage
 - FastApi for backend
 
-
 ## Setup
 - Make sure Postgres and Qdrant are running
   ```
@@ -22,20 +21,6 @@ We appreciate a star ⭐ at [CocoIndex Github](https://github.com/cocoindex-io/c
   export COCOINDEX_DATABASE_URL="postgres://cocoindex:cocoindex@localhost/cocoindex"
   ```
 
-- Create Qdrant Collection
-  ```
-  curl -X PUT 'http://localhost:6333/collections/image_search' \
-    -H 'Content-Type: application/json' \
-    -d '{
-      "vectors": {
-        "embedding": {
-          "size": 768,
-          "distance": "Cosine"
-        }
-      }
-    }'
-  ```
-
 ## Run
 - Install dependencies:
   ```

{cocoindex-0.1.48 → cocoindex-0.1.49}/examples/product_recommendation/README.md

@@ -9,7 +9,8 @@ Please drop [CocoIndex on Github](https://github.com/cocoindex-io/cocoindex) a s
 
 ## Prerequisite
 *   [Install Postgres](https://cocoindex.io/docs/getting_started/installation#-install-postgres) if you don't have one.
-*   [Install Neo4j](https://cocoindex.io/docs/ops/storages#neo4j) if you don't have one.
+*   Install [Neo4j](https://cocoindex.io/docs/ops/storages#neo4j-dev-instance) or [Kuzu](https://cocoindex.io/docs/ops/storages#kuzu-dev-instance) if you don't have one.
+    *   The example uses Neo4j by default for now. If you want to use Kuzu, find out the "SELECT ONE GRAPH DATABASE TO USE" section and switch the active branch.
 *   [Configure your OpenAI API key](https://cocoindex.io/docs/ai/llm#openai).
 
 ## Documentation
@@ -39,18 +40,17 @@ cocoindex update main.py
 
 ### Browse the knowledge graph
 
-After the knowledge graph is built, you can explore the knowledge graph you built in Neo4j Browser.
+After the knowledge graph is built, you can explore the knowledge graph.
 
-For the dev enviroment, you can connect neo4j browser using credentials:
-- username: `neo4j`
-- password: `cocoindex`
-which is pre-configured in the our docker compose [config.yaml](https://raw.githubusercontent.com/cocoindex-io/cocoindex/refs/heads/main/dev/neo4j.yaml).
+* If you're using Neo4j, you can open the explorer at [http://localhost:7474](http://localhost:7474), with username `neo4j` and password `cocoindex`.
+* If you're using Kuzu, you can start a Kuzu explorer locally. See [Kuzu dev instance](https://cocoindex.io/docs/ops/storages#kuzu-dev-instance) for more details.
 
-You can open it at [http://localhost:7474](http://localhost:7474), and run the following Cypher query to get all relationships:
+You can run the following Cypher query to get all relationships:
 
 ```cypher
 MATCH p=()-->() RETURN p
 ```
+
 ![Neo4j Browser Screenshot](img/neo4j.png)
 
 ## CocoInsight 

{cocoindex-0.1.48 → cocoindex-0.1.49}/examples/product_recommendation/main.py

@@ -7,6 +7,38 @@ import datetime
 import cocoindex
 from jinja2 import Template
 
+neo4j_conn_spec = cocoindex.add_auth_entry(
+    "Neo4jConnection",
+    cocoindex.storages.Neo4jConnection(
+        uri="bolt://localhost:7687",
+        user="neo4j",
+        password="cocoindex",
+    ),
+)
+kuzu_conn_spec = cocoindex.add_auth_entry(
+    "KuzuConnection",
+    cocoindex.storages.KuzuConnection(
+        api_server_url="http://localhost:8123",
+    ),
+)
+
+# SELECT ONE GRAPH DATABASE TO USE
+# This example can use either Neo4j or Kuzu as the graph database.
+# Please make sure only one branch is live and others are commented out.
+
+# Use Neo4j
+GraphDbSpec = cocoindex.storages.Neo4j
+GraphDbConnection = cocoindex.storages.Neo4jConnection
+GraphDbDeclaration = cocoindex.storages.Neo4jDeclaration
+conn_spec = neo4j_conn_spec
+
+# Use Kuzu
+#  GraphDbSpec = cocoindex.storages.Kuzu
+#  GraphDbConnection = cocoindex.storages.KuzuConnection
+#  GraphDbDeclaration = cocoindex.storages.KuzuDeclaration
+#  conn_spec = kuzu_conn_spec
+
+
 # Template for rendering product information as markdown to provide information to LLMs
 PRODUCT_TEMPLATE = """
 # {{ title }}
@@ -77,34 +109,6 @@ def extract_product_info(product: cocoindex.Json, filename: str) -> ProductInfo:
     )
 
 
-neo4j_conn_spec = cocoindex.add_auth_entry(
-    "Neo4jConnection",
-    cocoindex.storages.Neo4jConnection(
-        uri="bolt://localhost:7687",
-        user="neo4j",
-        password="cocoindex",
-    ),
-)
-kuzu_conn_spec = cocoindex.add_auth_entry(
-    "KuzuConnection",
-    cocoindex.storages.KuzuConnection(
-        api_server_url="http://localhost:8123",
-    ),
-)
-
-# Use Neo4j as the graph database
-GraphDbSpec = cocoindex.storages.Neo4j
-GraphDbConnection = cocoindex.storages.Neo4jConnection
-GraphDbDeclaration = cocoindex.storages.Neo4jDeclaration
-conn_spec = neo4j_conn_spec
-
-# Use Kuzu as the graph database
-#  GraphDbSpec = cocoindex.storages.Kuzu
-#  GraphDbConnection = cocoindex.storages.KuzuConnection
-#  GraphDbDeclaration = cocoindex.storages.KuzuDeclaration
-#  conn_spec = kuzu_conn_spec
-
-
 @cocoindex.flow_def(name="StoreProduct")
 def store_product_flow(
     flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope

{cocoindex-0.1.48 → cocoindex-0.1.49}/examples/text_embedding_qdrant/README.md

@@ -19,7 +19,6 @@ We appreciate a star ⭐ at [CocoIndex Github](https://github.com/cocoindex-io/c
 ### Query
 We use Qdrant client to query the index, and reuse the embedding operation in the indexing flow.
 
-
 ## Pre-requisites
 
 - [Install Postgres](https://cocoindex.io/docs/getting_started/installation#-install-postgres) if you don't have one. Although the target store is Qdrant, CocoIndex uses Postgress to track the data lineage for incremental processing.
@@ -30,24 +29,6 @@ We use Qdrant client to query the index, and reuse the embedding operation in th
    docker run -d -p 6334:6334 -p 6333:6333 qdrant/qdrant
    ```
 
-- [Create a collection](https://qdrant.tech/documentation/concepts/vectors/#named-vectors) to export the embeddings to.
-
-   ```bash
-   curl  -X PUT \
-     'http://localhost:6333/collections/cocoindex' \
-     --header 'Content-Type: application/json' \
-     --data-raw '{
-     "vectors": {
-       "text_embedding": {
-         "size": 384,
-         "distance": "Cosine"
-       }
-     }
-   }'
-   ```
-
-   You can view the collections and data with the Qdrant dashboard at <http://localhost:6333/dashboard>.
-
 ## Run
 
 - Install dependencies:
@@ -62,6 +43,9 @@ We use Qdrant client to query the index, and reuse the embedding operation in th
    cocoindex setup main.py
    ```
 
+   It will automatically create a collection in Qdrant.
+   You can view the collections and data with the Qdrant dashboard at <http://localhost:6333/dashboard>.
+
 - Update index:
 
    ```bash

{cocoindex-0.1.48 → cocoindex-0.1.49}/python/cocoindex/functions.py

@@ -1,6 +1,7 @@
 """All builtin functions."""
 
 from typing import Annotated, Any, TYPE_CHECKING
+import dataclasses
 
 from .typing import Float32, Vector, TypeAttr
 from . import op, llm
@@ -14,9 +15,20 @@ class ParseJson(op.FunctionSpec):
     """Parse a text into a JSON object."""
 
 
+@dataclasses.dataclass
+class CustomLanguageSpec:
+    """Custom language specification."""
+
+    language_name: str
+    separators_regex: list[str]
+    aliases: list[str] = dataclasses.field(default_factory=list)
+
+
 class SplitRecursively(op.FunctionSpec):
     """Split a document (in string) recursively."""
 
+    custom_languages: list[CustomLanguageSpec] = dataclasses.field(default_factory=list)
+
 
 class ExtractByLlm(op.FunctionSpec):
     """Extract information from a text using a LLM."""