cocoindex 0.1.41__tar.gz → 0.1.43__tar.gz

{cocoindex-0.1.41 → cocoindex-0.1.43}/Cargo.lock

@@ -993,7 +993,7 @@ dependencies = [
 
 [[package]]
 name = "cocoindex"
-version = "0.1.41"
+version = "0.1.43"
 dependencies = [
  "anyhow",
  "async-openai",

{cocoindex-0.1.41 → cocoindex-0.1.43}/Cargo.toml

@@ -2,7 +2,7 @@
 name = "cocoindex"
 # Version used for local development is always higher than others to take precedence.
 # Will be overridden for specific release versions.
-version = "0.1.41"
+version = "0.1.43"
 edition = "2024"
 
 [profile.release]

{cocoindex-0.1.41 → cocoindex-0.1.43}/PKG-INFO

@@ -1,9 +1,10 @@
 Metadata-Version: 2.4
 Name: cocoindex
-Version: 0.1.41
+Version: 0.1.43
 Requires-Dist: sentence-transformers>=3.3.1
 Requires-Dist: click>=8.1.8
 Requires-Dist: rich>=14.0.0
+Requires-Dist: python-dotenv>=1.1.0
 Requires-Dist: pytest ; extra == 'test'
 Provides-Extra: test
 License-File: LICENSE
@@ -154,7 +155,7 @@ It defines an index flow like this:
 | [Embeddings to Qdrant](examples/text_embedding_qdrant) | Index documents in a Qdrant collection for semantic search |
 | [FastAPI Server with Docker](examples/fastapi_server_docker) | Run the semantic search server in a Dockerized FastAPI setup | 
 | [Product Recommendation](examples/product_recommendation) | Build real-time product recommendations with LLM and graph database| 
-| [Image Search with Vision API](examples/image_search_example) | Generates detailed captions for images using a vision model, embeds them, enables live-updating semantic search via FastAPI and served on a React frontend|
+| [Image Search with Vision API](examples/image_search) | Generates detailed captions for images using a vision model, embeds them, enables live-updating semantic search via FastAPI and served on a React frontend|
 
 More coming and stay tuned 👀!
 

{cocoindex-0.1.41 → cocoindex-0.1.43}/README.md

@@ -138,7 +138,7 @@ It defines an index flow like this:
 | [Embeddings to Qdrant](examples/text_embedding_qdrant) | Index documents in a Qdrant collection for semantic search |
 | [FastAPI Server with Docker](examples/fastapi_server_docker) | Run the semantic search server in a Dockerized FastAPI setup | 
 | [Product Recommendation](examples/product_recommendation) | Build real-time product recommendations with LLM and graph database| 
-| [Image Search with Vision API](examples/image_search_example) | Generates detailed captions for images using a vision model, embeds them, enables live-updating semantic search via FastAPI and served on a React frontend|
+| [Image Search with Vision API](examples/image_search) | Generates detailed captions for images using a vision model, embeds them, enables live-updating semantic search via FastAPI and served on a React frontend|
 
 More coming and stay tuned 👀!
 

{cocoindex-0.1.41 → cocoindex-0.1.43}/docs/docs/core/basics.md

@@ -101,4 +101,4 @@ As an indexing flow is long-lived, it needs to store intermediate data to keep t
 CocoIndex uses internal storage for this purpose.
 
 Currently, CocoIndex uses Postgres database as the internal storage.
-See [Initialization](initialization) for configuring its location, and `cocoindex setup` CLI command (see [CocoIndex CLI](cli)) creates tables for the internal storage.
+See [Settings](settings#databaseconnectionspec) for configuring its location, and `cocoindex setup` CLI command (see [CocoIndex CLI](cli)) creates tables for the internal storage.

cocoindex-0.1.43/docs/docs/core/cli.mdx

@@ -0,0 +1,75 @@
+---
+title: CLI
+description: CocoIndex CLI
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# CocoIndex CLI
+
+CocoIndex CLI is a standalone tool for easily managing and inspecting your flows and indexes.
+
+## Invoke the CLI
+
+Once CocoIndex is installed, you can invoke the CLI directly using the `cocoindex` command. Most commands require an `APP_TARGET` argument, which tells the CLI where your flow definitions are located.
+
+### APP_TARGET Format
+
+The `APP_TARGET` can be:
+1.  A **path to a Python file** defining your flows (e.g., `main.py`, `path/to/my_flows.py`).
+2.  An **installed Python module name** that contains your flow definitions (e.g., `my_package.flows`).
+3.  For commands that operate on a *specific flow* (like `show`, `update`, `evaluate`), you can combine the application reference with a flow name:
+    * `path/to/my_flows.py:MyFlow`
+    * `my_package.flows:MyFlow`
+
+### Environment Variables
+
+Environment variables are needed as CocoIndex library settings, as described in [CocoIndex Settings](settings#list-of-environment-variables).
+
+You can set environment variables in an environment file.
+
+*   By default, the `cocoindex` CLI searches upward from the current directory for a `.env` file.
+*   You can use `--env-file <path>` to specify one explicitly:
+
+    ```sh
+    cocoindex --env-file path/to/custom.env <COMMAND> ...
+    ```
+
+Loaded variables do *NOT* override existing system ones.
+If no file is found, only existing system environment variables are used.
+
+### Global Options
+
+CocoIndex CLI supports the following global options:
+
+* `--env-file <path>`: Load environment variables from a specified `.env` file. If not provided, `.env` in the current directory is loaded if it exists.
+* `--version`: Show the CocoIndex version and exit.
+* `--help`: Show the main help message and exit.
+
+:::caution Deprecated Usage
+
+The old method of invoking the CLI using `python main.py cocoindex ...` via the `@cocoindex.main_fn()` decorator is now deprecated. Please remove `@cocoindex.main_fn()` from your scripts and use the standalone cocoindex command as described.
+
+:::
+
+## Subcommands
+
+The following subcommands are available:
+
+| Subcommand | Description |
+| ---------- | ----------- |
+| `ls` | List all flows present in the given file/module. Or list all persisted flows under the current app namespace if no file/module specified. |
+| `show` | Show the spec and schema for a specific flow. |
+| `setup` | Check and apply backend setup changes for flows, including the internal and target storage (to export). |
+| `drop` | Drop the backend setup for specified flows. |
+| `update` | Update the index defined by the flow. |
+| `evaluate` | Evaluate the flow and dump flow outputs to files.  Instead of updating the index, it dumps what should be indexed to files. Mainly used for evaluation purpose. |
+| `server` | Start a HTTP server providing REST APIs. It will allow tools like CocoInsight to access the server. |
+
+Use `--help` to see the full list of subcommands, and `subcommand --help` to see the usage of a specific one.
+
+```sh
+cocoindex --help       # Show all subcommands
+cocoindex show --help  # Show usage of "show" subcommand
+```

{cocoindex-0.1.41 → cocoindex-0.1.43}/docs/docs/core/data_types.mdx

@@ -35,6 +35,7 @@ This is the list of all basic types supported by CocoIndex:
 | Time | | `datetime.time` | `datetime.time` |
 | LocalDatetime | Date and time without timezone | `cocoindex.LocalDateTime` | `datetime.datetime` |
 | OffsetDatetime | Date and time with a timezone offset | `cocoindex.OffsetDateTime` | `datetime.datetime` |
+| TimeDelta | A duration of time | `datetime.timedelta` | `datetime.timedelta` |
 | Vector[*T*, *Dim*?] | *T* must be basic type. *Dim* is a positive integer and optional. |`cocoindex.Vector[T]` or `cocoindex.Vector[T, Dim]` | `list[T]` | 
 | Json | | `cocoindex.Json` | Any data convertible to JSON by `json` package | 
 

{cocoindex-0.1.41 → cocoindex-0.1.43}/docs/docs/core/flow_def.mdx

@@ -313,7 +313,7 @@ Following metrics are supported:
 
 ### Getting App Namespace
 
-You can use the [`app_namespace` setting](initialization#app-namespace) or `COCOINDEX_APP_NAMESPACE` environment variable to specify the app namespace,
+You can use the [`app_namespace` setting](settings#app-namespace) or `COCOINDEX_APP_NAMESPACE` environment variable to specify the app namespace,
 to organize flows across different environments (e.g., dev, staging, production), team members, etc.
 
 In the code, You can call `flow.get_app_namespace()` to get the app namespace, and use it to name certain backends. It takes the following arguments:

{cocoindex-0.1.41 → cocoindex-0.1.43}/docs/docs/core/flow_methods.mdx

@@ -1,5 +1,5 @@
 ---
-title: Flow Running
+title: Run a Flow
 toc_max_heading_level: 4
 description: Run a CocoIndex Flow, including build / update data in the target storage and evaluate the flow without changing the target storage.
 ---
@@ -7,7 +7,7 @@ description: Run a CocoIndex Flow, including build / update data in the target s
 import Tabs from '@theme/Tabs';
 import TabItem from '@theme/TabItem';
 
-# Running a CocoIndex Flow
+# Run a CocoIndex Flow
 
 After a flow is defined as discussed in [Flow Definition](/docs/core/flow_def), you can start to transform data with it.
 
@@ -30,17 +30,7 @@ def demo_flow(flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataSco
 ```
 
 It creates a `demo_flow` object in `cocoindex.Flow` type.
-To enable CLI, you also need to make sure you have a main function decorated with `@cocoindex.main_fn()`:
 
-
-```python title="main.py"
-@cocoindex.main_fn()
-def main():
-    ...
-
-if __name__ == "__main__":
-    main()
-```
 </TabItem>
 </Tabs>
 
@@ -78,7 +68,7 @@ The `cocoindex update` subcommand creates/updates data in the target storage.
 Once it's done, the target data is fresh up to the moment when the function is called.
 
 ```sh
-python main.py cocoindex update
+cocoindex update main.py
 ```
 
 #### Library API
@@ -115,7 +105,7 @@ Change capture mechanisms enable CocoIndex to continuously capture changes from
 To perform live update, run the `cocoindex update` subcommand with `-L` option:
 
 ```sh
-python main.py cocoindex update -L
+cocoindex update main.py -L
 ```
 
 If there's at least one data source with change capture mechanism enabled, it will keep running until the aborted (e.g. by `Ctrl-C`).
@@ -232,7 +222,7 @@ It takes the following options:
 Example:
 
 ```sh
-python main.py cocoindex evaluate --output-dir ./eval_output
+cocoindex evaluate main.py --output-dir ./eval_output
 ```
 
 ### Library API

cocoindex-0.1.43/docs/docs/core/settings.mdx

@@ -0,0 +1,116 @@
+---
+title: CocoIndex Settings
+description: Provide settings for CocoIndex, e.g. database connection, app namespace, etc.
+---
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# CocoIndex Settings
+
+Certain settings need to be provided for CocoIndex to work, e.g. database connections, app namespace, etc.
+
+## Launch CocoIndex
+
+You have two ways to launch CocoIndex:
+
+*   Use [Cocoindex CLI](cli). It's handy for most routine indexing building and management tasks.
+    It will load settings from environment variables, either already set in your environment, or specified in `.env` file.
+    See [CLI](cli#environment-variables) for more details.
+
+*   Call CocoIndex functionality from your own Python application or library.
+    It's needed when you want to leverage CocoIndex support for query, or have your custom logic to trigger indexing, etc.
+
+    <Tabs>
+    <TabItem value="python" label="Python" default>
+
+    You need to explicitly call `cocoindex.init()` before doing anything with CocoIndex, and settings will be loaded during the call.
+
+    *   If it's called without any argument, it will load settings from environment variables.
+        Only existing environment variables already set in your environment will be used.
+        If you want to load environment variables from a specific `.env` file, consider call `load_dotenv()` provided by the [`python-dotenv`](https://github.com/theskumar/python-dotenv) package.
+
+        ```py
+        from dotenv import load_dotenv
+        import cocoindex
+
+        load_dotenv()
+        cocoindex.init()
+        ```
+
+    *   It takes an optional `cocoindex.Settings` dataclass object as argument, so you can also construct settings explicitly and pass to it:
+
+        ```py
+        import cocoindex
+
+        cocoindex.init(
+            cocoindex.Settings(
+                database=cocoindex.DatabaseConnectionSpec(
+                    url="postgres://cocoindex:cocoindex@localhost/cocoindex"
+                )
+            )
+        )
+        ```
+    </TabItem>
+    </Tabs>
+
+## List of Settings
+
+`cocoindex.Settings` is a dataclass that contains the following fields:
+
+*   `app_namespace` (type: `str`, required): The namespace of the application.
+*   `database` (type: `DatabaseConnectionSpec`, required): The connection to the Postgres database.
+
+### App Namespace
+
+The `app_namespace` field helps organize flows across different environments (e.g., dev, staging, production), team members, etc. When set, it prefixes flow names with the namespace.
+
+For example, if the namespace is `Staging`, for a flow with name specified as `Flow1` in code, the full name of the flow will be `Staging.Flow1`.
+You can also get the current app namespace by calling `cocoindex.get_app_namespace()` (see [Getting App Namespace](flow_def#getting-app-namespace) for more details).
+
+If not set, all flows are in a default unnamed namespace.
+
+*Environment variable*: `COCOINDEX_APP_NAMESPACE`
+
+### DatabaseConnectionSpec
+
+`DatabaseConnectionSpec` configures the connection to a database. Only Postgres is supported for now. It has the following fields:
+
+*   `url` (type: `str`, required): The URL of the Postgres database to use as the internal storage, e.g. `postgres://cocoindex:cocoindex@localhost/cocoindex`.
+
+    *Environment variable* for `Settings.database.url`: `COCOINDEX_DATABASE_URL`
+
+*   `user` (type: `str`, optional): The username for the Postgres database. If not provided, username will come from `url`.
+
+    *Environment variable* for `Settings.database.user`: `COCOINDEX_DATABASE_USER`
+
+*   `password` (type: `str`, optional): The password for the Postgres database. If not provided, password will come from `url`.
+
+    *Environment variable* for `Settings.database.password`: `COCOINDEX_DATABASE_PASSWORD`
+
+:::tip
+
+Please be careful that all values in `url` needs to be url-encoded if they contain special characters.
+For this reason, prefer to use the separated `user` and `password` fields for username and password.
+
+:::
+
+:::info
+
+If you use the Postgres database hosted by [Supabase](https://supabase.com/), please click **Connect** on your project dashboard and find the following URL:
+
+*    If you're on a IPv6 network, use the URL under **Direct connection**. You can visit [IPv6 test](https://test-ipv6.com/) to see if you have IPv6 Internet connection.
+*    Otherwise, use the URL under **Session pooler**.
+
+:::
+
+## List of Environment Variables
+
+This is the list of environment variables, each of which has a corresponding field in `Settings`:
+
+| environment variable | corresponding field in `Settings` | required? |
+|---------------------|-------------------|----------|
+| `COCOINDEX_DATABASE_URL` | `database.url` | Yes |
+| `COCOINDEX_DATABASE_USER` | `database.user` | No |
+| `COCOINDEX_DATABASE_PASSWORD` | `database.password` | No |
+| `COCOINDEX_APP_NAMESPACE` | `app_namespace` | No |

{cocoindex-0.1.41 → cocoindex-0.1.43}/docs/docs/getting_started/quickstart.md

@@ -46,7 +46,7 @@ We'll need to install a bunch of dependencies for this project.
 2.  Prepare input files for the index. Put them in a directory, e.g. `markdown_files`.
     If you don't have any files at hand, you may download the example [markdown_files.zip](markdown_files.zip) and unzip it in the current directory.
 
-## Step 2: Create the Python file `quickstart.py`
+## Step 2: Define the indexing flow
 
 Create a new file `quickstart.py` and import the `cocoindex` library:
 
@@ -54,11 +54,7 @@ Create a new file `quickstart.py` and import the `cocoindex` library:
 import cocoindex
 ```
 
-Then we'll create the indexing flow.
-
-### Step 2.1: Define the indexing flow
-
-Starting from the indexing flow:
+Then we'll create the indexing flow as follows.
 
 ```python title="quickstart.py"
 @cocoindex.flow_def(name="TextEmbedding")
@@ -117,24 +113,6 @@ Notes:
 
 6. In CocoIndex, a *collector* collects multiple entries of data together. In this example, the `doc_embeddings` collector collects data from all `chunk`s across all `doc`s, and using the collected data to build a vector index `"doc_embeddings"`, using `Postgres`.
 
-### Step 2.2: Define the main function
-
-We can provide an empty main function for now, with a `@cocoindex.main_fn()` decorator:
-
-```python title="quickstart.py"
-@cocoindex.main_fn()
-def _main():
-    pass
-
-if __name__ == "__main__":
-    _main()
-```
-
-The `@cocoindex.main_fn` declares a function as the main function for an indexing application. This achieves the following effects:
-
-*   Initialize the CocoIndex library states. Settings (e.g. database URL) are loaded from environment variables by default.
-*   When the CLI is invoked with `cocoindex` subcommand, `cocoindex CLI` takes over the control, which provides convenient ways to manage the index. See the next step for more details.
-
 ## Step 3: Run the indexing pipeline and queries
 
 Specify the database URL by environment variable:
@@ -148,7 +126,7 @@ export COCOINDEX_DATABASE_URL="postgresql://cocoindex:cocoindex@localhost:5432/c
 We need to setup the index:
 
 ```bash
-python quickstart.py cocoindex setup
+cocoindex setup quickstart.py
 ```
 
 Enter `yes` and it will automatically create a few tables in the database.
@@ -160,7 +138,7 @@ Now we have tables needed by this CocoIndex flow.
 Now we're ready to build the index:
 
 ```bash
-python quickstart.py cocoindex update
+cocoindex update quickstart.py
 ```
 
 It will run for a few seconds and output the following statistics:
@@ -260,13 +238,15 @@ There're two CocoIndex-specific logic:
     It's done by the `eval()` method of the transform flow `text_to_embedding`.
     The return type of this method is `list[float]` as declared in the `text_to_embedding()` function (`cocoindex.DataSlice[list[float]]`).
 
-### Step 4.3: Update the main function
+### Step 4.3: Add the main script logic
 
-Now we can update the main function to use the query function we just defined:
+Now we can add the main logic to the program. It uses the query function we just defined:
 
 ```python title="quickstart.py"
-@cocoindex.main_fn()
-def _run():
+if __name__ == "__main__":
+    # Initialize CocoIndex library states
+    cocoindex.init()
+
     # Initialize the database connection pool.
     pool = ConnectionPool(os.getenv("COCOINDEX_DATABASE_URL"))
     # Run queries in a loop to demonstrate the query capabilities.
@@ -291,7 +271,7 @@ It interacts with users and search the database by calling the `search()` method
 
 ### Step 4.4: Run queries against the index
 
-Now we can run the same Python file, which will run the new main function:
+Now we can run the same Python file, which will run the new added main logic:
 
 ```bash
 python quickstart.py

{cocoindex-0.1.41 → cocoindex-0.1.43}/docs/docs/ops/storages.md

@@ -37,7 +37,7 @@ It should be a unique table, meaning that no other export target should export t
 The spec takes the following fields:
 
 *   `database` (type: [auth reference](../core/flow_def#auth-registry) to `DatabaseConnectionSpec`, optional): The connection to the Postgres database.
-    See [DatabaseConnectionSpec](../core/initialization#databaseconnectionspec) for its specific fields.
+    See [DatabaseConnectionSpec](../core/settings#databaseconnectionspec) for its specific fields.
     If not provided, will use the same database as the [internal storage](/docs/core/basics#internal-storage).
 
 *   `table_name` (type: `str`, optional): The name of the table to store to. If unspecified, will use the table name `[${AppNamespace}__]${FlowName}__${TargetName}`, e.g. `DemoFlow__doc_embeddings` or `Staging__DemoFlow__doc_embeddings`.
@@ -419,7 +419,7 @@ The `Neo4j` storage exports each row as a relationship to Neo4j Knowledge Graph.
 Neo4j also provides a declaration spec `Neo4jDeclaration`, to configure indexing options for nodes only referenced by relationships. It has the following fields:
 
 *   `connection` (type: auth reference to `Neo4jConnectionSpec`)
-*   Fields for [nodes to declare](#nodes-to-declare), including
+*   Fields for [nodes to declare](#declare-extra-node-labels), including
     *   `nodes_label` (required)
     *   `primary_key_fields` (required)
     *   `vector_indexes` (optional)

{cocoindex-0.1.41 → cocoindex-0.1.43}/docs/docusaurus.config.ts

@@ -49,6 +49,17 @@ const config: Config = {
         ],
       }),
     }),
+    [
+      '@docusaurus/plugin-client-redirects',
+      {
+        redirects: [
+          {
+            from: '/core/initialization',
+            to: '/core/settings',
+          },
+        ],
+      },
+    ],
   ],
 
   presets: [

{cocoindex-0.1.41 → cocoindex-0.1.43}/docs/package.json

@@ -16,6 +16,7 @@
   },
   "dependencies": {
     "@docusaurus/core": "3.7.0",
+    "@docusaurus/plugin-client-redirects": "^3.7.0",
     "@docusaurus/preset-classic": "3.7.0",
     "@docusaurus/theme-mermaid": "^3.7.0",
     "@mdx-js/react": "^3.0.0",

{cocoindex-0.1.41 → cocoindex-0.1.43}/docs/sidebars.ts

@@ -19,8 +19,8 @@ const sidebars: SidebarsConfig = {
       items: [
         'core/basics',
         'core/data_types',
-        'core/initialization',
         'core/flow_def',
+        'core/settings',
         'core/flow_methods',
         'core/cli',
         'core/custom_function',

{cocoindex-0.1.41 → cocoindex-0.1.43}/docs/yarn.lock

@@ -1604,6 +1604,21 @@
     react-helmet-async "npm:@slorber/react-helmet-async@*"
     react-loadable "npm:@docusaurus/react-loadable@6.0.0"
 
+"@docusaurus/plugin-client-redirects@^3.7.0":
+  version "3.7.0"
+  resolved "https://registry.yarnpkg.com/@docusaurus/plugin-client-redirects/-/plugin-client-redirects-3.7.0.tgz#b5cf92529768c457c01ad350bfc50862c6149463"
+  integrity sha512-6B4XAtE5ZVKOyhPgpgMkb7LwCkN+Hgd4vOnlbwR8nCdTQhLjz8MHbGlwwvZ/cay2SPNRX5KssqKAlcHVZP2m8g==
+  dependencies:
+    "@docusaurus/core" "3.7.0"
+    "@docusaurus/logger" "3.7.0"
+    "@docusaurus/utils" "3.7.0"
+    "@docusaurus/utils-common" "3.7.0"
+    "@docusaurus/utils-validation" "3.7.0"
+    eta "^2.2.0"
+    fs-extra "^11.1.1"
+    lodash "^4.17.21"
+    tslib "^2.6.0"
+
 "@docusaurus/plugin-content-blog@3.7.0":
   version "3.7.0"
   resolved "https://registry.yarnpkg.com/@docusaurus/plugin-content-blog/-/plugin-content-blog-3.7.0.tgz#7bd69de87a1f3adb652e1473ef5b7ccc9468f47e"

{cocoindex-0.1.41 → cocoindex-0.1.43}/examples/amazon_s3_embedding/README.md

@@ -40,7 +40,7 @@ pip install -e .
 Setup:
 
 ```sh
-python main.py cocoindex setup
+cocoindex setup main.py
 ```
 
 Run:
@@ -59,13 +59,13 @@ CocoInsight is in Early Access now (Free) 😊 You found us! A quick 3 minute vi
 Run CocoInsight to understand your RAG data pipeline:
 
 ```sh
-python main.py cocoindex server -ci
+cocoindex server -ci main.py
 ```
 
 You can also add a `-L` flag to make the server keep updating the index to reflect source changes at the same time:
 
 ```sh
-python main.py cocoindex server -ci -L
+cocoindex server -ci -L main.py
 ```
 
 Then open the CocoInsight UI at [https://cocoindex.io/cocoinsight](https://cocoindex.io/cocoinsight).

{cocoindex-0.1.41 → cocoindex-0.1.43}/examples/amazon_s3_embedding/main.py

@@ -52,8 +52,7 @@ query_handler = cocoindex.query.SimpleSemanticsQueryHandler(
             model="sentence-transformers/all-MiniLM-L6-v2")),
     default_similarity_metric=cocoindex.VectorSimilarityMetric.COSINE_SIMILARITY)
 
-@cocoindex.main_fn()
-def _run():
+def _main():
     # Use a `FlowLiveUpdater` to keep the flow data updated.
     with cocoindex.FlowLiveUpdater(amazon_s3_text_embedding_flow):
         # Run queries in a loop to demonstrate the query capabilities.
@@ -73,5 +72,6 @@ def _run():
                 break
 
 if __name__ == "__main__":
-    load_dotenv(override=True)
-    _run()
+    load_dotenv()
+    cocoindex.init()
+    _main()

{cocoindex-0.1.41 → cocoindex-0.1.43}/examples/amazon_s3_embedding/pyproject.toml

@@ -3,7 +3,7 @@ name = "amazon-s3-text-embedding"
 version = "0.1.0"
 description = "Simple example for cocoindex: build embedding index based on Amazon S3 files."
 requires-python = ">=3.11"
-dependencies = ["cocoindex>=0.1.39", "python-dotenv>=1.0.1"]
+dependencies = ["cocoindex>=0.1.42", "python-dotenv>=1.0.1"]
 
 [tool.setuptools]
 packages = []

cocoindex-0.1.43/examples/code_embedding/README.md

@@ -0,0 +1,71 @@
+# Build real-time index for codebase 
+[![GitHub](https://img.shields.io/github/stars/cocoindex-io/cocoindex?color=5B5BD6)](https://github.com/cocoindex-io/cocoindex)
+
+CocoIndex provides built-in support for code base chunking, using Tree-sitter to keep syntax boundary. In this example, we will build real-time index for codebase using CocoIndex. 
+
+We appreciate a star ⭐ at [CocoIndex Github](https://github.com/cocoindex-io/cocoindex) if this is helpful.
+
+![Build embedding index for codebase](https://github.com/user-attachments/assets/6dc5ce89-c949-41d4-852f-ad95af163dbd)
+
+[Tree-sitter](https://en.wikipedia.org/wiki/Tree-sitter_%28parser_generator%29) is a parser generator tool and an incremental parsing library. It is available in Rust 🦀 - [GitHub](https://github.com/tree-sitter/tree-sitter). CocoIndex has built-in Rust integration with Tree-sitter to efficiently parse code and extract syntax trees for various programming languages. Check out the list of supported languages [here](https://cocoindex.io/docs/ops/functions#splitrecursively) - in the `language` section.
+
+
+## Tutorials
+- Step by step tutorial - Check out the [blog](https://cocoindex.io/blogs/index-code-base-for-rag).
+- Video tutorial - [Youtube](https://youtu.be/G3WstvhHO24?si=Bnxu67Ax5Lv8b-J2).
+
+## Steps
+
+### Indexing Flow
+<p align='center'>
+  <img width="434" alt="Screenshot 2025-05-19 at 10 14 36 PM" src="https://github.com/user-attachments/assets/3a506034-698f-480a-b653-22184dae4e14" />
+</p>
+
+1. We will ingest CocoIndex codebase.
+2. For each file, perform chunking (Tree-sitter) and then embedding. 
+3. We will save the embeddings and the metadata in Postgres with PGVector.
+   
+### Query:
+We will match against user-provided text by a SQL query, reusing the embedding operation in the indexing flow.
+
+
+## Prerequisite
+[Install Postgres](https://cocoindex.io/docs/getting_started/installation#-install-postgres) if you don't have one.
+
+## Run
+
+- Install dependencies:
+  ```bash
+  pip install -e .
+  ```
+
+- Setup:
+
+  ```bash
+  cocoindex setup main.py
+  ```
+
+- Update index:
+  
+  ```bash
+  cocoindex update main.py
+  ```
+
+- Run:
+
+  ```bash
+  python main.py
+  ```
+
+## CocoInsight
+I used CocoInsight (Free beta now) to troubleshoot the index generation and understand the data lineage of the pipeline. 
+It just connects to your local CocoIndex server, with Zero pipeline data retention. Run the following command to start CocoInsight:
+
+```
+cocoindex server -ci main.py
+```
+
+Then open the CocoInsight UI at [https://cocoindex.io/cocoinsight](https://cocoindex.io/cocoinsight).
+
+<img width="1305" alt="Chunking Visualization" src="https://github.com/user-attachments/assets/8e83b9a4-2bed-456b-83e5-b5381b28b84a" />
+