cocoindex 0.2.19__tar.gz → 0.2.20__tar.gz

{cocoindex-0.2.19 → cocoindex-0.2.20}/Cargo.lock

@@ -1283,7 +1283,7 @@ dependencies = [
 
 [[package]]
 name = "cocoindex"
-version = "0.2.19"
+version = "0.2.20"
 dependencies = [
  "anyhow",
  "async-openai",

{cocoindex-0.2.19 → cocoindex-0.2.20}/Cargo.toml

@@ -2,7 +2,7 @@
 name = "cocoindex"
 # Version used for local development is always higher than others to take precedence.
 # Will be overridden for specific release versions.
-version = "0.2.19"
+version = "0.2.20"
 edition = "2024"
 rust-version = "1.89"
 license = "Apache-2.0"

{cocoindex-0.2.19 → cocoindex-0.2.20}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cocoindex
-Version: 0.2.19
+Version: 0.2.20
 Classifier: Development Status :: 3 - Alpha
 Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Operating System :: OS Independent
@@ -75,7 +75,6 @@ Project-URL: Homepage, https://cocoindex.io/
     <a href="https://trendshift.io/repositories/13939" target="_blank"><img src="https://trendshift.io/api/badge/repositories/13939" alt="cocoindex-io%2Fcocoindex | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
 </div>
 
-
 Ultra performant data transformation framework for AI, with core engine written in Rust. Support incremental processing and data lineage out-of-box.  Exceptional developer velocity. Production-ready at day 0.
 
 ⭐ Drop a star to help us grow!
@@ -113,9 +112,8 @@ CocoIndex makes it effortless to transform data with AI, and keep source data an
 
 </br>
 
-
-
 ## Exceptional velocity
+
 Just declare transformation in dataflow with ~100 lines of python
 
 ```python
@@ -139,6 +137,7 @@ CocoIndex follows the idea of [Dataflow](https://en.wikipedia.org/wiki/Dataflow_
 **Particularly**, developers don't explicitly mutate data by creating, updating and deleting. They just need to define transformation/formula for a set of source data.
 
 ## Plug-and-Play Building Blocks
+
 Native builtins for different source, targets and transformations. Standardize interface, make it 1-line code switch between different components - as easy as assembling building blocks.
 
 <p align="center">
@@ -146,6 +145,7 @@ Native builtins for different source, targets and transformations. Standardize i
 </p>
 
 ## Data Freshness
+
 CocoIndex keep source data and target in sync effortlessly.
 
 <p align="center">
@@ -153,11 +153,14 @@ CocoIndex keep source data and target in sync effortlessly.
 </p>
 
 It has out-of-box support for incremental indexing:
+
 - minimal recomputation on source or logic change.
 - (re-)processing necessary portions; reuse cache when possible
 
-## Quick Start:
+## Quick Start
+
 If you're new to CocoIndex, we recommend checking out
+
 - 📖 [Documentation](https://cocoindex.io/docs)
 - ⚡  [Quick Start Guide](https://cocoindex.io/docs/getting_started/quickstart)
 - 🎬 [Quick Start Video Tutorial](https://youtu.be/gv5R8nOXsWU?si=9ioeKYkMEnYevTXT)
@@ -172,7 +175,6 @@ pip install -U cocoindex
 
 2. [Install Postgres](https://cocoindex.io/docs/getting_started/installation#-install-postgres) if you don't have one. CocoIndex uses it for incremental processing.
 
-
 ## Define data flow
 
 Follow [Quick Start Guide](https://cocoindex.io/docs/getting_started/quickstart) to define your first indexing flow. An example flow looks like:
@@ -228,6 +230,7 @@ It defines an index flow like this:
 | [Text Embedding](examples/text_embedding) | Index text documents with embeddings for semantic search |
 | [Code Embedding](examples/code_embedding) | Index code embeddings for semantic search |
 | [PDF Embedding](examples/pdf_embedding) | Parse PDF and index text embeddings for semantic search |
+| [PDF Elements Embedding](examples/pdf_elements_embedding) | Extract text and images from PDFs; embed text with SentenceTransformers and images with CLIP; store in Qdrant for multimodal search |
 | [Manuals LLM Extraction](examples/manuals_llm_extraction) | Extract structured information from a manual using LLM |
 | [Amazon S3 Embedding](examples/amazon_s3_embedding) | Index text documents from Amazon S3 |
 | [Azure Blob Storage Embedding](examples/azure_blob_embedding) | Index text documents from Azure Blob Storage |
@@ -244,16 +247,18 @@ It defines an index flow like this:
 | [Custom Output Files](examples/custom_output_files) | Convert markdown files to HTML files and save them to a local directory, using *CocoIndex Custom Targets* |
 | [Patient intake form extraction](examples/patient_intake_extraction) | Use LLM to extract structured data from patient intake forms with different formats |
 
-
 More coming and stay tuned 👀!
 
 ## 📖 Documentation
+
 For detailed documentation, visit [CocoIndex Documentation](https://cocoindex.io/docs), including a [Quickstart guide](https://cocoindex.io/docs/getting_started/quickstart).
 
 ## 🤝 Contributing
+
 We love contributions from our community ❤️. For details on contributing or running the project for development, check out our [contributing guide](https://cocoindex.io/docs/about/contributing).
 
 ## 👥 Community
+
 Welcome with a huge coconut hug 🥥⋆｡˚🤗. We are super excited for community contributions of all kinds - whether it's code improvements, documentation updates, issue reports, feature requests, and discussions in our Discord.
 
 Join our community here:
@@ -263,9 +268,11 @@ Join our community here:
 - ▶️ [Subscribe to our YouTube channel](https://www.youtube.com/@cocoindex-io)
 - 📜 [Read our blog posts](https://cocoindex.io/blogs/)
 
-## Support us:
+## Support us
+
 We are constantly improving, and more features and examples are coming soon. If you love this project, please drop us a star ⭐ at GitHub repo [![GitHub](https://img.shields.io/github/stars/cocoindex-io/cocoindex?color=5B5BD6)](https://github.com/cocoindex-io/cocoindex) to stay tuned and help us grow.
 
 ## License
+
 CocoIndex is Apache 2.0 licensed.
 

{cocoindex-0.2.19 → cocoindex-0.2.20}/README.md

@@ -22,7 +22,6 @@
     <a href="https://trendshift.io/repositories/13939" target="_blank"><img src="https://trendshift.io/api/badge/repositories/13939" alt="cocoindex-io%2Fcocoindex | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
 </div>
 
-
 Ultra performant data transformation framework for AI, with core engine written in Rust. Support incremental processing and data lineage out-of-box.  Exceptional developer velocity. Production-ready at day 0.
 
 ⭐ Drop a star to help us grow!
@@ -60,9 +59,8 @@ CocoIndex makes it effortless to transform data with AI, and keep source data an
 
 </br>
 
-
-
 ## Exceptional velocity
+
 Just declare transformation in dataflow with ~100 lines of python
 
 ```python
@@ -86,6 +84,7 @@ CocoIndex follows the idea of [Dataflow](https://en.wikipedia.org/wiki/Dataflow_
 **Particularly**, developers don't explicitly mutate data by creating, updating and deleting. They just need to define transformation/formula for a set of source data.
 
 ## Plug-and-Play Building Blocks
+
 Native builtins for different source, targets and transformations. Standardize interface, make it 1-line code switch between different components - as easy as assembling building blocks.
 
 <p align="center">
@@ -93,6 +92,7 @@ Native builtins for different source, targets and transformations. Standardize i
 </p>
 
 ## Data Freshness
+
 CocoIndex keep source data and target in sync effortlessly.
 
 <p align="center">
@@ -100,11 +100,14 @@ CocoIndex keep source data and target in sync effortlessly.
 </p>
 
 It has out-of-box support for incremental indexing:
+
 - minimal recomputation on source or logic change.
 - (re-)processing necessary portions; reuse cache when possible
 
-## Quick Start:
+## Quick Start
+
 If you're new to CocoIndex, we recommend checking out
+
 - 📖 [Documentation](https://cocoindex.io/docs)
 - ⚡  [Quick Start Guide](https://cocoindex.io/docs/getting_started/quickstart)
 - 🎬 [Quick Start Video Tutorial](https://youtu.be/gv5R8nOXsWU?si=9ioeKYkMEnYevTXT)
@@ -119,7 +122,6 @@ pip install -U cocoindex
 
 2. [Install Postgres](https://cocoindex.io/docs/getting_started/installation#-install-postgres) if you don't have one. CocoIndex uses it for incremental processing.
 
-
 ## Define data flow
 
 Follow [Quick Start Guide](https://cocoindex.io/docs/getting_started/quickstart) to define your first indexing flow. An example flow looks like:
@@ -175,6 +177,7 @@ It defines an index flow like this:
 | [Text Embedding](examples/text_embedding) | Index text documents with embeddings for semantic search |
 | [Code Embedding](examples/code_embedding) | Index code embeddings for semantic search |
 | [PDF Embedding](examples/pdf_embedding) | Parse PDF and index text embeddings for semantic search |
+| [PDF Elements Embedding](examples/pdf_elements_embedding) | Extract text and images from PDFs; embed text with SentenceTransformers and images with CLIP; store in Qdrant for multimodal search |
 | [Manuals LLM Extraction](examples/manuals_llm_extraction) | Extract structured information from a manual using LLM |
 | [Amazon S3 Embedding](examples/amazon_s3_embedding) | Index text documents from Amazon S3 |
 | [Azure Blob Storage Embedding](examples/azure_blob_embedding) | Index text documents from Azure Blob Storage |
@@ -191,16 +194,18 @@ It defines an index flow like this:
 | [Custom Output Files](examples/custom_output_files) | Convert markdown files to HTML files and save them to a local directory, using *CocoIndex Custom Targets* |
 | [Patient intake form extraction](examples/patient_intake_extraction) | Use LLM to extract structured data from patient intake forms with different formats |
 
-
 More coming and stay tuned 👀!
 
 ## 📖 Documentation
+
 For detailed documentation, visit [CocoIndex Documentation](https://cocoindex.io/docs), including a [Quickstart guide](https://cocoindex.io/docs/getting_started/quickstart).
 
 ## 🤝 Contributing
+
 We love contributions from our community ❤️. For details on contributing or running the project for development, check out our [contributing guide](https://cocoindex.io/docs/about/contributing).
 
 ## 👥 Community
+
 Welcome with a huge coconut hug 🥥⋆｡˚🤗. We are super excited for community contributions of all kinds - whether it's code improvements, documentation updates, issue reports, feature requests, and discussions in our Discord.
 
 Join our community here:
@@ -210,8 +215,10 @@ Join our community here:
 - ▶️ [Subscribe to our YouTube channel](https://www.youtube.com/@cocoindex-io)
 - 📜 [Read our blog posts](https://cocoindex.io/blogs/)
 
-## Support us:
+## Support us
+
 We are constantly improving, and more features and examples are coming soon. If you love this project, please drop us a star ⭐ at GitHub repo [![GitHub](https://img.shields.io/github/stars/cocoindex-io/cocoindex?color=5B5BD6)](https://github.com/cocoindex-io/cocoindex) to stay tuned and help us grow.
 
 ## License
+
 CocoIndex is Apache 2.0 licensed.

{cocoindex-0.2.19 → cocoindex-0.2.20}/THIRD_PARTY_NOTICES.html

@@ -2428,7 +2428,7 @@ Software.
                 <h3 id="Apache-2.0">Apache License 2.0</h3>
                 <h4>Used by:</h4>
                 <ul class="license-used-by">
-                    <li><a href=" https://crates.io/crates/cocoindex ">cocoindex 0.2.19</a></li>
+                    <li><a href=" https://crates.io/crates/cocoindex ">cocoindex 0.2.20</a></li>
                     <li><a href=" https://github.com/awesomized/crc-fast-rust ">crc-fast 1.3.0</a></li>
                     <li><a href=" https://github.com/qdrant/rust-client ">qdrant-client 1.15.0</a></li>
                 </ul>

{cocoindex-0.2.19 → cocoindex-0.2.20}/docs/docs/ai/llm.mdx

@@ -28,6 +28,7 @@ We support the following types of LLM APIs:
 | [LiteLLM](#litellm) | `LlmApiType.LITE_LLM` | ✅ | ❌ |
 | [OpenRouter](#openrouter) | `LlmApiType.OPEN_ROUTER` | ✅ | ❌ |
 | [vLLM](#vllm) | `LlmApiType.VLLM` | ✅ | ❌ |
+| [Bedrock](#bedrock) | `LlmApiType.BEDROCK` | ✅ | ❌ |
 
 ## LLM Tasks
 
@@ -440,3 +441,28 @@ cocoindex.LlmSpec(
 
 </TabItem>
 </Tabs>
+
+### Bedrock
+
+To use the Bedrock API, you need to set up AWS credentials. You can do this by setting the following environment variables:
+
+- `AWS_ACCESS_KEY_ID`
+- `AWS_SECRET_ACCESS_KEY`
+- `AWS_SESSION_TOKEN` (optional)
+
+A spec for Bedrock looks like this:
+
+<Tabs>
+<TabItem value="python" label="Python" default>
+
+```python
+cocoindex.LlmSpec(
+    api_type=cocoindex.LlmApiType.BEDROCK,
+    model="us.anthropic.claude-3-5-haiku-20241022-v1:0",
+)
+```
+
+</TabItem>
+</Tabs>
+
+You can find the full list of models supported by Bedrock [here](https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html).

{cocoindex-0.2.19 → cocoindex-0.2.20}/docs/docs/examples/examples/academic_papers_index.md

@@ -64,7 +64,7 @@ def paper_metadata_flow(
 ```
 
 `flow_builder.add_source` will create a table with sub fields (`filename`, `content`).
-<DocumentationButton url="https://cocoindex.io/docs/ops/sources" text="Sources" margin="0 0 16px 0" />
+<DocumentationButton url="https://cocoindex.io/docs/sources" text="Sources" margin="0 0 16px 0" />
 
 ## Extract and collect metadata
 

{cocoindex-0.2.19 → cocoindex-0.2.20}/docs/docs/examples/examples/codebase_index.md

@@ -70,7 +70,7 @@ def code_embedding_flow(flow_builder: cocoindex.FlowBuilder, data_scope: cocoind
 - Exclude files and directories starting `.`,  `target` in the root and `node_modules` under any directory.
 
 `flow_builder.add_source` will create a table with sub fields (`filename`, `content`).
-<DocumentationButton url="https://cocoindex.io/docs/ops/sources" text="Sources" />
+<DocumentationButton url="https://cocoindex.io/docs/sources" text="Sources" />
 
 
 ## Process each file and collect the information

{cocoindex-0.2.19 → cocoindex-0.2.20}/docs/docs/examples/examples/custom_targets.md

@@ -36,7 +36,7 @@ flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope
 	)
 ```
 This ingestion creates a table with `filename` and `content` fields.
-<DocumentationButton url="https://cocoindex.io/docs/ops/sources" text="Sources" />
+<DocumentationButton url="https://cocoindex.io/docs/sources" text="Sources" />
 
 ## Process each file and collect
 

{cocoindex-0.2.19 → cocoindex-0.2.20}/docs/docs/examples/examples/docs_to_knowledge_graph.md

@@ -66,7 +66,7 @@ def docs_to_kg_flow(flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.D
 Here `flow_builder.add_source` creates a [KTable](https://cocoindex.io/docs/core/data_types#KTable).
 `filename` is the key of the KTable.
 
-<DocumentationButton url="https://cocoindex.io/docs/ops/sources" text="Sources" margin="0 0 16px 0" />
+<DocumentationButton url="https://cocoindex.io/docs/sources" text="Sources" margin="0 0 16px 0" />
 
 
 ### Add data collectors

{cocoindex-0.2.19 → cocoindex-0.2.20}/docs/docs/examples/examples/document_ai.md

@@ -98,7 +98,7 @@ data_scope["documents"] = flow_builder.add_source(
 doc_embeddings = data_scope.add_collector()
 ```
 
-<DocumentationButton url="https://cocoindex.io/docs/ops/sources" text="Source" margin="0 0 16px 0" />
+<DocumentationButton url="https://cocoindex.io/docs/sources" text="Source" margin="0 0 16px 0" />
 
 <DocumentationButton url="https://cocoindex.io/docs/ops/collectors" text="Collector" margin="0 0 16px 0" />
 
@@ -154,4 +154,4 @@ For a step-by-step walkthrough of each indexing stage and the query path, check
 
 CocoIndex natively supports Google Drive, Amazon S3, Azure Blob Storage, and more with native incremental processing out of box - when new or updated files are detected, the pipeline will capture the changes and only process what's changed.
 
-<DocumentationButton url="https://cocoindex.io/docs/ops/sources" text="Sources" margin="0 0 16px 0" />
+<DocumentationButton url="https://cocoindex.io/docs/sources" text="Sources" margin="0 0 16px 0" />

{cocoindex-0.2.19 → cocoindex-0.2.20}/docs/docs/examples/examples/image_search.md

@@ -66,7 +66,7 @@ def image_object_embedding_flow(flow_builder, data_scope):
 
 The `add_source` function sets up a table with fields like `filename` and `content`. Images are automatically re-scanned every minute.
 
-<DocumentationButton url="https://cocoindex.io/docs/ops/sources/localfile" text="LocalFile" />
+<DocumentationButton url="https://cocoindex.io/docs/sources/localfile" text="LocalFile" />
 
 
 ## Process Each Image and Collect the Embedding
@@ -266,6 +266,6 @@ One of CocoIndex’s core strengths is its ability to connect to your existing d
 - Amazon S3 / SQS
 - Azure Blob Storage
 
-<DocumentationButton url="https://cocoindex.io/docs/ops/sources" text="Sources" margin="0 0 16px 0" />
+<DocumentationButton url="https://cocoindex.io/docs/sources" text="Sources" margin="0 0 16px 0" />
 
 Once connected, CocoIndex continuously watches for changes — new uploads, updates, or deletions — and applies them to your index in real time.

{cocoindex-0.2.19 → cocoindex-0.2.20}/docs/docs/examples/examples/manual_extraction.md

@@ -67,7 +67,7 @@ def manual_extraction_flow(
 - `filename` (key, type: `str`): the filename of the file, e.g. `dir1/file1.md`
 - `content` (type: `str` if `binary` is `False`, otherwise `bytes`): the content of the file
 
-<DocumentationButton url="https://cocoindex.io/docs/ops/sources" text="LocalFile" margin="0 0 16px 0" />
+<DocumentationButton url="https://cocoindex.io/docs/sources" text="LocalFile" margin="0 0 16px 0" />
 
 ## Parse Markdown
 

{cocoindex-0.2.19 → cocoindex-0.2.20}/docs/docs/examples/examples/multi_format_index.md

@@ -52,7 +52,7 @@ data_scope["documents"] = flow_builder.add_source(
     cocoindex.sources.LocalFile(path="source_files", binary=True)
 )
 ```
-<DocumentationButton url="https://cocoindex.io/docs/ops/sources/localfile" text="LocalFile" margin="0 0 16px 0" />
+<DocumentationButton url="https://cocoindex.io/docs/sources/localfile" text="LocalFile" margin="0 0 16px 0" />
 
 
 ## Convert Files to Pages
@@ -203,4 +203,4 @@ Follow the url `https://cocoindex.io/cocoinsight`.  It connects to your local Co
 ## Connect to other sources
 CocoIndex natively supports Google Drive, Amazon S3, Azure Blob Storage, and more.
 
-<DocumentationButton url="https://cocoindex.io/docs/ops/sources" text="Sources" margin="0 0 16px 0" />
+<DocumentationButton url="https://cocoindex.io/docs/sources" text="Sources" margin="0 0 16px 0" />

{cocoindex-0.2.19 → cocoindex-0.2.20}/docs/docs/examples/examples/patient_form_extraction.md

@@ -66,7 +66,7 @@ def patient_intake_extraction_flow(
 
 `flow_builder.add_source` will create a table with a few sub fields.
 
-<DocumentationButton url="https://cocoindex.io/docs/ops/sources" text="Sources" margin="0 0 16px 0" />
+<DocumentationButton url="https://cocoindex.io/docs/sources" text="Sources" margin="0 0 16px 0" />
 
 
 ##  Parse documents with different formats to Markdown
@@ -298,4 +298,4 @@ Click on the `markdown` column for `Patient_Intake_Form_Joe.pdf`, you could see
 ## Connect to other sources
 CocoIndex natively supports Google Drive, Amazon S3, Azure Blob Storage, and more.
 
-<DocumentationButton url="https://cocoindex.io/docs/ops/sources" text="Sources" margin="0 0 16px 0" />
+<DocumentationButton url="https://cocoindex.io/docs/sources" text="Sources" margin="0 0 16px 0" />

{cocoindex-0.2.19 → cocoindex-0.2.20}/docs/docs/examples/examples/photo_search.md

@@ -65,8 +65,8 @@ def face_recognition_flow(flow_builder, data_scope):
 This creates a table with `filename` and `content` fields. 📂
 
 
-You can connect it to your [S3 Buckets](https://cocoindex.io/docs/ops/sources/amazons3) (with SQS integration, [example](https://cocoindex.io/blogs/s3-incremental-etl))
-or [Azure Blob store](https://cocoindex.io/docs/ops/sources/azureblob).
+You can connect it to your [S3 Buckets](https://cocoindex.io/docs/sources/amazons3) (with SQS integration, [example](https://cocoindex.io/blogs/s3-incremental-etl))
+or [Azure Blob store](https://cocoindex.io/docs/sources/azureblob).
 
 ## Detect and Extract Faces
 

{cocoindex-0.2.19 → cocoindex-0.2.20}/docs/docs/examples/examples/postgres_source.md

@@ -59,7 +59,7 @@ CocoIndex incrementally sync data from Postgres. When new or updated rows are fo
 - `notification` enables change capture based on Postgres LISTEN/NOTIFY. Each change triggers an incremental processing on the specific row immediately.
 - Regardless if `notification` is provided or not, CocoIndex still needs to scan the full table to detect changes in some scenarios (e.g. between two `update` invocation), and the `ordinal_column` provides a field that CocoIndex can use to quickly detect which row has changed without reading value columns.
 
-Check [Postgres source](https://cocoindex.io/docs/ops/sources/postgres) for more details.
+Check [Postgres source](https://cocoindex.io/docs/sources/postgres) for more details.
 
 If you use the Postgres database hosted by Supabase, please click Connect on your project dashboard and find the URL there. Check [DatabaseConnectionSpec](https://cocoindex.io/docs/core/settings#databaseconnectionspec)
 for more details.

{cocoindex-0.2.19 → cocoindex-0.2.20}/docs/docs/examples/examples/product_recommendation.md

@@ -30,7 +30,7 @@ Product taxonomy is a way to organize product catalogs in a logical and hierarch
 
 ## Prerequisites
 *   [Install PostgreSQL](https://cocoindex.io/docs/getting_started/installation#-install-postgres). CocoIndex uses PostgreSQL internally for incremental processing.
-*   [Install Neo4j](https://cocoindex.io/docs/ops/storages#Neo4j), a graph database.
+*   [Install Neo4j](https://cocoindex.io/docs/targets/neo4j), a graph database.
 *   -  [Configure your OpenAI API key](https://cocoindex.io/docs/ai/llm#openai). Create a `.env` file from `.env.example`, and fill `OPENAI_API_KEY`.
 
 Alternatively, we have native support for Gemini, Ollama, LiteLLM. You can choose your favorite LLM provider and work completely on-premises.

{cocoindex-0.2.19 → cocoindex-0.2.20}/docs/docs/examples/examples/simple_vector_index.md

@@ -51,7 +51,7 @@ def text_embedding_flow(flow_builder: cocoindex.FlowBuilder, data_scope: cocoind
 ```
 
 `flow_builder.add_source` will create a table with sub fields (`filename`, `content`)
-<DocumentationButton url="https://cocoindex.io/docs/ops/sources" text="Source" />
+<DocumentationButton url="https://cocoindex.io/docs/sources" text="Source" />
 
 
 ## Process each file and collect the embeddings

{cocoindex-0.2.19 → cocoindex-0.2.20}/docs/docs/getting_started/quickstart.md

@@ -64,7 +64,7 @@ doc_embeddings = data_scope.add_collector()
 
 `flow_builder.add_source` will create a table with sub fields (`filename`, `content`)
 
-<DocumentationButton url="https://cocoindex.io/docs/ops/sources" text="Source" />
+<DocumentationButton url="https://cocoindex.io/docs/sources" text="Source" />
 
 <DocumentationButton url="https://cocoindex.io/docs/core/flow_def#data-collector" text="Data Collector" />
 

cocoindex-0.2.20/docs/docs/sources/index.md

@@ -0,0 +1,22 @@
+---
+title: Sources
+toc_max_heading_level: 4
+description: CocoIndex Built-in Sources
+---
+
+# CocoIndex Built-in Sources
+
+In CocoIndex, a source is the data origin you import from (e.g., files, databases, APIs) that feeds into an indexing flow for transformation and retrieval.
+
+| Source Type    | Description                        |
+|----------------|------------------------------------|
+| [LocalFile](/docs/sources/localfile)     | Local file system                        |
+| [AmazonS3](/docs/sources/amazons3)       | Object store (Amazon S3 bucket)          |
+| [AzureBlob](/docs/sources/azureblob)     | Object store (Azure Blob Storage)        |
+| [GoogleDrive](/docs/sources/googledrive) | Cloud file system (Google Drive)         |
+| [Postgres](/docs/sources/postgres)       | Relational database (Postgres)           |
+
+Related:
+- [Life cycle of a indexing flow](/docs/core/basics#life-cycle-of-an-indexing-flow) 
+- [Live Update Tutorial](/docs/tutorials/live_updates) 
+for change capture mechanisms.

{cocoindex-0.2.19 → cocoindex-0.2.20}/docs/docs/targets/index.md

@@ -18,7 +18,6 @@ The way to map data from a data collector to a target depends on data model of t
 | [Qdrant](/docs/targets/qdrant)     | Vector Database, Keyword Search       |
 | [LanceDB](/docs/targets/lancedb)   | Vector Database, Keyword Search         |
 | [Neo4j](/docs/targets/neo4j)       | [Property graph](#property-graph-targets)        |
-| [Kuzu](/docs/targets/kuzu)         | [Property graph](#property-graph-targets)        |
 
 If you are looking for targets beyond here, you can always use [custom targets](/docs/custom_ops/custom_targets) as building blocks.
 

{cocoindex-0.2.19 → cocoindex-0.2.20}/docs/docs/targets/kuzu.md

@@ -5,7 +5,9 @@ toc_max_heading_level: 4
 ---
 import { ExampleButton } from '../../src/components/GitHubButton';
 
-# Kuzu
+# Kuzu (Archived)
+
+Note:[Kuzu](https://github.com/kuzudb/kuzu) - embedded graph database is no longer maintained.
 
 Exports data to a [Kuzu](https://kuzu.com/) graph database.
 

{cocoindex-0.2.19 → cocoindex-0.2.20}/examples/docs_to_knowledge_graph/README.md

@@ -14,12 +14,11 @@ Please drop [Cocoindex on Github](https://github.com/cocoindex-io/cocoindex) a s
 
 ## Prerequisite
 *   [Install Postgres](https://cocoindex.io/docs/getting_started/installation#-install-postgres) if you don't have one.
-*   Install [Neo4j](https://cocoindex.io/docs/ops/targets#neo4j-dev-instance) or [Kuzu](https://cocoindex.io/docs/ops/targets#kuzu-dev-instance) if you don't have one.
-    *   The example uses Neo4j by default for now. If you want to use Kuzu, find out the "SELECT ONE GRAPH DATABASE TO USE" section and switch the active branch.
+*   Install [Neo4j](https://cocoindex.io/docs/targets/neo4j).
 *   Install / configure LLM API. In this example we use Ollama, which runs LLM model locally. You need to get it ready following [this guide](https://cocoindex.io/docs/ai/llm#ollama). Alternatively, you can also follow the comments in source code to switch to OpenAI, and [configure OpenAI API key](https://cocoindex.io/docs/ai/llm#openai) before running the example.
 
 ## Documentation
-You can read the official CocoIndex Documentation for Property Graph Targets [here](https://cocoindex.io/docs/ops/targets#property-graph-targets).
+You can read the official CocoIndex Documentation for Property Graph Targets [here](https://cocoindex.io/docs/targets#property-graph-targets).
 
 ## Run
 
@@ -48,8 +47,6 @@ cocoindex update main
 After the knowledge graph is built, you can explore the knowledge graph.
 
 * If you're using Neo4j, you can open the explorer at [http://localhost:7474](http://localhost:7474), with username `neo4j` and password `cocoindex`.
-* If you're using Kuzu, you can start a Kuzu explorer locally. See [Kuzu dev instance](https://cocoindex.io/docs/ops/targets#kuzu-dev-instance) for more details.
-
 You can run the following Cypher query to get all relationships:
 
 ```cypher

{cocoindex-0.2.19 → cocoindex-0.2.20}/examples/docs_to_knowledge_graph/main.py

@@ -14,29 +14,12 @@ neo4j_conn_spec = cocoindex.add_auth_entry(
         password="cocoindex",
     ),
 )
-kuzu_conn_spec = cocoindex.add_auth_entry(
-    "KuzuConnection",
-    cocoindex.targets.KuzuConnection(
-        api_server_url="http://localhost:8123",
-    ),
-)
 
-# SELECT ONE GRAPH DATABASE TO USE
-# This example can use either Neo4j or Kuzu as the graph database.
-# Please make sure only one branch is live and others are commented out.
-
-# Use Neo4j
 GraphDbSpec = cocoindex.targets.Neo4j
 GraphDbConnection = cocoindex.targets.Neo4jConnection
 GraphDbDeclaration = cocoindex.targets.Neo4jDeclaration
 conn_spec = neo4j_conn_spec
 
-# Use Kuzu
-#  GraphDbSpec = cocoindex.targets.Kuzu
-#  GraphDbConnection = cocoindex.targets.KuzuConnection
-#  GraphDbDeclaration = cocoindex.targets.KuzuDeclaration
-#  conn_spec = kuzu_conn_spec
-
 
 @dataclasses.dataclass
 class DocumentSummary:

{cocoindex-0.2.19 → cocoindex-0.2.20}/examples/manuals_llm_extraction/main.py

@@ -118,6 +118,9 @@ def manual_extraction_flow(
                 # Replace by this spec below, to use Anthropic API model
                 #   llm_spec=cocoindex.LlmSpec(
                 #       api_type=cocoindex.LlmApiType.ANTHROPIC, model="claude-3-5-sonnet-latest"),
+                # Replace by this spec below, to use Bedrock API model
+                #   llm_spec=cocoindex.LlmSpec(
+                #       api_type=cocoindex.LlmApiType.BEDROCK, model="us.anthropic.claude-3-5-haiku-20241022-v1:0"),
                 output_type=ModuleInfo,
                 instruction="Please extract Python module information from the manual.",
             )

{cocoindex-0.2.19 → cocoindex-0.2.20}/examples/patient_intake_extraction/README.md

@@ -4,7 +4,7 @@ We appreciate a star ⭐ at [CocoIndex Github](https://github.com/cocoindex-io/c
 
 
 This repo shows how to use LLM to extract structured data from patient intake forms with different formats - like PDF, Docx, etc.
-CocoIndex supports multiple [sources](https://cocoindex.io/docs/ops/sources) and [LLM models](https://cocoindex.io/docs/ai/llm) natively.
+CocoIndex supports multiple [sources](https://cocoindex.io/docs/sources) and [LLM models](https://cocoindex.io/docs/ai/llm) natively.
 
 ![Structured Data From Patient Intake Forms](https://github.com/user-attachments/assets/1f6afb69-d26d-4a08-8774-13982d6aec1e)
 

cocoindex-0.2.20/examples/pdf_elements_embedding/.gitignore

@@ -0,0 +1 @@
+/source_files

cocoindex-0.2.20/examples/pdf_elements_embedding/README.md

@@ -0,0 +1,71 @@
+# Extract text and images from PDFs and build multimodal search
+
+[![GitHub](https://img.shields.io/github/stars/cocoindex-io/cocoindex?color=5B5BD6)](https://github.com/cocoindex-io/cocoindex)
+
+In this example, we extract texts and images from PDF pages, embed them with two models, and store them in Qdrant for multimodal search:
+
+- Text: SentenceTransformers `all-MiniLM-L6-v2`
+- Images: CLIP `openai/clip-vit-large-patch14` (ViT-L/14, 768-dim)
+
+We appreciate a star ⭐ at [CocoIndex Github](https://github.com/cocoindex-io/cocoindex) if this is helpful.
+
+## Steps
+
+### Indexing Flow
+
+1. Ingest PDF files from the `source_files` directory.
+2. For each PDF page:
+   - Extract page text and images using `pypdf`.
+   - Skip very small images and create thumbnails up to 512×512 for consistency.
+   - Split text into chunks with `SplitRecursively` (language="text", chunk_size=600, chunk_overlap=100).
+   - Embed text chunks with SentenceTransformers (`all-MiniLM-L6-v2`).
+   - Embed images with CLIP (`openai/clip-vit-large-patch14`).
+3. Save embeddings and metadata in Qdrant:
+   - Text collection: `PdfElementsEmbeddingText`
+   - Image collection: `PdfElementsEmbeddingImage`
+
+## Prerequisite
+
+[Install Qdrant](https://qdrant.tech/documentation/guides/installation/) if you don't have one running locally.
+
+Start Qdrant with Docker (exposes HTTP 6333 and gRPC 6334):
+
+```bash
+docker run -p 6333:6333 -p 6334:6334 qdrant/qdrant
+```
+
+Note: This example connects via gRPC at `http://localhost:6334`.
+
+## Input Data Preparation
+
+Download a few sample PDFs (all are board game manuals) and put them into the `source_files` directory by running:
+
+```bash
+./fetch_manual_urls.sh
+```
+
+You can also put your favorite PDFs into the `source_files` directory.
+
+## Run
+
+Install dependencies:
+
+```bash
+pip install -e .
+```
+
+Update index, which will also setup the tables at the first time:
+
+```bash
+cocoindex update --setup main
+```
+
+## CocoInsight
+
+I used CocoInsight (Free beta now) to troubleshoot the index generation and understand the data lineage of the pipeline. It just connects to your local CocoIndex server, with Zero pipeline data retention. Run following command to start CocoInsight:
+
+```bash
+cocoindex server -ci main
+```
+
+Then open the CocoInsight UI at [https://cocoindex.io/cocoinsight](https://cocoindex.io/cocoinsight).