vector-graph-rag 0.2.0__tar.gz → 0.2.2__tar.gz

{vector_graph_rag-0.2.0 → vector_graph_rag-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: vector-graph-rag
-Version: 0.2.0
+Version: 0.2.2
 Summary: A Graph RAG implementation using pure vector search with Milvus
 Project-URL: Homepage, https://github.com/zilliztech/vector-graph-rag
 Project-URL: Documentation, https://zilliztech.github.io/vector-graph-rag/
@@ -42,6 +42,8 @@ Requires-Dist: mistralai>=1.0; extra == 'all'
 Requires-Dist: ollama>=0.4; extra == 'all'
 Requires-Dist: onnxruntime<1.24,>=1.17; (python_version == '3.10') and extra == 'all'
 Requires-Dist: onnxruntime>=1.17; (python_version >= '3.11') and extra == 'all'
+Requires-Dist: opentelemetry-api>=1.0; extra == 'all'
+Requires-Dist: opentelemetry-sdk>=1.0; extra == 'all'
 Requires-Dist: pytest-asyncio>=0.21.0; extra == 'all'
 Requires-Dist: pytest>=7.0.0; extra == 'all'
 Requires-Dist: ruff>=0.1.0; extra == 'all'
@@ -55,9 +57,14 @@ Provides-Extra: api
 Requires-Dist: fastapi>=0.109.0; extra == 'api'
 Requires-Dist: uvicorn[standard]>=0.27.0; extra == 'api'
 Provides-Extra: dev
+Requires-Dist: opentelemetry-sdk>=1.0; extra == 'dev'
 Requires-Dist: pytest-asyncio>=0.21.0; extra == 'dev'
 Requires-Dist: pytest>=7.0.0; extra == 'dev'
 Requires-Dist: ruff>=0.1.0; extra == 'dev'
+Provides-Extra: docling
+Requires-Dist: docling>=2.0.0; extra == 'docling'
+Requires-Dist: lxml-html-clean; extra == 'docling'
+Requires-Dist: trafilatura>=2.0.0; extra == 'docling'
 Provides-Extra: docs
 Requires-Dist: mkdocs-material>=9.0.0; extra == 'docs'
 Provides-Extra: google
@@ -75,8 +82,15 @@ Provides-Extra: local
 Requires-Dist: einops>=0.8.2; extra == 'local'
 Requires-Dist: sentence-transformers>=3.0; extra == 'local'
 Requires-Dist: torch>=2.0.0; extra == 'local'
+Provides-Extra: mineru
+Requires-Dist: lxml-html-clean; extra == 'mineru'
+Requires-Dist: markitdown[docx,pdf]>=0.1.4; extra == 'mineru'
+Requires-Dist: mineru[pipeline]>=3.0.0; extra == 'mineru'
+Requires-Dist: trafilatura>=2.0.0; extra == 'mineru'
 Provides-Extra: mistral
 Requires-Dist: mistralai>=1.0; extra == 'mistral'
+Provides-Extra: observability
+Requires-Dist: opentelemetry-api>=1.0; extra == 'observability'
 Provides-Extra: ollama
 Requires-Dist: ollama>=0.4; extra == 'ollama'
 Provides-Extra: onnx
@@ -155,6 +169,17 @@ uv add "vector-graph-rag[all]"
 
 </details>
 
+<details>
+<summary><b>With OpenTelemetry tracing</b></summary>
+
+```bash
+pip install "vector-graph-rag[observability]"
+# or
+uv add "vector-graph-rag[observability]"
+```
+
+</details>
+
 ## 🚀 Quick Start
 
 ```python
@@ -199,6 +224,11 @@ Use `upsert_documents_by_source()` when a source file, message, or page is
 created or modified. In Vector Graph RAG, a `Document` is a passage/chunk; the
 source object is identified by `metadata["source"]` or the explicit `source`
 argument. The method replaces only that source's chunks and graph references.
+Source-level writes are not transactionally atomic, but the same upsert/delete
+operation can be retried after an interruption to converge the source back to a
+consistent state.
+See the [Incremental Updates guide](docs/incremental-updates.md) for parser
+integration, source key design, and retry recommendations.
 
 ```python
 from langchain_core.documents import Document
@@ -208,7 +238,7 @@ rag.upsert_documents_by_source(
         Document(
             page_content="Einstein developed relativity at Princeton.",
             metadata={
-                "source": "sharepoint:file-123",
+                "source": "file:file-123",
                 "triplets": [
                     ["Einstein", "developed", "relativity"],
                     ["Einstein", "worked at", "Princeton"],
@@ -219,7 +249,7 @@ rag.upsert_documents_by_source(
     extract_triplets=False,
 )
 
-rag.delete_documents_by_source("sharepoint:file-123")
+rag.delete_documents_by_source("file:file-123")
 ```
 
 > **Migration note:** v0.1.5 exposed `upsert_documents(document_id=...)` and
@@ -233,6 +263,34 @@ for removal in v1.0.0. For explicit full refreshes, use `rebuild_texts()`,
 
 </details>
 
+<details>
+<summary>📈 <b>OpenTelemetry tracing</b> — click to expand</summary>
+
+Vector Graph RAG can emit OpenTelemetry spans for ingestion, loaders,
+embeddings, Milvus operations, retrieval, and generation. Configure the
+OpenTelemetry SDK/exporter in your application, then attach request context
+around RAG calls:
+
+```python
+from vector_graph_rag import VectorGraphRAG, observability_context
+
+rag = VectorGraphRAG(collection_prefix="my_project")
+
+with observability_context(
+    request_id="req-123",
+    tenant_id="tenant-a",
+    graph_name="my_project",
+    source="file-123",
+):
+    rag.upsert_documents_by_source(chunks, source="file-123")
+```
+
+The built-in spans avoid document text, prompts, query text, generated answers,
+filters, and full URLs by default. See the
+[Observability guide](docs/observability.md) for setup details.
+
+</details>
+
 <details>
 <summary>🌐 <b>Import from URLs and files</b> — click to expand</summary>
 

{vector_graph_rag-0.2.0 → vector_graph_rag-0.2.2}/README.md

@@ -65,6 +65,17 @@ uv add "vector-graph-rag[all]"
 
 </details>
 
+<details>
+<summary><b>With OpenTelemetry tracing</b></summary>
+
+```bash
+pip install "vector-graph-rag[observability]"
+# or
+uv add "vector-graph-rag[observability]"
+```
+
+</details>
+
 ## 🚀 Quick Start
 
 ```python
@@ -109,6 +120,11 @@ Use `upsert_documents_by_source()` when a source file, message, or page is
 created or modified. In Vector Graph RAG, a `Document` is a passage/chunk; the
 source object is identified by `metadata["source"]` or the explicit `source`
 argument. The method replaces only that source's chunks and graph references.
+Source-level writes are not transactionally atomic, but the same upsert/delete
+operation can be retried after an interruption to converge the source back to a
+consistent state.
+See the [Incremental Updates guide](docs/incremental-updates.md) for parser
+integration, source key design, and retry recommendations.
 
 ```python
 from langchain_core.documents import Document
@@ -118,7 +134,7 @@ rag.upsert_documents_by_source(
         Document(
             page_content="Einstein developed relativity at Princeton.",
             metadata={
-                "source": "sharepoint:file-123",
+                "source": "file:file-123",
                 "triplets": [
                     ["Einstein", "developed", "relativity"],
                     ["Einstein", "worked at", "Princeton"],
@@ -129,7 +145,7 @@ rag.upsert_documents_by_source(
     extract_triplets=False,
 )
 
-rag.delete_documents_by_source("sharepoint:file-123")
+rag.delete_documents_by_source("file:file-123")
 ```
 
 > **Migration note:** v0.1.5 exposed `upsert_documents(document_id=...)` and
@@ -143,6 +159,34 @@ for removal in v1.0.0. For explicit full refreshes, use `rebuild_texts()`,
 
 </details>
 
+<details>
+<summary>📈 <b>OpenTelemetry tracing</b> — click to expand</summary>
+
+Vector Graph RAG can emit OpenTelemetry spans for ingestion, loaders,
+embeddings, Milvus operations, retrieval, and generation. Configure the
+OpenTelemetry SDK/exporter in your application, then attach request context
+around RAG calls:
+
+```python
+from vector_graph_rag import VectorGraphRAG, observability_context
+
+rag = VectorGraphRAG(collection_prefix="my_project")
+
+with observability_context(
+    request_id="req-123",
+    tenant_id="tenant-a",
+    graph_name="my_project",
+    source="file-123",
+):
+    rag.upsert_documents_by_source(chunks, source="file-123")
+```
+
+The built-in spans avoid document text, prompts, query text, generated answers,
+filters, and full URLs by default. See the
+[Observability guide](docs/observability.md) for setup details.
+
+</details>
+
 <details>
 <summary>🌐 <b>Import from URLs and files</b> — click to expand</summary>
 

{vector_graph_rag-0.2.0 → vector_graph_rag-0.2.2}/docs/faq.md

@@ -61,7 +61,7 @@ Frequently asked questions about Vector Graph RAG — covering when to use it, c
     rag.delete_documents_by_source("file-123")
     ```
 
-    Incremental updates replace only the chunks and graph references for that source value. They are not transactionally atomic, so avoid interrupting or concurrently mutating the same collection prefix during an update.
+    Incremental updates replace only the chunks and graph references for that source value. They are not transactionally atomic. If a write is interrupted, rerun the same source operation to converge the source back to a consistent state. See [Incremental Updates](incremental-updates.md) for the full parser, CUD, and retry pattern.
 
 ??? note "Can I use local/open-source LLMs?"
     Yes. Vector Graph RAG uses the OpenAI-compatible API format, so any LLM that exposes an OpenAI-compatible endpoint will work. This includes local models served via [Ollama](https://ollama.com/), [vLLM](https://github.com/vllm-project/vllm), [LM Studio](https://lmstudio.ai/), or any other OpenAI-compatible server. You can configure the base URL and model name when initializing the RAG instance. Keep in mind that triplet extraction and reranking quality depend heavily on the LLM's capability — weaker models may produce incomplete or inaccurate triplets, which directly affects retrieval quality. For best results, use a model with strong instruction-following and reasoning abilities.

{vector_graph_rag-0.2.0 → vector_graph_rag-0.2.2}/docs/getting-started.md

@@ -32,6 +32,12 @@
     pip install "vector-graph-rag[all]"
     ```
 
+=== "With observability"
+
+    ```bash
+    pip install "vector-graph-rag[observability]"
+    ```
+
 !!! note "Prerequisites"
     - Python 3.9+
     - An OpenAI API key (set `OPENAI_API_KEY` environment variable)
@@ -180,12 +186,32 @@ rag.rebuild_documents(result.documents, extract_triplets=True)
 ```
 
 !!! warning "Loader dependencies"
-    Install with `pip install "vector-graph-rag[loaders]"` to enable URL fetching and document conversion.
+    Install with `uv add "vector-graph-rag[loaders]"` to enable URL fetching and document conversion.
+
+To use a different local parser, pass a converter into `DocumentImporter`:
+
+```python
+from vector_graph_rag.loaders import DoclingConverter, DocumentImporter
+
+importer = DocumentImporter(
+    converter=DoclingConverter(),
+    chunk_size=1000,
+    chunk_overlap=200,
+)
+result = importer.import_sources(["/path/to/document.pdf"])
+```
+
+Install Docling support with `uv add "vector-graph-rag[docling]"`.
 
 ### Incremental Updates
 
 Use `upsert_documents_by_source()` when a source file, page, or message is created or modified. In Vector Graph RAG, a `Document` is a passage/chunk, and `metadata["source"]` identifies the source object whose chunks should be replaced.
 
+For parser and loader workflows, attach the same stable source value to every
+chunk produced from one file, page, or message. See
+[Incremental Updates](incremental-updates.md) for the full source-level CUD and
+retry pattern.
+
 ```python
 from langchain_core.documents import Document
 
@@ -194,7 +220,7 @@ rag.upsert_documents_by_source(
         Document(
             page_content="Einstein developed relativity at Princeton.",
             metadata={
-                "source": "sharepoint:file-123",
+                "source": "file:file-123",
                 "triplets": [
                     ["Einstein", "developed", "relativity"],
                     ["Einstein", "worked at", "Princeton"],
@@ -205,14 +231,34 @@ rag.upsert_documents_by_source(
     extract_triplets=False,
 )
 
-rag.delete_documents_by_source("sharepoint:file-123")
+rag.delete_documents_by_source("file:file-123")
 ```
 
 !!! warning "v0.2.0 migration"
     `upsert_documents(document_id=...)` and `delete_documents(document_id)` were removed because `Document` means passage/chunk in this project. Use `upsert_documents_by_source()` and `delete_documents_by_source()` with a stable `metadata["source"]` value instead.
 
 !!! note "Consistency model"
-    Incremental updates perform a source-level cascade across passages, entities, and relations. They are not transactionally atomic, so avoid interrupting or concurrently mutating the same collection prefix during an update.
+    Incremental updates perform a source-level cascade across passages, entities, and relations. They are not transactionally atomic, and queries may observe intermediate state if a write is interrupted. If an upsert or delete fails, rerun the same operation for the same source to converge the source back to a consistent state. Avoid concurrently mutating the same collection prefix during an update.
+
+## Observability
+
+Install `vector-graph-rag[observability]` to enable OpenTelemetry trace instrumentation. The library creates spans for ingestion, loading, embeddings, Milvus operations, retrieval, and generation, while your application configures the OpenTelemetry SDK and exporter.
+
+```python
+from vector_graph_rag import VectorGraphRAG, observability_context
+
+rag = VectorGraphRAG(collection_prefix="my_project")
+
+with observability_context(
+    request_id="req-123",
+    tenant_id="tenant-a",
+    graph_name="my_project",
+    source="file-123",
+):
+    rag.upsert_documents_by_source(chunks, source="file-123")
+```
+
+See [Observability](observability.md) for setup, span coverage, and data-safety notes.
 
 ## Querying
 

{vector_graph_rag-0.2.0 → vector_graph_rag-0.2.2}/docs/how-it-works.md

@@ -223,4 +223,4 @@ flowchart TD
     Triplet extraction and reranking quality depends heavily on the underlying LLM's capability. Weaker models may produce incomplete or inaccurate triplets, which directly affects retrieval quality.
 
 !!! warning "Graph Consistency"
-    The knowledge graph topology is maintained as a logical layer on top of Milvus via cross-referenced ID fields. Since Milvus does not support transactions, multi-step mutations (e.g., source-level cascade updates) are not atomic and may leave inconsistent state if interrupted. Use `rebuild_documents()` for initial bulk indexing or full refreshes, and use `upsert_documents_by_source()` / `delete_documents_by_source()` for source updates when the write operation is expected to run to completion.
+    The knowledge graph topology is maintained as a logical layer on top of Milvus via cross-referenced ID fields. Since Milvus does not support transactions, multi-step mutations (e.g., source-level cascade updates) are not atomic, and queries may observe intermediate state if a write is interrupted. Source-level `upsert_documents_by_source()` and `delete_documents_by_source()` operations are designed to be retryable: rerun the same operation for the same source after a failure to converge the source back to a consistent state. Use `rebuild_documents()` for initial bulk indexing or full refreshes.

vector_graph_rag-0.2.2/docs/incremental-updates.md

@@ -0,0 +1,245 @@
+# Incremental Updates
+
+Use source-level incremental updates when your upstream system can tell you
+which external object changed. A source can be a file, web page, email message,
+database row, or any other stable business object.
+
+In Vector Graph RAG, a LangChain `Document` represents one passage or chunk.
+The source object is tracked through metadata, not through `Document.id`.
+
+## Core Contract
+
+Each incremental update call must contain chunks from exactly one source.
+
+```python
+from langchain_core.documents import Document
+
+chunks = [
+    Document(
+        page_content="Q2 revenue increased in the enterprise segment.",
+        metadata={
+            "source": "file:file-123",
+            "page": 1,
+            "chunk_index": 0,
+        },
+    ),
+    Document(
+        page_content="Renewal rates improved after support response times dropped.",
+        metadata={
+            "source": "file:file-123",
+            "page": 2,
+            "chunk_index": 1,
+        },
+    ),
+]
+
+rag.upsert_documents_by_source(chunks)
+```
+
+The default grouping field is `metadata["source"]`. You can also pass the
+source explicitly:
+
+```python
+rag.upsert_documents_by_source(
+    documents=chunks,
+    source="file:file-123",
+)
+```
+
+Use a custom field when your application already has a different metadata name:
+
+```python
+rag.upsert_documents_by_source(
+    documents=chunks,
+    source="file-123",
+    source_field="file_id",
+)
+
+rag.delete_documents_by_source("file-123", source_field="file_id")
+```
+
+## Parser Or Loader Output
+
+Document parsing is intentionally outside the core incremental API. Use your
+existing loader or parser to produce text chunks, then attach the same stable
+source value to every chunk from the same external object.
+
+```python
+from langchain_core.documents import Document
+from vector_graph_rag import VectorGraphRAG
+
+
+def parse_file_to_chunks(path: str) -> list[str]:
+    """Return text chunks from your parser or document processing pipeline."""
+    return [
+        "The handbook describes the support escalation policy.",
+        "Escalated issues must include the customer impact and owner.",
+    ]
+
+
+def index_file(rag: VectorGraphRAG, path: str, file_id: str) -> None:
+    chunks = [
+        Document(
+            page_content=text,
+            metadata={
+                "source": f"file:{file_id}",
+                "chunk_index": index,
+                "parser": "internal",
+            },
+        )
+        for index, text in enumerate(parse_file_to_chunks(path))
+    ]
+
+    rag.upsert_documents_by_source(chunks, extract_triplets=True)
+```
+
+If your parser also extracts knowledge graph triplets, store them in each
+chunk's `metadata["triplets"]` and disable LLM triplet extraction:
+
+```python
+chunks = [
+    Document(
+        page_content="The handbook describes the support escalation policy.",
+        metadata={
+            "source": "file:file-123",
+            "triplets": [
+                ["handbook", "describes", "support escalation policy"],
+                ["escalated issues", "include", "customer impact"],
+            ],
+        },
+    )
+]
+
+rag.upsert_documents_by_source(chunks, extract_triplets=False)
+```
+
+## Create, Update, Delete
+
+Use the same upsert API for both create and update.
+
+```python
+# Create or replace one source.
+rag.upsert_documents_by_source(
+    documents=chunks,
+    source="file:file-123",
+)
+
+# Delete one source.
+rag.delete_documents_by_source("file:file-123")
+```
+
+When a source already exists, `upsert_documents_by_source()` removes the old
+chunks and graph references for that source, then inserts the new chunks. Other
+sources under the same collection prefix are preserved.
+
+For multi-source updates, call the API once per source:
+
+```python
+for source, chunks in changed_sources:
+    rag.upsert_documents_by_source(chunks, source=source)
+```
+
+## Retry Behavior
+
+Source-level updates perform a multi-step cascade across passages, relations,
+and entities. Milvus does not provide a transaction that spans those logical
+records, so these writes are not transactionally atomic. If the process is
+interrupted, queries may observe intermediate state until the update is retried.
+
+The source-level APIs are designed to be retryable. If an upsert or delete
+raises an exception, rerun the same operation for the same source.
+
+```python
+try:
+    rag.upsert_documents_by_source(
+        documents=chunks,
+        source="file:file-123",
+    )
+except Exception:
+    # Log the failed source in your application job state, then retry it.
+    rag.upsert_documents_by_source(
+        documents=chunks,
+        source="file:file-123",
+    )
+```
+
+For delete:
+
+```python
+try:
+    rag.delete_documents_by_source("file:file-123")
+except Exception:
+    rag.delete_documents_by_source("file:file-123")
+```
+
+A production ingestion job should track which source keys succeeded or failed.
+If a retry sees no matching passages during delete, the source may already have
+been removed by the previous attempt.
+
+## ID Guidance
+
+Use `source` for the external object identity. Use `Document.id` only when you
+need to control the passage ID.
+
+If `Document.id` is omitted, `upsert_documents_by_source()` generates stable
+passage IDs from the source value and chunk index. If you provide `Document.id`,
+make sure it is unique to that passage and does not belong to another source.
+
+Good source values are stable and globally meaningful in your application:
+
+```text
+file:file-123
+url:https://example.com/docs/pricing
+message:message-456
+record:account-789
+```
+
+Avoid using a chunk ID, page number, or transient parser output path as the
+source. Those values identify a passage or a local processing artifact, not the
+external object that should be replaced as a unit.
+
+## Metadata And Filtering
+
+Additional metadata is stored on passages and can be used for filtering.
+
+```python
+rag.upsert_documents_by_source(
+    documents=chunks,
+    source="file:file-123",
+    metadata={
+        "tenant_id": "tenant-a",
+        "workspace_id": "finance",
+    },
+)
+
+result = rag.query(
+    "What should escalated issues include?",
+    filter='tenant_id == "tenant-a" and workspace_id == "finance"',
+)
+```
+
+The source metadata is also stored on each passage, so you can filter by source
+when needed:
+
+```python
+result = rag.query(
+    "What should escalated issues include?",
+    filter='source == "file:file-123"',
+)
+```
+
+## Initial Bulk Load
+
+For a first-time corpus load, you can either use `rebuild_documents()` or call
+`upsert_documents_by_source()` once per source. Use `rebuild_documents()` when
+you intentionally want a full refresh of the collection prefix. Use
+`upsert_documents_by_source()` when you want the same ingestion path for initial
+load and later CUD events.
+
+```python
+for source, chunks in all_sources:
+    rag.upsert_documents_by_source(chunks, source=source)
+```
+
+Avoid the legacy `add_*` helpers for new ingestion code. They keep their old
+full-rebuild behavior and are planned for removal in v1.0.0.

{vector_graph_rag-0.2.0 → vector_graph_rag-0.2.2}/docs/index.md

@@ -114,7 +114,7 @@ print(result.answer)
 ```
 
 !!! note "Ingestion semantics"
-    The legacy `add_*` ingestion helpers rebuild the full knowledge base and are planned for removal in v1.0.0. Use `rebuild_texts()`, `rebuild_documents()`, or `rebuild_documents_with_triplets()` for full refreshes, and use `upsert_documents_by_source()` / `delete_documents_by_source()` when a source file, page, or message changes later.
+    The legacy `add_*` ingestion helpers rebuild the full knowledge base and are planned for removal in v1.0.0. Use `rebuild_texts()`, `rebuild_documents()`, or `rebuild_documents_with_triplets()` for full refreshes, and use `upsert_documents_by_source()` / `delete_documents_by_source()` when a source file, page, or message changes later. See [Incremental Updates](incremental-updates.md) for parser integration, source key design, and retry recommendations.
 
 !!! tip "Getting Started"
     See the [Getting Started](getting-started.md) guide for installation and configuration options.