vector-graph-rag 0.2.0__tar.gz → 0.2.1__tar.gz

{vector_graph_rag-0.2.0 → vector_graph_rag-0.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: vector-graph-rag
-Version: 0.2.0
+Version: 0.2.1
 Summary: A Graph RAG implementation using pure vector search with Milvus
 Project-URL: Homepage, https://github.com/zilliztech/vector-graph-rag
 Project-URL: Documentation, https://zilliztech.github.io/vector-graph-rag/
@@ -199,6 +199,9 @@ Use `upsert_documents_by_source()` when a source file, message, or page is
 created or modified. In Vector Graph RAG, a `Document` is a passage/chunk; the
 source object is identified by `metadata["source"]` or the explicit `source`
 argument. The method replaces only that source's chunks and graph references.
+Source-level writes are not transactionally atomic, but the same upsert/delete
+operation can be retried after an interruption to converge the source back to a
+consistent state.
 
 ```python
 from langchain_core.documents import Document

{vector_graph_rag-0.2.0 → vector_graph_rag-0.2.1}/README.md

@@ -109,6 +109,9 @@ Use `upsert_documents_by_source()` when a source file, message, or page is
 created or modified. In Vector Graph RAG, a `Document` is a passage/chunk; the
 source object is identified by `metadata["source"]` or the explicit `source`
 argument. The method replaces only that source's chunks and graph references.
+Source-level writes are not transactionally atomic, but the same upsert/delete
+operation can be retried after an interruption to converge the source back to a
+consistent state.
 
 ```python
 from langchain_core.documents import Document

{vector_graph_rag-0.2.0 → vector_graph_rag-0.2.1}/docs/getting-started.md

@@ -212,7 +212,7 @@ rag.delete_documents_by_source("sharepoint:file-123")
     `upsert_documents(document_id=...)` and `delete_documents(document_id)` were removed because `Document` means passage/chunk in this project. Use `upsert_documents_by_source()` and `delete_documents_by_source()` with a stable `metadata["source"]` value instead.
 
 !!! note "Consistency model"
-    Incremental updates perform a source-level cascade across passages, entities, and relations. They are not transactionally atomic, so avoid interrupting or concurrently mutating the same collection prefix during an update.
+    Incremental updates perform a source-level cascade across passages, entities, and relations. They are not transactionally atomic, and queries may observe intermediate state if a write is interrupted. If an upsert or delete fails, rerun the same operation for the same source to converge the source back to a consistent state. Avoid concurrently mutating the same collection prefix during an update.
 
 ## Querying
 

{vector_graph_rag-0.2.0 → vector_graph_rag-0.2.1}/docs/how-it-works.md

@@ -223,4 +223,4 @@ flowchart TD
     Triplet extraction and reranking quality depends heavily on the underlying LLM's capability. Weaker models may produce incomplete or inaccurate triplets, which directly affects retrieval quality.
 
 !!! warning "Graph Consistency"
-    The knowledge graph topology is maintained as a logical layer on top of Milvus via cross-referenced ID fields. Since Milvus does not support transactions, multi-step mutations (e.g., source-level cascade updates) are not atomic and may leave inconsistent state if interrupted. Use `rebuild_documents()` for initial bulk indexing or full refreshes, and use `upsert_documents_by_source()` / `delete_documents_by_source()` for source updates when the write operation is expected to run to completion.
+    The knowledge graph topology is maintained as a logical layer on top of Milvus via cross-referenced ID fields. Since Milvus does not support transactions, multi-step mutations (e.g., source-level cascade updates) are not atomic, and queries may observe intermediate state if a write is interrupted. Source-level `upsert_documents_by_source()` and `delete_documents_by_source()` operations are designed to be retryable: rerun the same operation for the same source after a failure to converge the source back to a consistent state. Use `rebuild_documents()` for initial bulk indexing or full refreshes.

{vector_graph_rag-0.2.0 → vector_graph_rag-0.2.1}/docs/python-api.md

@@ -403,6 +403,8 @@ def upsert_documents_by_source(
 
 If the source does not exist, the method inserts a new source. If it already exists, the method deletes the previous chunks for that source, updates graph references, and inserts the new chunks without rebuilding unrelated sources.
 
+Source-level writes are not transactionally atomic. If an upsert fails during the multi-step cascade, rerun the same `upsert_documents_by_source()` call for the same source to converge the source back to the requested state. Queries may observe intermediate state until the retry succeeds.
+
 The method accepts exactly one source per call. If `source` is not provided and the documents contain multiple `metadata[source_field]` values, it raises `ValueError`.
 
 ```python
@@ -447,6 +449,8 @@ def delete_documents_by_source(
 
 Shared entities and relations are preserved when other sources still reference them. Orphaned relations and entities are removed.
 
+Source-level deletes are retryable. If a delete fails after partially cleaning graph records, rerun `delete_documents_by_source()` with the same `source` and `source_field` to finish the cascade.
+
 ```python
 deleted = rag.delete_documents_by_source("sharepoint:file-123")
 ```

{vector_graph_rag-0.2.0 → vector_graph_rag-0.2.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "vector-graph-rag"
-version = "0.2.0"
+version = "0.2.1"
 description = "A Graph RAG implementation using pure vector search with Milvus"
 readme = "README.md"
 license = { text = "MIT" }

{vector_graph_rag-0.2.0 → vector_graph_rag-0.2.1}/src/vector_graph_rag/__init__.py

@@ -18,7 +18,7 @@ from vector_graph_rag.rag import VectorGraphRAG, create_rag
 from vector_graph_rag.storage.embeddings import EmbeddingModel
 from vector_graph_rag.storage.milvus import MilvusStore
 
-__version__ = "0.2.0"
+__version__ = "0.2.1"
 
 __all__ = [
     "Settings",

{vector_graph_rag-0.2.0 → vector_graph_rag-0.2.1}/src/vector_graph_rag/rag.py

@@ -233,7 +233,7 @@ class VectorGraphRAG:
     def _normalize_source_value(value: Any, field_name: str) -> str:
         """Validate and normalize a source metadata value."""
         if not isinstance(value, str) or not value.strip():
-            raise ValueError(f'{field_name} must be a non-empty string.')
+            raise ValueError(f"{field_name} must be a non-empty string.")
         return value.strip()
 
     @staticmethod
@@ -277,7 +277,7 @@ class VectorGraphRAG:
             if conflicting_sources:
                 conflicts = ", ".join(sorted(conflicting_sources))
                 raise ValueError(
-                    f'documents contain {source_field} values that differ from source '
+                    f"documents contain {source_field} values that differ from source "
                     f'"{explicit_source}": {conflicts}'
                 )
             return explicit_source
@@ -513,6 +513,23 @@ class VectorGraphRAG:
             rid: existing_relations.get(builder.relations[rid], {}).get("id", rid)
             for rid in builder.relation_ids
         }
+        current_passage_ids = set(builder.passage_ids)
+        known_relation_ids = set(relation_id_map.values())
+        suspect_existing_relation_ids = sorted(
+            {
+                relation_id
+                for entity in existing_entities.values()
+                if current_passage_ids.intersection(entity.get("passage_ids", []))
+                for relation_id in entity.get("relation_ids", [])
+                if relation_id not in known_relation_ids
+            }
+        )
+        live_suspect_relation_ids: set[str] = set()
+        if suspect_existing_relation_ids:
+            live_suspect_relation_ids = {
+                relation["id"]
+                for relation in self._store._get_relations_by_ids(suspect_existing_relation_ids)
+            }
 
         existing_passages = self._store.get_passages_by_ids(
             builder.passage_ids,
@@ -520,9 +537,7 @@ class VectorGraphRAG:
         )
         for passage in existing_passages:
             if passage.get(source_field) != source:
-                raise ValueError(
-                    f'Passage ID "{passage["id"]}" already belongs to another source.'
-                )
+                raise ValueError(f'Passage ID "{passage["id"]}" already belongs to another source.')
 
         if show_progress:
             logger.info("Upserting incremental graph into Milvus...")
@@ -534,6 +549,7 @@ class VectorGraphRAG:
         entity_insert_texts: List[str] = []
         entity_insert_embeddings: List[List[float]] = []
         entity_insert_metadatas: List[Dict[str, Any]] = []
+        entity_update_records: List[Dict[str, Any]] = []
 
         for index, eid in enumerate(builder.entity_ids):
             stored_id = entity_id_map[eid]
@@ -548,12 +564,28 @@ class VectorGraphRAG:
             }
             existing = existing_entities.get(entity_text_by_id[eid])
             if existing:
-                self._store._update_entity(
-                    stored_id,
-                    text=entity_text_by_id[eid],
-                    embedding=entity_embeddings[index],
-                    relation_ids=self._merge_unique(existing.get("relation_ids", []), relation_ids),
-                    passage_ids=self._merge_unique(existing.get("passage_ids", []), passage_ids),
+                existing_relation_ids = existing.get("relation_ids", [])
+                if current_passage_ids.intersection(existing.get("passage_ids", [])):
+                    existing_relation_ids = [
+                        relation_id
+                        for relation_id in existing_relation_ids
+                        if relation_id in known_relation_ids
+                        or relation_id in live_suspect_relation_ids
+                    ]
+                entity_update_records.append(
+                    {
+                        "id": stored_id,
+                        "text": entity_text_by_id[eid],
+                        "vector": entity_embeddings[index],
+                        "relation_ids": self._merge_unique(
+                            existing_relation_ids,
+                            relation_ids,
+                        ),
+                        "passage_ids": self._merge_unique(
+                            existing.get("passage_ids", []),
+                            passage_ids,
+                        ),
+                    }
                 )
             else:
                 entity_insert_ids.append(stored_id)
@@ -561,6 +593,9 @@ class VectorGraphRAG:
                 entity_insert_embeddings.append(entity_embeddings[index])
                 entity_insert_metadatas.append(metadata)
 
+        if entity_update_records:
+            self._store._upsert_entity_records(entity_update_records)
+
         if entity_insert_ids:
             self._store._insert_entities(
                 entity_insert_texts,
@@ -574,6 +609,7 @@ class VectorGraphRAG:
         relation_insert_texts: List[str] = []
         relation_insert_embeddings: List[List[float]] = []
         relation_insert_metadatas: List[Dict[str, Any]] = []
+        relation_update_records: List[Dict[str, Any]] = []
 
         for index, rid in enumerate(builder.relation_ids):
             stored_id = relation_id_map[rid]
@@ -594,22 +630,31 @@ class VectorGraphRAG:
 
             existing = existing_relations.get(relation_text_by_id[rid])
             if existing:
-                self._store._update_relation(
-                    stored_id,
-                    text=relation_text_by_id[rid],
-                    embedding=relation_embeddings[index],
-                    entity_ids=entity_ids,
-                    passage_ids=self._merge_unique(existing.get("passage_ids", []), passage_ids),
-                    subject=metadata.get("subject"),
-                    predicate=metadata.get("predicate"),
-                    object_=metadata.get("object"),
-                )
+                update_record = {
+                    "id": stored_id,
+                    "text": relation_text_by_id[rid],
+                    "vector": relation_embeddings[index],
+                    "entity_ids": entity_ids,
+                    "passage_ids": self._merge_unique(
+                        existing.get("passage_ids", []),
+                        passage_ids,
+                    ),
+                }
+                for field in ["subject", "predicate", "object"]:
+                    if metadata.get(field) is not None:
+                        update_record[field] = metadata[field]
+                    elif field in existing:
+                        update_record[field] = existing[field]
+                relation_update_records.append(update_record)
             else:
                 relation_insert_ids.append(stored_id)
                 relation_insert_texts.append(relation_text_by_id[rid])
                 relation_insert_embeddings.append(relation_embeddings[index])
                 relation_insert_metadatas.append(metadata)
 
+        if relation_update_records:
+            self._store._upsert_relation_records(relation_update_records)
+
         if relation_insert_ids:
             self._store._insert_relations(
                 relation_insert_texts,
@@ -915,6 +960,8 @@ class VectorGraphRAG:
         In this project, a LangChain Document is a passage/chunk. This method
         uses a stable source metadata value to replace all chunks that belong to
         one source object, such as a file, URL, message, or business record.
+        The write is not transactionally atomic, but rerunning the same source
+        upsert after a failure converges the source to the requested state.
 
         Args:
             documents: Parsed chunks/passages for one source.
@@ -1011,6 +1058,9 @@ class VectorGraphRAG:
         """
         Incrementally delete all documents that belong to one source.
 
+        The write is not transactionally atomic, but rerunning the same source
+        delete after a failure finishes the source-level cascade.
+
         Args:
             source: Stable source value previously used with
                     upsert_documents_by_source().
@@ -1037,7 +1087,10 @@ class VectorGraphRAG:
         )
 
         relations = self._store._get_relations_by_ids(relation_ids)
-        deleted_relation_ids: set[str] = set()
+        existing_relation_ids = {relation["id"] for relation in relations}
+        relation_update_records: List[Dict[str, Any]] = []
+        relation_delete_ids: List[str] = []
+        deleted_relation_ids: set[str] = set(relation_ids) - existing_relation_ids
         for relation in relations:
             relation_id = relation["id"]
             new_passage_ids = self._remove_many(
@@ -1045,15 +1098,29 @@ class VectorGraphRAG:
                 removed_passage_ids,
             )
             if new_passage_ids:
-                self._store._update_relation(
-                    relation_id,
-                    passage_ids=new_passage_ids,
-                )
+                update_record = {
+                    "id": relation_id,
+                    "text": relation["text"],
+                    "vector": self._embedding_model.embed(relation["text"]),
+                    "entity_ids": relation.get("entity_ids", []),
+                    "passage_ids": new_passage_ids,
+                }
+                for field in ["subject", "predicate", "object"]:
+                    if field in relation:
+                        update_record[field] = relation[field]
+                relation_update_records.append(update_record)
             else:
-                self._store._delete_relation(relation_id)
+                relation_delete_ids.append(relation_id)
                 deleted_relation_ids.add(relation_id)
 
+        if relation_update_records:
+            self._store._upsert_relation_records(relation_update_records)
+        if relation_delete_ids:
+            self._store._delete_relations(relation_delete_ids)
+
         entities = self._store._get_entities_by_ids(entity_ids)
+        entity_update_records: List[Dict[str, Any]] = []
+        entity_delete_ids: List[str] = []
         for entity in entities:
             entity_id = entity["id"]
             new_passage_ids = self._remove_many(
@@ -1066,13 +1133,22 @@ class VectorGraphRAG:
             )
 
             if new_passage_ids or new_relation_ids:
-                self._store._update_entity(
-                    entity_id,
-                    passage_ids=new_passage_ids,
-                    relation_ids=new_relation_ids,
+                entity_update_records.append(
+                    {
+                        "id": entity_id,
+                        "text": entity["text"],
+                        "vector": self._embedding_model.embed(entity["text"]),
+                        "passage_ids": new_passage_ids,
+                        "relation_ids": new_relation_ids,
+                    }
                 )
             else:
-                self._store._delete_entity(entity_id)
+                entity_delete_ids.append(entity_id)
+
+        if entity_update_records:
+            self._store._upsert_entity_records(entity_update_records)
+        if entity_delete_ids:
+            self._store._delete_entities(entity_delete_ids)
 
         self._store.delete_passages(passage_ids)
         self._retriever = None

{vector_graph_rag-0.2.0 → vector_graph_rag-0.2.1}/src/vector_graph_rag/storage/milvus.py

@@ -13,7 +13,7 @@ Users should interact with passages through the Graph abstraction layer.
 import logging
 import re
 import uuid
-from typing import Any, Dict, List, Optional
+from typing import Any, Dict, Iterator, List, Optional
 
 from pymilvus import DataType, MilvusClient
 from tqdm import tqdm
@@ -97,6 +97,77 @@ class MilvusStore:
             )
         return field_name
 
+    def _batch_items(
+        self,
+        items: List[Any],
+        batch_size: Optional[int] = None,
+    ) -> Iterator[List[Any]]:
+        """Yield items in batches using the configured batch size."""
+        effective_batch_size = batch_size or self.settings.batch_size
+        for start_idx in range(0, len(items), effective_batch_size):
+            yield items[start_idx : start_idx + effective_batch_size]
+
+    @staticmethod
+    def _unique_preserve_order(values: List[str]) -> List[str]:
+        """Return unique strings without changing first-seen order."""
+        seen = set()
+        unique_values = []
+        for value in values:
+            if value in seen:
+                continue
+            seen.add(value)
+            unique_values.append(value)
+        return unique_values
+
+    def _query_by_texts(
+        self,
+        collection_name: str,
+        texts: List[str],
+        output_fields: List[str],
+    ) -> Dict[str, Dict[str, Any]]:
+        """Query records by exact text in batches."""
+        records_by_text: Dict[str, Dict[str, Any]] = {}
+        unique_texts = self._unique_preserve_order(texts)
+        if not unique_texts:
+            return records_by_text
+
+        for batch_texts in self._batch_items(unique_texts):
+            quoted_texts = ", ".join(self._quote_string(text) for text in batch_texts)
+            results = self.client.query(
+                collection_name=collection_name,
+                filter=f"text in [{quoted_texts}]",
+                output_fields=output_fields,
+            )
+            for record in results:
+                text = record.get("text")
+                if isinstance(text, str) and text not in records_by_text:
+                    records_by_text[text] = record
+
+        return records_by_text
+
+    def _upsert_records(
+        self,
+        collection_name: str,
+        records: List[Dict[str, Any]],
+    ) -> None:
+        """Upsert records in batches using the configured batch size."""
+        if not records:
+            return
+
+        for batch_records in self._batch_items(records):
+            self.client.upsert(
+                collection_name=collection_name,
+                data=batch_records,
+            )
+
+    def _upsert_entity_records(self, records: List[Dict[str, Any]]) -> None:
+        """Upsert fully materialized entity records."""
+        self._upsert_records(self.entity_collection, records)
+
+    def _upsert_relation_records(self, records: List[Dict[str, Any]]) -> None:
+        """Upsert fully materialized relation records."""
+        self._upsert_records(self.relation_collection, records)
+
     def _create_collection(
         self,
         collection_name: str,
@@ -500,16 +571,11 @@ class MilvusStore:
         Returns:
             Mapping from entity text to the first matching entity record.
         """
-        entities_by_text: Dict[str, Dict[str, Any]] = {}
-        for text in entity_texts:
-            results = self.client.query(
-                collection_name=self.entity_collection,
-                filter=f"text == {self._quote_string(text)}",
-                output_fields=["id", "text", "relation_ids", "passage_ids"],
-            )
-            if results:
-                entities_by_text[text] = results[0]
-        return entities_by_text
+        return self._query_by_texts(
+            self.entity_collection,
+            entity_texts,
+            output_fields=["id", "text", "relation_ids", "passage_ids"],
+        )
 
     def _get_relations_by_ids(
         self,
@@ -562,24 +628,19 @@ class MilvusStore:
         Returns:
             Mapping from relation text to the first matching relation record.
         """
-        relations_by_text: Dict[str, Dict[str, Any]] = {}
-        for text in relation_texts:
-            results = self.client.query(
-                collection_name=self.relation_collection,
-                filter=f"text == {self._quote_string(text)}",
-                output_fields=[
-                    "id",
-                    "text",
-                    "entity_ids",
-                    "passage_ids",
-                    "subject",
-                    "predicate",
-                    "object",
-                ],
-            )
-            if results:
-                relations_by_text[text] = results[0]
-        return relations_by_text
+        return self._query_by_texts(
+            self.relation_collection,
+            relation_texts,
+            output_fields=[
+                "id",
+                "text",
+                "entity_ids",
+                "passage_ids",
+                "subject",
+                "predicate",
+                "object",
+            ],
+        )
 
     def get_passages_by_ids(
         self,