graphrag-codeproperty-graph 0.5.0__tar.gz → 0.5.2__tar.gz

graphrag_codeproperty_graph-0.5.2/CHANGELOG.md

@@ -0,0 +1,82 @@
+# Changelog
+
+## [Unreleased]
+
+## [0.5.2] - 2026-09-16
+
+### Fixed
+- **`VectorLoader` no longer writes the raw embedding vector as a Neptune node
+  property.** The 0.5.1 on-node write (`SET n += {embedding: <array>, ...}`) was a
+  Neo4j-era pattern: Neo4j stores list-valued properties, but the graph backend is
+  now Neptune Database, whose openCypher rejects arrays ("Property value must be a
+  simple literal"). Every per-node write failed with `BadRequestException`, flooding
+  the ingest log with warnings. The embedding **vector** already lives in the vector
+  store (AOSS/OpenSearch), which is the sole surface used for kNN retrieval — nothing
+  reads the on-node vector. `VectorLoader.load` now writes only **scalar embedding
+  provenance** to the node (`hasEmbedding`, `embeddingModel`, `embeddingDimensions`,
+  `embeddingTarget`, `embeddedTime`), which Neptune accepts, preserving the
+  graph-side "this node has an embedding" signal without duplicating unqueryable data.
+  Covered by `tests/test_delta_ingestor_embedding_property.py`
+  (`test_vector_loader_writes_scalar_provenance_not_array`). (PSAPI-66782.)
+
+## [0.5.1] - 2026-09-15
+
+### Fixed
+- **`DeltaIngestor.ingest_artifact` now passes `graph_store` to `VectorLoader`.**
+  Previously `VectorLoader` was constructed with only `vector_store`, so the
+  `graph_store`-guarded step in `VectorLoader.load` — writing each embedding as an
+  `embedding` property on the Neptune node (`update_node_properties`) — never
+  fired. This restores the client-schema behaviour where CPG nodes carry their
+  embedding on the graph node (matching the original cpg-embedding-agent's
+  `SET n.embedding`), in addition to the OpenSearch vector index. One-line wiring
+  fix; the `SummaryOverlay`/`CodeSliceStore`/`GraphLoader` in the same method
+  already received `graph_store`. Covered by
+  `tests/test_delta_ingestor_embedding_property.py`. (PSAPI-66782, alignment wave W2.2.)
+
+## [0.5.0] - 2026-08-20
+
+### Added
+- **CPGService** — node-level delta filtering for Neptune ingestion
+  - Only writes changed/added METHOD subtrees (not all nodes)
+  - AST edge traversal identifies child node ownership
+  - Surgical removal of deleted methods via `remove_fn` callback
+  - First run (no manifest): full ingest; incremental: filtered by diff
+  - 60-90% reduction in Neptune write volume on incremental runs
+- Full artifact pipeline modules (previously in private wheel):
+  - `ArtifactReader` — S3 artifact reading and parsing
+  - `ArtifactValidator` — schema compliance validation before ingest
+  - `GraphLoader` — Neptune batch write with retry cascade
+  - `VectorLoader` — OpenSearch Serverless vector ingestion
+  - `SummaryOverlay` — LLM summary → Neptune node properties
+  - `CodeSliceStore` — source evidence → Neptune node properties
+  - `GraphSONConverter` — GraphSON format conversion
+  - `Schema` — full Joern CPG schema (20 node types, 14+ edge types)
+- Optimistic locking in ManifestManager (ETag-based)
+- `ManifestConflictError` for concurrent write detection
+- `delete_domain()` and `list_domains()` in tenant_ops
+- Full CPG schema support in models (VectorRecord, SummaryRecord, CodeSliceRecord)
+- 5 new tests for CPGService delta filtering (14 total)
+
+### Changed
+- DeltaIngestor extended with full artifact pipeline (`ingest_artifact`)
+- ManifestManager now supports rollback and optimistic locking
+- Models expanded with full Joern node/edge type documentation
+
+### Dependencies
+- Added `boto3>=1.28.0`
+- `graphrag-document-graph>=3.1.1` (bumped from 3.0.8)
+
+## [0.4.3] - 2026-08-03
+
+### Added
+- Initial public release
+- Joern/Semgrep CPG extraction and ingestion
+- Delta ingestion — skip-or-replace logic to minimize Neptune writes
+- Graph diff comparison between CPG states
+- S3-backed manifest management for repository state tracking
+- AI vulnerability enrichment and risk scoring
+- Tenant lifecycle management (create, purge, replace)
+- CPGNode and CPGEdge Pydantic models (Joern-specific types)
+
+### Dependencies
+- graphrag-document-graph>=3.0.8

{graphrag_codeproperty_graph-0.5.0 → graphrag_codeproperty_graph-0.5.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: graphrag-codeproperty-graph
-Version: 0.5.0
+Version: 0.5.2
 Summary: Code Property Graphs — Joern/Semgrep extraction, AI vulnerability enrichment, risk scoring
 Project-URL: Repository, https://github.com/kanjani-ai-research/codeproperty-graph
 Author-email: Evan Erwee <evan@erwee.com>
@@ -40,9 +40,8 @@ pip install graphrag-codeproperty-graph
 
 ## Dependencies
 
-- `graphrag-document-graph>=3.0.8` — Document graph infrastructure (typed nodes, Cypher builders, multi-tenancy)
-
-This package does **not** depend on `bona`.
+- `graphrag-document-graph>=3.1.1` — Document graph infrastructure (typed nodes, Cypher builders, multi-tenancy)
+- `boto3>=1.28.0` — AWS SDK for S3 manifest storage and service integrations
 
 ## Dependency Chain
 
@@ -57,12 +56,38 @@ graphrag-codeproperty-graph
 
 ## Quick Start
 
-### Python API Example
+### CPGService — Delta Node-Level Filtering (Recommended)
+
+```python
+from codeproperty_graph import CPGService
+
+# Only writes changed/added method subtrees to Neptune
+service = CPGService(bucket="graphrag-artifacts-705909755305")
+
+result = await service.ingest(
+    repo="my-service",
+    job_id="build-456",
+    tenant_id="tenant_abc123",
+    nodes_data=joern_nodes,
+    edges_data=joern_edges,
+    nodes_path="s3://bucket/cpg-exports/my-service/build-456/nodes.json",
+    edges_path="s3://bucket/cpg-exports/my-service/build-456/edges.json",
+    graph_store=neptune_store,
+    write_fn=batch_write_function,
+    remove_fn=remove_method_subgraph,  # optional: surgical deletion
+)
+
+# result: {"status": "SKIPPED"} — no changes
+# result: {"status": "INGESTED", "delta": "+5 -2 ~3 =150",
+#           "filtered_nodes": 47, "total_nodes": 5123}
+```
+
+### DeltaIngestor — Full Artifact Pipeline
 
 ```python
-from codeproperty_graph import DeltaIngestor, CPGNode, CPGEdge, GraphDiff
+from codeproperty_graph import DeltaIngestor
 
-# Delta ingestion — only writes to Neptune when code actually changed
+# Full pipeline: graph + vectors + summaries + code slices
 ingestor = DeltaIngestor(bucket="graphrag-artifacts-705909755305")
 
 result = await ingestor.ingest(
@@ -116,12 +141,21 @@ previous = await manager.load(repo="my-service")
 
 ```
 src/codeproperty_graph/
-├── __init__.py           # Public API: CPGNode, CPGEdge, DeltaIngestor, etc.
-├── models.py             # CPGNode, CPGEdge, Manifest — Joern-specific types
+├── __init__.py           # Public API (30 exports)
+├── models.py             # CPGNode, CPGEdge, Manifest, VectorRecord, SummaryRecord, CodeSliceRecord
+├── schema.py             # Full Joern CPG schema (20 node types, 14 edge types)
 ├── graph_diff.py         # Compare two CPG states by method signature
-├── manifest_manager.py   # S3-backed state tracking per repository
-├── delta_ingestor.py     # Skip-or-replace orchestration with tenant purge
-└── tenant_ops.py         # Clean lifecycle management (delete_tenant)
+├── manifest_manager.py   # S3-backed state tracking with optimistic locking
+├── delta_ingestor.py     # Full artifact pipeline orchestration
+├── cpg_service.py        # Node-level delta filtering (only write changed methods)
+├── tenant_ops.py         # Lifecycle management (delete_tenant, delete_domain)
+├── artifact_reader.py    # S3 artifact reading and parsing
+├── artifact_validator.py # Schema compliance validation
+├── graph_loader.py       # Neptune batch write with retry cascade
+├── vector_loader.py      # OpenSearch Serverless vector ingestion
+├── summary_overlay.py    # LLM summaries → Neptune node properties
+├── code_slice_store.py   # Source evidence → Neptune node properties
+└── graphson_converter.py # GraphSON format conversion
 ```
 
 ## Integration
@@ -149,7 +183,13 @@ src/codeproperty_graph/
 2. Extract METHOD node signatures: `{full_name: hash}`
 3. Compare against previous manifest in S3
 4. If identical → **SKIP** (no Neptune writes, saves cost)
-5. If changed → **INGEST** full graph under new tenant, purge old tenant, update manifest
+5. If changed → **CPGService filters to changed methods only**:
+   - Identify added/modified METHOD nodes via `GraphDiff.compare()`
+   - Walk AST edges to collect child nodes of changed methods
+   - Filter edges to changed subgraph
+   - Write only the delta (60-90% fewer Neptune writes)
+   - Surgically remove deleted method subgraphs
+   - Update manifest with full current state
 
 ### With Document Graph
 

{graphrag_codeproperty_graph-0.5.0 → graphrag_codeproperty_graph-0.5.2}/README.md

@@ -15,9 +15,8 @@ pip install graphrag-codeproperty-graph
 
 ## Dependencies
 
-- `graphrag-document-graph>=3.0.8` — Document graph infrastructure (typed nodes, Cypher builders, multi-tenancy)
-
-This package does **not** depend on `bona`.
+- `graphrag-document-graph>=3.1.1` — Document graph infrastructure (typed nodes, Cypher builders, multi-tenancy)
+- `boto3>=1.28.0` — AWS SDK for S3 manifest storage and service integrations
 
 ## Dependency Chain
 
@@ -32,12 +31,38 @@ graphrag-codeproperty-graph
 
 ## Quick Start
 
-### Python API Example
+### CPGService — Delta Node-Level Filtering (Recommended)
+
+```python
+from codeproperty_graph import CPGService
+
+# Only writes changed/added method subtrees to Neptune
+service = CPGService(bucket="graphrag-artifacts-705909755305")
+
+result = await service.ingest(
+    repo="my-service",
+    job_id="build-456",
+    tenant_id="tenant_abc123",
+    nodes_data=joern_nodes,
+    edges_data=joern_edges,
+    nodes_path="s3://bucket/cpg-exports/my-service/build-456/nodes.json",
+    edges_path="s3://bucket/cpg-exports/my-service/build-456/edges.json",
+    graph_store=neptune_store,
+    write_fn=batch_write_function,
+    remove_fn=remove_method_subgraph,  # optional: surgical deletion
+)
+
+# result: {"status": "SKIPPED"} — no changes
+# result: {"status": "INGESTED", "delta": "+5 -2 ~3 =150",
+#           "filtered_nodes": 47, "total_nodes": 5123}
+```
+
+### DeltaIngestor — Full Artifact Pipeline
 
 ```python
-from codeproperty_graph import DeltaIngestor, CPGNode, CPGEdge, GraphDiff
+from codeproperty_graph import DeltaIngestor
 
-# Delta ingestion — only writes to Neptune when code actually changed
+# Full pipeline: graph + vectors + summaries + code slices
 ingestor = DeltaIngestor(bucket="graphrag-artifacts-705909755305")
 
 result = await ingestor.ingest(
@@ -91,12 +116,21 @@ previous = await manager.load(repo="my-service")
 
 ```
 src/codeproperty_graph/
-├── __init__.py           # Public API: CPGNode, CPGEdge, DeltaIngestor, etc.
-├── models.py             # CPGNode, CPGEdge, Manifest — Joern-specific types
+├── __init__.py           # Public API (30 exports)
+├── models.py             # CPGNode, CPGEdge, Manifest, VectorRecord, SummaryRecord, CodeSliceRecord
+├── schema.py             # Full Joern CPG schema (20 node types, 14 edge types)
 ├── graph_diff.py         # Compare two CPG states by method signature
-├── manifest_manager.py   # S3-backed state tracking per repository
-├── delta_ingestor.py     # Skip-or-replace orchestration with tenant purge
-└── tenant_ops.py         # Clean lifecycle management (delete_tenant)
+├── manifest_manager.py   # S3-backed state tracking with optimistic locking
+├── delta_ingestor.py     # Full artifact pipeline orchestration
+├── cpg_service.py        # Node-level delta filtering (only write changed methods)
+├── tenant_ops.py         # Lifecycle management (delete_tenant, delete_domain)
+├── artifact_reader.py    # S3 artifact reading and parsing
+├── artifact_validator.py # Schema compliance validation
+├── graph_loader.py       # Neptune batch write with retry cascade
+├── vector_loader.py      # OpenSearch Serverless vector ingestion
+├── summary_overlay.py    # LLM summaries → Neptune node properties
+├── code_slice_store.py   # Source evidence → Neptune node properties
+└── graphson_converter.py # GraphSON format conversion
 ```
 
 ## Integration
@@ -124,7 +158,13 @@ src/codeproperty_graph/
 2. Extract METHOD node signatures: `{full_name: hash}`
 3. Compare against previous manifest in S3
 4. If identical → **SKIP** (no Neptune writes, saves cost)
-5. If changed → **INGEST** full graph under new tenant, purge old tenant, update manifest
+5. If changed → **CPGService filters to changed methods only**:
+   - Identify added/modified METHOD nodes via `GraphDiff.compare()`
+   - Walk AST edges to collect child nodes of changed methods
+   - Filter edges to changed subgraph
+   - Write only the delta (60-90% fewer Neptune writes)
+   - Surgically remove deleted method subgraphs
+   - Update manifest with full current state
 
 ### With Document Graph
 

{graphrag_codeproperty_graph-0.5.0 → graphrag_codeproperty_graph-0.5.2}/pyproject.toml

@@ -7,7 +7,7 @@ packages = ["src/codeproperty_graph"]
 
 [project]
 name = "graphrag-codeproperty-graph"
-version = "0.5.0"
+version = "0.5.2"
 description = "Code Property Graphs — Joern/Semgrep extraction, AI vulnerability enrichment, risk scoring"
 readme = "README.md"
 requires-python = ">=3.11"

{graphrag_codeproperty_graph-0.5.0 → graphrag_codeproperty_graph-0.5.2}/src/codeproperty_graph/delta_ingestor.py

@@ -166,6 +166,7 @@ class DeltaIngestor:
         vector_loader = VectorLoader(
             vector_store=vector_store,
             expected_dimensions=manifest.get("embedding_dimensions", 0),
+            graph_store=graph_store,
             tenant_id=tenant_id,
         )
         vector_result = await vector_loader.load(enriched_vectors)

{graphrag_codeproperty_graph-0.5.0 → graphrag_codeproperty_graph-0.5.2}/src/codeproperty_graph/vector_loader.py

@@ -163,7 +163,17 @@ class VectorLoader:
             result.total_loaded += count
             logger.info(f"Loaded {count} vectors for target '{target}'")
 
-        # Also write embedding as a node property on Neptune (client schema conformance)
+        # Write embedding PROVENANCE (scalar metadata) onto the graph node.
+        #
+        # The embedding VECTOR itself lives in the vector store (AOSS/OpenSearch),
+        # which is the sole surface used for kNN retrieval. We deliberately do NOT
+        # write the raw vector as a node property: the original Neo4j schema stored
+        # `SET n.embedding = <array>` (Neo4j supports list-valued properties), but
+        # the graph backend here is Neptune Database, whose openCypher rejects
+        # array/list property values ("Property value must be a simple literal").
+        # Storing scalar provenance (model, dimensions, timestamp, presence flag)
+        # keeps the "this node has an embedding" graph-side signal — queryable and
+        # filterable — without duplicating the vector where nothing reads it.
         if self._graph_store and records:
             written = 0
             for raw in records:
@@ -171,13 +181,21 @@ class VectorLoader:
                 try:
                     await self._graph_store.update_node_properties(
                         record.cpg_node_id,
-                        {"embedding": record.vector, "embeddedTime": record.metadata.get("embedded_time", "")},
+                        {
+                            "hasEmbedding": True,
+                            "embeddingModel": record.embedding_model,
+                            "embeddingDimensions": (
+                                record.embedding_dimensions or len(record.vector)
+                            ),
+                            "embeddingTarget": record.embedding_target,
+                            "embeddedTime": record.metadata.get("embedded_time", ""),
+                        },
                     )
                     written += 1
                 except Exception as e:
-                    logger.debug(f"Failed to write embedding property for {record.cpg_node_id}: {e}")
+                    logger.debug(f"Failed to write embedding provenance for {record.cpg_node_id}: {e}")
             if written:
-                logger.info(f"Wrote embedding property to {written} Neptune nodes")
+                logger.info(f"Wrote embedding provenance to {written} Neptune nodes")
 
         if result.errors:
             logger.warning(f"Encountered {len(result.errors)} validation errors during load")

graphrag_codeproperty_graph-0.5.2/tests/test_delta_ingestor_embedding_property.py

@@ -0,0 +1,129 @@
+"""Wave W2.2 — DeltaIngestor must pass graph_store to VectorLoader.
+
+When graph_store is passed to VectorLoader, VectorLoader.load also writes the
+embedding as a property on the Neptune node (update_node_properties), which is
+the client-schema behaviour the Azure cpg-embedding-agent had (SET n.embedding).
+Previously ingest_artifact constructed VectorLoader WITHOUT graph_store, so that
+Neptune write silently never fired.
+
+This test asserts the wiring: VectorLoader is constructed with the same
+graph_store that GraphLoader / SummaryOverlay receive.
+"""
+
+import pytest
+from unittest.mock import AsyncMock, MagicMock, patch
+
+from codeproperty_graph import DeltaIngestor
+
+
+def _manifest():
+    return {"repo_id": "test-repo", "analysis_run_id": "job-1", "embedding_dimensions": 768}
+
+
+@pytest.mark.asyncio
+async def test_vector_loader_receives_graph_store():
+    ingestor = DeltaIngestor.__new__(DeltaIngestor)
+    # Manifest manager: force "changed" so the pipeline proceeds to the loaders.
+    mm = MagicMock()
+    mm.has_changes.return_value = (True, None)
+    ingestor._manifest_mgr = mm
+
+    graph_store = MagicMock(name="neptune_store")
+    vector_store = MagicMock(name="opensearch_store")
+
+    nodes = [{"id": "m1", "node_type": "METHOD", "full_name": "com.app.A.m", "hash": "h1"}]
+    vectors = [{"cpg_node_id": "m1", "embedding_target": "code", "vector": [0.0] * 768, "text": "x"}]
+
+    dm = "codeproperty_graph.delta_ingestor"
+    with patch(f"{dm}.ArtifactValidator") as V, \
+         patch(f"{dm}.GraphLoader") as GL, \
+         patch(f"{dm}.VectorLoader") as VL, \
+         patch(f"{dm}.SummaryOverlay") as SO, \
+         patch(f"{dm}.CodeSliceStore") as CS:
+
+        V.return_value.validate.return_value = MagicMock(valid=True, warnings=[], errors=[])
+        GL.return_value.load = AsyncMock(return_value={"nodes_written": 1})
+        VL.return_value.load = AsyncMock(return_value=MagicMock(total_loaded=1))
+        SO.return_value.apply = AsyncMock(return_value=1)
+        CS.return_value.store = AsyncMock(return_value=0)
+
+        await ingestor.ingest_artifact(
+            manifest=_manifest(),
+            nodes_data=nodes,
+            edges_data=[],
+            vectors_data=vectors,
+            summaries_data=[],
+            code_slices_data=[],
+            graph_store=graph_store,
+            vector_store=vector_store,
+            validate=False,
+        )
+
+        # The crux of W2.2: VectorLoader must be constructed WITH graph_store,
+        # so the Neptune embedding-property write can fire.
+        assert VL.called, "VectorLoader was not constructed"
+        _, kwargs = VL.call_args
+        assert kwargs.get("graph_store") is graph_store, (
+            "VectorLoader must receive the graph_store (else the Neptune "
+            "embedding-property write never fires — the W2.2 regression)"
+        )
+        # And it should be the SAME store GraphLoader got.
+        assert GL.call_args.kwargs.get("graph_store") is graph_store
+
+
+# ─── Wave (Neptune fix): on-node write must be SCALAR provenance, not the array ──
+#
+# Neptune Database openCypher rejects list/array property values ("Property value
+# must be a simple literal"). The raw embedding vector lives in the vector store
+# (AOSS); the graph node must carry only scalar provenance. This test drives
+# VectorLoader.load directly and asserts what actually reaches update_node_properties.
+
+from codeproperty_graph import VectorLoader
+
+
+@pytest.mark.asyncio
+async def test_vector_loader_writes_scalar_provenance_not_array():
+    graph_store = MagicMock(name="neptune_store")
+    graph_store.update_node_properties = AsyncMock(return_value=True)
+
+    vector_store = MagicMock(name="opensearch_store")
+    vector_store.put_vectors = AsyncMock(return_value=1)
+
+    loader = VectorLoader(vector_store=vector_store, graph_store=graph_store)
+
+    records = [{
+        "cpg_node_id": "n1",
+        "embedding_target": "class_summary",
+        "vector": [0.1] * 768,
+        "text": "summary text",
+        "embedding_model": "nomic-embed-text",
+        "embedding_dimensions": 768,
+        "metadata": {"embedded_time": "2026-09-16T07:50:43Z"},
+    }]
+
+    await loader.load(records)
+
+    # The on-node write must have happened...
+    assert graph_store.update_node_properties.await_count == 1
+    node_id, props = graph_store.update_node_properties.await_args.args
+
+    assert node_id == "n1"
+    # ...and it must NOT include the raw vector array (the Neptune-rejecting bug).
+    assert "embedding" not in props, (
+        "VectorLoader must not write the raw embedding array to the Neptune node "
+        "(Neptune Database rejects list-valued properties)"
+    )
+    # ...every value written must be a Neptune-safe scalar literal.
+    for key, value in props.items():
+        assert not isinstance(value, (list, dict, tuple, set)), (
+            f"Node property {key!r} must be a scalar literal, got {type(value).__name__}"
+        )
+    # ...and the useful provenance is present.
+    assert props["hasEmbedding"] is True
+    assert props["embeddingModel"] == "nomic-embed-text"
+    assert props["embeddingDimensions"] == 768
+    assert props["embeddingTarget"] == "class_summary"
+    assert props["embeddedTime"] == "2026-09-16T07:50:43Z"
+
+    # The VECTOR itself still goes to the vector store (AOSS).
+    assert vector_store.put_vectors.await_count >= 1

graphrag_codeproperty_graph-0.5.0/CHANGELOG.md

@@ -1,16 +0,0 @@
-# Changelog
-
-## [0.4.3] - 2026-08-03
-
-### Added
-- Initial public release
-- Joern/Semgrep CPG extraction and ingestion
-- Delta ingestion — skip-or-replace logic to minimize Neptune writes
-- Graph diff comparison between CPG states
-- S3-backed manifest management for repository state tracking
-- AI vulnerability enrichment and risk scoring
-- Tenant lifecycle management (create, purge, replace)
-- CPGNode and CPGEdge Pydantic models (Joern-specific types)
-
-### Dependencies
-- graphrag-document-graph>=3.0.8