graph-knowledge-doc-parser 0.2.4__tar.gz → 0.2.5__tar.gz

{graph_knowledge_doc_parser-0.2.4 → graph_knowledge_doc_parser-0.2.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: graph-knowledge-doc-parser
-Version: 0.2.4
+Version: 0.2.5
 Summary: doc parser using llm driven engine with graph knowledge awareness
 Author: humblemat810
 Author-email: 67593116+humblemat810@users.noreply.github.com
@@ -14,11 +14,18 @@ Classifier: Programming Language :: Python :: 3.14
 Classifier: Programming Language :: Python :: 3.15
 Classifier: Programming Language :: Python :: 3 :: Only
 Classifier: Topic :: Text Processing :: General
+Provides-Extra: azure
+Provides-Extra: cloud
+Provides-Extra: gemini
+Provides-Extra: openai
+Provides-Extra: vertex
 Requires-Dist: diskcache (>=5.6,<6.0) ; platform_python_implementation == "PyPy"
 Requires-Dist: joblib (>=1.5.3,<2.0.0) ; platform_python_implementation == "CPython"
-Requires-Dist: kogwistar (==0.6.3)
+Requires-Dist: kogwistar (==0.6.4)
 Requires-Dist: langchain-core (>=1.2.5,<2.0.0)
-Requires-Dist: langchain-google-genai (>=4.1.2,<5.0.0)
+Requires-Dist: langchain-google-genai (>=4.1.2,<5.0.0) ; extra == "gemini" or extra == "cloud"
+Requires-Dist: langchain-google-vertexai (>=3.2.4,<4.0.0) ; extra == "vertex" or extra == "cloud"
+Requires-Dist: langchain-openai (>=1.6.7,<2.0.0) ; extra == "openai" or extra == "azure" or extra == "cloud"
 Requires-Dist: mcp (>=2.2.0,<3.0.0)
 Requires-Dist: pathspec (>=0.12.1,<0.13.0)
 Requires-Dist: pdf2image (>=1.17.0,<2.0.0)
@@ -73,7 +80,7 @@ disabled for that layer and the workflow routes through the remaining methods
 before reaching explicit parse failure. PageIndex is a one-layer structural
 fallback that preserves exact source pointers and returns expandable children
 to normal strategy selection. See the
-[`0.2.4` release note](doc/release_0.2.4.md) and the
+[`0.2.5` release note](doc/release_0.2.5.md) and the
 [progressive refinement ADR](doc/adr_progressive_refinement_strategy_arbitration.md).
 The deterministic and text-only Bonsai fixture evaluation is recorded in the
 [`PageIndex adversarial fixture report`](doc/page_index_adversarial_fixture_report.md).
@@ -211,7 +218,7 @@ The project currently expects or optionally uses:
 - `split_raw_file_list`: optional allow-list file for PDF splitting runs
 - `answer_export_list`: optional export list path used by local workflows
 
-An example template is provided in [`.env.example`](/c:/Users/chanh/Documents/kg_doc_parser/.env.example).
+An example template is provided in [`.env.example`](.env.example).
 
 ## Provider Guide
 
@@ -255,6 +262,20 @@ and structured extraction.
   - `KG_DOC_PARSER_PROJECT=my-project`
   - `KG_DOC_PARSER_LOCATION=us-central1`
 
+Gemini, OpenAI/Azure, and Vertex adapters are optional so local-only
+installations do not pull their cloud SDK trees:
+
+```powershell
+poetry install -E openai -E gemini -E vertex
+# or, after building/installing the package:
+python -m pip install "graph-knowledge-doc-parser[openai,gemini,vertex]"
+```
+
+Installing an adapter does not create credentials, a cloud project, or a paid
+service. Use the no-charge offline provider contract tests for CI validation;
+only run a live provider smoke test when the account, endpoint, model, and
+zero-cost allowance are explicitly confirmed.
+
 ### Recipe Parsing Example
 
 If you are parsing a cooking recipe, one practical split is:
@@ -343,7 +364,7 @@ There is now a manual workflow-ingest demo harness that can run the end-to-end f
 - an already running external Kogwistar server
 
 The legacy semantic-smoke test in
-[`tests/test_semantic_layerwise_doc_parsing.py`](/c:/Users/chanh/Documents/kg_doc_parser/tests/test_semantic_layerwise_doc_parsing.py)
+[`tests/test_semantic_layerwise_doc_parsing.py`](tests/test_semantic_layerwise_doc_parsing.py)
 also expects a live Kogwistar server at `http://127.0.0.1:28110`. It does not
 start that server for you, so use the VS Code server launch config or start it
 manually before running the Ollama case.

{graph_knowledge_doc_parser-0.2.4 → graph_knowledge_doc_parser-0.2.5}/README.md

@@ -40,7 +40,7 @@ disabled for that layer and the workflow routes through the remaining methods
 before reaching explicit parse failure. PageIndex is a one-layer structural
 fallback that preserves exact source pointers and returns expandable children
 to normal strategy selection. See the
-[`0.2.4` release note](doc/release_0.2.4.md) and the
+[`0.2.5` release note](doc/release_0.2.5.md) and the
 [progressive refinement ADR](doc/adr_progressive_refinement_strategy_arbitration.md).
 The deterministic and text-only Bonsai fixture evaluation is recorded in the
 [`PageIndex adversarial fixture report`](doc/page_index_adversarial_fixture_report.md).
@@ -178,7 +178,7 @@ The project currently expects or optionally uses:
 - `split_raw_file_list`: optional allow-list file for PDF splitting runs
 - `answer_export_list`: optional export list path used by local workflows
 
-An example template is provided in [`.env.example`](/c:/Users/chanh/Documents/kg_doc_parser/.env.example).
+An example template is provided in [`.env.example`](.env.example).
 
 ## Provider Guide
 
@@ -222,6 +222,20 @@ and structured extraction.
   - `KG_DOC_PARSER_PROJECT=my-project`
   - `KG_DOC_PARSER_LOCATION=us-central1`
 
+Gemini, OpenAI/Azure, and Vertex adapters are optional so local-only
+installations do not pull their cloud SDK trees:
+
+```powershell
+poetry install -E openai -E gemini -E vertex
+# or, after building/installing the package:
+python -m pip install "graph-knowledge-doc-parser[openai,gemini,vertex]"
+```
+
+Installing an adapter does not create credentials, a cloud project, or a paid
+service. Use the no-charge offline provider contract tests for CI validation;
+only run a live provider smoke test when the account, endpoint, model, and
+zero-cost allowance are explicitly confirmed.
+
 ### Recipe Parsing Example
 
 If you are parsing a cooking recipe, one practical split is:
@@ -310,7 +324,7 @@ There is now a manual workflow-ingest demo harness that can run the end-to-end f
 - an already running external Kogwistar server
 
 The legacy semantic-smoke test in
-[`tests/test_semantic_layerwise_doc_parsing.py`](/c:/Users/chanh/Documents/kg_doc_parser/tests/test_semantic_layerwise_doc_parsing.py)
+[`tests/test_semantic_layerwise_doc_parsing.py`](tests/test_semantic_layerwise_doc_parsing.py)
 also expects a live Kogwistar server at `http://127.0.0.1:28110`. It does not
 start that server for you, so use the VS Code server launch config or start it
 manually before running the Ollama case.

{graph_knowledge_doc_parser-0.2.4 → graph_knowledge_doc_parser-0.2.5}/kg_doc_parser/document_ingester_logger.py

@@ -82,7 +82,6 @@ NOTES
 
 from __future__ import annotations
 
-import json
 import os
 import queue
 import sqlite3
@@ -100,6 +99,8 @@ from langchain_core.outputs.llm_result import LLMResult
 from langchain_core.messages import BaseMessage
 from uuid import UUID
 
+from kg_doc_parser.workflow_ingest.serialization import safe_json_dumps
+
 # ---------------------------
 # Pricing / cost calculation
 # ---------------------------
@@ -537,7 +538,7 @@ class DocumentIngestSQLiteCallback(BaseCallbackHandler):
                 token_count=0,
                 cost_usd=0.0,
                 n_try=n_try,
-                metadata_json=json.dumps(payload, ensure_ascii=False),
+                metadata_json=safe_json_dumps(payload, ensure_ascii=False),
             )
         )
     def on_llm_start(
@@ -594,7 +595,7 @@ class DocumentIngestSQLiteCallback(BaseCallbackHandler):
                 token_count=0,
                 cost_usd=0.0,
                 n_try = payload.get('n_try', 0),
-                metadata_json=json.dumps(payload, ensure_ascii=False),
+                metadata_json=safe_json_dumps(payload, ensure_ascii=False),
             )
         )
 
@@ -710,7 +711,7 @@ class DocumentIngestSQLiteCallback(BaseCallbackHandler):
                 token_count=token_count,
                 cost_usd=float(cost_usd),
                 n_try=float(payload.get('n_try', 0)),
-                metadata_json=json.dumps(payload, ensure_ascii=False),
+                metadata_json=safe_json_dumps(payload, ensure_ascii=False),
             )
         )
 
@@ -777,6 +778,6 @@ class DocumentIngestSQLiteCallback(BaseCallbackHandler):
                 token_count=0,
                 cost_usd=0.0,
                 n_try=float(payload.get('n_try', 0)),
-                metadata_json=json.dumps(payload, ensure_ascii=False),
+                metadata_json=safe_json_dumps(payload, ensure_ascii=False),
             )
         )

{graph_knowledge_doc_parser-0.2.4 → graph_knowledge_doc_parser-0.2.5}/kg_doc_parser/workflow_ingest/__init__.py

@@ -53,6 +53,8 @@ from .page_index import (
     BlockAssignment,
     BlockAssignmentBatch,
     CandidateBlock,
+    HierarchicalSummaryAssignment,
+    HierarchicalSummaryBatch,
     PageIndexBlockSpec,
     PageIndexParseResult,
     PageIndexValidationResult,
@@ -88,6 +90,7 @@ from .providers import (
     build_chat_model,
     build_chat_model_for_role,
     build_embedding_function,
+    provider_call_metrics_snapshot,
 )
 from .runners import (
     LayerwiseWorkflowCommandResult,
@@ -137,6 +140,8 @@ __all__ = [
     "EmbeddingProviderConfig",
     "FakeChatModel",
     "GroundedSourceRecord",
+    "HierarchicalSummaryAssignment",
+    "HierarchicalSummaryBatch",
     "HydratedTextPointer",
     "IngestExecutionClient",
     "IngestRunHandle",
@@ -210,6 +215,7 @@ __all__ = [
     "prepare_layer_frontier",
     "prepare_ocr_workflow_input",
     "propose_layer_breakdown",
+    "provider_call_metrics_snapshot",
     "review_layer",
     "run_demo_harness",
     "run_demo_harness_workflow",

{graph_knowledge_doc_parser-0.2.4 → graph_knowledge_doc_parser-0.2.5}/kg_doc_parser/workflow_ingest/cli.py

@@ -50,6 +50,9 @@ def _provider_settings_from_args(args: argparse.Namespace) -> WorkflowProviderSe
     parse_strategy_order = getattr(args, "parse_strategy_order", None)
     triage_enabled = getattr(args, "triage_enabled", None)
     page_index_summary_enabled = getattr(args, "page_index_summary_enabled", None)
+    page_index_hierarchical_summary_enabled = getattr(
+        args, "page_index_hierarchical_summary_enabled", None
+    )
     ocr_override = any(value is not None for value in ocr_values.values())
     parser_override = any(value is not None for value in parser_values.values())
     proposal_override = proposal_mode is not None
@@ -57,6 +60,7 @@ def _provider_settings_from_args(args: argparse.Namespace) -> WorkflowProviderSe
     strategy_order_override = parse_strategy_order is not None
     triage_override = triage_enabled is not None
     summary_override = page_index_summary_enabled is not None
+    hierarchical_summary_override = page_index_hierarchical_summary_enabled is not None
     if (
         not ocr_override
         and not parser_override
@@ -65,6 +69,7 @@ def _provider_settings_from_args(args: argparse.Namespace) -> WorkflowProviderSe
         and not strategy_order_override
         and not triage_override
         and not summary_override
+        and not hierarchical_summary_override
     ):
         return None
     settings = WorkflowProviderSettings.from_env()
@@ -80,6 +85,12 @@ def _provider_settings_from_args(args: argparse.Namespace) -> WorkflowProviderSe
         settings = settings.model_copy(update={"triage_enabled": triage_enabled})
     if summary_override:
         settings = settings.model_copy(update={"page_index_summary_enabled": page_index_summary_enabled})
+    if hierarchical_summary_override:
+        settings = settings.model_copy(
+            update={
+                "page_index_hierarchical_summary_enabled": page_index_hierarchical_summary_enabled
+            }
+        )
     if ocr_override:
         settings = settings.model_copy(
             update={
@@ -125,6 +136,12 @@ def _add_provider_args(parser: argparse.ArgumentParser) -> None:
         default=None,
         help="Enable or disable summaries on PageIndex nodes for this parse call",
     )
+    group.add_argument(
+        "--page-index-hierarchical-summary-enabled",
+        action=argparse.BooleanOptionalAction,
+        default=None,
+        help="Enable or disable the bounded parent-summary context pass",
+    )
     group.add_argument("--ocr-provider", default=None)
     group.add_argument("--ocr-model", default=None)
     group.add_argument("--ocr-base-url", default=None)

{graph_knowledge_doc_parser-0.2.4 → graph_knowledge_doc_parser-0.2.5}/kg_doc_parser/workflow_ingest/handlers.py

@@ -2,7 +2,7 @@ from __future__ import annotations
 
 import logging
 from collections.abc import Callable, Mapping
-from typing import Protocol, TypedDict, cast
+from typing import Literal, Protocol, TypedDict, cast
 
 from kogwistar.runtime import MappingStepResolver
 from kogwistar.runtime.models import RunFailure, RunSuccess, RunSuspended, StepRunResult
@@ -28,7 +28,7 @@ from .models import (
     WorkflowExportBundle,
     WorkflowIngestInput,
 )
-from .page_index import parse_page_index_layer
+from .page_index import PageIndexSourceFormat, parse_page_index_layer
 from .parser_core import (
     ParseSemanticFn,
     ProposeLayerFn,
@@ -64,6 +64,21 @@ from .strategy import (
 _LOGGER = logging.getLogger(__name__)
 
 
+def _page_index_source_format(inp: WorkflowIngestInput) -> PageIndexSourceFormat:
+    """Read the document format carried by the normalized collection metadata."""
+
+    collection = select_primary_collection(inp)
+    candidates: list[object] = [collection.metadata.get("source_format")]
+    for page in collection.pages:
+        candidates.append(page.metadata.get("source_format"))
+        for unit in page.units:
+            candidates.append(unit.metadata.get("source_format"))
+    for value in candidates:
+        if value in {"text", "markdown"}:
+            return cast(Literal["text", "markdown"], value)
+    return "text"
+
+
 class StepHandler(Protocol):
     """Execute one parser workflow step against the runtime context."""
 
@@ -92,9 +107,11 @@ class WorkflowRuntimeDeps(TypedDict, total=False):
     split_strategy: SplitStrategy
     fallback_split_strategy: SplitStrategy
     max_review_retries: int
+    layer_frontier_batch_size: int
     coverage_threshold: float
     provider_settings: WorkflowProviderSettings
     triage_strategy_fn: Callable[[dict[str, object]], object]
+    provider_diagnostics_sink: Callable[[dict[str, object]], None]
 
 
 def _build_export_bundle(
@@ -363,10 +380,22 @@ def register_layerwise_parser_steps(
             if normalized_input.page_index_summary_enabled is not None
             else (settings.page_index_summary_enabled if settings is not None else True)
         )
+        page_index_hierarchical_summary_enabled = (
+            normalized_input.page_index_hierarchical_summary_enabled
+            if normalized_input.page_index_hierarchical_summary_enabled is not None
+            else (
+                settings.page_index_hierarchical_summary_enabled
+                if settings is not None
+                else False
+            )
+        )
         triage_build_error: str | None = None
         if triage_fn is None and settings is not None and triage_enabled and requested == "auto":
             try:
-                triage_fn = build_llm_strategy_triage(settings)
+                triage_fn = build_llm_strategy_triage(
+                    settings,
+                    diagnostics_sink=runtime_deps.get("provider_diagnostics_sink"),
+                )
             except Exception as exc:  # noqa: BLE001 - unavailable providers use deterministic fallback.
                 triage_build_error = f"triage provider unavailable: {type(exc).__name__}: {exc}"
         parent_context: list[dict[str, object]] = []
@@ -434,6 +463,7 @@ def register_layerwise_parser_steps(
                 "parse_strategy_fallback_order": list(decision.fallback_order),
                 "disabled_strategies": sorted(disabled_strategies),
                     "page_index_summary_enabled": page_index_summary_enabled,
+                    "page_index_hierarchical_summary_enabled": page_index_hierarchical_summary_enabled,
             }
         )
         selected_split_strategy = (
@@ -483,6 +513,7 @@ def register_layerwise_parser_steps(
                     "disabled_strategies": sorted(disabled_strategies),
                     "page_index_attempted": False,
                     "page_index_summary_enabled": page_index_summary_enabled,
+                    "page_index_hierarchical_summary_enabled": page_index_hierarchical_summary_enabled,
                 },
                 "retry_count": 0,
             }
@@ -515,7 +546,9 @@ def register_layerwise_parser_steps(
     @_register_step(resolver, step_name="page_index_layer", runtime_deps=runtime_deps)
     def _page_index_layer(ctx: StepContext) -> StepRunResult:
         current_layer_context = CurrentLayerContext.model_validate(ctx.state_view["current_layer_context"])
+        normalized_input = WorkflowIngestInput.model_validate(ctx.state_view["normalized_input"])
         parser_source_map = ctx.state_view.get("parser_source_map") or {}
+        source_format = _page_index_source_format(normalized_input)
         candidates = []
         for parent_id, parent_title in zip(
             current_layer_context.parent_node_ids,
@@ -527,7 +560,7 @@ def register_layerwise_parser_steps(
                     parent_title=parent_title,
                     parent_pointers=current_layer_context.parent_content_pointers_by_id.get(parent_id, []),
                     parser_source_map=parser_source_map,
-                    source_format="text",
+                    source_format=source_format,
                     summary_enabled=bool(current_layer_context.metadata.get("page_index_summary_enabled", True)),
                 )
             )
@@ -568,6 +601,14 @@ def register_layerwise_parser_steps(
     def _prepare_layer_frontier(ctx: StepContext) -> StepRunResult:
         parse_session = ParseSessionState.model_validate(ctx.state_view["parse_session"])
         semantic_tree = SemanticNode.model_validate(ctx.state_view["semantic_tree"])
+        configured_batch_size = runtime_deps.get("layer_frontier_batch_size")
+        if configured_batch_size is None:
+            settings = runtime_deps.get("provider_settings")
+            configured_batch_size = (
+                getattr(settings, "layer_frontier_batch_size", None)
+                if settings is not None
+                else 1
+            )
         context, remaining, updated_session = prepare_layer_frontier(
             parse_session=parse_session,
             frontier_queue=[
@@ -576,6 +617,7 @@ def register_layerwise_parser_steps(
             ],
             semantic_tree=semantic_tree,
             max_retries=int(runtime_deps.get("max_review_retries", 3)),
+            max_items=(int(configured_batch_size) if configured_batch_size is not None else None),
         )
         with ctx.state_write as st:
             st["parse_session"] = updated_session.model_dump(field_mode="backend", dump_format="json")
@@ -731,6 +773,7 @@ def register_layerwise_parser_steps(
                 parent_node_ids=list(current_layer_context.parent_node_ids),
                 attempt=int((ctx.state_view.get("strategy_attempt_counts") or {}).get(strategy, 1)),
                 event="failed",
+                failure_type="retry",
                 reasons=reasons[:12],
             )
             if remaining:
@@ -844,7 +887,10 @@ def register_layerwise_parser_steps(
 
     @_register_step(resolver, step_name="finalize_semantic_tree", runtime_deps=runtime_deps)
     def _finalize_semantic_tree(ctx: StepContext) -> StepRunResult:
-        tree = finalize_semantic_tree(SemanticNode.model_validate(ctx.state_view["semantic_tree"]))
+        tree = finalize_semantic_tree(
+            SemanticNode.model_validate(ctx.state_view["semantic_tree"]),
+            parser_source_map=ctx.state_view.get("parser_source_map") or {},
+        )
         with ctx.state_write as st:
             st["semantic_tree"] = tree.model_dump()
         return _success("validate_tree")
@@ -872,7 +918,7 @@ def register_postparse_steps(
             validation_notes=[],
         )
         bundle = _build_export_bundle(ctx=ctx, runtime_deps=runtime_deps)
-        threshold = float(runtime_deps.get("coverage_threshold", 0.99))
+        threshold = float(runtime_deps.get("coverage_threshold", 1.0))
         if report.overall_text_coverage < threshold:
             error_message = (
                 f"text coverage below threshold: {report.overall_text_coverage:.3f} < {threshold:.3f}"