graph-knowledge-doc-parser 0.1.0__tar.gz → 0.2.2__tar.gz

graph_knowledge_doc_parser-0.1.0/README.md → graph_knowledge_doc_parser-0.2.2/PKG-INFO

@@ -1,50 +1,123 @@
-# Kogwistar-docparser 
-
-If you want the shortest path into the project, start with [QUICKSTART.md](QUICKSTART.md).
-
-## Graph Knowledge Doc Parser
-
-Utilities and experiments for document ingestion, PDF splitting, OCR, and page-level parsing. Refactored from the kogwistar project as a stand alone ingestor.
-
-## Status
-
-This repository is still being refactored and should be treated as work in progress.
-
-The document ingestion pipeline is currently being extracted and consolidated into the main `kogwistar` repository. Until that refactor is complete, this repo should be considered an active staging area for parser and ingestion-related work.
-
-## What Is Here
-
-- PDF splitting and image generation helpers in `src/pdf2png.py`
-- Gemini-based OCR and page parsing flows in `src/ocr.py`
-- SQLite-based ingestion telemetry in `src/document_ingester_logger.py`
-- File discovery and filtering helpers in `src/utils/file_loaders.py`
-- Experimental and regression-style tests under `tests/`
-
-## Workflow Surface
-
-The reusable workflow-ingest code now has three layers:
-
-- Python APIs, which are the primary contract for tests and orchestration
-- CLI entrypoints, which are thin wrappers around those APIs
-- composable subworkflows for OCR, page-index parsing, and recursive layerwise parsing
-
-The reusable helpers live under `src/workflow_ingest/` and are designed so the
-same core logic can be called from tests, scripts, and higher-level workflow
-code without duplicating orchestration.
-
-### CLI Commands
-
+Metadata-Version: 2.4
+Name: graph-knowledge-doc-parser
+Version: 0.2.2
+Summary: doc parser using llm driven engine with graph knowledge awareness
+Author: humblemat810
+Author-email: 67593116+humblemat810@users.noreply.github.com
+Requires-Python: >=3.12,<4.0
+Classifier: License :: Other/Proprietary License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: 3.15
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Topic :: Text Processing :: General
+Requires-Dist: diskcache (>=5.6,<6.0) ; platform_python_implementation == "PyPy"
+Requires-Dist: joblib (>=1.5.3,<2.0.0) ; platform_python_implementation == "CPython"
+Requires-Dist: kogwistar (==0.6.3)
+Requires-Dist: langchain-core (>=1.2.5,<2.0.0)
+Requires-Dist: langchain-google-genai (>=4.1.2,<5.0.0)
+Requires-Dist: mcp (>=2.2.0,<3.0.0)
+Requires-Dist: pathspec (>=0.12.1,<0.13.0)
+Requires-Dist: pdf2image (>=1.17.0,<2.0.0)
+Requires-Dist: pikepdf (>=10.1.0,<11.0.0) ; platform_python_implementation != "PyPy"
+Requires-Dist: pydantic (>=2.12.5,<3.0.0)
+Requires-Dist: pydantic-extension (>=0.0.7,<0.0.8)
+Requires-Dist: pypdf (>=6.5.0,<7.0.0)
+Requires-Dist: rapidfuzz (>=3.14.3,<4.0.0)
+Project-URL: Homepage, https://github.com/humblemat810/kg_doc_parser
+Project-URL: Repository, https://github.com/humblemat810/kg_doc_parser
+Description-Content-Type: text/markdown
+
+# Kogwistar Graph-Knowledge Doc Pipeline
+
+If you want the shortest path into the project, start with [QUICKSTART.md](QUICKSTART.md).
+
+## Graph Knowledge Doc Pipeline
+
+Utilities and experiments for document ingestion, PDF splitting, OCR, page-level parsing, and conversion into graph-knowledge artifacts that the Obsidian sink and downstream Kogwistar workflows can consume. Refactored from the kogwistar project as a standalone ingestion-to-graph pipeline.
+
+## Status
+
+This repository is still being refactored and should be treated as work in progress.
+
+The document ingestion pipeline is currently being extracted and consolidated into the main `kogwistar` repository. Until that refactor is complete, this repo should be considered an active staging area for parser and ingestion-related work.
+
+## What Is Here
+
+- PDF splitting and image generation helpers in `src/pdf2png.py`
+- Gemini-based OCR and page parsing flows in `src/ocr.py`
+- SQLite-based ingestion telemetry in `src/document_ingester_logger.py`
+- File discovery and filtering helpers in `src/utils/file_loaders.py`
+- Experimental and regression-style tests under `tests/`
+
+## Workflow Surface
+
+The reusable workflow-ingest code now has three layers:
+
+- Python APIs, which are the primary contract for tests and orchestration
+- CLI entrypoints, which are thin wrappers around those APIs
+- composable subworkflows for OCR, page-index parsing, recursive layerwise parsing, and graph-knowledge conversion
+
+The reusable helpers live under `src/workflow_ingest/` and are designed so the
+same core logic can be called from tests, scripts, and higher-level workflow
+code without duplicating orchestration.
+
+For an adoption path that starts with parser-grounded source units and later
+adds LLM-Wiki cross-document maintenance, see
+[`doc/progressive_adoption_guide.md`](doc/progressive_adoption_guide.md). The
+guide also explains which vector-RAG and source-import responsibilities remain
+with the integrating application.
+
+## CLI Cheatsheet
+
+| Command | What it does | Main tests |
+|---|---|---|
+| `workflow-ingest ocr <source> --output-dir <dir>` | Runs OCR over a file or folder and writes OCR artifacts | `tests/test_workflow_ingest_parsing_api.py`, `tests/test_workflow_ingest_cli.py` |
+| `workflow-ingest page-index <source> --output-dir <dir>` | Parses page-index text into structured graph input | `tests/test_workflow_ingest_parsing_api.py`, `tests/test_workflow_ingest_cli.py` |
+| `workflow-ingest layerwise <source> --output-dir <dir>` | Runs the recursive layerwise parser workflow | `tests/test_workflow_ingest_layerwise_parser.py`, `tests/test_workflow_ingest_cli.py` |
+| `workflow-ingest demo --output-dir <dir>` | Runs the end-to-end demo harness and writes run artifacts | `tests/test_workflow_ingest_demo_harness.py` |
+| `workflow-ingest ocr-smoke-assets --output-dir <dir>` | Generates local OCR smoke assets for manual or automated runs | `tests/test_workflow_ingest_cli.py` |
+
+For a quick sanity check, `workflow-ingest --help` shows the full command tree,
+and `workflow-ingest demo --help` shows the demo-specific options without
+writing any artifacts.
+
+### CLI Commands
+
 After `poetry install`, the repo exposes a `workflow-ingest` command family:
-
-```powershell
-workflow-ingest --help
-workflow-ingest ocr --help
-workflow-ingest page-index --help
-workflow-ingest layerwise --help
-workflow-ingest demo --help
+
+```powershell
+workflow-ingest --help
+workflow-ingest ocr --help
+workflow-ingest page-index --help
+workflow-ingest layerwise --help
+workflow-ingest demo --help
 workflow-ingest ocr-smoke-assets --help
 ```
 
+To inspect the demo harness options:
+
+```powershell
+workflow-ingest demo --help
+```
+
+To actually run the demo harness, use:
+
+```powershell
+workflow-ingest demo --output-dir logs\workflow_ingest_demo
+```
+
+Then open the output directory:
+
+```powershell
+explorer logs\workflow_ingest_demo
+```
+
+That directory contains the probe trail, summary JSON, cache directory, and the local engine/server data for the run.
+
 If you want the local checked-out `./kogwistar` subtree to win over the GitHub
 dependency during development, run the Bash bootstrap helper after install:
 
@@ -59,178 +132,197 @@ That script does two things:
 
 If you do not run the bootstrap helper, the repo keeps using the GitHub-sourced
 `kogwistar` dependency declared in `pyproject.toml`.
-
-Typical examples:
-
-```powershell
-workflow-ingest ocr tests\.tmp_workflow_ingest_ocr\generated_smoke_assets\ocr_smoke_document.pdf --output-dir logs\ocr_run
-workflow-ingest ocr tests\.tmp_workflow_ingest_ocr\generated_smoke_assets --output-dir logs\ocr_batch
-workflow-ingest page-index tests\fixtures\page_index\sample_page_index.txt --output-dir logs\page_index
-workflow-ingest layerwise tests\.tmp_workflow_ingest_ocr\manual_cases\ollama\glm-ocr_latest\image\artifacts\legacy_split_pages\ocr-manual-ollama-image --output-dir logs\layerwise
-workflow-ingest ocr-smoke-assets --output-dir tests\.tmp_workflow_ingest_ocr\generated_smoke_assets
-```
-
-### Reusable Artifacts
-
-The workflow outputs are intentionally inspectable on disk:
-
-- `workflow-events.jsonl`: readable step trail for the outer orchestration layer
-- `ocr-state.sqlite`: authoritative OCR/render resume state
-- `ocr-progress.json`: human-readable mirror of the current OCR state
-- `ocr-summary.json`: final OCR run summary
-- `legacy_split_pages/<document>/page_N.json`: legacy-compatible OCR page artifacts
-- `rendered_pages/<document>/page_N.png`: rasterized page images
-- `page-index-summary.json`: page-index run summary
-- `layerwise-summary.json`: recursive layerwise parser summary
-- `layerwise-graph.json`: legacy recursive layerwise graph payload
-
-### Python APIs
-
-If you want to embed the pipelines directly, the main helpers are:
-
-- `src.workflow_ingest.run_ocr_source_workflow(...)`
-- `src.workflow_ingest.run_ocr_batch_workflow(...)`
-- `src.workflow_ingest.parse_page_index_document(...)`
-- `src.workflow_ingest.run_page_index_source_workflow(...)`
-- `src.workflow_ingest.run_layerwise_source_workflow(...)`
-- `src.workflow_ingest.run_demo_harness_workflow(...)`
-
-Those helpers are meant to stay stable and are what the CLI layer calls under
-the hood.
-
-## Setup
-
-1. Create and activate a Python 3.13 environment.
-2. Install dependencies with Poetry:
-
-```powershell
-poetry install
-```
-
-3. Create a local env file from the example:
-
-```powershell
-Copy-Item .env.example .env
-```
-
-4. Add your local `GOOGLE_API_KEY` and any optional file-list paths needed for your workflow.
-
-## Environment Variables
-
-The project currently expects or optionally uses:
-
-- `GOOGLE_API_KEY`: required for Gemini OCR and LLM-backed parsing flows
-- `LANGSMITH_TRACING`: optional LangSmith tracing toggle
-- `ocr_file_list`: optional allow-list file for OCR runs
-- `split_raw_file_list`: optional allow-list file for PDF splitting runs
-- `answer_export_list`: optional export list path used by local workflows
-
-An example template is provided in [`.env.example`](/c:/Users/chanh/Documents/kg_doc_parser/.env.example).
-
-## Provider Guide
-
-The workflow layer is vendor-neutral, but the concrete OCR, parser, and embedding
-backends are selected by config.
-
-### OCR Provider Examples
-
-- Google GenAI OCR:
-  - `KG_DOC_OCR_PROVIDER=gemini`
-  - `KG_DOC_OCR_MODEL=gemini-2.5-flash`
-- Ollama OCR or vision-capable local model:
-  - `KG_DOC_OCR_PROVIDER=ollama`
-  - `KG_DOC_OCR_MODEL=llava:latest`
-  - `KG_DOC_OCR_BASE_URL=http://127.0.0.1:11434`
-- Vertex AI OCR:
-  - `KG_DOC_OCR_PROVIDER=vertex`
-  - `KG_DOC_OCR_MODEL=gemini-2.5-pro`
-  - `KG_DOC_OCR_PROJECT=my-project`
-  - `KG_DOC_OCR_LOCATION=us-central1`
-
-### Parser / LLM Provider Examples
-
-The parser provider is the chat model used for semantic parsing, layer review,
-and structured extraction.
-
-- LangChain Google GenAI:
-  - `KG_DOC_PARSER_PROVIDER=gemini`
-  - `KG_DOC_PARSER_MODEL=gemini-2.5-flash`
-- ChatGPT / OpenAI REST:
-  - `KG_DOC_PARSER_PROVIDER=openai`
-  - `KG_DOC_PARSER_MODEL=gpt-4.1-mini`
-  - `KG_DOC_PARSER_API_KEY_ENV=OPENAI_API_KEY`
-- LangChain Ollama:
-  - `KG_DOC_PARSER_PROVIDER=ollama`
-  - `KG_DOC_PARSER_MODEL=llama3.1`
-  - `KG_DOC_PARSER_BASE_URL=http://127.0.0.1:11434`
-- LangChain Vertex AI:
-  - `KG_DOC_PARSER_PROVIDER=vertex`
-  - `KG_DOC_PARSER_MODEL=gemini-2.5-pro`
-  - `KG_DOC_PARSER_PROJECT=my-project`
-  - `KG_DOC_PARSER_LOCATION=us-central1`
-
-### Recipe Parsing Example
-
-If you are parsing a cooking recipe, one practical split is:
-
-- OCR on Gemini or another vision model
-- parser on OpenAI, Ollama, or Vertex AI
-
-For example:
-
-```powershell
-KG_DOC_OCR_PROVIDER=gemini
-KG_DOC_OCR_MODEL=gemini-2.5-flash
-KG_DOC_PARSER_PROVIDER=openai
-KG_DOC_PARSER_MODEL=gpt-4.1-mini
-KG_DOC_PARSER_API_KEY_ENV=OPENAI_API_KEY
-KG_DOC_EMBED_PROVIDER=fake
-```
-
-That setup can extract a recipe into structured graph data such as:
-- ingredients
-- steps
-- tools
-- timers
-- inferred sections like `prep`, `cook`, and `serve`
-
-### Embedding Examples
-
-- Fake deterministic CI embedding:
-  - `KG_DOC_EMBED_PROVIDER=fake`
-- OpenAI embeddings:
-  - `KG_DOC_EMBED_PROVIDER=openai`
-  - `KG_DOC_EMBED_MODEL=text-embedding-3-small`
-- Vertex AI embeddings:
-  - `KG_DOC_EMBED_PROVIDER=vertex`
-  - `KG_DOC_EMBED_MODEL=text-embedding-004`
-- Ollama embeddings:
-  - `KG_DOC_EMBED_PROVIDER=ollama`
-  - `KG_DOC_EMBED_MODEL=nomic-embed-text`
-
-Note:
-
-- `embedding_space` in the workflow ingest models is currently a metadata and
-  routing-intent label.
-- It does not yet imply that the engine is using a separate embedder per space.
-- The current engine bootstrap still wires one embedding function per engine
-  instance, while the multi-space routing proposal remains a future Kogwistar
-  core concern.
-
-## Running Tests
-
-Some tests are integration-style and expect local document folders and API credentials to exist. That means not every test is portable in a clean checkout.
-
-To run the test suite:
-
-```powershell
-pytest -q
-```
-
-If you only want to work on isolated units, review the test files first and run a narrower subset.
-
-## Demo Harness
-
+
+Typical examples:
+
+```powershell
+workflow-ingest ocr tests\.tmp_workflow_ingest_ocr\generated_smoke_assets\ocr_smoke_document.pdf --output-dir logs\ocr_run
+workflow-ingest ocr tests\.tmp_workflow_ingest_ocr\generated_smoke_assets --output-dir logs\ocr_batch
+workflow-ingest page-index tests\fixtures\page_index\sample_page_index.txt --output-dir logs\page_index
+workflow-ingest layerwise tests\.tmp_workflow_ingest_ocr\manual_cases\ollama\glm-ocr_latest\image\artifacts\legacy_split_pages\ocr-manual-ollama-image --output-dir logs\layerwise
+workflow-ingest ocr-smoke-assets --output-dir tests\.tmp_workflow_ingest_ocr\generated_smoke_assets
+```
+
+### Reusable Artifacts
+
+The workflow outputs are intentionally inspectable on disk:
+
+- `workflow-events.jsonl`: readable step trail for the outer orchestration layer
+- `ocr-state.sqlite`: authoritative OCR/render resume state
+- `ocr-progress.json`: human-readable mirror of the current OCR state
+- `ocr-summary.json`: final OCR run summary
+- `legacy_split_pages/<document>/page_N.json`: legacy-compatible OCR page artifacts
+- `rendered_pages/<document>/page_N.png`: rasterized page images
+- `page-index-summary.json`: page-index run summary
+- `layerwise-summary.json`: recursive layerwise parser summary
+- `layerwise-graph.json`: legacy recursive layerwise graph payload
+
+### Python APIs
+
+If you want to embed the pipelines directly, the main helpers are:
+
+- `src.workflow_ingest.run_ocr_source_workflow(...)`
+- `src.workflow_ingest.run_ocr_batch_workflow(...)`
+- `src.workflow_ingest.parse_page_index_document(...)`
+- `src.workflow_ingest.run_page_index_source_workflow(...)`
+- `src.workflow_ingest.run_layerwise_source_workflow(...)`
+- `src.workflow_ingest.run_demo_harness_workflow(...)`
+
+Those helpers are meant to stay stable and are what the CLI layer calls under
+the hood.
+
+## Setup
+
+1. Create and activate a Python 3.13 environment.
+2. Install dependencies with Poetry:
+
+```powershell
+poetry install
+```
+
+3. Create a local env file from the example:
+
+```powershell
+Copy-Item .env.example .env
+```
+
+4. Add your local `GOOGLE_API_KEY` and any optional file-list paths needed for your workflow.
+
+## Environment Variables
+
+The project currently expects or optionally uses:
+
+- `GOOGLE_API_KEY`: required for Gemini OCR and LLM-backed parsing flows
+- `LANGSMITH_TRACING`: optional LangSmith tracing toggle
+- `ocr_file_list`: optional allow-list file for OCR runs
+- `split_raw_file_list`: optional allow-list file for PDF splitting runs
+- `answer_export_list`: optional export list path used by local workflows
+
+An example template is provided in [`.env.example`](/c:/Users/chanh/Documents/kg_doc_parser/.env.example).
+
+## Provider Guide
+
+The workflow layer is vendor-neutral, but the concrete OCR, parser, and embedding
+backends are selected by config.
+
+### OCR Provider Examples
+
+- Google GenAI OCR:
+  - `KG_DOC_OCR_PROVIDER=gemini`
+  - `KG_DOC_OCR_MODEL=gemini-2.5-flash`
+- Ollama OCR or vision-capable local model:
+  - `KG_DOC_OCR_PROVIDER=ollama`
+  - `KG_DOC_OCR_MODEL=llava:latest`
+  - `KG_DOC_OCR_BASE_URL=http://127.0.0.1:11434`
+- Vertex AI OCR:
+  - `KG_DOC_OCR_PROVIDER=vertex`
+  - `KG_DOC_OCR_MODEL=gemini-2.5-pro`
+  - `KG_DOC_OCR_PROJECT=my-project`
+  - `KG_DOC_OCR_LOCATION=us-central1`
+
+### Parser / LLM Provider Examples
+
+The parser provider is the chat model used for semantic parsing, layer review,
+and structured extraction.
+
+- LangChain Google GenAI:
+  - `KG_DOC_PARSER_PROVIDER=gemini`
+  - `KG_DOC_PARSER_MODEL=gemini-2.5-flash`
+- ChatGPT / OpenAI REST:
+  - `KG_DOC_PARSER_PROVIDER=openai`
+  - `KG_DOC_PARSER_MODEL=gpt-4.1-mini`
+  - `KG_DOC_PARSER_API_KEY_ENV=OPENAI_API_KEY`
+- LangChain Ollama:
+  - `KG_DOC_PARSER_PROVIDER=ollama`
+  - `KG_DOC_PARSER_MODEL=llama3.1`
+  - `KG_DOC_PARSER_BASE_URL=http://127.0.0.1:11434`
+- LangChain Vertex AI:
+  - `KG_DOC_PARSER_PROVIDER=vertex`
+  - `KG_DOC_PARSER_MODEL=gemini-2.5-pro`
+  - `KG_DOC_PARSER_PROJECT=my-project`
+  - `KG_DOC_PARSER_LOCATION=us-central1`
+
+### Recipe Parsing Example
+
+If you are parsing a cooking recipe, one practical split is:
+
+- OCR on Gemini or another vision model
+- parser on OpenAI, Ollama, or Vertex AI
+
+For example:
+
+```powershell
+KG_DOC_OCR_PROVIDER=gemini
+KG_DOC_OCR_MODEL=gemini-2.5-flash
+KG_DOC_PARSER_PROVIDER=openai
+KG_DOC_PARSER_MODEL=gpt-4.1-mini
+KG_DOC_PARSER_API_KEY_ENV=OPENAI_API_KEY
+KG_DOC_EMBED_PROVIDER=fake
+```
+
+That setup can extract a recipe into structured graph data such as:
+- ingredients
+- steps
+- tools
+- timers
+- inferred sections like `prep`, `cook`, and `serve`
+
+### Embedding Examples
+
+- Fake deterministic CI embedding:
+  - `KG_DOC_EMBED_PROVIDER=fake`
+- OpenAI embeddings:
+  - `KG_DOC_EMBED_PROVIDER=openai`
+  - `KG_DOC_EMBED_MODEL=text-embedding-3-small`
+- Vertex AI embeddings:
+  - `KG_DOC_EMBED_PROVIDER=vertex`
+  - `KG_DOC_EMBED_MODEL=text-embedding-004`
+- Ollama embeddings:
+  - `KG_DOC_EMBED_PROVIDER=ollama`
+  - `KG_DOC_EMBED_MODEL=nomic-embed-text`
+
+Note:
+
+- `embedding_space` in the workflow ingest models is currently a metadata and
+  routing-intent label.
+- It does not yet imply that the engine is using a separate embedder per space.
+- The current engine bootstrap still wires one embedding function per engine
+  instance, while the multi-space routing proposal remains a future Kogwistar
+  core concern.
+
+## Running Tests
+
+Some tests are integration-style and expect local document folders and API credentials to exist. That means not every test is portable in a clean checkout.
+
+Deterministic CI covers parser contracts with fake or injected model responses.
+Model choice and output quality are parser-repository concerns, not core migration
+gates. Any test that invokes Ollama, Gemini, OpenAI, Azure, Vertex, or another real
+provider must use `llm_real` or `manual`; Ollama tests must also use
+`requires_ollama`. Such tests must not carry `ci` or `ci_full`. Collection fails if
+these marker classes are mixed, preventing accidental live-model calls in CI.
+
+Run deterministic contract CI with:
+
+```powershell
+pytest -m "(ci or ci_full) and not manual and not llm_real and not requires_ollama"
+```
+
+Run provider/model checks only when explicitly requested, for example:
+
+```powershell
+pytest -m "llm_real and requires_ollama"
+```
+
+To run the test suite:
+
+```powershell
+pytest -q
+```
+
+If you only want to work on isolated units, review the test files first and run a narrower subset.
+
+## Demo Harness
+
 There is now a manual workflow-ingest demo harness that can run the end-to-end flow against:
 
 - an in-process isolated FastAPI server
@@ -242,53 +334,63 @@ The legacy semantic-smoke test in
 also expects a live Kogwistar server at `http://127.0.0.1:28110`. It does not
 start that server for you, so use the VS Code server launch config or start it
 manually before running the Ollama case.
-
-Example:
-
-```powershell
-.venv\Scripts\python.exe scripts\run_workflow_ingest_demo.py --output-dir logs\workflow_ingest_demo
-```
-
-External live server example:
-
-```powershell
-.venv\Scripts\python.exe scripts\run_workflow_ingest_demo.py --server-mode external_http --external-base-url http://127.0.0.1:28110
-```
-
-Demo artifacts are written into the chosen output directory:
-
-- `probe-events.jsonl`: demo-friendly step and lifecycle probe events
-- `demo-summary.json`: run summary, persistence result, and artifact pointers
-- `llm-cache/`: workflow-native cached proposal/review call results
-- `engines/`: local workflow and conversation graph storage for the run
-- `server-data/`: isolated server-side persistence directory when the harness boots its own server
-
-Notes:
-
-- The workflow-native layer proposal/review path uses deterministic file-backed caching to reduce repeated token cost and compute time.
-- The legacy parser path also supports a redirected `joblib` cache via `KG_DOC_PARSER_JOBLIB_CACHE_DIR`.
-- Probe logging is separate from CDC and conversation graph traces, so demos can show a short readable event trail without digging into runtime internals.
-
-## OCR And Parsing Workflows
-
-The newer workflow-first paths are designed as reusable subworkflows:
-
-- OCR image/PDF ingest
-  - resumable via `ocr-state.sqlite`
-  - emits `workflow-events.jsonl`
-  - keeps legacy OCR page artifacts on disk
-- page-index parsing
-  - heuristic mode for deterministic structure extraction
-  - Ollama mode for local parser-backed parsing
-- recursive layerwise parsing
-  - wraps the legacy recursive parser in a reusable workflow runner
-
-These can be invoked from Python directly or through the `workflow-ingest`
-CLI family, depending on whether you want reusable orchestration or a quick
-shell command.
-
-## Notes
-
-- `README.md`, env handling, and ingestion boundaries are still being cleaned up as part of the ongoing refactor.
-- Runtime outputs such as `logs/`, local `.env`, caches, and generated artifacts should remain uncommitted.
-- If behavior diverges between this repo and `kogwistar`, prefer the direction of the ongoing migration and refactor work.
+
+Example:
+
+```powershell
+.venv\Scripts\python.exe scripts\run_workflow_ingest_demo.py --output-dir logs\workflow_ingest_demo
+```
+
+External live server example:
+
+```powershell
+.venv\Scripts\python.exe scripts\run_workflow_ingest_demo.py --server-mode external_http --external-base-url http://127.0.0.1:28110
+```
+
+Demo artifacts are written into the chosen output directory:
+
+- `probe-events.jsonl`: demo-friendly step and lifecycle probe events
+- `demo-summary.json`: run summary, persistence result, and artifact pointers
+- `llm-cache/`: workflow-native cached proposal/review call results
+- `engines/`: local workflow and conversation graph storage for the run
+- `server-data/`: isolated server-side persistence directory when the harness boots its own server
+
+Notes:
+
+- The workflow-native layer proposal/review path uses deterministic file-backed caching to reduce repeated token cost and compute time.
+- The legacy parser path uses the shared Kogwistar cache facade. Set
+  `KG_DOC_PARSER_CACHE_BACKEND=auto|joblib|diskcache|none` and
+  `KG_DOC_PARSER_CACHE_DIR` to choose the provider and location. `auto` selects
+  DiskCache on PyPy and Joblib on CPython. The older
+  `KG_DOC_PARSER_JOBLIB_CACHE_DIR` variable remains a compatibility alias.
+- Probe logging is separate from CDC and conversation graph traces, so demos can show a short readable event trail without digging into runtime internals.
+
+## OCR And Parsing Workflows
+
+The newer workflow-first paths are designed as reusable subworkflows:
+
+- OCR image/PDF ingest
+  - resumable via `ocr-state.sqlite`
+  - emits `workflow-events.jsonl`
+  - keeps legacy OCR page artifacts on disk
+- page-index parsing
+  - heuristic mode for deterministic structure extraction
+  - Ollama mode for local parser-backed parsing
+- recursive layerwise parsing
+  - wraps the legacy recursive parser in a reusable workflow runner
+
+These can be invoked from Python directly or through the `workflow-ingest`
+CLI family, depending on whether you want reusable orchestration or a quick
+shell command.
+
+## Notes
+
+- `README.md`, env handling, and ingestion boundaries are still being cleaned up as part of the ongoing refactor.
+- Runtime outputs such as `logs/`, local `.env`, caches, and generated artifacts should remain uncommitted.
+- If behavior diverges between this repo and `kogwistar`, prefer the direction of the ongoing migration and refactor work.
+
+Two-stage conversation materialization is an engine-owned capability. The
+parser writes grounded stage-one artifacts through the supplied conversation
+engine and does not create a second embedding queue. See
+[`doc/adr_conversation_two_stage_parser_contract.md`](doc/adr_conversation_two_stage_parser_contract.md).
+