ski-kg-extractor 3.0.3__tar.gz

ski_kg_extractor-3.0.3/PKG-INFO

@@ -0,0 +1,120 @@
+Metadata-Version: 2.4
+Name: ski-kg-extractor
+Version: 3.0.3
+Summary: Extract compliance rules from regulatory documents using LLM-assisted parsing. Phase 1 only.
+Author-email: KpiFinity <hello@kpifinity.com>
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/kpifinity/ski-framework
+Project-URL: Documentation, https://kpifinity.github.io/ski-framework/
+Project-URL: Issues, https://github.com/kpifinity/ski-framework/issues
+Keywords: ski-framework,compliance,knowledge-graph,extraction
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Legal Industry
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: pydantic<3,>=2.13.4
+Requires-Dist: python-dotenv<2,>=1.2.2
+Requires-Dist: click<9,>=8.1.7
+Requires-Dist: pyyaml<7,>=6.0.1
+Requires-Dist: httpx<0.29,>=0.27
+Requires-Dist: pypdf<7,>=6.12.2
+Requires-Dist: python-docx<2,>=1.1.0
+Requires-Dist: anthropic<1,>=0.21.3
+Requires-Dist: openai<2,>=1.16.2
+
+# kg-extractor
+
+> Install from PyPI: `pip install ski-kg-extractor` (publishing starts with the first release after June 2026).
+
+> **Status:** v3.0 — first production-target release.
+
+Extract structured compliance rules from regulatory documents
+(LLM-assisted) and emit a **v3 Knowledge Graph** ready for
+`kg-validator` and the SKI Model runtime.
+
+`kg-extractor` is a **Phase 1 (compilation)** tool. The output goes
+through `kg-validator` (schema + §3.6 validation passes) and is then
+signed before crossing the sovereign boundary. The extractor's LLM
+backend may be local or cloud — its output is not part of the runtime
+path.
+
+## Highlights (v3.0)
+
+- **Emits v3 KG JSON by default.** Each extracted rule is wrapped into
+  the typed-graph shape from spec v3.0 §3: a Rule node, an Obligation
+  with a guessed `obligation_type` and numeric `value`/`unit`, a
+  Subject, a Citation, and the four edges connecting them
+  (`applies_to`, `consists_of`, `scoped_to`, `cited_by`).
+- **`--jurisdiction` flag.** Every KG carries a Jurisdiction node and
+  every rule is `scoped_to` it. Defaults to `global`.
+- **Pluggable LLM backend.** Default is **Ollama** (local). Cloud
+  backends (`anthropic`, `openai`) exist for compilation-phase use only.
+- **Deterministic by default.** Temperature 0, fixed seed, recorded
+  prompt SHA-256 in extraction metadata for reproducibility audits.
+- **`IMPLIED` is prohibited (B2.1 Anchor Constraint).** The prompt
+  instructs the model not to emit `IMPLIED` and the parser downgrades
+  any that slip through to `DISCRETIONARY` with a warning. Rules
+  cannot be inferred beyond source text.
+- **`extraction_quality` replaces `confidence`.** The per-rule trust
+  signal is now `extraction_quality` (EXPLICIT / DISCRETIONARY /
+  CONFLICTING) — a Phase-1 authoring concept, separate from the
+  runtime's categorical verdict (Axiom 2 prohibits confidence scores
+  in the audit trail).
+
+## Installation
+
+```bash
+pip install -e tools/kg-extractor
+```
+
+## Quick start
+
+```bash
+# Extract a v3 KG from a regulation:
+kg-extractor extract --file regulation.txt --sector energy \
+    --jurisdiction us.federal \
+    --output kg-extracted-v3.json
+
+# Validate the emitted KG:
+kg-validator validate --input kg-extracted-v3.json
+
+# Batch:
+kg-extractor batch --input-dir ./regulations --output-dir ./kgs \
+    --sector energy --jurisdiction us.federal
+
+# Debugging — dump the raw flat-rule list before wrapping:
+kg-extractor extract --file regulation.txt --emit-raw --output flat.json
+```
+
+## Architecture
+
+```
+kg_extractor/
+├── __init__.py     public API: Extractor, ExtractionResult, ExtractionQuality, emit_v3_kg
+├── extractor.py    chunk → LLM → flat ComplianceRule list
+├── models.py       ComplianceRule, ExtractionResult, ExtractionQuality enum
+├── backends.py     Ollama / Anthropic / OpenAI backends (compilation phase only)
+├── v3_emitter.py   wraps ExtractionResult into a v3 KG dict (spec §3 shape)
+├── utils.py        chunking, document parsing, rule validation
+└── cli.py          click-based CLI (`extract`, `batch`, `filter`, `examples`, `version`)
+```
+
+## Notes on `extraction_quality`
+
+- **EXPLICIT** — the LLM produced a verbatim source quote in
+  `reasoning`. The rule is safe to ship without further review.
+- **DISCRETIONARY** — the LLM could not anchor the rule in a verbatim
+  source quote. A human reviewer must approve before shipping.
+- **CONFLICTING** — the LLM detected internal contradiction in the
+  source. Surface for legal review.
+
+This is **not** a confidence score in the probabilistic sense. The
+runtime never reads it. It exists solely to drive the Phase-1
+human-review queue.

ski_kg_extractor-3.0.3/README.md

@@ -0,0 +1,89 @@
+# kg-extractor
+
+> Install from PyPI: `pip install ski-kg-extractor` (publishing starts with the first release after June 2026).
+
+> **Status:** v3.0 — first production-target release.
+
+Extract structured compliance rules from regulatory documents
+(LLM-assisted) and emit a **v3 Knowledge Graph** ready for
+`kg-validator` and the SKI Model runtime.
+
+`kg-extractor` is a **Phase 1 (compilation)** tool. The output goes
+through `kg-validator` (schema + §3.6 validation passes) and is then
+signed before crossing the sovereign boundary. The extractor's LLM
+backend may be local or cloud — its output is not part of the runtime
+path.
+
+## Highlights (v3.0)
+
+- **Emits v3 KG JSON by default.** Each extracted rule is wrapped into
+  the typed-graph shape from spec v3.0 §3: a Rule node, an Obligation
+  with a guessed `obligation_type` and numeric `value`/`unit`, a
+  Subject, a Citation, and the four edges connecting them
+  (`applies_to`, `consists_of`, `scoped_to`, `cited_by`).
+- **`--jurisdiction` flag.** Every KG carries a Jurisdiction node and
+  every rule is `scoped_to` it. Defaults to `global`.
+- **Pluggable LLM backend.** Default is **Ollama** (local). Cloud
+  backends (`anthropic`, `openai`) exist for compilation-phase use only.
+- **Deterministic by default.** Temperature 0, fixed seed, recorded
+  prompt SHA-256 in extraction metadata for reproducibility audits.
+- **`IMPLIED` is prohibited (B2.1 Anchor Constraint).** The prompt
+  instructs the model not to emit `IMPLIED` and the parser downgrades
+  any that slip through to `DISCRETIONARY` with a warning. Rules
+  cannot be inferred beyond source text.
+- **`extraction_quality` replaces `confidence`.** The per-rule trust
+  signal is now `extraction_quality` (EXPLICIT / DISCRETIONARY /
+  CONFLICTING) — a Phase-1 authoring concept, separate from the
+  runtime's categorical verdict (Axiom 2 prohibits confidence scores
+  in the audit trail).
+
+## Installation
+
+```bash
+pip install -e tools/kg-extractor
+```
+
+## Quick start
+
+```bash
+# Extract a v3 KG from a regulation:
+kg-extractor extract --file regulation.txt --sector energy \
+    --jurisdiction us.federal \
+    --output kg-extracted-v3.json
+
+# Validate the emitted KG:
+kg-validator validate --input kg-extracted-v3.json
+
+# Batch:
+kg-extractor batch --input-dir ./regulations --output-dir ./kgs \
+    --sector energy --jurisdiction us.federal
+
+# Debugging — dump the raw flat-rule list before wrapping:
+kg-extractor extract --file regulation.txt --emit-raw --output flat.json
+```
+
+## Architecture
+
+```
+kg_extractor/
+├── __init__.py     public API: Extractor, ExtractionResult, ExtractionQuality, emit_v3_kg
+├── extractor.py    chunk → LLM → flat ComplianceRule list
+├── models.py       ComplianceRule, ExtractionResult, ExtractionQuality enum
+├── backends.py     Ollama / Anthropic / OpenAI backends (compilation phase only)
+├── v3_emitter.py   wraps ExtractionResult into a v3 KG dict (spec §3 shape)
+├── utils.py        chunking, document parsing, rule validation
+└── cli.py          click-based CLI (`extract`, `batch`, `filter`, `examples`, `version`)
+```
+
+## Notes on `extraction_quality`
+
+- **EXPLICIT** — the LLM produced a verbatim source quote in
+  `reasoning`. The rule is safe to ship without further review.
+- **DISCRETIONARY** — the LLM could not anchor the rule in a verbatim
+  source quote. A human reviewer must approve before shipping.
+- **CONFLICTING** — the LLM detected internal contradiction in the
+  source. Surface for legal review.
+
+This is **not** a confidence score in the probabilistic sense. The
+runtime never reads it. It exists solely to drive the Phase-1
+human-review queue.

ski_kg_extractor-3.0.3/pyproject.toml

@@ -0,0 +1,49 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "ski-kg-extractor"
+version = "3.0.3"
+description = "Extract compliance rules from regulatory documents using LLM-assisted parsing. Phase 1 only."
+readme = "README.md"
+requires-python = ">=3.10"
+license = "Apache-2.0"
+authors = [{ name = "KpiFinity", email = "hello@kpifinity.com" }]
+keywords = ["ski-framework", "compliance", "knowledge-graph", "extraction"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Legal Industry",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Software Development :: Libraries :: Python Modules",
+]
+dependencies = [
+    "pydantic>=2.13.4,<3",
+    "python-dotenv>=1.2.2,<2",
+    "click>=8.1.7,<9",
+    "pyyaml>=6.0.1,<7",
+    "httpx>=0.27,<0.29",
+    "pypdf>=6.12.2,<7",
+    "python-docx>=1.1.0,<2",
+    "anthropic>=0.21.3,<1",
+    "openai>=1.16.2,<2",
+]
+
+[project.urls]
+Homepage = "https://github.com/kpifinity/ski-framework"
+Documentation = "https://kpifinity.github.io/ski-framework/"
+Issues = "https://github.com/kpifinity/ski-framework/issues"
+
+[project.scripts]
+kg-extractor = "kg_extractor.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+kg_extractor = ["py.typed"]

ski_kg_extractor-3.0.3/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

ski_kg_extractor-3.0.3/src/kg_extractor/__init__.py

@@ -0,0 +1,16 @@
+"""kg-extractor — extract compliance rules from regulatory text and emit v3 KGs."""
+
+from .extractor import Extractor
+from .models import ComplianceRule, ExtractionQuality, ExtractionResult
+from .v3_emitter import emit_v3_kg
+
+__version__ = "3.0.3"
+__author__ = "KpiFinity"
+
+__all__ = [
+    "ComplianceRule",
+    "ExtractionQuality",
+    "ExtractionResult",
+    "Extractor",
+    "emit_v3_kg",
+]

ski_kg_extractor-3.0.3/src/kg_extractor/backends.py

@@ -0,0 +1,195 @@
+"""LLM backend abstraction for kg-extractor.
+
+Phase 1 (compilation) is permitted to use cloud LLMs because the output
+is signed and validated before the boundary crossing. The extractor
+defaults to a local backend, but `anthropic` and `openai` are also
+available for compilation-phase use.
+
+NONE of these backends are conformant when used at the SKI Model runtime.
+The extractor refuses to talk to runtime endpoints.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import os
+from dataclasses import dataclass
+from typing import Optional, Protocol
+
+
+@dataclass
+class ExtractionCallResult:
+    text: str
+    backend: str
+    model: str
+    seed: Optional[int]
+    temperature: float
+    prompt_sha256: str
+
+
+class ExtractorBackend(Protocol):
+    name: str
+    model: str
+
+    def extract_json(self, prompt: str, max_tokens: int) -> ExtractionCallResult: ...
+
+
+def _hash_prompt(prompt: str) -> str:
+    return hashlib.sha256(prompt.encode("utf-8")).hexdigest()
+
+
+# ---------------------------------------------------------------------------
+# Local Ollama (default, preferred for reproducibility)
+# ---------------------------------------------------------------------------
+
+
+class OllamaBackend:
+    name = "ollama"
+
+    def __init__(self, *, model: str, base_url: str, seed: int = 42, temperature: float = 0.0):
+        self.model = model
+        self._base_url = base_url.rstrip("/")
+        self._seed = seed
+        self._temperature = temperature
+
+    def extract_json(self, prompt: str, max_tokens: int) -> ExtractionCallResult:
+        import httpx
+
+        payload = {
+            "model": self.model,
+            "prompt": prompt,
+            "stream": False,
+            "format": "json",
+            "options": {
+                "temperature": self._temperature,
+                "seed": self._seed,
+                "num_predict": max_tokens,
+                "top_p": 1.0,
+                "top_k": 1,
+            },
+        }
+        resp = httpx.post(f"{self._base_url}/api/generate", json=payload, timeout=120.0)
+        resp.raise_for_status()
+        return ExtractionCallResult(
+            text=resp.json().get("response", ""),
+            backend=self.name,
+            model=self.model,
+            seed=self._seed,
+            temperature=self._temperature,
+            prompt_sha256=_hash_prompt(prompt),
+        )
+
+
+# ---------------------------------------------------------------------------
+# Anthropic (compilation-phase only)
+# ---------------------------------------------------------------------------
+
+
+class AnthropicBackend:
+    name = "anthropic"
+
+    def __init__(self, *, model: str, api_key: str, temperature: float = 0.0, seed: Optional[int] = None):
+        try:
+            import anthropic  # type: ignore
+        except ImportError as exc:
+            raise RuntimeError(
+                "anthropic backend requires `pip install anthropic`. "
+                "Or — preferably — use the ollama backend."
+            ) from exc
+        self._client = anthropic.Anthropic(api_key=api_key)
+        self.model = model
+        self._temperature = temperature
+        self._seed = seed  # Anthropic API does not currently honour `seed`; recorded for audit only.
+
+    def extract_json(self, prompt: str, max_tokens: int) -> ExtractionCallResult:
+        msg = self._client.messages.create(
+            model=self.model,
+            max_tokens=max_tokens,
+            temperature=self._temperature,
+            messages=[{"role": "user", "content": prompt}],
+        )
+        text = msg.content[0].text if msg.content else ""
+        return ExtractionCallResult(
+            text=text,
+            backend=self.name,
+            model=self.model,
+            seed=self._seed,
+            temperature=self._temperature,
+            prompt_sha256=_hash_prompt(prompt),
+        )
+
+
+# ---------------------------------------------------------------------------
+# OpenAI (compilation-phase only)
+# ---------------------------------------------------------------------------
+
+
+class OpenAIBackend:
+    name = "openai"
+
+    def __init__(self, *, model: str, api_key: str, temperature: float = 0.0, seed: Optional[int] = 42):
+        try:
+            from openai import OpenAI  # type: ignore
+        except ImportError as exc:
+            raise RuntimeError(
+                "openai backend requires `pip install openai`. Or — preferably — use the ollama backend."
+            ) from exc
+        self._client = OpenAI(api_key=api_key)
+        self.model = model
+        self._temperature = temperature
+        self._seed = seed
+
+    def extract_json(self, prompt: str, max_tokens: int) -> ExtractionCallResult:
+        resp = self._client.chat.completions.create(
+            model=self.model,
+            max_tokens=max_tokens,
+            temperature=self._temperature,
+            seed=self._seed,
+            response_format={"type": "json_object"},
+            messages=[{"role": "user", "content": prompt}],
+        )
+        text = resp.choices[0].message.content or ""
+        return ExtractionCallResult(
+            text=text,
+            backend=self.name,
+            model=self.model,
+            seed=self._seed,
+            temperature=self._temperature,
+            prompt_sha256=_hash_prompt(prompt),
+        )
+
+
+# ---------------------------------------------------------------------------
+# Factory
+# ---------------------------------------------------------------------------
+
+
+def build_backend(name: Optional[str] = None) -> ExtractorBackend:
+    name = (name or os.getenv("KG_EXTRACTOR_LLM_BACKEND", "ollama")).lower()
+    if name == "ollama":
+        return OllamaBackend(
+            model=os.getenv("KG_EXTRACTOR_MODEL", "qwen2.5:7b-instruct"),
+            base_url=os.getenv("OLLAMA_BASE_URL", "http://localhost:11434"),
+            seed=int(os.getenv("KG_EXTRACTOR_SEED", "42")),
+            temperature=float(os.getenv("KG_EXTRACTOR_TEMPERATURE", "0")),
+        )
+    if name == "anthropic":
+        api_key = os.getenv("ANTHROPIC_API_KEY", "")
+        if not api_key:
+            raise RuntimeError("ANTHROPIC_API_KEY required for the anthropic backend.")
+        return AnthropicBackend(
+            model=os.getenv("KG_EXTRACTOR_MODEL", "claude-haiku-4-5-20251001"),
+            api_key=api_key,
+            temperature=float(os.getenv("KG_EXTRACTOR_TEMPERATURE", "0")),
+        )
+    if name == "openai":
+        api_key = os.getenv("OPENAI_API_KEY", "")
+        if not api_key:
+            raise RuntimeError("OPENAI_API_KEY required for the openai backend.")
+        return OpenAIBackend(
+            model=os.getenv("KG_EXTRACTOR_MODEL", "gpt-4o-mini"),
+            api_key=api_key,
+            temperature=float(os.getenv("KG_EXTRACTOR_TEMPERATURE", "0")),
+            seed=int(os.getenv("KG_EXTRACTOR_SEED", "42")),
+        )
+    raise RuntimeError(f"Unknown KG_EXTRACTOR_LLM_BACKEND: {name!r}")