plm-knowledge 1.0.0__tar.gz

plm_knowledge-1.0.0/PKG-INFO

@@ -0,0 +1,93 @@
+Metadata-Version: 2.4
+Name: plm-knowledge
+Version: 1.0.0
+Summary: TracePulse PLM Knowledge product-line — BM25 retrieval, RAG, business rules, lifecycle FSM, knowledge-pack manifests (Wave 6.5 Conv B / PRD §3 #6). Full description in README.
+Author: TracePulse
+License: Proprietary
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+Requires-Dist: plm-shared
+Requires-Dist: pydantic>=2.5.0
+Requires-Dist: PyYAML>=6.0
+Requires-Dist: rank-bm25>=0.2.2
+Requires-Dist: openai>=1.16.0
+Requires-Dist: httpx>=0.27.0
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+Requires-Dist: pytest-asyncio; extra == "test"
+
+# plm-knowledge
+
+TracePulse PLM Knowledge product-line. Wave 6.5 Conv B sibling-package
+carve per PRD §3 #6.
+
+## Scope (PRD §3 #6 verbatim — 4 sub-layers)
+
+| Sub-layer | Source | Carve disposition |
+|-----------|--------|-------------------|
+| **Foundation** | `01_Dictionnaries/**` (146 .md files at repo root) + `plm_taxonomy` + `plm_evaluation_grids` | **Path stays at repo root through Wave 6.5** per Decision #171 = (c) defer relocation to Wave 7. `plm_knowledge.foundation_loader` is a thin reader that resolves the path through `config.dictionnaries_path` (already configurable); no `git mv` of the dictionary tree. |
+| **Rule & Policy** | `02_App/backend/services/issue_classification.py` + `02_App/backend/agents/business_validator.py` | Lifted via `git mv` to `02_App/plm-knowledge/plm_knowledge/{issue_classification,business_validator}.py`. Re-export shims at every old monolith path. |
+| **Retrieval** | `02_App/backend/services/rag_service.py` + `02_App/backend/skills/knowledge_pack_loader.py` + `02_App/backend/skills/knowledge_search_engine.py` (BM25) | Lifted via `git mv` to `02_App/plm-knowledge/plm_knowledge/{rag_service,knowledge_pack_loader,knowledge_search_engine}.py`. Re-export shims at every old monolith path. |
+| **Operational** | V1 placeholder per D-LOCKED-15 + Decision #172 = (a) keep in-process state V1 | `plm_knowledge.operational` exposes a stable but inert public surface so downstream callers can target the future operational telemetry hooks without an interface bump. |
+
+## Public surface (frozen V1 — `KNOWLEDGE_PACKAGE_VERSION = "v1.0.0-wave6.5"`)
+
+* The 5 lifted modules at top level of the package (`rag_service`,
+  `issue_classification`, `business_validator`, `knowledge_pack_loader`,
+  `knowledge_search_engine`).
+* `plm_knowledge.foundation_loader` — thin reader for the Foundation
+  layer (path resolution + .md enumeration).
+* `plm_knowledge.operational` — V1 placeholder (per D-LOCKED-15 +
+  Decision #172).
+* `plm_knowledge.lifecycle` — 4-state FSM aligned verbatim with PRD
+  §8.11.3 (mirror of Wave 6.5 Conv A `plm-skill-packages` per Decision
+  #167).
+* `plm_knowledge.manifests` — YAML manifest loader (mirror of Decision
+  #168 precedent — YAML + filesystem; no SQL, no admin UI in V1).
+* `plm_knowledge.contracts` — wire-contract Pydantic v2 frozen models
+  for the public surface (Decision #160 interface-freeze pattern).
+* `plm_knowledge.registry` — manifest+lifecycle registry exposed to
+  the kernel (capability registry callers).
+
+## Lifecycle FSM (PRD §8.11.3 verbatim — Decision #167 mirrored)
+
+```
+draft ──► released ──► deprecated ──► retired
+                                          │
+                                          └──► (terminal)
+```
+
+States are **strictly forward-flowing**; the FSM rejects skip
+transitions, out-of-terminal moves, and unknown states. Mirrors the
+4-state contract in `plm_shared.capability_registry`.
+
+## Transitional dependency note (Wave 6.5 → Wave 7 Extraction)
+
+Lifted code retains its existing `from services.X` / `from agents.base_agent` /
+`from config import settings` imports. These resolve at runtime in the
+backend process via PYTHONPATH (also via `pip install -e ../plm-knowledge`
+declared in `02_App/backend/requirements.txt`); sibling tests configure
+`tests/conftest.py` to extend `sys.path`. Wave 7 Extraction physically
+separates the repos and at that point inbound deps refactor to
+ports/Protocols. Pattern matches Wave 6.5 Conv A `plm-skill-packages`
+precedent verbatim.
+
+## Re-export shims
+
+The following monolith paths are thin re-export shims pointing to the
+canonical sibling-package modules. They keep legacy callers working
+through Wave 7 Extraction (drop after 90-day soak per US-DC.1).
+
+* `02_App/backend/services/rag_service.py`
+* `02_App/backend/services/issue_classification.py`
+* `02_App/backend/agents/business_validator.py`
+* `02_App/backend/skills/knowledge_pack_loader.py`
+* `02_App/backend/skills/knowledge_search_engine.py`
+
+## References
+
+* PRD §3 #6 — `plm-knowledge` product-line definition
+* PRD §8.11.3 — 4-state capability lifecycle FSM
+* WAVE-6-READINESS-SWEEP.md §11.2 — Wave 6.5 conv-by-conv
+* W6.5-CONV-A-FOLLOWUPS.md — Wave 6.5 Conv B warm-up
+* Decisions #170-#173 — Wave 6.5 Conv B Q-W6.5B-* answers

plm_knowledge-1.0.0/README.md

@@ -0,0 +1,75 @@
+# plm-knowledge
+
+TracePulse PLM Knowledge product-line. Wave 6.5 Conv B sibling-package
+carve per PRD §3 #6.
+
+## Scope (PRD §3 #6 verbatim — 4 sub-layers)
+
+| Sub-layer | Source | Carve disposition |
+|-----------|--------|-------------------|
+| **Foundation** | `01_Dictionnaries/**` (146 .md files at repo root) + `plm_taxonomy` + `plm_evaluation_grids` | **Path stays at repo root through Wave 6.5** per Decision #171 = (c) defer relocation to Wave 7. `plm_knowledge.foundation_loader` is a thin reader that resolves the path through `config.dictionnaries_path` (already configurable); no `git mv` of the dictionary tree. |
+| **Rule & Policy** | `02_App/backend/services/issue_classification.py` + `02_App/backend/agents/business_validator.py` | Lifted via `git mv` to `02_App/plm-knowledge/plm_knowledge/{issue_classification,business_validator}.py`. Re-export shims at every old monolith path. |
+| **Retrieval** | `02_App/backend/services/rag_service.py` + `02_App/backend/skills/knowledge_pack_loader.py` + `02_App/backend/skills/knowledge_search_engine.py` (BM25) | Lifted via `git mv` to `02_App/plm-knowledge/plm_knowledge/{rag_service,knowledge_pack_loader,knowledge_search_engine}.py`. Re-export shims at every old monolith path. |
+| **Operational** | V1 placeholder per D-LOCKED-15 + Decision #172 = (a) keep in-process state V1 | `plm_knowledge.operational` exposes a stable but inert public surface so downstream callers can target the future operational telemetry hooks without an interface bump. |
+
+## Public surface (frozen V1 — `KNOWLEDGE_PACKAGE_VERSION = "v1.0.0-wave6.5"`)
+
+* The 5 lifted modules at top level of the package (`rag_service`,
+  `issue_classification`, `business_validator`, `knowledge_pack_loader`,
+  `knowledge_search_engine`).
+* `plm_knowledge.foundation_loader` — thin reader for the Foundation
+  layer (path resolution + .md enumeration).
+* `plm_knowledge.operational` — V1 placeholder (per D-LOCKED-15 +
+  Decision #172).
+* `plm_knowledge.lifecycle` — 4-state FSM aligned verbatim with PRD
+  §8.11.3 (mirror of Wave 6.5 Conv A `plm-skill-packages` per Decision
+  #167).
+* `plm_knowledge.manifests` — YAML manifest loader (mirror of Decision
+  #168 precedent — YAML + filesystem; no SQL, no admin UI in V1).
+* `plm_knowledge.contracts` — wire-contract Pydantic v2 frozen models
+  for the public surface (Decision #160 interface-freeze pattern).
+* `plm_knowledge.registry` — manifest+lifecycle registry exposed to
+  the kernel (capability registry callers).
+
+## Lifecycle FSM (PRD §8.11.3 verbatim — Decision #167 mirrored)
+
+```
+draft ──► released ──► deprecated ──► retired
+                                          │
+                                          └──► (terminal)
+```
+
+States are **strictly forward-flowing**; the FSM rejects skip
+transitions, out-of-terminal moves, and unknown states. Mirrors the
+4-state contract in `plm_shared.capability_registry`.
+
+## Transitional dependency note (Wave 6.5 → Wave 7 Extraction)
+
+Lifted code retains its existing `from services.X` / `from agents.base_agent` /
+`from config import settings` imports. These resolve at runtime in the
+backend process via PYTHONPATH (also via `pip install -e ../plm-knowledge`
+declared in `02_App/backend/requirements.txt`); sibling tests configure
+`tests/conftest.py` to extend `sys.path`. Wave 7 Extraction physically
+separates the repos and at that point inbound deps refactor to
+ports/Protocols. Pattern matches Wave 6.5 Conv A `plm-skill-packages`
+precedent verbatim.
+
+## Re-export shims
+
+The following monolith paths are thin re-export shims pointing to the
+canonical sibling-package modules. They keep legacy callers working
+through Wave 7 Extraction (drop after 90-day soak per US-DC.1).
+
+* `02_App/backend/services/rag_service.py`
+* `02_App/backend/services/issue_classification.py`
+* `02_App/backend/agents/business_validator.py`
+* `02_App/backend/skills/knowledge_pack_loader.py`
+* `02_App/backend/skills/knowledge_search_engine.py`
+
+## References
+
+* PRD §3 #6 — `plm-knowledge` product-line definition
+* PRD §8.11.3 — 4-state capability lifecycle FSM
+* WAVE-6-READINESS-SWEEP.md §11.2 — Wave 6.5 conv-by-conv
+* W6.5-CONV-A-FOLLOWUPS.md — Wave 6.5 Conv B warm-up
+* Decisions #170-#173 — Wave 6.5 Conv B Q-W6.5B-* answers

plm_knowledge-1.0.0/plm_knowledge/__init__.py

@@ -0,0 +1,92 @@
+"""TracePulse PLM Knowledge — Wave 6.5 Conv B.
+
+PRD §3 #6 product-line carve. Hosts the 4-sub-layer knowledge stack:
+
+  * **Foundation** — `01_Dictionnaries/**` (146 .md files at repo root)
+    + `plm_taxonomy` + `plm_evaluation_grids`. Path stays at repo root
+    through Wave 6.5 per Decision #171 = (c) defer relocation to Wave 7;
+    the ``foundation_loader`` module is a thin reader.
+  * **Rule & Policy** — ``issue_classification`` + ``business_validator``
+    (lifted via `git mv` from the monolith).
+  * **Retrieval** — ``rag_service`` + ``knowledge_pack_loader`` +
+    ``knowledge_search_engine`` (BM25; lifted via `git mv` from the
+    monolith).
+  * **Operational** — V1 placeholder per D-LOCKED-15 + Decision #172.
+
+Public surface:
+  * The 5 lifted modules at top level (``rag_service``,
+    ``issue_classification``, ``business_validator``,
+    ``knowledge_pack_loader``, ``knowledge_search_engine``).
+  * ``foundation_loader`` — Foundation layer thin reader.
+  * ``operational`` — Operational layer V1 placeholder.
+  * ``lifecycle`` — 4-state FSM aligned verbatim with PRD §8.11.3
+    (Decision #167 mirrored).
+  * ``manifests`` — YAML manifest loader (Decision #168 precedent =
+    YAML + filesystem).
+  * ``contracts`` — wire-contract Pydantic v2 frozen models for the
+    public surface (Decision #160 interface-freeze pattern).
+  * ``registry`` — manifest+lifecycle registry exposed to the kernel
+    (capability registry callers).
+
+Re-export shims at every monolith path keep legacy callers working
+through Wave 7 Extraction (Wave 1 W1.4 RunTaskTracker template + Wave
+6.5 Conv A precedent).
+"""
+
+from plm_knowledge.contracts import (
+    KNOWLEDGE_PACKAGE_VERSION,
+    KnowledgeKind,
+    KnowledgeLifecycleState,
+    KnowledgePackManifest,
+)
+from plm_knowledge.lifecycle import (
+    LIFECYCLE_STATES,
+    LIFECYCLE_TRANSITIONS,
+    InvalidLifecycleTransition,
+    can_transition,
+    is_terminal,
+    next_state,
+)
+from plm_knowledge.manifests import (
+    MANIFESTS_DIR,
+    ManifestLoadError,
+    load_all,
+    load_manifest,
+)
+from plm_knowledge.registry import (
+    KnowledgePackRegistry,
+    get_knowledge_registry,
+)
+from plm_knowledge.foundation_loader import (
+    list_foundation_documents,
+    resolve_foundation_root,
+)
+from plm_knowledge.operational import (
+    OPERATIONAL_LAYER_VERSION,
+    OperationalAck,
+    record_operational_event,
+)
+
+__all__ = [
+    "KNOWLEDGE_PACKAGE_VERSION",
+    "KnowledgeKind",
+    "KnowledgeLifecycleState",
+    "KnowledgePackManifest",
+    "LIFECYCLE_STATES",
+    "LIFECYCLE_TRANSITIONS",
+    "InvalidLifecycleTransition",
+    "can_transition",
+    "is_terminal",
+    "next_state",
+    "MANIFESTS_DIR",
+    "ManifestLoadError",
+    "load_all",
+    "load_manifest",
+    "KnowledgePackRegistry",
+    "get_knowledge_registry",
+    "list_foundation_documents",
+    "resolve_foundation_root",
+    "OPERATIONAL_LAYER_VERSION",
+    "OperationalAck",
+    "record_operational_event",
+]

plm_knowledge-1.0.0/plm_knowledge/business_validator.py

@@ -0,0 +1,279 @@
+"""
+Business Validator Agent
+Validates generated BPMN processes against original source data.
+"""
+
+import json
+from typing import Dict, Any, Optional
+
+from plm_shared.agents.base_agent import BaseAgent
+from plm_shared.protocols.services.llm import llm_service
+
+
+# Define the response schema for validation as a plain dict
+VALIDATION_SCHEMA = {
+    "type": "object",
+    "properties": {
+        "isValid": {
+            "type": "boolean",
+            "description": "True ONLY if the process perfectly matches ALL source data patterns and is syntactically perfect."
+        },
+        "score": {
+            "type": "number",
+            "description": "A confidence score from 0 to 100 based on data coverage."
+        },
+        "keywords": {
+            "type": "array",
+            "items": {"type": "string"},
+            "description": "3 to 10 short thematic keywords (domains, roles, themes) for the process. Lowercase. No duplicates."
+        },
+        "criticalIssues": {
+            "type": "array",
+            "items": {"type": "string"},
+            "description": "List of logical errors, missing activities from CSV, BPMN violations, or BPMN elements missing a precise expert description in documentation/persona/system."
+        },
+        "improvementSuggestions": {
+            "type": "string",
+            "description": "Technical instructions for the analyst to fix the issues, including completing expert-level documentation for each BPMN element."
+        }
+    },
+    "required": ["isValid", "score", "criticalIssues", "improvementSuggestions", "keywords"]
+}
+
+
+SYSTEM_INSTRUCTION = """You are a Senior Process Auditor & BPMN Validator. 
+Your goal is to find ANY discrepancy between the raw source data (CSV/Text) and the AI-generated model.
+
+STRICT CRITERIA:
+1. EXHAUSTIVE COVERAGE: Every unique activity found in the CSV 'Activity' or 'Event' column MUST be represented as a Task in the BPMN.
+2. SEQUENCE INTEGRITY: Timestamps in CSV must justify the flow. If 'Invoice' always follows 'Receipt', the BPMN must show this.
+3. BPMN 2.0 SYNTAX: 
+   - Must have exactly ONE Start Event.
+   - Must have at least ONE End Event.
+   - All nodes must be connected (no islands).
+   - Gateways must have clear labels for their outgoing flows.
+4. METRICS REALISM: Execution counts in 'miningData' must sum up logically (e.g., sum of branches = entry count).
+5. BUSINESS METADATA COMPLETENESS: Every step in the BPMN diagram (tasks, start/end events, gateways) MUST have a precise, expert-level description in <bpmn:documentation>. The description must be specific to the element (what it represents in the process, business meaning), not generic. For each <bpmn:task>: require <bpmn:documentation>; tasks that involve a responsible role or an IT system must have <bpmn:extensionElements> with <persona> and/or <system>. For start event: documentation should state what triggers the process. For end event(s): what outcome is reached. For gateways: what decision or split is represented. If any element lacks a precise, expert description or missing persona/system where relevant, add a critical issue and in improvementSuggestions ask to complete these properties with expert-level wording.
+
+BE PEDANTIC. If a single task is missing or a flow is illogical, set isValid to false.
+
+KEYWORDS REQUIREMENT:
+- Provide 3 to 10 short thematic keywords for the process.
+- Derive them from the source data and the generated BPMN (domains, roles, themes).
+- Use lowercase, avoid duplicates.
+"""
+
+
+class BusinessValidatorAgent(BaseAgent):
+    """
+    Business Validator Agent
+    
+    Validates generated BPMN processes against original source data.
+    Acts as a Senior Process Auditor checking:
+    
+    1. EXHAUSTIVE COVERAGE: Every activity in source data must be in BPMN
+    2. SEQUENCE INTEGRITY: Timestamps must justify the flow
+    3. BPMN 2.0 SYNTAX: Valid structure, connected nodes
+    4. METRICS REALISM: Execution counts add up logically
+    5. BUSINESS METADATA: Precise descriptions and persona/system assignments
+    
+    Input:
+        - original_content: Original CSV or text source data
+        - generated_process: Generated process dictionary to validate
+    
+    Output:
+        - isValid: Boolean validation result
+        - score: Confidence score (0-100)
+        - keywords: Extracted thematic keywords
+        - criticalIssues: List of found issues
+        - improvementSuggestions: Instructions for fixing issues
+    """
+
+    def __init__(self):
+        super().__init__(
+            name="business_validator",
+            description="Validates generated BPMN processes against original source data"
+        )
+
+    def get_output_schema(self) -> Dict[str, Any]:
+        """Define business validator output schema."""
+        return {
+            "type": "object",
+            "properties": {
+                "isValid": {"type": "boolean"},
+                "score": {"type": "number"},
+                "keywords": {"type": "array", "items": {"type": "string"}},
+                "criticalIssues": {"type": "array", "items": {"type": "string"}},
+                "improvementSuggestions": {"type": "string"},
+            },
+            "required": ["isValid", "score", "criticalIssues", "improvementSuggestions", "keywords"],
+        }
+
+    def get_required_fields(self) -> list:
+        """Return required input fields."""
+        return ["original_content", "generated_process"]
+
+    def validate_input(self, input_data: Dict[str, Any]) -> bool:
+        """Validate business validator input."""
+        if "original_content" not in input_data:
+            raise ValueError("Missing required field: original_content")
+        
+        if "generated_process" not in input_data:
+            raise ValueError("Missing required field: generated_process")
+        
+        original_content = input_data.get("original_content")
+        if not original_content or not str(original_content).strip():
+            raise ValueError("original_content must be non-empty string")
+        
+        generated_process = input_data.get("generated_process")
+        if not generated_process or not isinstance(generated_process, (dict, str)):
+            raise ValueError("generated_process must be dict or string")
+        
+        return True
+
+    async def execute(self, input_data: Dict[str, Any]) -> Dict[str, Any]:
+        """Execute business validation workflow."""
+        self.validate_input(input_data)
+        
+        original_content = input_data.get("original_content")
+        generated_process = input_data.get("generated_process")
+        
+        return await self._validate_process(original_content, generated_process)
+
+    async def _validate_process(
+        self,
+        original_content: str,
+        generated_process: Dict[str, Any]
+    ) -> Dict[str, Any]:
+        """
+        Validate a generated BPMN process against original source data using ADK multi-LLM.
+        
+        Args:
+            original_content: Original CSV or text content
+            generated_process: Generated process dictionary to validate
+            
+        Returns:
+            Dictionary containing isValid, score, criticalIssues, improvementSuggestions, keywords
+            
+        Raises:
+            Exception: If validation fails
+        """
+        try:
+            print(f"[SCAN] Validating with provider: {llm_service.provider}")
+            
+            # Phase 1: Preprocess large CSV to avoid token overflow
+            source_content = await self._preprocess_source_data(original_content)
+            
+            # Phase 2: Build validation prompt
+            prompt = self._build_validation_prompt(source_content, generated_process)
+            
+            # Phase 3: Call ADK service
+            result = await llm_service.generate_structured(
+                system_prompt=SYSTEM_INSTRUCTION,
+                user_prompt=prompt,
+                output_schema=VALIDATION_SCHEMA,
+                temperature=0.3
+            )
+            
+            print(f"[OK] Validation complete - Score: {result.get('score', 0)}%")
+            return result
+            
+        except Exception as error:
+            print(f"[ERR] Validator Agent Error: {error}")
+            import traceback
+            traceback.print_exc()
+            # Return permissive validation on error
+            return self._get_default_validation()
+
+    async def _preprocess_source_data(self, original_content: str) -> str:
+        """
+        Preprocess large CSV to avoid token overflow.
+        
+        Args:
+            original_content: Original source data (CSV or text)
+            
+        Returns:
+            Preprocessed source data or original if small enough
+        """
+        if len(original_content) > 50000 and "Employee_ID" in original_content:
+            print("[SCAN] Applying CSV preprocessing for validation...")
+            try:
+                from plm_shared.utils.csv_preprocessor import smart_csv_preprocessor
+                return smart_csv_preprocessor(original_content)
+            except ImportError:
+                print("[WARN] CSV preprocessor not available, using original content")
+                return original_content[:50000]
+        
+        return original_content
+
+    def _build_validation_prompt(
+        self,
+        source_content: str,
+        generated_process: Dict[str, Any]
+    ) -> str:
+        """
+        Build validation prompt combining source data and generated process.
+        
+        Args:
+            source_content: Preprocessed source data
+            generated_process: Generated process to validate
+            
+        Returns:
+            Formatted validation prompt
+        """
+        return f"""[SOURCE_DATA_EXTRACT]
+{source_content[:10000]}
+
+[GENERATED_BPMN_AND_METRICS]
+{json.dumps(generated_process) if isinstance(generated_process, dict) else str(generated_process)}
+"""
+
+    def _get_default_validation(self) -> Dict[str, Any]:
+        """
+        Return permissive default validation result on error.
+        
+        Returns:
+            Default validation dictionary
+        """
+        return {
+            "isValid": True,
+            "score": 100,
+            "keywords": [],
+            "criticalIssues": [],
+            "improvementSuggestions": ""
+        }
+
+
+# Singleton instance
+business_validator_agent = BusinessValidatorAgent()
+
+
+# ==================== BACKWARD COMPATIBILITY ====================
+
+async def validate_process(
+    original_content: str,
+    generated_process: Dict[str, Any]
+) -> Dict[str, Any]:
+    """
+    Backward compatibility wrapper for original function.
+    
+    Validate a generated BPMN process against original source data using ADK multi-LLM
+    
+    Args:
+        original_content: Original CSV or text content
+        generated_process: Generated process dictionary to validate
+        
+    Returns:
+        Dictionary containing isValid, score, criticalIssues, improvementSuggestions
+        
+    Raises:
+        Exception: If validation fails
+    
+    Delegates to business_validator_agent singleton.
+    """
+    result = await business_validator_agent.execute({
+        "original_content": original_content,
+        "generated_process": generated_process,
+    })
+    
+    return result