kmds-data-helper 0.1.0__tar.gz

kmds_data_helper-0.1.0/PKG-INFO

@@ -0,0 +1,110 @@
+Metadata-Version: 2.4
+Name: kmds-data-helper
+Version: 0.1.0
+Summary: Repository-grounded KMDS helper that analyzes project artifacts and builds a KMDS knowledge graph
+Author: KMDS Data Helper Team
+License: Proprietary
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Operating System :: OS Independent
+Requires-Python: <3.13,>=3.12
+Description-Content-Type: text/markdown
+Requires-Dist: kmds
+Requires-Dist: pymupdf4llm>=0.0.12
+Requires-Dist: ollama>=0.1.0
+Requires-Dist: pandas>=2.0.0
+Requires-Dist: setuptools<70.0.0
+Requires-Dist: urllib3<2.0.0
+Requires-Dist: rich>=13.0.0
+Requires-Dist: nbformat>=5.10.4
+Requires-Dist: fastapi>=0.136.1
+Requires-Dist: uvicorn>=0.46.0
+Requires-Dist: requests>=2.32.5
+Requires-Dist: pytest>=9.0.3
+Requires-Dist: pypdf>=6.11.0
+Requires-Dist: fg-data-profiling>=4.19.1
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+Requires-Dist: pytest-xdist; extra == "test"
+Requires-Dist: requests; extra == "test"
+
+---
+# KMDS Data Helper: Repo Architect Framework
+
+A modular, multi-persona framework for analyzing data science repositories. Uses local LLMs (via Ollama) to synthesize insights from documentation, data schemas, and Jupyter notebooks.
+
+## 📂 Project Structure
+KMDS-Helper follows a strict modular architecture to separate concerns:
+- `src/kmds_data_helper/`: Core logic modules (Config, Processing, LLM, Engine).
+- `documents/`: Project documentation (.pdf, .txt).
+- `data/`: Physical data assets (CSVs) - isolated from output.
+- `notebooks/`: Experimental code (.ipynb).
+- `output/`: Isolated directory for generated reports.
+
+## 🛠️ Installation & Setup
+1. **Environment**: Ensure you are using the local virtual environment.
+   ```bash
+   source .venv/bin/activate
+   ```
+2. **LLM Engine**: Requires [Ollama](https://ollama.com) running locally with the `qwen2.5-coder:7b` model.
+3. **Dependencies**:
+   ```bash
+   pip install rich ollama dataprofiler pymupdf4llm nbformat pyyaml
+   ```
+
+## ⚙️ Configuration
+The framework is controlled by `kmds_config.yaml` in the root directory. You can toggle persona behaviors (Scientist, Tech Lead, Architect) and pathing without changing Python code.
+
+## 🚀 Usage
+Run the main orchestrator from the project root:
+```bash
+python3 main.py
+```
+
+## 📦 Packaged Usage (v1)
+This first version assumes a fixed repository structure. A user can install the package, run the knowledge-graph aggregator in a cloned repo, and produce a KMDS knowledge graph.
+
+### Required folders in the cloned repo
+- `documents/`
+- `notebooks/`
+- `data_dictionary/`
+- `output/`
+
+### Expected helper output artifacts
+At least one of these files should exist in `output/`:
+- `full_service_report.json`
+- `kmds_summary.json`
+- `kmds_strategic_summary.json`
+
+### Install
+From the project root:
+```bash
+pip install -e .
+```
+
+### Generate knowledge graph from helper outputs
+```bash
+kmds-kb --workspace . --project-file project_knowledge_graph.xml --mode auto
+```
+
+The command validates the required folders, ingests the helper output artifacts, and writes:
+- `project_knowledge_graph.xml`
+
+### Adapter command (direct use)
+You can also run the output adapter directly for a single file:
+```bash
+kmds-analyze --input output/full_service_report.json --project-file project_knowledge_graph.xml --create-project --workflow-name kmds_project_workflow --mode auto
+```
+
+### Backward-compatible template script
+If you are using the template script path, this remains supported:
+```bash
+python kb_aggregator.py --workspace . --project-file project_knowledge_graph.xml --mode auto
+```
+
+### Common failure messages
+- Missing folder(s): one or more required directories are absent.
+- No helper output files found: none of the expected JSON artifacts are present in `output/`.
+- Project file already exists in create mode: rerun with update mode or choose a new target path.
+
+---

kmds_data_helper-0.1.0/README.md

@@ -0,0 +1,80 @@
+---
+# KMDS Data Helper: Repo Architect Framework
+
+A modular, multi-persona framework for analyzing data science repositories. Uses local LLMs (via Ollama) to synthesize insights from documentation, data schemas, and Jupyter notebooks.
+
+## 📂 Project Structure
+KMDS-Helper follows a strict modular architecture to separate concerns:
+- `src/kmds_data_helper/`: Core logic modules (Config, Processing, LLM, Engine).
+- `documents/`: Project documentation (.pdf, .txt).
+- `data/`: Physical data assets (CSVs) - isolated from output.
+- `notebooks/`: Experimental code (.ipynb).
+- `output/`: Isolated directory for generated reports.
+
+## 🛠️ Installation & Setup
+1. **Environment**: Ensure you are using the local virtual environment.
+   ```bash
+   source .venv/bin/activate
+   ```
+2. **LLM Engine**: Requires [Ollama](https://ollama.com) running locally with the `qwen2.5-coder:7b` model.
+3. **Dependencies**:
+   ```bash
+   pip install rich ollama dataprofiler pymupdf4llm nbformat pyyaml
+   ```
+
+## ⚙️ Configuration
+The framework is controlled by `kmds_config.yaml` in the root directory. You can toggle persona behaviors (Scientist, Tech Lead, Architect) and pathing without changing Python code.
+
+## 🚀 Usage
+Run the main orchestrator from the project root:
+```bash
+python3 main.py
+```
+
+## 📦 Packaged Usage (v1)
+This first version assumes a fixed repository structure. A user can install the package, run the knowledge-graph aggregator in a cloned repo, and produce a KMDS knowledge graph.
+
+### Required folders in the cloned repo
+- `documents/`
+- `notebooks/`
+- `data_dictionary/`
+- `output/`
+
+### Expected helper output artifacts
+At least one of these files should exist in `output/`:
+- `full_service_report.json`
+- `kmds_summary.json`
+- `kmds_strategic_summary.json`
+
+### Install
+From the project root:
+```bash
+pip install -e .
+```
+
+### Generate knowledge graph from helper outputs
+```bash
+kmds-kb --workspace . --project-file project_knowledge_graph.xml --mode auto
+```
+
+The command validates the required folders, ingests the helper output artifacts, and writes:
+- `project_knowledge_graph.xml`
+
+### Adapter command (direct use)
+You can also run the output adapter directly for a single file:
+```bash
+kmds-analyze --input output/full_service_report.json --project-file project_knowledge_graph.xml --create-project --workflow-name kmds_project_workflow --mode auto
+```
+
+### Backward-compatible template script
+If you are using the template script path, this remains supported:
+```bash
+python kb_aggregator.py --workspace . --project-file project_knowledge_graph.xml --mode auto
+```
+
+### Common failure messages
+- Missing folder(s): one or more required directories are absent.
+- No helper output files found: none of the expected JSON artifacts are present in `output/`.
+- Project file already exists in create mode: rerun with update mode or choose a new target path.
+
+---

kmds_data_helper-0.1.0/pyproject.toml

@@ -0,0 +1,72 @@
+[project]
+name = "kmds-data-helper"
+version = "0.1.0"
+description = "Repository-grounded KMDS helper that analyzes project artifacts and builds a KMDS knowledge graph"
+readme = "README.md"
+license = { text = "Proprietary" }
+authors = [
+    { name = "KMDS Data Helper Team" }
+]
+requires-python = ">=3.12,<3.13"
+dependencies = [
+    "kmds",
+    "pymupdf4llm>=0.0.12",
+    "ollama>=0.1.0",
+    "pandas>=2.0.0",
+    "setuptools<70.0.0",
+    "urllib3<2.0.0",
+    "rich>=13.0.0",
+    "nbformat>=5.10.4",
+    "fastapi>=0.136.1",
+    "uvicorn>=0.46.0",
+    "requests>=2.32.5",
+    "pytest>=9.0.3",
+    "pypdf>=6.11.0",
+    "fg-data-profiling>=4.19.1",
+]
+
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.12",
+    "Operating System :: OS Independent",
+]
+
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project.optional-dependencies]
+test = [
+    "pytest",
+    "pytest-xdist",
+    "requests"
+]
+
+[project.scripts]
+# Maps kmds-check to your verified, production pre-flight entry point
+kmds-check = "kmds_data_helper.kmds_check:main"
+# NEW: Maps kmds-analyze to your unified application CLI execution entry point
+kmds-analyze = "kmds_data_helper.helper_output_adapter:main"
+kmds-kb = "kmds_data_helper.kb_aggregator:main"
+kmds-test = "pytest:main"
+
+[tool.pytest.ini_options]
+# Ensures pytest only runs the new persona tests and stays away from broken legacy files
+testpaths = ["tests/test_personas.py"]
+python_files = "test_*.py"
+
+# -n 3: Runs 3 parallel workers
+# -v: Verbose output
+# --disable-warnings: Hides those annoying urllib3 version mismatch logs
+addopts = "-n 3 -v --disable-warnings"
+
+filterwarnings = [
+    "ignore:urllib3.*match a supported version:requests.RequestsDependencyWarning",
+]
+
+
+[dependency-groups]
+dev = [
+    "pytest-asyncio>=1.3.0",
+    "pytest-xdist>=3.8.0",
+]

kmds_data_helper-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

kmds_data_helper-0.1.0/src/kmds_data_helper/__init__.py

@@ -0,0 +1,12 @@
+from .engine import KMDSEngine
+from .utils import parse_notebook_with_outputs, save_kmds_json
+from .aggregator import PersonaAggregator
+from .llm_client import LLMClient
+
+__all__ = [
+    "KMDSEngine", 
+    "parse_notebook_with_outputs", 
+    "save_kmds_json",
+    "PersonaAggregator",
+    "LLMClient"
+]

kmds_data_helper-0.1.0/src/kmds_data_helper/aggregator.py

@@ -0,0 +1,44 @@
+import json
+from typing import Dict, Any, List
+
+class PersonaAggregator:
+    """
+    Knowledge Dictionary Builder.
+    Consolidates individual persona reports into a single grounded 
+    knowledge base for strategic synthesis.
+    """
+    def __init__(self):
+        # Stores the grounded findings for the full project
+        self.knowledge_dict: Dict[str, Any] = {}
+
+    def add_audit_result(self, result: Dict[str, Any]):
+        """
+        Ingests a result from KMDSEngine and maps it to the knowledge dictionary.
+        Input format: {"notebook": "...", "persona": "...", "analysis": {...}}
+        """
+        persona = result.get("persona")
+        notebook = result.get("notebook")
+        data = result.get("analysis", {})
+
+        # Grounding Safeguard: Only aggregate successful, non-error findings
+        if "error" not in data:
+            if persona not in self.knowledge_dict:
+                self.knowledge_dict[persona] = {}
+            
+            # Index findings by notebook for multi-notebook project tracking
+            self.knowledge_dict[persona][notebook] = data
+
+    def get_grounded_stats(self) -> str:
+        """
+        Formats the current knowledge dictionary into a string for the 
+        Strategic Lead's RUN_STATS input.
+        """
+        if not self.knowledge_dict:
+            return "NO PRIOR AUDIT EVIDENCE FOUND."
+
+        summary = "KMDS GROUNDED EVIDENCE DICTIONARY:\n"
+        for persona, notebooks in self.knowledge_dict.items():
+            for nb_name, findings in notebooks.items():
+                summary += f"[{persona}] Evidence from {nb_name}: {json.dumps(findings)}\n"
+        
+        return summary

kmds_data_helper-0.1.0/src/kmds_data_helper/cli.py

@@ -0,0 +1,12 @@
+"""Backward-compatible CLI shim.
+
+The canonical implementation lives in helper_output_adapter.py.
+"""
+
+from .helper_output_adapter import ingest_helper_output, main
+
+__all__ = ["ingest_helper_output", "main"]
+
+
+if __name__ == "__main__":
+	raise SystemExit(main())

kmds_data_helper-0.1.0/src/kmds_data_helper/config_manager.py

@@ -0,0 +1,43 @@
+import yaml
+from pathlib import Path
+from typing import Dict, Any
+
+class ConfigManager:
+    """
+    The central source of truth for the KMDS Workspace configuration.
+    Resolves explicit layout directory maps from kmds_config.yaml.
+    """
+    def __init__(self, workspace_path: str = "."):
+        self.workspace = Path(workspace_path)
+        self.config_path = self.workspace / "kmds_config.yaml"
+        self.config = self._load_config()
+        
+        # Centralized pathing map: Separates KMDS 'documents' from Sphinx 'docs'
+        self.paths = {
+            "notebooks": self.get_directory_path("notebooks"),
+            "personas": self.get_directory_path("personas"),
+            "documents": self.get_directory_path("documents"), 
+            "data_dictionary": self.get_directory_path("data_dictionary"),
+            "data": self.get_directory_path("data"),
+            "sphinx_docs": self.workspace / "docs", # Isolated Sphinx tree
+            "output": self.workspace / "output"
+        }
+
+    def _load_config(self) -> Dict[str, Any]:
+        if not self.config_path.exists():
+            return {}
+        with open(self.config_path, 'r', encoding='utf-8') as f:
+            return yaml.safe_load(f) or {}
+
+    def get_directory_path(self, key: str) -> Path:
+        """Resolves the configured folder path against the workspace root."""
+        dirs = self.config.get("directories", {})
+        fallback_map = {
+            "notebooks": "notebooks",
+            "personas": "personas",
+            "documents": "documents",
+            "data_dictionary": "data_dictionary",
+            "data": "data"
+        }
+        folder_name = dirs.get(key, fallback_map.get(key, key))
+        return self.workspace / folder_name

kmds_data_helper-0.1.0/src/kmds_data_helper/data_processor.py

@@ -0,0 +1,200 @@
+import pandas as pd
+import csv as native_csv
+from data_profiling import ProfileReport
+import nbformat
+from pathlib import Path
+
+class KMDSDataProcessor:
+    def __init__(self, config_manager):
+        self.cfg = config_manager
+        
+        self.active_features = {
+            "pdf_processing": False,
+            "data_profiling": False,
+            "notebook_analysis": False
+        }
+        self._run_system_checks()
+
+    def _run_system_checks(self):
+        """Internal guardrail to verify if directories contain valid files."""
+        if any(self.cfg.paths["documents"].glob("*.pdf")):
+            self.active_features["pdf_processing"] = True
+            
+        if any(self.cfg.paths["data"].glob("**/*.csv")):
+             self.active_features["data_profiling"] = True
+             
+        if any(self.cfg.paths["notebooks"].glob("*.ipynb")):
+            self.active_features["notebook_analysis"] = True
+
+    def get_ground_truth(self):
+        """
+        Scans KMDS documents and data directories while protecting Sphinx tree structure.
+        """
+        truth = []
+        
+        # 1. Ingest text instructions from your KMDS 'documents' folder safely
+        for txt in self.cfg.paths["documents"].glob("*.txt"):
+            with open(txt, 'r', encoding='utf-8') as f:
+                truth.append({"source": txt.name, "type": "doc", "content": f.read()[:5000]})
+
+        # 2. Ingest tabular data schemas cleanly
+        for csv_path in self.cfg.paths["data"].glob("**/*.csv"):
+            if str(self.cfg.paths["output"].resolve()) in str(csv_path.resolve()):
+                continue
+                
+            try:
+                print(f"🔬 [PROFILER] Reading native file structures for {csv_path.name}...")
+                
+                with open(csv_path, 'r', encoding='utf-8-sig') as f:
+                    sample = f.readline()
+                    delimiter = ';' if ';' in sample else ','
+                    f.seek(0)
+                    reader = native_csv.reader(f, delimiter=delimiter)
+                    columns_found = next(reader)
+                
+                columns_found = [c.strip().strip('"').strip("'") for c in columns_found if c.strip()]
+
+                # Load a slice into pandas to inspect types
+                df = pd.read_csv(
+                    csv_path, 
+                    header=0, 
+                    names=columns_found, 
+                    nrows=20, 
+                    on_bad_lines='skip'
+                )
+                
+                profile = ProfileReport(df, minimal=True, progress_bar=False)
+                description = profile.get_description()
+                
+                type_insights = {}
+                variables_map = description.get("variables", {})
+                for col in columns_found:
+                    col_type = variables_map.get(col, {}).get("type", "unknown")
+                    type_insights[col] = str(col_type)
+
+                truth.append({
+                    "source": csv_path.name,
+                    "type": "physical_schema",
+                    "columns": columns_found,
+                    "data_types": type_insights
+                })
+                print(f"✅ [PROFILER] Identified all {len(columns_found)} schema column parameters.")
+                
+            except Exception as e:
+                print(f"[-] Profile extraction failed for {csv_path.name}: {e}")
+                
+        return truth
+
+    def read_notebook(self, nb_path):
+        with open(nb_path, 'r', encoding='utf-8') as f:
+            nb = nbformat.read(f, as_version=4)
+        return {
+            "markdown": [c.source for c in nb.cells if c.cell_type == 'markdown'],
+            "code": [c.source for c in nb.cells if c.cell_type == 'code']
+        }
+import pandas as pd
+import csv as native_csv
+import nbformat
+from pathlib import Path
+
+class KMDSDataProcessor:
+    def __init__(self, config_manager):
+        self.cfg = config_manager
+        
+        self.active_features = {
+            "pdf_processing": False,
+            "data_profiling": False,
+            "notebook_analysis": False
+        }
+        self._run_system_checks()
+
+    def _run_system_checks(self):
+        """Internal guardrail to verify if directories contain valid files."""
+        if any(self.cfg.paths["documents"].glob("*.pdf")):
+            self.active_features["pdf_processing"] = True
+            
+        if any(self.cfg.paths["data"].glob("**/*.csv")):
+             self.active_features["data_profiling"] = True
+             
+        if any(self.cfg.paths["notebooks"].glob("*.ipynb")):
+            self.active_features["notebook_analysis"] = True
+
+    def get_ground_truth(self):
+        """
+        Scans KMDS documents and data directories, ensuring full schema metrics
+        are written without triggering internal pandas profiling type-errors.
+        """
+        truth = []
+        
+        # 1. Ingest text instructions from your KMDS 'documents' folder
+        for txt in self.cfg.paths["documents"].glob("*.txt"):
+            try:
+                with open(txt, 'r', encoding='utf-8') as f:
+                    truth.append({"source": txt.name, "type": "doc", "content": f.read()[:5000]})
+            except Exception as e:
+                print(f"[-] Failed reading text asset {txt.name}: {e}")
+
+        # 2. Tabular Data Schema Profiling
+        for csv_path in self.cfg.paths["data"].glob("**/*.csv"):
+            if str(self.cfg.paths["output"].resolve()) in str(csv_path.resolve()):
+                continue
+                
+            try:
+                print(f"🔬 [PROFILER] Parsing file structure for {csv_path.name}...")
+                
+                # Step A: Native CSV stream lookahead to extract absolute raw headers
+                with open(csv_path, 'r', encoding='utf-8-sig') as f:
+                    sample = f.readline()
+                    delimiter = ';' if ';' in sample else ','
+                    f.seek(0)
+                    reader = native_csv.reader(f, delimiter=delimiter)
+                    columns_found = next(reader)
+                
+                # Trim quotation tokens and whitespaces cleanly
+                columns_found = [c.strip().strip('"').strip("'") for c in columns_found if c.strip()]
+
+                # Step B: Read Data Sample using standard pandas mapping engine
+                df = pd.read_csv(
+                    csv_path, 
+                    header=0, 
+                    names=columns_found, 
+                    nrows=100, 
+                    low_memory=False,
+                    on_bad_lines='skip'
+                )
+                
+                # Step C: Generate Type Insights explicitly using standard DataFrame analysis
+                # This guarantees full column tracking without crashing on complex data rows
+                type_insights = {}
+                for col in df.columns:
+                    # Cleanly deduce column profiling metric categories natively
+                    if pd.api.types.is_numeric_dtype(df[col]):
+                        type_insights[col] = "Numeric"
+                    elif pd.api.types.is_bool_dtype(df[col]):
+                        type_insights[col] = "Boolean"
+                    elif pd.api.types.is_datetime64_any_dtype(df[col]):
+                        type_insights[col] = "DateTime"
+                    else:
+                        type_insights[col] = "Categorical"
+
+                truth.append({
+                    "source": csv_path.name,
+                    "type": "physical_schema",
+                    "columns": list(df.columns),
+                    "data_types": type_insights
+                })
+                print(f"✅ [PROFILER] Extracted all {len(df.columns)} active schema parameters.")
+                
+            except Exception as e:
+                print(f"[-] Profile extraction failed for {csv_path.name}: {e}")
+                
+        return truth
+
+    def read_notebook(self, nb_path):
+        """Parses .ipynb files into markdown and code chunks."""
+        with open(nb_path, 'r', encoding='utf-8') as f:
+            nb = nbformat.read(f, as_version=4)
+        return {
+            "markdown": [c.source for c in nb.cells if c.cell_type == 'markdown'],
+            "code": [c.source for c in nb.cells if c.cell_type == 'code']
+        }