pipediag 0.1.1__tar.gz

pipediag-0.1.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Dathi Likhith
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pipediag-0.1.1/MANIFEST.in

@@ -0,0 +1,23 @@
+include README.md
+include LICENSE
+include pyproject.toml
+
+recursive-include src/pipediag *.py
+recursive-include src/pipediag/calibration *.json
+recursive-include web/backend *.py
+recursive-include web/frontend/dist *.html *.svg *.js *.css
+
+
+exclude *.py[cod]
+global-exclude SOURCES.txt
+recursive-exclude tests *
+recursive-exclude experiments *
+recursive-exclude __pycache__ *
+
+prune build
+prune dist
+prune .pytest_cache
+prune .venv
+prune venv
+prune src/pipediag.egg-info
+prune web/frontend/node_modules

pipediag-0.1.1/PKG-INFO

@@ -0,0 +1,540 @@
+Metadata-Version: 2.4
+Name: pipediag
+Version: 0.1.1
+Summary: An explainable diagnostic framework for scikit-learn machine learning pipelines
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/LikhithDathi/PipeDiag
+Project-URL: Repository, https://github.com/LikhithDathi/PipeDiag
+Project-URL: Issues, https://github.com/LikhithDathi/PipeDiag/issues
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=2.0
+Requires-Dist: scikit-learn>=1.8
+Provides-Extra: llm
+Requires-Dist: ollama>=0.4; extra == "llm"
+Provides-Extra: web
+Requires-Dist: fastapi>=0.115; extra == "web"
+Requires-Dist: uvicorn[standard]>=0.30; extra == "web"
+Dynamic: license-file
+
+# PipeDiag
+
+**An explainable diagnostic framework for scikit-learn machine learning pipelines.**
+
+PipeDiag observes supported scikit-learn pipeline execution, records runtime events and dataset references, and provides components for evidence-based diagnosis, Bayesian inference, structured explanations, and human-readable reports.
+
+The framework separates runtime observation from diagnostic inference so that applications can configure how evidence is collected and how diagnostic results are produced.
+
+## Features
+
+- **Runtime instrumentation:** Observe operations performed by supported scikit-learn pipelines.
+- **Execution events:** Record component names, operations, event status, execution duration, dataset references, and event relationships.
+- **Evidence-based diagnosis:** Evaluate diagnostic evidence against calibrated likelihood models.
+- **Bayesian inference:** Produce a posterior distribution over diagnostic hypotheses.
+- **Explainable results:** Build structured explanations and render diagnostic reports.
+- **Configurable architecture:** Supply an evidence provider and diagnostic engine through the application layer.
+- **Packaged calibration data:** Include the default likelihood-model artifact with the Python package.
+- **Optional web dashboard:** Explore diagnostic runs through a locally hosted React dashboard.
+- **Optional LLM integration:** Support Ollama-related explanation workflows through an optional dependency.
+
+## Requirements
+
+- Python 3.12 or later
+- NumPy 2.0 or later
+- scikit-learn 1.8 or later
+
+## Installation
+
+### Install from PyPI
+
+Install the core PipeDiag package:
+
+```bash
+python -m pip install pipediag
+```
+
+The core installation includes runtime instrumentation, evidence-processing components, calibrated Bayesian diagnosis, explanation generation, and reporting functionality.
+
+### Optional dependencies
+
+PipeDiag provides optional integrations that can be installed separately.
+
+**Web dashboard:**
+
+```bash
+python -m pip install "pipediag[web]"
+```
+
+Launch the dashboard:
+
+```bash
+pipediag dashboard
+```
+
+The dashboard is available at:
+
+```text
+http://127.0.0.1:8000
+```
+
+To customize the host and port:
+
+```bash
+pipediag dashboard --host 127.0.0.1 --port 8000 --no-browser
+```
+
+The dashboard runs locally and provides access to the PipeDiag backend and React frontend.
+
+**Ollama integration:**
+
+```bash
+python -m pip install "pipediag[llm]"
+```
+
+**Install both integrations:**
+
+```bash
+python -m pip install "pipediag[web,llm]"
+```
+
+The `llm` extra installs Ollama-related support. The `web` extra installs FastAPI and Uvicorn. These extras are optional and are not part of the base dependency set.
+
+### Instgit clone https://github.com/LikhithDathi/PipeDiag.git
+cd PipeDiag
+repository and install the package:
+
+```bash
+git clone <REPOSITORY_URL>
+cd pipediag
+python -m pip install .
+```
+
+For development, where source changes should be available without reinstalling:
+
+```bash
+python -m pip install -e .
+```
+
+### Install from a built wheel
+
+If you have downloaded or built a wheel, install it with:
+
+```bash
+python -m pip install pipediag-0.1.0-py3-none-any.whl
+```
+
+Replace the filename with the wheel you have.
+
+## Quick start: end-to-end pipeline diagnosis
+
+PipeDiag provides the public `pipediag.start()` API for observing supported scikit-learn pipeline execution and connecting the captured runtime context to a configured diagnostic application.
+
+### 1. Configure the diagnostic application
+
+The diagnostic application coordinates evidence acquisition, Bayesian inference, explanation generation, and report rendering.
+
+```python
+from pipediag.application import (
+    DiagnosticApplication,
+    DiagnosticEvidenceProvider,
+)
+
+from pipediag.diagnostics.engine_factory import (
+    build_calibrated_diagnostic_engine,
+)
+
+from pipediag.evidence.runtime_provider import (
+    RuntimeEvidenceProvider,
+)
+
+application = DiagnosticApplication(
+    evidence_provider=DiagnosticEvidenceProvider(
+        RuntimeEvidenceProvider()
+    ),
+    diagnostic_engine=build_calibrated_diagnostic_engine(),
+)
+```
+
+### 2. Instrument and execute a pipeline
+
+```python
+import numpy as np
+import pipediag
+
+from sklearn.linear_model import LogisticRegression
+from sklearn.model_selection import train_test_split
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import StandardScaler
+
+X = np.array([
+    [1.0, 2.0],
+    [2.0, 1.0],
+    [3.0, 4.0],
+    [4.0, 3.0],
+    [5.0, 6.0],
+    [6.0, 5.0],
+    [7.0, 8.0],
+    [8.0, 7.0],
+])
+
+y = np.array([0, 0, 0, 0, 1, 1, 1, 1])
+
+X_train, X_test, y_train, y_test = train_test_split(
+    X,
+    y,
+    test_size=0.25,
+    random_state=42,
+    stratify=y,
+)
+
+pipeline = Pipeline([
+    ("scaler", StandardScaler()),
+    ("classifier", LogisticRegression()),
+])
+
+with pipediag.start(application=application) as run:
+    pipeline = run.instrument(pipeline)
+
+    pipeline.fit(X_train, y_train)
+    predictions = pipeline.predict(X_test)
+
+    result = run.result()
+```
+
+### 3. Inspect the diagnostic result
+
+```python
+diagnostic = result.diagnostic_result
+
+print("Run ID:", run.run_id)
+print("Runtime events:", run.context.event_count())
+print("Predictions:", predictions)
+
+print("Posterior:", diagnostic.posterior)
+print("Uncertainty:", diagnostic.uncertainty)
+
+print("\nDiagnostic report:")
+print(result.report)
+```
+
+The diagnostic result exposes:
+
+- `posterior`: probability distribution over diagnostic hypotheses.
+- `uncertainty`: uncertainty associated with the posterior distribution.
+- `evidence`: evidence groups supplied to the diagnostic engine.
+- `metadata`: contextual information associated with the diagnosis.
+
+### What happens during a run?
+
+1. `pipediag.start()` creates a `DiagnosticRun` and its `RunContext`.
+2. `run.instrument(pipeline)` instruments the supported pipeline.
+3. Pipeline operations execute normally while runtime events are recorded.
+4. `RuntimeEvidenceProvider` acquires execution evidence from the completed runtime context.
+5. `DiagnosticEvidenceProvider` adapts the acquired evidence into `EvidenceGroup` objects.
+6. The calibrated diagnostic engine evaluates the evidence and performs Bayesian inference.
+7. The application builds a structured explanation and renders a diagnostic report.
+
+The context manager does not suppress exceptions raised by the user's pipeline.
+
+### Current runtime evidence coverage
+
+The current `RuntimeEvidenceProvider` implementation acquires execution evidence (G8) from captured runtime events. It calculates the execution failure rate, total event count, and failed event count.
+
+The provider also contains a generalization-evidence helper, but that helper is not currently invoked by `collect()`. Other evidence groups are not automatically acquired by this provider.
+
+Consequently, the end-to-end example demonstrates the configured diagnostic workflow using the currently implemented G8 runtime evidence. It should not be interpreted as automatic acquisition of all eight evidence groups.
+
+## Runtime observation versus diagnostic inference
+
+PipeDiag separates runtime observation from diagnostic inference. These capabilities can be used independently or combined through the public `DiagnosticRun` API.
+
+### Runtime observation only
+
+Use this mode when you want to instrument a pipeline and inspect its execution without performing Bayesian diagnosis.
+
+```python
+with pipediag.start() as run:
+    run.instrument(pipeline)
+    pipeline.fit(X, y)
+    predictions = pipeline.predict(X)
+
+events = run.context.get_events()
+
+for event in events:
+    print(
+        event.event_id,
+        event.operation,
+        event.status,
+        event.duration_ms,
+    )
+```
+
+This mode does not require a diagnostic application.
+
+### End-to-end diagnosis
+
+Use this mode when you want PipeDiag to acquire evidence and produce a diagnostic result.
+
+Configure a `DiagnosticApplication` with:
+
+- An evidence provider responsible for acquiring diagnostic evidence.
+- A diagnostic engine responsible for Bayesian inference.
+
+Pass the configured application to `pipediag.start()` and call `run.result()` after pipeline execution.
+
+```python
+with pipediag.start(application=application) as run:
+    run.instrument(pipeline)
+    pipeline.fit(X, y)
+
+    result = run.result()
+```
+
+Alternatively, the application can be supplied directly to `run.result(application=application)`.
+
+Calling `run.result()` without a configured application raises a `RuntimeError`.
+
+### Diagnostic result structure
+
+The application returns a `DiagnosticRunResult` containing:
+
+- `diagnostic_result`: posterior distribution, uncertainty, evidence groups, and metadata.
+- `explanation`: structured diagnostic interpretation.
+- `report`: rendered human-readable report.
+
+The posterior represents probabilistic diagnostic support. It should not be interpreted as proof of a causal root cause.
+
+## Building a calibrated diagnostic engine
+
+PipeDiag provides a factory for constructing a diagnostic engine from the packaged calibration artifact.
+
+```python
+from pipediag.diagnostics.engine_factory import (
+    build_calibrated_diagnostic_engine,
+)
+
+engine = build_calibrated_diagnostic_engine()
+```
+
+The default calibration loader looks for the JSON artifact inside the installed package first. It also supports source-tree and shared-data fallback paths.
+
+The loader validates the artifact schema, declared model count, required model fields, and duplicate model keys before returning a likelihood-model registry.
+
+The calibrated engine can be used by a configured `DiagnosticApplication` or with the lower-level evidence interface when you already have compatible evidence groups.
+
+## Working with diagnostic results
+
+The lower-level diagnostic engine accepts evidence groups:
+
+```python
+from pipediag.diagnostics.evidence_group import EvidenceGroup
+from pipediag.diagnostics.engine_factory import (
+    build_calibrated_diagnostic_engine,
+)
+
+engine = build_calibrated_diagnostic_engine()
+
+evidence = [
+    EvidenceGroup(
+        name="G1",
+        value=0.8,
+        status="observed",
+    )
+]
+
+result = engine.diagnose(evidence)
+
+print("Posterior:", result.posterior)
+print("Uncertainty:", result.uncertainty)
+```
+
+This is a minimal API example, not a validated real-world diagnostic scenario. The meaning and validity of evidence values depend on the evidence-generation process and the calibration models. Do not interpret a single synthetic value as proof of a pipeline defect.
+
+A diagnostic result includes:
+
+- `posterior`: probabilities associated with diagnostic hypotheses.
+- `uncertainty`: the engine's uncertainty measure.
+- `evidence`: the evidence groups supplied to the engine.
+- `metadata`: optional contextual information.
+
+The current hypothesis set includes data leakage, generalization failure, distribution shift, class imbalance, data-quality problems, evaluation problems, underfitting, feature-related problems, and pipeline/execution problems.
+
+## Calibration data
+
+The package includes a default artifact at:
+
+```text
+pipediag/calibration/likelihood_models.json
+```
+
+The artifact uses schema version 1 and contains calibrated likelihood-model entries for the diagnostic engine.
+
+The default loader resolves the artifact from the installed package when available. It also supports fallback locations for source checkouts and shared data installations.
+
+If you supply a custom artifact path, ensure that the JSON follows the schema expected by the installed version of PipeDiag.
+
+A successful calibration load establishes that the artifact can be parsed and used to construct the registry. It does not, by itself, establish that the resulting probabilities are reliable for every application or dataset. Calibration quality must be assessed against appropriate evaluation scenarios.
+
+## Regenerating calibration artifacts
+
+PipeDiag provides a script for regenerating the calibrated likelihood-model artifact.
+
+The calibration models are generated using the calibration workflow in `experiments/full_system_validation.py`.
+
+To regenerate the artifacts, run the following command from the project root:
+
+```bash
+python scripts/export_calibration.py
+```
+
+The script:
+
+1. Builds the calibrated likelihood-model registry.
+2. Extracts the likelihood parameters for each model.
+3. Validates that exactly 11 models have been generated.
+4. Serializes the models into the versioned JSON artifact format.
+5. Writes the artifact to both calibration locations.
+6. Reloads and verifies both generated JSON files.
+
+The generated artifacts are stored at:
+
+```text
+artifacts/calibration/likelihood_models.json
+src/pipediag/calibration/likelihood_models.json
+```
+
+The package-level artifact under `src/pipediag/calibration/` is tracked in the repository and included in the Python distribution.
+
+The repository-level artifact under `artifacts/calibration/` is a generated development artifact. It is excluded from Git and can be regenerated by running the export script.
+
+Both files should remain synchronized when the export script is run. The script updates and verifies both locations automatically.
+
+The calibration artifact uses schema version 1 and currently contains 11 likelihood models.
+
+Regenerating the artifact does not itself establish the scientific validity of the calibration methodology or its underlying data. Those require separate validation.
+
+## Architecture overview
+
+PipeDiag separates runtime observation from inference and presentation.
+
+```text
+Scikit-learn pipeline
+        |
+        v
+Runtime instrumentation
+        |
+        v
+RunContext and execution events
+        |
+        v
+Configured evidence provider
+        |
+        v
+Evidence groups
+        |
+        v
+Calibrated diagnostic engine
+        |
+        v
+Bayesian posterior and uncertainty
+        |
+        v
+Structured explanation
+        |
+        v
+Human-readable report
+```
+
+The diagram describes the high-level application workflow. Runtime events are not automatically equivalent to diagnostic evidence; a configured provider is responsible for obtaining and adapting the evidence required by the diagnostic engine.
+
+## Web dashboard
+
+PipeDiag includes an optional local web dashboard built with React and powered by a FastAPI backend.
+
+Install the web dependencies:
+
+```bash
+python -m pip install "pipediag[web]"
+```
+
+Launch the dashboard:
+
+```bash
+pipediag dashboard
+```
+
+By default, the dashboard is served at:
+
+```text
+http://127.0.0.1:8000
+```
+
+The dashboard provides an interface for exploring diagnostic runs, viewing diagnostic results, and accessing saved run history.
+
+The backend exposes API endpoints for health checks, diagnostic runs, and example execution.
+
+To stop the dashboard, press `Ctrl+C` in the terminal running the server.
+
+The dashboard is optional. Users who only need the Python diagnostic framework do not need to install the web dependencies.
+
+## Development and tests
+
+To install the project for development:
+
+```bash
+python -m pip install -e .
+```
+
+Install the optional web dependencies when working on the dashboard:
+
+```bash
+python -m pip install -e ".[web]"
+```
+
+Install the optional LLM dependencies when working on the Ollama integration:
+
+```bash
+python -m pip install -e ".[llm]"
+```
+
+Run the test suite with:
+
+```bash
+python -m pytest -q
+```
+
+Build the distribution artifacts with:
+
+```bash
+python -m pip install build
+python -m build
+```
+
+The wheel and source distribution are written to the `dist/` directory.
+
+For release verification, install the built wheel in a fresh virtual environment and check both runtime instrumentation and calibration loading.
+
+## Scope and limitations
+
+- Runtime instrumentation targets supported scikit-learn pipelines; it is not a general-purpose Python execution tracer.
+- Runtime events and dataset references provide execution evidence, but they do not independently establish the semantic meaning of every feature or the root cause of a model problem.
+- Full diagnostic inference requires a compatible evidence provider and configured diagnostic engine.
+- The current runtime evidence provider automatically acquires execution evidence (G8); other evidence groups require their own acquisition and integration.
+- Diagnostic outputs depend on the quality, coverage, and applicability of the evidence and calibration models.
+- A confident posterior should not be interpreted as a guarantee of correctness without appropriate empirical validation.
+- Optional integrations may require additional dependencies and their own configuration.
+
+## Project status
+
+PipeDiag includes runtime instrumentation, evidence-processing components, calibrated Bayesian diagnosis, explanation and reporting components, and optional integration support.
+
+The project is under active development. Before using it in a production decision-making workflow, validate the complete diagnostic process against representative scenarios, verify probability calibration, and document the supported deployment configuration.
+
+## License
+
+PipeDiag is released under the MIT License.
+
+Copyright (c) 2026 Dathi Likhith.
+
+See the [LICENSE](LICENSE) file for details.