m8flow 1.1.2 → 1.1.4

package/README.md

@@ -1,15 +1,62 @@
-# M8Flow CLI
+<p align="center">
+  <img src="https://cdn.jsdelivr.net/npm/m8flow@latest/bundled/frontend-dist/assets/logo_full.png" alt="M8Flow" height="60" />
+</p>
 
-Visual ML Pipeline Builder — install globally, run anywhere.
+<h3 align="center">AI-powered visual machine learning workflow builder with local execution, pipeline orchestration, and zero-config setup.</h3>
 
-```
+<p align="center">
+  <a href="https://www.npmjs.com/package/m8flow"><img src="https://img.shields.io/npm/v/m8flow?color=6366f1&label=npm" alt="npm version" /></a>
+  <a href="https://www.npmjs.com/package/m8flow"><img src="https://img.shields.io/npm/dm/m8flow?color=22c55e&label=downloads" alt="downloads" /></a>
+  <a href="https://www.npmjs.com/package/m8flow"><img src="https://img.shields.io/npm/l/m8flow?color=f59e0b" alt="license" /></a>
+  <img src="https://img.shields.io/badge/Python-3.8%2B-blue" alt="python" />
+  <img src="https://img.shields.io/badge/Node.js-18%2B-green" alt="node" />
+</p>
+
+---
+
+## Quick Start
+
+```bash
 npm install -g m8flow
 m8flow run
 ```
 
+Your browser opens automatically. Start building ML pipelines in seconds — no Docker, no config files, no cloud accounts.
+
+---
+
+## Why M8Flow?
+
+Most ML tooling forces you to write boilerplate, manage environments manually, or pay for cloud compute. M8Flow takes a different approach:
+
+| Problem | M8Flow Solution |
+|---|---|
+| Writing pipeline code from scratch | Visual drag-and-drop node editor |
+| Complex environment setup | Automatic Python venv + dependency install |
+| No AI assistance for ML workflows | Built-in AI pipeline generation (OpenRouter / Gemini / Mistral) |
+| Vendor lock-in and cloud costs | 100% local-first — your machine, your data |
+| Slow iteration cycles | Live pipeline execution with real-time logs |
+
+---
+
+## Features
+
+- **Visual drag-and-drop pipeline builder** — connect nodes with edges, no boilerplate
+- **AI-assisted workflow generation** — describe your pipeline in plain English
+- **Multi-provider AI support** — OpenRouter, Google Gemini, Mistral La Plateforme
+- **Local-first architecture** — all data stays on your machine
+- **Automatic environment setup** — Python virtualenv + dependencies on first run
+- **60+ built-in ML nodes** — preprocessing, models, evaluation, visualization
+- **Custom Python components** — add your own nodes with AI code generation
+- **CSV dataset handling** — upload files, inspect schemas, feed into pipelines
+- **Pipeline versioning** — save checkpoints, compare versions, roll back
+- **Real-time execution monitoring** — live logs, node status, output previews
+- **Cross-browser state sync** — flows persist across tabs and browsers via local server
+- **Port conflict auto-resolution** — never manually kill a port again
+
 ---
 
-## System requirements
+## Requirements
 
 | Tool   | Minimum |
 |--------|---------|
@@ -23,38 +70,33 @@ m8flow run
 
 ### `m8flow run`
 
-Starts the full stack and opens the browser.
+Launches the complete local M8Flow environment.
 
 ```
 Options:
   -p, --port <n>      Frontend port (default: 3000)
   -b, --backend <n>   Backend port  (default: 8000)
-  --no-browser        Don't open the browser automatically
-  --verbose           Stream backend and frontend logs
+  --no-browser        Skip opening the browser automatically
+  --verbose           Stream full backend logs
   -h, --help          Show help
 ```
 
-**Examples**
-
 ```bash
-m8flow run                          # defaults (3000 / 8000)
-m8flow run --port 4000              # custom frontend port
-m8flow run --backend 9000 --verbose # custom backend + full logs
-m8flow run --no-browser             # headless / CI mode
+m8flow run                           # standard launch
+m8flow run --port 4000               # custom port
+m8flow run --no-browser --verbose    # headless with logs
 ```
 
 ### `m8flow doctor`
 
-Checks system requirements before you try to run.
+Diagnoses your environment before running.
 
 ```bash
 m8flow doctor
 ```
 
-Output example:
-
 ```
-  M8Flow Doctor  v1.0.0
+  M8Flow Doctor  v1.1.3
 
     ✔  Node.js  (20.11.0)
     ✔  Python   (Python 3.11.5)
@@ -62,77 +104,109 @@ Output example:
     ✔  Python dependencies
     ✔  Bundled assets
     ✔  Port Frontend (3000)  — free
-    ✖  Port Backend  (8000)  — busy — will auto-shift to 8001
+    ✔  Port Backend  (8000)  — free
 ```
 
 ---
 
-## What `m8flow run` does
-
-| Step | Action |
-|------|--------|
-| 1 | Detect Python 3.8+ on PATH (`python3`, `python`, `py`) |
-| 2 | Create `~/.m8flow/{uploads,models,pipelines}` |
-| 3 | Create Python virtualenv at `~/.m8flow/venv` |
-| 4 | `pip install -r requirements.txt` (only when file changes) |
-| 5 | Auto-resolve port conflicts with `detect-port` |
-| 6 | Spawn `uvicorn main:app` via **execa** |
-| 7 | Serve React build via built-in Node HTTP server |
-| 8 | TCP-poll the backend until it accepts connections |
-| 9 | Open `http://localhost:3000` with **open** |
-| 10 | Clean `SIGINT` / `SIGTERM` shutdown |
+## What happens when you run `m8flow run`
+
+| Step | What M8Flow does |
+|------|-----------------|
+| 1 | Locates Python 3.8+ on your PATH |
+| 2 | Creates `~/.m8flow/` storage directories |
+| 3 | Creates an isolated Python virtualenv |
+| 4 | Installs all ML dependencies (once only — hash-cached) |
+| 5 | Auto-resolves port conflicts |
+| 6 | Starts FastAPI backend via uvicorn |
+| 7 | Serves the React frontend via Node HTTP server |
+| 8 | Waits for backend health check (up to 120 s) |
+| 9 | Opens `http://localhost:3000` in your browser |
 
 ---
 
-## Data storage
+## Architecture
 
-Everything stays on your machine:
+```
+m8flow run
+├── Node.js CLI (bin/m8flow.js)
+│   ├── Python venv setup  →  ~/.m8flow/venv/
+│   ├── Frontend server    →  localhost:3000   (Node static)
+│   └── Backend server     →  localhost:8000   (uvicorn)
+│
+├── Frontend  — React + XYFlow canvas, Zustand state
+│   └── Visual node editor, AI chat, dataset manager
+│
+└── Backend   — FastAPI + Python runtime
+    ├── Pipeline executor  (topological DAG execution)
+    ├── LLM service        (OpenRouter / Gemini / Mistral)
+    ├── Self-healer        (AI auto-fix on node errors)
+    └── Template library   (60+ pre-built ML components)
+```
 
-| Path | Purpose |
-|------|---------|
-| `~/.m8flow/uploads/`  | Uploaded CSV files |
-| `~/.m8flow/models/`   | Trained models |
-| `~/.m8flow/pipelines/`| Saved pipelines |
-| `~/.m8flow/venv/`     | Python virtualenv |
-| `~/.m8flow/.env`      | API keys (optional) |
+| Layer | Technology |
+|---|---|
+| Frontend | React, XYFlow, Zustand, Vite |
+| Backend | FastAPI, Python, uvicorn |
+| Runtime | Isolated Python virtualenv |
+| CLI | Node.js 18+, ESM, execa |
+| AI Layer | OpenRouter · Google Gemini · Mistral La Plateforme |
+| Storage | Local filesystem (`~/.m8flow/`) |
 
-### Setting your OpenRouter API key
+---
 
-```bash
-echo "OPENROUTER_API_KEY=sk-or-..." >> ~/.m8flow/.env
-```
+## AI Keys
 
-The backend loads this file on every startup.
+M8Flow works without any API key (demo mode with limited models). For full AI pipeline generation, add at least one key in **Settings → API Keys**:
+
+| Provider | Key prefix | Models |
+|---|---|---|
+| OpenRouter | `sk-or-...` | Llama 3.3, NVIDIA Nemotron, Gemma, and 200+ more |
+| Google Gemini | `AIza...` | Gemini 2.5 Flash, Flash Lite, Pro |
+| Mistral | any | Codestral, Mistral Small, Mixtral |
+
+Keys are stored locally in your browser and on your machine — never sent to M8Flow servers.
 
 ---
 
-## Build from source
+## Data & Storage
+
+Everything lives on your machine:
+
+| Path | Contents |
+|------|---------|
+| `~/.m8flow/uploads/` | Uploaded CSV datasets |
+| `~/.m8flow/models/` | Trained model files |
+| `~/.m8flow/pipelines/` | Saved pipeline state |
+| `~/.m8flow/venv/` | Python virtualenv |
+| `~/.m8flow/app_state.json` | Flows, projects, settings |
+
+---
+
+## Build from Source
 
 ```bash
-git clone https://github.com/your-org/m8flow
+git clone https://github.com/mursaleen231213/m8flow
 cd m8flow
 
-# 1. Bundle backend + frontend
+# Build React frontend + bundle Python backend
 node cli/scripts/build.js
 
-# 2. Install globally from local build
+# Install locally
 cd cli && npm install -g .
 
-# 3. Run
+# Launch
 m8flow run
 ```
 
----
-
-## Publish to npm
+### Push updates to npm
 
 ```bash
-# Inside cli/
-npm publish
+cd cli
+npm version patch          # bump version
+npm publish --otp=<code>   # publish (OTP from authenticator)
 ```
 
-`prepublishOnly` runs `build.js` automatically.
-
 ---
 
 ## Uninstall
@@ -140,6 +214,13 @@ npm publish
 ```bash
 npm uninstall -g m8flow
 
-# Optional — remove all data:
-rm -rf ~/.m8flow
+# Remove all local data (optional):
+rm -rf ~/.m8flow          # Mac/Linux
+Remove-Item -Recurse -Force "$env:USERPROFILE\.m8flow"  # Windows
 ```
+
+---
+
+<p align="center">
+  Built with ❤️ for ML engineers who value speed and simplicity.
+</p>

package/bundled/backend/api/routes/flows.py

@@ -1,3 +1,4 @@
+import json
 from fastapi import APIRouter, HTTPException, Request
 from pydantic import BaseModel
 from domain.models import FlowSchema
@@ -86,9 +87,20 @@ def _to_canvas_node(node: dict) -> dict:
 
 
 def _inject_api_key(http_request: Request) -> None:
-    """Read X-OpenRouter-Key header and set it as the per-request ContextVar."""
-    key = http_request.headers.get("X-OpenRouter-Key")
-    llm_service._request_api_key.set(key or None)
+    """Read provider key + per-agent model headers and set per-request ContextVars."""
+    llm_service._request_api_key.set(http_request.headers.get("X-OpenRouter-Key") or None)
+    llm_service._request_gemini_key.set(http_request.headers.get("X-Gemini-Key") or None)
+    llm_service._request_mistral_key.set(http_request.headers.get("X-Mistral-Key") or None)
+
+    # Parse per-agent model matrix sent as JSON: {"architect":"gemini-2.5-flash", ...}
+    raw_agents = http_request.headers.get("X-Agent-Models")
+    if raw_agents:
+        try:
+            llm_service._request_agent_models.set(json.loads(raw_agents))
+        except Exception:
+            llm_service._request_agent_models.set(None)
+    else:
+        llm_service._request_agent_models.set(None)
 
 
 @router.post("/generate")
@@ -257,6 +269,34 @@ async def ask_flow(http_request: Request, req: AskRequest):
     }
 
 
+class InterviewRequest(BaseModel):
+    """Trigger Phase 1 analysis when a CSV is first uploaded."""
+    context: str  # dataset summary string from the upload response
+
+
+@router.post("/interview")
+async def interview_flow(http_request: Request, req: InterviewRequest):
+    """
+    Phase 1 — Interactive interview entry point.
+
+    Called immediately after a CSV upload. Returns a conversational
+    analysis text (with [PLANNING], [ANALYSIS], [DEDUCTION],
+    [AWAITING CONFIRMATION] status labels) WITHOUT generating any nodes.
+    The frontend displays this as an assistant message and waits for user
+    confirmation before proceeding to pipeline generation.
+    """
+    _inject_api_key(http_request)
+    if not req.context.strip():
+        raise HTTPException(status_code=422, detail="Context cannot be empty")
+    try:
+        text = await llm_service.interview_dataset(req.context)
+        return {"result_type": "interview", "message": text}
+    except RuntimeError as exc:
+        raise HTTPException(status_code=503, detail=str(exc))
+    except Exception as exc:
+        raise HTTPException(status_code=500, detail=f"{type(exc).__name__}: {exc}")
+
+
 @router.post("/execute")
 def execute_flow(flow: FlowSchema):
     try:

package/bundled/backend/api/routes/nodes.py

@@ -77,9 +77,9 @@ async def generate_node_code_route(http_request: Request, req: GenerateCodeReque
     """Generate M8Flow-compatible Python node code from a plain-English description."""
     from services import llm_service
 
-    # Honour the per-request OpenRouter key set by the frontend
-    key = http_request.headers.get("X-OpenRouter-Key")
-    llm_service._request_api_key.set(key or None)
+    llm_service._request_api_key.set(http_request.headers.get("X-OpenRouter-Key") or None)
+    llm_service._request_gemini_key.set(http_request.headers.get("X-Gemini-Key") or None)
+    llm_service._request_mistral_key.set(http_request.headers.get("X-Mistral-Key") or None)
 
     if not req.description.strip():
         raise HTTPException(status_code=422, detail="Description cannot be empty")

package/bundled/backend/config.py

@@ -10,9 +10,11 @@ if _global_env.exists():
     load_dotenv(_global_env, override=False)   # don't override already-set vars
 
 class Config:
-    MISTRAL_API_KEY    = os.getenv("MISTRAL_API_KEY", "")
-    OPENAI_API_KEY     = os.getenv("OPENAI_API_KEY", "")
+    MISTRAL_API_KEY    = os.getenv("MISTRAL_API_KEY",    "")
+    OPENAI_API_KEY     = os.getenv("OPENAI_API_KEY",     "")
     OPENROUTER_API_KEY = os.getenv("OPENROUTER_API_KEY", "")
+    GEMINI_API_KEY     = os.getenv("GEMINI_API_KEY",     "")  # Google AI Studio key
+    MISTRAL_API_KEY    = os.getenv("MISTRAL_API_KEY",    "")  # Mistral La Plateforme key
     STORAGE_DIR        = os.getenv("M8FLOW_UPLOAD_DIR", os.getenv("STORAGE_DIR", "./storage"))
 
 config = Config()

package/bundled/backend/core/code_validator.py

@@ -41,7 +41,7 @@ ALLOWED_IMPORTS = frozenset({
     "numpy", "pandas", "scipy", "sklearn", "xgboost", "lightgbm",
     "statsmodels", "imblearn",
     # Plotting
-    "matplotlib", "seaborn", "plotly",
+    "matplotlib", "seaborn", "plotly", "mpl_toolkits",
     # Standard safe libs
     "math", "statistics", "itertools", "functools", "collections",
     "json", "re", "datetime", "typing",

package/bundled/backend/core/executor.py

@@ -62,6 +62,49 @@ def _serialize_value(val: Any) -> Any:
             "shape": list(val.shape),
             "dtype": str(val.dtype),
         }
+
+    # ── sklearn displays ─────────────────────────────────────────────────────
+    is_display = hasattr(val, "figure_") and hasattr(val.figure_, "savefig")
+    if is_display:
+        try:
+            import io
+            import base64
+            buf = io.BytesIO()
+            val.figure_.savefig(buf, format="png", bbox_inches="tight")
+            b64 = base64.b64encode(buf.getvalue()).decode("utf-8")
+            return {
+                "image_base64": b64,
+                "title": getattr(val, "estimator_name", type(val).__name__)
+            }
+        except Exception:
+            pass
+
+    # ── matplotlib / seaborn ─────────────────────────────────────────────────
+    is_figure = hasattr(val, "savefig") and type(val).__module__.startswith("matplotlib")
+    if is_figure:
+        try:
+            import io
+            import base64
+            buf = io.BytesIO()
+            val.savefig(buf, format="png", bbox_inches="tight")
+            b64 = base64.b64encode(buf.getvalue()).decode("utf-8")
+            return {
+                "image_base64": b64,
+                "title": "Matplotlib / Seaborn Plot"
+            }
+        except Exception:
+            pass
+
+    # ── plotly ───────────────────────────────────────────────────────────────
+    if hasattr(val, "to_html") and type(val).__module__.startswith("plotly"):
+        try:
+            return {
+                "plotly_html": val.to_html(full_html=False, include_plotlyjs="cdn"),
+                "title": "Plotly Visualization"
+            }
+        except Exception:
+            pass
+
     # sklearn / any estimator
     if hasattr(val, "predict") and hasattr(val, "fit"):
         return {
@@ -75,9 +118,33 @@ def _serialize_value(val: Any) -> Any:
     if isinstance(val, (np.bool_,)):
         return bool(val)
     if isinstance(val, dict):
+        if "data" in val and isinstance(val.get("data"), list) and "layout" in val:
+            try:
+                import json
+                import plotly.io as pio
+                val_json = json.dumps(val)
+                fig = pio.from_json(val_json)
+                return {
+                    "plotly_html": fig.to_html(full_html=False, include_plotlyjs="cdn"),
+                    "title": "Plotly Visualization"
+                }
+            except Exception:
+                pass
         return {k: _serialize_value(v) for k, v in val.items()}
     if isinstance(val, (list, tuple)):
         return [_serialize_value(v) for v in val]
+    if isinstance(val, str):
+        if val.startswith('{') and '"data":' in val and '"layout":' in val:
+            try:
+                import plotly.io as pio
+                fig = pio.from_json(val)
+                return {
+                    "plotly_html": fig.to_html(full_html=False, include_plotlyjs="cdn"),
+                    "title": "Plotly Visualization"
+                }
+            except Exception:
+                pass
+        return val
     return val
 
 

package/bundled/backend/core/runtime.py

@@ -3,6 +3,13 @@ import inspect
 import functools
 from typing import Any
 
+# Ensure matplotlib uses a non-interactive backend
+try:
+    import matplotlib
+    matplotlib.use('Agg')
+except ImportError:
+    pass
+
 
 @functools.lru_cache(maxsize=128)
 def _compile(code: str):
@@ -45,5 +52,33 @@ def execute_node_code(code: str, inputs: dict[str, Any]) -> dict[str, Any]:
 
     result = run_fn(**filtered)
     if not isinstance(result, dict):
-        return {"output": result}
+        result = {"output": result}
+
+    # ── Automatically capture unreturned matplotlib figures ──────────────────
+    import sys
+    if "matplotlib.pyplot" in sys.modules:
+        import matplotlib.pyplot as plt
+        figs = plt.get_fignums()
+        if figs:
+            # Identify figures already returned explicitly to avoid duplicates
+            returned_fig_ids = {id(v) for v in result.values() if hasattr(v, "savefig")}
+            
+            import io, base64
+            for i, num in enumerate(figs):
+                fig = plt.figure(num)
+                if id(fig) in returned_fig_ids:
+                    continue
+
+                buf = io.BytesIO()
+                fig.savefig(buf, format="png", bbox_inches="tight")
+                b64 = base64.b64encode(buf.getvalue()).decode("utf-8")
+                # Avoid overwriting explicit returns
+                key = f"plot_{i}" if i > 0 else "plot"
+                if key not in result:
+                    result[key] = {
+                        "image_base64": b64,
+                        "title": f"Figure {num}"
+                    }
+            plt.close("all")
+
     return result

package/bundled/backend/main.py

@@ -1,6 +1,16 @@
 from fastapi import FastAPI, Request
 from fastapi.middleware.cors import CORSMiddleware
 from fastapi.responses import JSONResponse
+
+# Force matplotlib to use a non-interactive backend (Agg)
+# This prevents "main thread is not in main loop" errors when running 
+# inside background threads (like FastAPI/uvicorn).
+try:
+    import matplotlib
+    matplotlib.use('Agg')
+except ImportError:
+    pass
+
 from api.routes import flows, nodes, appstate
 import os
 import time
@@ -60,6 +70,17 @@ app.include_router(nodes.router,    prefix="/api/nodes",    tags=["Nodes"])
 app.include_router(appstate.router, prefix="/api/app/state", tags=["AppState"])
 
 
+@app.get("/v1/models")
+def openai_compat_models():
+    """
+    OpenAI-compatible /v1/models stub.
+    External tools (VS Code extensions, Cursor, Continue.dev …) probe this
+    endpoint to check if the server speaks the OpenAI protocol.
+    Return a minimal valid response so they get a 200 instead of a noisy 404.
+    """
+    return {"object": "list", "data": []}
+
+
 @app.get("/api/health")
 def health_check():
     from config import config

package/bundled/backend/requirements.txt

@@ -1,35 +1,60 @@
 # ── Core API ─────────────────────────────────────────────────────────────────
-fastapi
+fastapi==0.136.1
 uvicorn[standard]
-python-multipart
-pydantic
-python-dotenv
-httpx
-
+starlette==1.0.0
+python-multipart==0.0.27
+python-dotenv==1.2.2
+pydantic==2.13.3
+pydantic_core==2.46.3
+annotated-types==0.7.0
+# ── HTTP ─────────────────────────────────────────────────────────────────────
+httpx==0.28.1
+httpcore==1.0.9
+h11==0.16.0
+anyio==4.13.0
+sniffio==1.3.1
+certifi==2026.4.22
+idna==3.13
 # ── Data ─────────────────────────────────────────────────────────────────────
-pandas
-numpy
-scipy
-joblib
-
+pandas==3.0.2
+numpy==2.4.4
+scipy==1.17.1
+joblib==1.5.3
+threadpoolctl==3.6.0
+python-dateutil==2.9.0.post0
+six==1.17.0
+tzdata==2026.2
 # ── ML ───────────────────────────────────────────────────────────────────────
-scikit-learn
-xgboost
-lightgbm
-statsmodels
-
-# ── Imbalanced learning ───────────────────────────────────────────────────────
-imbalanced-learn
-
-# ── Explainability ────────────────────────────────────────────────────────────
-shap
-
-# ── Dimensionality reduction ──────────────────────────────────────────────────
-umap-learn
-
-# ── Visualisation (server-side, not imported at runtime) ─────────────────────
-matplotlib
-seaborn
-
-# ── Legacy / compat ───────────────────────────────────────────────────────────
-openai
+scikit-learn==1.8.0
+# ── Visualisation ────────────────────────────────────────────────────────────
+matplotlib==3.10.9
+seaborn==0.13.2
+plotly==6.7.0
+pillow==12.2.0
+contourpy==1.3.3
+cycler==0.12.1
+fonttools==4.62.1
+kiwisolver==1.5.0
+pyparsing==3.3.2
+narwhals==2.21.0
+# ── AI / LLM clients ─────────────────────────────────────────────────────────
+openai==2.33.0
+mistralai==2.4.4
+httpx==0.28.1
+jiter==0.14.0
+distro==1.9.0
+tqdm==4.67.3
+# ── Telemetry ────────────────────────────────────────────────────────────────
+opentelemetry-api==1.39.1
+opentelemetry-semantic-conventions==0.60b1
+# ── Utils ────────────────────────────────────────────────────────────────────
+click==8.3.3
+colorama==0.4.6
+annotated-doc==0.0.4
+eval_type_backport==0.3.1
+importlib_metadata==8.7.1
+jsonpath-python==1.1.5
+packaging==26.2
+typing_extensions==4.15.0
+typing-inspection==0.4.2
+zipp==3.23.1