vesper-wizard 2.0.4 → 2.0.6

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Vesper Team
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,60 +1,323 @@
-# vesper-wizard
+# Vesper MCP Server 🚀
 
-Zero-friction setup wizard for [Vesper](https://github.com/vesper/mcp-server) — your local MCP-native dataset intelligence layer.
+**AI-powered dataset discovery, quality analysis, and preparation** with multimodal support (text, image, audio, video).
 
-## Install
+Vesper is a Model Context Protocol (MCP) server that helps you find, analyze, and prepare high-quality datasets for machine learning projects. It integrates seamlessly with AI assistants like Claude, providing autonomous dataset workflows.
+
+## ✨ Features
+
+### 🔍 **Dataset Discovery**
+- Search across HuggingFace, Kaggle, UCI ML Repository, and more
+- Intelligent ranking based on quality, safety, and relevance
+- Automatic metadata extraction and enrichment
+
+### 📊 **Quality Analysis**
+- **Text**: Missing data, duplicates, column profiling
+- **Images**: Resolution, corruption, blur detection
+- **Audio**: Sample rate, duration, silence detection
+- **Video**: FPS, frame validation, corruption risk
+- **Unified Reports**: Consolidated quality scores (0-100) with recommendations
+
+### 🛠️ **Data Preparation**
+- Automated cleaning pipelines
+- Format conversion (CSV, JSON, Parquet)
+- Train/test/validation splitting
+- Automatic installation to project directories
+
+### 🎯 **Multimodal Support**
+- Analyze mixed datasets (text + images + audio)
+- Media-specific quality metrics
+- Intelligent modality detection
+
+## 📦 Installation
+
+## 🚀 Quick Start (VS Code + Copilot)
+
+The fastest way to install Vesper and configure it for **GitHub Copilot Chat** or **Cursor** is to run the automated setup:
+
+```bash
+npx -y -p @vespermcp/mcp-server@latest vespermcp --setup
+```
+
+1.  Select **Visual Studio Code (Settings.json)** from the list.
+2.  Restart VS Code.
+3.  Open Copilot Chat and look for the **MCP Servers** section.
+
+## 🛠️ Configuration
+Vesper supports:
+- **GitHub Copilot Chat**: Automated setup via `settings.json`.
+- **Cursor**: Automated setup via `mcp.json`.
+- **Claude Desktop**: Automated setup via `claude_desktop_config.json`.
+
+### Manual Python Setup (if needed)
+
+```bash
+pip install opencv-python pillow numpy librosa soundfile
+```
+
+## ⚙️ MCP Configuration
+
+### For Cursor
+1. Go to **Settings** > **Features** > **MCP**
+2. Click **Add New MCP Server**
+3. Enter:
+   - **Name**: `vesper`
+   - **Type**: `command`
+   - **Command**: `vesper`
+
+### For Claude Desktop
+Vesper attempts to auto-configure itself! Restart Claude and check. If not:
+
+```json
+{
+  "mcpServers": {
+    "vesper": {
+      "command": "vesper",
+      "args": [],
+      "env": {
+        "HF_TOKEN": "your-huggingface-token"
+      }
+    }
+  }
+}
+```
+
+> **Note**: If the `vesper` command isn't found, you can stick to the absolute path method.
+
+### Environment Variables (Optional)
+
+- `KAGGLE_USERNAME` & `KAGGLE_KEY`: For Kaggle dataset access
+- `HF_TOKEN`: For private HuggingFace datasets
+
+### Optional Kaggle Setup (Not Required)
+
+Core Vesper works without any API keys. Keys are only needed when you explicitly use Kaggle or gated Hugging Face.
+
+Install optional Kaggle client only if you need Kaggle source access:
 
 ```bash
-npx vesper-wizard@latest
+pip install kaggle
 ```
 
-That's it. The wizard handles everything:
+```bash
+vespermcp config keys
+```
 
-1. Creates `~/.vesper/` directories and local API key
-2. Initializes a local credentials vault in unified-key mode (no external API keys required)
-3. Installs `@vespermcp/mcp-server` and auto-configures MCP for all detected agents (Claude, Cursor, VS Code, Codex, Gemini CLI)
-4. Verifies the installation
+The setup wizard supports skip and stores keys securely via OS keyring when available,
+with fallback to `~/.vesper/config.toml`.
 
-## What you get
+or use Kaggle's native file:
 
-After the wizard finishes, your AI assistant can immediately use Vesper tools:
+- `~/.kaggle/kaggle.json`
 
-| Tool | Description |
-|------|-------------|
-| `vesper_search` | Search 16,000+ datasets via natural language |
-| `discover_datasets` | Discover from HuggingFace, Kaggle, OpenML, data.world |
-| `download_dataset` | Download any dataset to local storage |
-| `prepare_dataset` | Full pipeline: analyze → clean → split → export |
-| `analyze_quality` | Deep quality analysis with recommendations |
-| `export_dataset` | Export to parquet, csv, feather, jsonl, arrow |
-| `fuse_datasets` | Combine multiple datasets with quality checks |
+If credentials are missing and you run Kaggle commands, Vesper shows:
 
-## Security
+`Kaggle support requires API key. Run 'vespermcp config keys' (30 seconds).`
 
-- **Local-only**: Uses one local key in `~/.vesper/config.toml`
-- **Keyring-backed**: Uses OS keyring when available, falls back to local TOML
-- **No cloud**: Zero external API calls during setup
-- **No external keys**: No HuggingFace/Kaggle/Nia key prompts during setup
+### CLI Examples
+
+```bash
+vespermcp discover --source kaggle "credit risk" --limit 10
+vespermcp discover --source huggingface "credit risk" --limit 10
+vespermcp download kaggle username/dataset-name
+vespermcp download kaggle https://www.kaggle.com/datasets/username/dataset-name --target-dir ./data
+```
 
-## Config file
+## 🚀 Quick Start
 
-The wizard generates `~/.vesper/config.toml`:
+After installation and configuration, restart your AI assistant and try:
+
+```
+search_datasets(query="sentiment analysis", limit=5)
+```
+
+```
+prepare_dataset(query="image classification cats vs dogs")
+```
 
-```toml
-api_key = "vesper_sk_local_..."
-auth_mode = "local_unified"
 ```
+generate_quality_report(
+  dataset_id="huggingface:imdb", 
+  dataset_path="/path/to/data"
+)
+```
+
+## 📚 Available Tools
 
-## Post-setup
+### Dataset Discovery
 
-Restart your IDE and try in your AI assistant:
+#### `unified_dataset_api`
+Single facade over multiple dataset backends. Use one tool for provider capability inspection, dataset discovery, dataset download, and dataset info lookup. The gateway prefers public/keyless providers and can also use server-managed credentials for connectors like Kaggle or data.world when configured by the operator.
 
+**Parameters:**
+- `operation` (string): `providers`, `discover`, `download`, or `info`
+- `source` (string, optional): `auto`, `huggingface`, `openml`, `kaggle`, `dataworld`, `s3`, `bigquery`
+- `query` (string, required for `discover`)
+- `dataset_id` (string, required for `download`/`info`)
+- `limit` (number, optional)
+- `target_dir` (string, optional)
+- `public_only` (boolean, optional)
+
+**Examples:**
 ```
-vesper_search(query="sentiment analysis")
-prepare_dataset(query="image classification cats dogs")
-analyze_quality(dataset_id="imdb")
+unified_dataset_api(operation="providers")
 ```
 
-## License
+```
+unified_dataset_api(operation="discover", query="credit risk", source="auto")
+```
+
+```
+unified_dataset_api(operation="download", dataset_id="huggingface:imdb")
+```
+
+---
+
+#### `search_datasets`
+Search for datasets across multiple sources.
+
+**Parameters:**
+- `query` (string): Search query
+- `limit` (number, optional): Max results (default: 10)
+- `min_quality_score` (number, optional): Minimum quality threshold
+
+**Example:**
+```
+search_datasets(query="medical imaging", limit=5, min_quality_score=70)
+```
+
+---
+
+### Data Preparation
+
+#### `prepare_dataset`
+Download, analyze, and prepare a dataset for use.
+
+**Parameters:**
+- `query` (string): Dataset search query or ID
+
+**Example:**
+```
+prepare_dataset(query="squad")
+```
+
+---
+
+#### `export_dataset`
+Export a prepared dataset to a custom directory with format conversion.
+
+**Parameters:**
+- `dataset_id` (string): Dataset identifier
+- `target_dir` (string): Export directory
+- `format` (string, optional): Output format (csv, json, parquet)
+
+**Example:**
+```
+export_dataset(
+  dataset_id="huggingface:imdb",
+  target_dir="./my-data",
+  format="csv"
+)
+```
+
+---
+
+### Quality Analysis
+
+#### `analyze_image_quality`
+Analyze image datasets for resolution, corruption, and blur.
+
+**Parameters:**
+- `path` (string): Path to image file or folder
+
+**Example:**
+```
+analyze_image_quality(path="/path/to/images")
+```
+
+---
+
+#### `analyze_media_quality`
+Analyze audio/video files for quality metrics.
+
+**Parameters:**
+- `path` (string): Path to media file or folder
+
+**Example:**
+```
+analyze_media_quality(path="/path/to/audio")
+```
+
+---
+
+#### `generate_quality_report`
+Generate a comprehensive unified quality report for multimodal datasets.
+
+**Parameters:**
+- `dataset_id` (string): Dataset identifier
+- `dataset_path` (string): Path to dataset directory
+
+**Example:**
+```
+generate_quality_report(
+  dataset_id="my-dataset",
+  dataset_path="/path/to/data"
+)
+```
+
+---
+
+### Data Splitting
+
+#### `split_dataset`
+Split a dataset into train/test/validation sets.
+
+**Parameters:**
+- `dataset_id` (string): Dataset identifier
+- `train_ratio` (number): Training set ratio (0-1)
+- `test_ratio` (number): Test set ratio (0-1)
+- `val_ratio` (number, optional): Validation set ratio (0-1)
+
+**Example:**
+```
+split_dataset(
+  dataset_id="my-dataset",
+  train_ratio=0.7,
+  test_ratio=0.2,
+  val_ratio=0.1
+)
+```
+
+## 🏗️ Architecture
+
+Vesper is built with:
+- **TypeScript** for the MCP server
+- **Python** for image/audio/video processing
+- **SQLite** for metadata storage
+- **Transformers.js** for semantic search
+
+## 🤝 Contributing
+
+Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
+
+## 📄 License
+
+MIT License - see [LICENSE](LICENSE) for details.
+
+## 🐛 Issues & Support
+
+- **Issues**: https://github.com/vesper/mcp-server/issues
+- **Discussions**: https://github.com/vesper/mcp-server/discussions
+
+## 🌟 Acknowledgments
+
+Built with:
+- [Model Context Protocol](https://modelcontextprotocol.io/)
+- [HuggingFace Hub](https://huggingface.co/)
+- [Kaggle API](https://www.kaggle.com/docs/api)
+- [OpenCV](https://opencv.org/)
+- [librosa](https://librosa.org/)
+
+---
 
-MIT
+Made with ❤️ by the Vesper Team

package/build/cache/cdn.js

@@ -0,0 +1,34 @@
+import fs from "fs";
+import path from "path";
+export class CDNService {
+    baseDir;
+    baseUrl;
+    constructor(baseDir = "data/cdn_mock", baseUrl = "https://cdn.vesper.ai") {
+        this.baseDir = path.resolve(baseDir);
+        this.baseUrl = baseUrl;
+        if (!fs.existsSync(this.baseDir)) {
+            fs.mkdirSync(this.baseDir, { recursive: true });
+        }
+    }
+    /**
+     * Uploads a file to the CDN.
+     * @param fileName Name of the file (including extension)
+     * @param content String or Buffer content
+     * @returns The public URL of the file
+     */
+    async upload(fileName, content) {
+        const filePath = path.join(this.baseDir, fileName);
+        fs.writeFileSync(filePath, content);
+        // Return a simulated cloud URL
+        return `${this.baseUrl}/${fileName}`;
+    }
+    /**
+     * Deletes a file from the CDN.
+     */
+    async delete(fileName) {
+        const filePath = path.join(this.baseDir, fileName);
+        if (fs.existsSync(filePath)) {
+            fs.unlinkSync(filePath);
+        }
+    }
+}

package/build/cache/service.js

@@ -0,0 +1,63 @@
+import crypto from "crypto";
+/**
+ * A simple in-memory cache provider simulating Redis for demonstration.
+ */
+export class MockRedisProvider {
+    store = new Map();
+    async get(key) {
+        const item = this.store.get(key);
+        if (!item)
+            return null;
+        if (item.expiry && Date.now() > item.expiry) {
+            this.store.delete(key);
+            return null;
+        }
+        return item.value;
+    }
+    async set(key, value, ttlSeconds) {
+        const expiry = ttlSeconds ? Date.now() + ttlSeconds * 1000 : null;
+        this.store.set(key, { value, expiry });
+    }
+    async delete(key) {
+        this.store.delete(key);
+    }
+}
+export class CacheService {
+    provider;
+    constructor(provider) {
+        this.provider = provider;
+    }
+    /**
+     * Caches quality reports (TTL: 24h)
+     */
+    async getReport(datasetId) {
+        const key = `report:${datasetId}`;
+        const data = await this.provider.get(key);
+        return data ? JSON.parse(data) : null;
+    }
+    async saveReport(datasetId, report) {
+        const key = `report:${datasetId}`;
+        await this.provider.set(key, JSON.stringify(report), 86400); // 24 hours
+    }
+    /**
+     * Caches cleaning plans by dataset ID and configuration hash
+     */
+    async getPlan(datasetId, config) {
+        const hash = this.generateHash(config);
+        const key = `plan:${datasetId}:${hash}`;
+        const data = await this.provider.get(key);
+        return data ? JSON.parse(data) : null;
+    }
+    async savePlan(datasetId, config, plan) {
+        const hash = this.generateHash(config);
+        const key = `plan:${datasetId}:${hash}`;
+        await this.provider.set(key, JSON.stringify(plan), 3600); // 1 hour
+    }
+    generateHash(obj) {
+        return crypto
+            .createHash("sha256")
+            .update(JSON.stringify(obj))
+            .digest("hex")
+            .substring(0, 16);
+    }
+}

package/build/cleaning/cleaner.js

@@ -0,0 +1,81 @@
+import { spawn } from "child_process";
+import path from "path";
+import fs from "fs";
+export class DataCleaner {
+    pythonPath = "python";
+    scriptPath;
+    constructor(buildDir = process.cwd()) {
+        const homeDir = process.env.HOME || process.env.USERPROFILE || buildDir;
+        const dataRoot = path.join(homeDir, ".vesper");
+        const scriptPath0 = path.resolve(dataRoot, "python", "cleaner.py");
+        const scriptPath1 = path.resolve(buildDir, "python", "cleaner.py");
+        const scriptPath2 = path.resolve(buildDir, "..", "src", "python", "cleaner.py");
+        const scriptPath3 = path.resolve(buildDir, "..", "python", "cleaner.py");
+        if (fs.existsSync(scriptPath0)) {
+            this.scriptPath = scriptPath0;
+        }
+        else if (fs.existsSync(scriptPath1)) {
+            this.scriptPath = scriptPath1;
+        }
+        else if (fs.existsSync(scriptPath2)) {
+            this.scriptPath = scriptPath2;
+        }
+        else if (fs.existsSync(scriptPath3)) {
+            this.scriptPath = scriptPath3;
+        }
+        else {
+            this.scriptPath = scriptPath0; // Final fallback
+        }
+        // Detect Python command (Windows may use 'py' instead of 'python')
+        if (process.platform === "win32") {
+            this.pythonPath = "py";
+        }
+    }
+    /**
+     * Execute a list of cleaning operations on a file
+     */
+    async clean(filePath, operations, format) {
+        return new Promise((resolve, reject) => {
+            const args = [
+                this.scriptPath,
+                filePath,
+                JSON.stringify(operations)
+            ];
+            if (format)
+                args.push(format);
+            const process = spawn(this.pythonPath, args);
+            let stdout = "";
+            let stderr = "";
+            process.stdout.on("data", (data) => {
+                stdout += data.toString();
+            });
+            process.stderr.on("data", (data) => {
+                stderr += data.toString();
+            });
+            process.on("close", (code) => {
+                if (code !== 0) {
+                    reject(new Error(`Data Cleaner failed (code ${code}): ${stderr}`));
+                    return;
+                }
+                try {
+                    const result = JSON.parse(stdout);
+                    if (!result.success) {
+                        reject(new Error(result.error));
+                    }
+                    else {
+                        resolve({
+                            success: true,
+                            rows_affected: Number(result.rows_affected ?? 0),
+                            columns_affected: Number(result.columns_affected ?? 0),
+                            output_path: result.output_path,
+                            logs: Array.isArray(result.logs) ? result.logs : [],
+                        });
+                    }
+                }
+                catch (e) {
+                    reject(new Error(`Failed to parse cleaner output: ${stdout}`));
+                }
+            });
+        });
+    }
+}

package/build/cleaning/evaluator.js

@@ -0,0 +1,89 @@
+import * as crypto from "crypto";
+export class RuleEvaluator {
+    /**
+     * Checks if a record matches a rule's condition.
+     */
+    matches(record, condition) {
+        const columnsToTest = condition.column === "*"
+            ? Object.keys(record)
+            : [condition.column];
+        for (const col of columnsToTest) {
+            const val = record[col];
+            if (val === undefined)
+                continue;
+            if (this.testValue(val, condition)) {
+                return true;
+            }
+        }
+        return false;
+    }
+    /**
+     * Applies a rule action to a record.
+     */
+    apply(record, rule) {
+        const newRecord = { ...record };
+        const action = rule.action;
+        const condition = rule.condition;
+        const columnsToApply = condition.column === "*"
+            ? Object.keys(record)
+            : [condition.column];
+        for (const col of columnsToApply) {
+            const val = record[col];
+            if (val === undefined)
+                continue;
+            // Optional: Re-test condition per column if using wildcard
+            if (condition.column === "*" && !this.testValue(val, condition)) {
+                continue;
+            }
+            switch (action.type) {
+                case "Replace":
+                    if (typeof val === "string") {
+                        const regex = new RegExp(action.params.pattern, "g");
+                        newRecord[col] = val.replace(regex, action.params.replacement);
+                    }
+                    break;
+                case "CustomMask":
+                    if (val !== null) {
+                        newRecord[col] = this.maskValue(String(val), action.params);
+                    }
+                    break;
+                case "NormalizeText":
+                    if (typeof val === "string") {
+                        newRecord[col] = action.params.case === "lower"
+                            ? val.toLowerCase()
+                            : val.toUpperCase();
+                    }
+                    break;
+                // Add more handlers as needed...
+            }
+        }
+        return newRecord;
+    }
+    testValue(val, condition) {
+        const { operator, value } = condition;
+        const strVal = String(val);
+        switch (operator) {
+            case "contains":
+                return strVal.includes(String(value));
+            case "equals":
+                return val === value;
+            case "starts_with":
+                return strVal.startsWith(String(value));
+            case "ends_with":
+                return strVal.endsWith(String(value));
+            case "is_null":
+                return (val === null || val === undefined) === value;
+            case "matches_regex":
+                return new RegExp(String(value)).test(strVal);
+            default:
+                return false;
+        }
+    }
+    maskValue(val, params) {
+        if (params.method === "hash") {
+            const salt = params.salt || "";
+            return crypto.createHash("sha256").update(val + salt).digest("hex").substring(0, 12) + "...";
+        }
+        return "MASKED";
+    }
+}