tryll-dataset-builder-mcp 1.2.0 → 1.3.0

package/README.md

@@ -1,9 +1,11 @@
 # Tryll Dataset Builder — MCP Server
 
-An MCP (Model Context Protocol) server for building structured RAG knowledge base datasets. Use it with Claude Code to create, manage, and export JSON datasets via natural language.
+An MCP (Model Context Protocol) server for building structured RAG knowledge base datasets. Use it with Claude Code to create, manage, and export JSON datasets via natural language — with optional real-time sync to the [Dataset Builder web app](https://trylljsoncreator.onrender.com).
 
 Built by [Tryll Engine](https://tryllengine.com) | [Discord](https://discord.gg/CMnMrmapyB)
 
+---
+
 ## Quick Start
 
 ### 1. Install
@@ -14,8 +16,6 @@ npm install -g tryll-dataset-builder-mcp
 
 ### 2. Add to Claude Code
 
-Run in your terminal:
-
 ```bash
 claude mcp add dataset-builder -- npx tryll-dataset-builder-mcp
 ```
@@ -42,55 +42,208 @@ Just talk to Claude:
 
 > "Create a knowledge base about Minecraft with categories: Mobs, Blocks, Biomes. Add 10 chunks to each category."
 
-> "Import my existing dataset from ./data/minecraft.json"
+> "Parse this wiki page and add it to my dataset: https://minecraft.wiki/w/Creeper"
+
+> "Show me the version history of my project"
 
-> "Search for all chunks mentioning 'diamond' in my project"
+---
 
 ## Configuration
 
 | Variable | Default | Description |
 |----------|---------|-------------|
-| `DATA_DIR` | `./datasets` | Directory where project JSON files are stored |
+| `DATA_DIR` | `./datasets` | Directory for project JSON files (local mode) |
+
+---
+
+## Two Modes of Operation
+
+### Local Mode (default)
+Data is stored as JSON files in `DATA_DIR`. No server needed.
+
+### Connected Mode (real-time sync)
+Connect to the [Dataset Builder web app](https://trylljsoncreator.onrender.com) for live collaboration. Changes made via MCP appear instantly in the browser, and vice versa.
+
+```
+You: "Connect to session ABC123"
+Claude: *connects via WebSocket*
+You: "Add 5 chunks about dragons"
+→ chunks appear in the browser in real-time
+```
+
+---
 
-## Available Tools (18)
+## Available Tools (27)
+
+### Session Management
+
+| Tool | Description |
+|------|-------------|
+| `connect_session` | Connect to the web app for real-time collaboration. Requires a 6-character session code from the browser UI |
+| `disconnect_session` | Disconnect from the web app, switch back to local storage |
 
 ### Project Management
+
 | Tool | Description |
 |------|-------------|
 | `create_project` | Create a new dataset project |
 | `list_projects` | List all projects with stats |
-| `delete_project` | Delete a project |
-| `get_project_stats` | Detailed statistics |
+| `delete_project` | Permanently delete a project |
+| `get_project_stats` | Detailed statistics (categories, chunks, text lengths) |
 
 ### Category Management
+
 | Tool | Description |
 |------|-------------|
-| `create_category` | Add a category to a project |
+| `create_category` | Add a category to organize chunks |
 | `list_categories` | List categories with chunk counts |
 | `rename_category` | Rename a category |
-| `delete_category` | Delete a category and its chunks |
+| `delete_category` | Delete a category and all its chunks |
 
 ### Chunk Operations
+
 | Tool | Description |
 |------|-------------|
-| `add_chunk` | Add a single knowledge chunk |
-| `bulk_add_chunks` | Add multiple chunks at once |
-| `get_chunk` | Get chunk content by ID |
-| `update_chunk` | Update chunk fields |
-| `delete_chunk` | Delete a chunk |
-| `duplicate_chunk` | Clone a chunk |
-| `move_chunk` | Move chunk between categories |
+| `add_chunk` | Add a single knowledge chunk with ID, text, and metadata |
+| `bulk_add_chunks` | Add multiple chunks at once (faster than one by one) |
+| `get_chunk` | Get full content of a chunk by ID |
+| `update_chunk` | Update chunk fields (ID, text, metadata) |
+| `delete_chunk` | Delete a chunk by ID |
+| `duplicate_chunk` | Clone a chunk (creates `id_copy`) |
+| `move_chunk` | Move a chunk between categories |
 
 ### Search & Export
+
+| Tool | Description |
+|------|-------------|
+| `search_chunks` | Search by chunk ID or text content |
+| `export_project` | Export as flat JSON array (RAG-ready) |
+| `import_json` | Import an existing JSON dataset |
+| `export_category` | Export a single category as JSON |
+
+### URL Parsing
+
+| Tool | Description |
+|------|-------------|
+| `parse_url` | Fetch a web page, extract text, auto-create chunks. Splits text > 2000 chars into multiple chunks. Extracts wiki infobox metadata |
+| `batch_parse_urls` | Parse multiple URLs at once |
+
+### Bulk Operations
+
 | Tool | Description |
 |------|-------------|
-| `search_chunks` | Search by ID or text content |
-| `export_project` | Export as flat JSON (RAG-ready) |
-| `import_json` | Import existing JSON dataset |
+| `bulk_update_metadata` | Set a metadata field across all chunks (or per category) |
+| `merge_projects` | Merge all data from one project into another |
+
+### Version History
+
+| Tool | Description |
+|------|-------------|
+| `get_history` | Get version history (last 50 commits) for a project |
+| `get_commit` | Get a specific commit with full snapshot data for diffing |
+| `rollback` | Rollback a project to a previous commit's state |
+
+---
+
+## Tool Details
+
+### `add_chunk`
+
+```
+project: "minecraft"
+category: "Mobs"
+id: "creeper"
+text: "A Creeper is a hostile mob that silently approaches players..."
+metadata:
+  page_title: "Creeper"
+  source: "Minecraft Wiki"
+  license: "CC BY-NC-SA 3.0"
+  health: "20"          ← custom metadata field
+  behavior: "explodes"  ← custom metadata field
+```
+
+Standard metadata fields: `page_title`, `source`, `license`. Any extra fields become custom metadata.
+
+### `parse_url`
+
+```
+project: "minecraft"
+category: "Mobs"
+url: "https://minecraft.wiki/w/Creeper"
+chunk_id: "creeper"
+license: "CC BY-NC-SA 3.0"
+```
 
-## Export Format
+- Fetches the page, extracts main text content
+- If text > 2000 chars → auto-splits into `creeper_1`, `creeper_2`, etc.
+- Extracts page title and source URL as metadata
+- For wiki pages: extracts infobox/sidebar data as custom metadata fields
 
-The exported JSON is a flat array, compatible with the [Dataset Builder web app](https://github.com/Skizziik/json_creator) and ready for RAG pipelines:
+### `get_history`
+
+```
+project: "minecraft"
+```
+
+Returns:
+```json
+[
+  {
+    "id": "uuid",
+    "timestamp": "2026-02-27T14:30:00.000Z",
+    "source": "mcp",
+    "action": "addChunk",
+    "summary": "Added chunk 'creeper' to 'Mobs'",
+    "stats": { "categories": 3, "chunks": 12 }
+  }
+]
+```
+
+### `rollback`
+
+```
+project: "minecraft"
+commit_id: "uuid-of-target-commit"
+```
+
+Restores the project to that commit's snapshot. Creates a new "rollback" commit so you can undo the rollback later.
+
+---
+
+## Data Formats
+
+### Project JSON (internal)
+
+```json
+{
+  "name": "minecraft",
+  "createdAt": "2026-02-27T10:00:00.000Z",
+  "categories": [
+    {
+      "id": "uuid",
+      "name": "Mobs",
+      "expanded": true,
+      "chunks": [
+        {
+          "_uid": "uuid",
+          "id": "creeper",
+          "text": "A Creeper is a hostile mob...",
+          "metadata": {
+            "page_title": "Creeper",
+            "source": "Minecraft Wiki",
+            "license": "CC BY-NC-SA 3.0"
+          },
+          "customFields": [
+            { "key": "health", "value": "20" }
+          ]
+        }
+      ]
+    }
+  ]
+}
+```
+
+### Export Format (RAG-ready)
 
 ```json
 [
@@ -101,20 +254,67 @@ The exported JSON is a flat array, compatible with the [Dataset Builder web app]
       "page_title": "Creeper",
       "source": "Minecraft Wiki",
       "license": "CC BY-NC-SA 3.0",
-      "type": "hostile_mob",
       "health": "20"
     }
   }
 ]
 ```
 
+### History Commit
+
+```json
+{
+  "id": "uuid",
+  "timestamp": "2026-02-27T14:30:00.000Z",
+  "source": "browser | mcp",
+  "action": "addChunk",
+  "summary": "Added chunk 'creeper' to 'Mobs'",
+  "stats": { "categories": 3, "chunks": 12 },
+  "snapshot": { "...full project state..." }
+}
+```
+
+---
+
+## Real-Time Collaboration
+
+```
+┌─────────────┐     WebSocket      ┌──────────────┐     REST API     ┌─────────────┐
+│   Browser    │ ◄──────────────► │  Web Server   │ ◄──────────────► │  MCP Server  │
+│  (Dataset    │   data:changed    │  (Express +   │   POST/PUT/DEL   │  (Claude     │
+│   Builder)   │   mcp:connected   │   WebSocket)  │   + source:mcp   │   Code)      │
+└─────────────┘                    └──────────────┘                    └─────────────┘
+```
+
+1. Open the [Dataset Builder](https://trylljsoncreator.onrender.com) in your browser
+2. Copy the 6-character session code from the top bar
+3. Tell Claude: *"Connect to session ABC123"*
+4. All changes sync in real-time between browser and Claude
+5. Version history tracks who made each change (browser vs MCP)
+
+---
+
 ## Example Prompts
 
 - *"Create a Dark Souls knowledge base with categories for Bosses, Weapons, and Locations"*
-- *"Add 15 chunks about Minecraft mobs with detailed descriptions"*
-- *"Export my project as JSON and save to file"*
-- *"Search for chunks about 'fire' in my dark_souls project"*
-- *"Move chunk 'ancient_dragon' from Bosses to Enemies category"*
+- *"Parse these wiki pages and add them to my Minecraft project: [url1], [url2], [url3]"*
+- *"Bulk update the license field to 'MIT' for all chunks in the Mobs category"*
+- *"Show me the version history of my project"*
+- *"Rollback my project to the commit before I deleted that category"*
+- *"Merge my test_data project into the main production project"*
+- *"Export the Bosses category as JSON"*
+- *"Connect to session XYZ789 and add 20 chunks about potions"*
+
+---
+
+## Links
+
+- **Web App**: [trylljsoncreator.onrender.com](https://trylljsoncreator.onrender.com)
+- **Web App Repo**: [github.com/Skizziik/json_creator](https://github.com/Skizziik/json_creator)
+- **MCP Repo**: [github.com/Skizziik/tryll_dataset_builder](https://github.com/Skizziik/tryll_dataset_builder)
+- **npm**: [tryll-dataset-builder-mcp](https://www.npmjs.com/package/tryll-dataset-builder-mcp)
+- **Tryll Engine**: [tryllengine.com](https://tryllengine.com)
+- **Discord**: [discord.gg/CMnMrmapyB](https://discord.gg/CMnMrmapyB)
 
 ## License
 

package/index.js

@@ -10,7 +10,7 @@ import * as cheerio from "cheerio";
 const store = new Store(process.env.DATA_DIR);
 
 const server = new Server(
-  { name: "tryll-dataset-builder", version: "1.2.0" },
+  { name: "tryll-dataset-builder", version: "1.3.0" },
   { capabilities: { tools: {} } }
 );
 
@@ -483,6 +483,43 @@ const TOOLS = [
       required: ["project", "category"],
     },
   },
+
+  // ---- History ----
+  {
+    name: "get_history",
+    description: "Get version history (last 50 commits) for a project. Each commit shows who made the change (browser/MCP), what was changed, and when. Returns lightweight list without snapshots.",
+    inputSchema: {
+      type: "object",
+      properties: {
+        project: { type: "string", description: "Project name" },
+      },
+      required: ["project"],
+    },
+  },
+  {
+    name: "get_commit",
+    description: "Get a specific commit with full snapshot data. Returns the commit's snapshot and the previous commit's snapshot for computing diffs.",
+    inputSchema: {
+      type: "object",
+      properties: {
+        project: { type: "string", description: "Project name" },
+        commit_id: { type: "string", description: "Commit UUID" },
+      },
+      required: ["project", "commit_id"],
+    },
+  },
+  {
+    name: "rollback",
+    description: "Rollback a project to a specific commit's state. Restores the project data from that commit's snapshot and creates a new 'rollback' commit in history. Safe: you can undo a rollback by rolling back to a later commit.",
+    inputSchema: {
+      type: "object",
+      properties: {
+        project: { type: "string", description: "Project name" },
+        commit_id: { type: "string", description: "Commit UUID to rollback to" },
+      },
+      required: ["project", "commit_id"],
+    },
+  },
 ];
 
 // ============================================
@@ -501,28 +538,28 @@ async function handleRemote(name, args) {
 
   switch (name) {
     case "create_project":
-      return apiCall('POST', '/api/projects', { name: args.name, session: s });
+      return apiCall('POST', '/api/projects', { name: args.name, session: s, source: 'mcp' });
     case "list_projects":
       return apiCall('GET', '/api/projects');
     case "delete_project":
-      return apiCall('DELETE', `/api/projects/${p(args.name)}?session=${s}`);
+      return apiCall('DELETE', `/api/projects/${p(args.name)}?session=${s}&source=mcp`);
     case "get_project_stats":
       return apiCall('GET', `/api/projects/${p(args.name)}/stats`);
     case "create_category":
-      return apiCall('POST', `/api/projects/${p(args.project)}/categories`, { name: args.name, session: s });
+      return apiCall('POST', `/api/projects/${p(args.project)}/categories`, { name: args.name, session: s, source: 'mcp' });
     case "list_categories":
       return apiCall('GET', `/api/projects/${p(args.project)}/categories`);
     case "rename_category":
-      return apiCall('PUT', `/api/projects/${p(args.project)}/categories/${p(args.old_name)}`, { newName: args.new_name, session: s });
+      return apiCall('PUT', `/api/projects/${p(args.project)}/categories/${p(args.old_name)}`, { newName: args.new_name, session: s, source: 'mcp' });
     case "delete_category":
-      return apiCall('DELETE', `/api/projects/${p(args.project)}/categories/${p(args.name)}?session=${s}`);
+      return apiCall('DELETE', `/api/projects/${p(args.project)}/categories/${p(args.name)}?session=${s}&source=mcp`);
     case "add_chunk":
       return apiCall('POST', `/api/projects/${p(args.project)}/categories/${p(args.category)}/chunks`, {
-        id: args.id, text: args.text, metadata: args.metadata, session: s,
+        id: args.id, text: args.text, metadata: args.metadata, session: s, source: 'mcp',
       });
     case "bulk_add_chunks":
       return apiCall('POST', `/api/projects/${p(args.project)}/categories/${p(args.category)}/chunks/bulk`, {
-        chunks: args.chunks, session: s,
+        chunks: args.chunks, session: s, source: 'mcp',
       });
     case "get_chunk": {
       const proj = await apiCall('GET', `/api/projects/${p(args.project)}`);
@@ -537,7 +574,7 @@ async function handleRemote(name, args) {
       for (const cat of proj2.categories) {
         const ch = cat.chunks.find(c => c.id === args.id);
         if (ch) {
-          const body = { session: s };
+          const body = { session: s, source: 'mcp' };
           if (args.new_id !== undefined) body.id = args.new_id;
           if (args.text !== undefined) body.text = args.text;
           const meta = {};
@@ -558,7 +595,7 @@ async function handleRemote(name, args) {
       for (const cat of proj3.categories) {
         const ch = cat.chunks.find(c => c.id === args.id);
         if (ch) {
-          return apiCall('DELETE', `/api/projects/${p(args.project)}/categories/${cat.id}/chunks/${ch._uid}?session=${s}`);
+          return apiCall('DELETE', `/api/projects/${p(args.project)}/categories/${cat.id}/chunks/${ch._uid}?session=${s}&source=mcp`);
         }
       }
       throw new Error(`Chunk "${args.id}" not found`);
@@ -568,14 +605,14 @@ async function handleRemote(name, args) {
       for (const cat of proj4.categories) {
         const ch = cat.chunks.find(c => c.id === args.id);
         if (ch) {
-          return apiCall('POST', `/api/projects/${p(args.project)}/categories/${cat.id}/chunks/${ch._uid}/duplicate`);
+          return apiCall('POST', `/api/projects/${p(args.project)}/categories/${cat.id}/chunks/${ch._uid}/duplicate`, { source: 'mcp' });
         }
       }
       throw new Error(`Chunk "${args.id}" not found`);
     }
     case "move_chunk":
       return apiCall('POST', `/api/projects/${p(args.project)}/chunks/${p(args.id)}/move`, {
-        targetCategory: args.target_category, session: s,
+        targetCategory: args.target_category, session: s, source: 'mcp',
       });
     case "search_chunks":
       return apiCall('GET', `/api/projects/${p(args.project)}/search?q=${encodeURIComponent(args.query)}`);
@@ -589,7 +626,7 @@ async function handleRemote(name, args) {
       }
       if (!jsonData) throw new Error('Provide either "json_path" or "data" parameter');
       return apiCall('POST', `/api/projects/${p(args.project)}/import`, {
-        data: jsonData, category: args.category, session: s,
+        data: jsonData, category: args.category, session: s, source: 'mcp',
       });
     }
     case "parse_url": {
@@ -601,7 +638,7 @@ async function handleRemote(name, args) {
         metadata: { page_title: parsed.pageTitle, source: parsed.source, license, ...parsed.infobox },
       }));
       const result = await apiCall('POST', `/api/projects/${p(args.project)}/categories/${p(args.category)}/chunks/bulk`, {
-        chunks: chunkData, session: s,
+        chunks: chunkData, session: s, source: 'mcp',
       });
       return { ...result, pageTitle: parsed.pageTitle, chunksCreated: chunks.length, infoboxFields: Object.keys(parsed.infobox) };
     }
@@ -617,7 +654,7 @@ async function handleRemote(name, args) {
             metadata: { page_title: parsed.pageTitle, source: parsed.source, license, ...parsed.infobox },
           }));
           const r = await apiCall('POST', `/api/projects/${p(args.project)}/categories/${p(args.category)}/chunks/bulk`, {
-            chunks: chunkData, session: s,
+            chunks: chunkData, session: s, source: 'mcp',
           });
           results.push({ url: entry.url, chunk_id: entry.chunk_id, chunks: chunks.length, added: r.added, errors: r.errors });
         } catch (err) {
@@ -628,14 +665,22 @@ async function handleRemote(name, args) {
     }
     case "bulk_update_metadata":
       return apiCall('POST', `/api/projects/${p(args.project)}/bulk-metadata`, {
-        field: args.field, value: args.value, category: args.category, session: s,
+        field: args.field, value: args.value, category: args.category, session: s, source: 'mcp',
       });
     case "merge_projects":
       return apiCall('POST', `/api/projects/${p(args.source)}/merge`, {
-        target: args.target, session: s,
+        target: args.target, session: s, source: 'mcp',
       });
     case "export_category":
       return apiCall('GET', `/api/projects/${p(args.project)}/categories/${p(args.category)}/export`);
+    case "get_history":
+      return apiCall('GET', `/api/projects/${p(args.project)}/history`);
+    case "get_commit":
+      return apiCall('GET', `/api/projects/${p(args.project)}/history/${args.commit_id}`);
+    case "rollback":
+      return apiCall('POST', `/api/projects/${p(args.project)}/history/${args.commit_id}/rollback`, {
+        session: s, source: 'mcp',
+      });
     default:
       throw new Error(`Unknown tool: ${name}`);
   }
@@ -866,6 +911,18 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {
           break;
         }
 
+        case "get_history":
+          result = store.getHistory(args.project);
+          break;
+
+        case "get_commit":
+          result = store.getCommit(args.project, args.commit_id);
+          break;
+
+        case "rollback":
+          result = store.rollback(args.project, args.commit_id, 'mcp');
+          break;
+
         default:
           throw new Error(`Unknown tool: ${name}`);
       }
@@ -890,7 +947,7 @@ server.setRequestHandler(CallToolRequestSchema, async (request) => {
 async function main() {
   const transport = new StdioServerTransport();
   await server.connect(transport);
-  console.error("Tryll Dataset Builder MCP server running (v1.2.0)");
+  console.error("Tryll Dataset Builder MCP server running (v1.3.0)");
 }
 
 main().catch((err) => {

package/lib/store.js

@@ -4,6 +4,7 @@ import { randomUUID } from 'crypto';
 
 const DEFAULT_LICENSE = 'CC BY-NC-SA 3.0';
 const STANDARD_META = ['page_title', 'source', 'license'];
+const MAX_HISTORY = 50;
 
 export class Store {
   constructor(dataDir) {
@@ -25,7 +26,7 @@ export class Store {
 
   listProjects() {
     this._ensureDir();
-    const files = readdirSync(this.dataDir).filter(f => f.endsWith('.json'));
+    const files = readdirSync(this.dataDir).filter(f => f.endsWith('.json') && !f.endsWith('.history.json'));
     return files.map(f => {
       const name = f.replace(/\.json$/, '');
       try {
@@ -439,6 +440,67 @@ export class Store {
     return false;
   }
 
+  // ---- HISTORY ----
+
+  _historyFilePath(name) {
+    return join(this.dataDir, `${name}.history.json`);
+  }
+
+  _loadHistory(name) {
+    const fp = this._historyFilePath(name);
+    if (!existsSync(fp)) return { project: name, commits: [] };
+    return JSON.parse(readFileSync(fp, 'utf-8'));
+  }
+
+  _saveHistory(name, history) {
+    this._ensureDir();
+    writeFileSync(this._historyFilePath(name), JSON.stringify(history, null, 2), 'utf-8');
+  }
+
+  _commit(projectName, action, summary, source) {
+    try {
+      const data = this._load(projectName);
+      const history = this._loadHistory(projectName);
+      const totalChunks = data.categories.reduce((sum, c) => sum + c.chunks.length, 0);
+      history.commits.unshift({
+        id: randomUUID(),
+        timestamp: new Date().toISOString(),
+        source: source || 'mcp',
+        action, summary,
+        stats: { categories: data.categories.length, chunks: totalChunks },
+        snapshot: JSON.parse(JSON.stringify(data)),
+      });
+      if (history.commits.length > MAX_HISTORY) history.commits.length = MAX_HISTORY;
+      this._saveHistory(projectName, history);
+    } catch { /* history logging should never break mutations */ }
+  }
+
+  getHistory(name) {
+    const history = this._loadHistory(name);
+    return history.commits.map(c => ({
+      id: c.id, timestamp: c.timestamp, source: c.source,
+      action: c.action, summary: c.summary, stats: c.stats,
+    }));
+  }
+
+  getCommit(name, commitId) {
+    const history = this._loadHistory(name);
+    const idx = history.commits.findIndex(c => c.id === commitId);
+    if (idx === -1) throw new Error('Commit not found');
+    const commit = history.commits[idx];
+    const prev = idx + 1 < history.commits.length ? history.commits[idx + 1].snapshot : null;
+    return { ...commit, prevSnapshot: prev };
+  }
+
+  rollback(name, commitId, source) {
+    const history = this._loadHistory(name);
+    const commit = history.commits.find(c => c.id === commitId);
+    if (!commit) throw new Error('Commit not found');
+    this._save(name, commit.snapshot);
+    this._commit(name, 'rollback', `Rolled back to commit from ${commit.timestamp}`, source || 'mcp');
+    return this._load(name);
+  }
+
   _parseCustomFields(metadata) {
     if (!metadata || typeof metadata !== 'object') return [];
     return Object.entries(metadata)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "tryll-dataset-builder-mcp",
-  "version": "1.2.0",
+  "version": "1.3.0",
   "description": "MCP server for building RAG knowledge base datasets. Create, manage and export structured JSON datasets via Claude Code.",
   "type": "module",
   "main": "index.js",