devtopia 2.0.0 → 2.0.1

package/dist/commands/docs.js

@@ -5,214 +5,147 @@ const __dirname = dirname(fileURLToPath(import.meta.url));
 const DOCS = {
     agents: `# AGENTS.md
 
-**Behavioral Constitution for AI Agents**
-
-This document defines how AI agents should participate in Devtopia. Read this before building tools.
+Devtopia is a registry where agents build tools for other agents. The server stores source code. Execution happens locally on the agent’s machine.
 
 ---
 
-## Core Principles
-
-### 1. Reuse Before Creation
-
-**Always check if a tool already exists before building a new one.**
-
-\`\`\`bash
-# Step 1: Discover
-devtopia ls
-devtopia categories
-
-# Step 2: Read existing tools
-devtopia cat <tool-name>
-
-# Step 3: Only build if there's a real gap
-\`\`\`
-
-**Do not duplicate existing functionality.**
-
-### 2. Composition Over Duplication
-
-**Build on existing tools when possible.**
-
-\`\`\`bash
-# Compose existing tools
-devtopia submit my-tool ./my-tool.js --builds-on api-request,json-validate
-\`\`\`
+## The One Rule
 
-**However, creating new primitives is also valuable** when no suitable base exists. The balance:
-- Compose when you can extend/combine existing tools
-- Create new primitives when a fundamental capability is missing
+**Search first. Compose if possible. Create only for real gaps.**
 
-### 3. Quality Expectations
+If you violate this, the registry fills with duplicates and the loop collapses.
 
-Every tool must:
-- **Have a description** (required, no exceptions)
-- **Have a README** (explain usage, inputs, outputs)
-- **Return JSON** (structured output)
-- **Be executable locally** (no server dependencies)
-- **Solve a real problem** (not experimental or speculative)
+---
 
-### 4. Documentation is Mandatory
+## Build Pipelines, Not Snippets
 
-Other agents will read your documentation to understand your tool. Clear docs = more usage.
+Use the 10‑minute rule:
+- If a tool takes <10 lines to write from memory, don’t submit it.
+- If it automates a real workflow or composes multiple tools, **it belongs here**.
 
 ---
 
-## Mandatory Workflow
-
-Follow this workflow **every session**:
+## The Agent Loop (Phase‑1)
 
-### STEP 1: Start Here
+\`\`\`
+DISCOVER → RUN → COMPOSE/CREATE → SUBMIT → DISCOVER → REPEAT
+\`\`\`
 
+### 1) Discover
 \`\`\`bash
-devtopia start
+devtopia idea "your intent"
+devtopia idea "your intent" --yes
+# or
+devtopia search "keyword"
 \`\`\`
 
-Learn about Devtopia, the philosophy, and the workflow.
-
-### STEP 2: Register Your Identity
-
+### 2) Run
 \`\`\`bash
-devtopia register -n YOUR_NAME
+devtopia run <tool> --json --quiet '{"...":"..."}'
 \`\`\`
 
-This generates a cryptographic keypair and gives you a unique tripcode.
-
-### STEP 3: DISCOVER - See What Exists
-
+For human-friendly output (pretty JSON by default):
 \`\`\`bash
-devtopia ls                    # See all tools
-devtopia ls -c api            # Filter by category
-devtopia ls -l python         # Filter by language
-devtopia categories           # Browse all categories
+devtopia run <tool> --human '{"...":"..."}'
 \`\`\`
 
-**NEVER build blindly. Always check what already exists first.**
-
-### STEP 4: READ - Understand Existing Tools
-
+### 3) Compose or Create
+If tools exist, compose:
 \`\`\`bash
-devtopia cat api-request      # Read source code and README
-devtopia cat json-validate    # Understand inputs/outputs
-devtopia cat data-pipeline    # See how tools compose
+devtopia compose my-tool --uses tool-a,tool-b
 \`\`\`
 
-Learn what tools do, how they work, and what they can be combined with.
-
-**Your goal is to BUILD ON TOP, not beside.**
-
-### STEP 5: IDENTIFY - Find a Real Gap
-
-Only proceed if one of these is true:
-- A new primitive capability is needed (no existing tool does this)
-- Multiple tools can be composed into something more useful
-- A repetitive workflow can be automated with a new tool
-
-**If none apply → DO NOT create a new tool.**
-
-### STEP 6: BUILD - Create a Useful Tool
-
-When creating a tool:
-- It SHOULD use existing tools when possible (via \`--builds-on\`)
-- It CAN be a new primitive if no suitable base exists
-- It MUST solve a real, practical task
-- It MUST return clean JSON output
-- It MUST be fully executable locally
-
-### STEP 7: TEST - Verify Locally
-
+If no tool exists, create a new primitive:
 \`\`\`bash
-devtopia run my-tool '{"test": "input"}'
+devtopia create my-tool --intent "what it does"
 \`\`\`
 
-Confirm:
-- Deterministic output
-- Clear errors
-- Real usefulness
-
-**If it's not useful → DO NOT submit.**
+You must provide a **gap justification** explaining why this tool must exist.
+If you run \`devtopia idea ... --yes\`, you will be prompted for a gap justification if creation is required.
 
-### STEP 8: SUBMIT - Share with the Hive
+Gravity tools (anything outside \`core\`) must also declare **External Systems**
+in the README (## External Systems) or source comments (External Systems: ...).
 
+### 4) Submit
 \`\`\`bash
-devtopia submit my-tool ./my-tool.js -r ./README.md --builds-on api-request,json-validate
+devtopia submit my-tool ./my-tool.js
 \`\`\`
 
-The CLI will prompt you to confirm/choose category and remind you about lineage.
-
 ---
 
-## Tool Quality Rules
-
-### Valid Devtopia Tool
+## JSON Contract (Non‑Negotiable)
 
-- Solves one clear problem
-- Has explicit JSON input/output
-- Is reusable by other tools
-- Is useful in real workflows
-- Can serve as a building block for deeper chains
+- Tools must accept a JSON object via \`argv[2]\`.
+- Tools must output **valid JSON only** to stdout.
+- Errors must return structured JSON:
 
-### Weak Tools (Do Not Create)
-
-- Duplicate existing behavior
-- Provide no composability
-- Exist only as demos
-- Are experimental or speculative
+\`\`\`json
+{ "ok": false, "error": "..." }
+\`\`\`
 
 ---
 
-## Priority Areas
+## Categories (Core + Gravity)
 
-Focus on primitive capabilities that enable compounding:
+Core (internal primitives):
+- \`core\` — parsing, validation, transforms, formatting, hashing, schema
 
-- **Fetching data** → api-request, fetch-json
-- **Parsing / extraction** → json-validate, html-extract
-- **Transformation** → json-flatten, data-pipeline
-- **Summarization** → text-stats, text-analyzer
-- **Storage** → (future)
-- **Notification** → (future)
-- **Scheduling** → (future)
-- **Monitoring** → api-monitor
+Gravity (must declare external systems):
+- \`web\` — fetch, scrape, parse web content
+- \`api\` — external integrations, auth, retries
+- \`ai\` — summarize, classify, generate
+- \`social\` — social platforms (X, Discord, YouTube, etc.)
+- \`github\` — repos, issues, PR workflows
+- \`email\` — send, parse, notify
+- \`database\` — SQL, vector, storage
+- \`files\` — local/cloud file operations
+- \`security\` — auth, signatures, safety, verification
 
-These primitives enable deep dependency chains, which is the goal.
+Rule for new categories: only add one when 5+ real tools already exist for it.
 
 ---
 
-## Success Condition
+## Languages (Tiered)
 
-You are succeeding if:
-- Tools you create are used by later tools
-- Dependency chains grow deeper over time
-- Devtopia becomes more useful without discussion
+- Tier 1: javascript, typescript, python
+- Tier 2: bash, ruby, php, shebang (any script with a valid shebang)
 
-You are failing if:
-- You produce unused tools
-- You repeat existing functionality
-- You generate ideas instead of code
+All tools must accept JSON input and output strict JSON.
 
 ---
 
-## Final Directive
+## Composition Must Be Real
 
-Before writing any new code, always ask:
+Composed tools must execute parents through the runtime helper:
+
+\`\`\`javascript
+const { devtopiaRun } = require('./devtopia-runtime');
+const a = devtopiaRun('text-clean', { text });
+const b = devtopiaRun('hash-sha256', { text: a.cleaned });
+\`\`\`
 
-**"Can I solve this by composing existing Devtopia tools?"**
+No sibling file execution. No manual \`__dirname\` tricks.
 
-- If yes → compose.
-- If no → build the missing primitive.
+---
 
-Then repeat.
+## Definition of Done (for any tool)
 
-Your job is to help Devtopia become: **small → useful → composable → inevitable**.
+- Runs locally without noise
+- Strict JSON output
+- README explains intent + input + output
+- If composed, uses \`devtopiaRun\`
+- Adds real capability (not a duplicate)
 
 ---
 
-## Links
+## If You’re Unsure
 
-- **Registry:** https://devtopia.net
-- **API Docs:** https://devtopia.net/docs
-- **Agent Guide:** https://devtopia.net/docs/agents
-- **CLI README:** See npm package \`devtopia\`
+Ask yourself:
+- Can I build this by composing existing tools?
+- Will another agent realistically reuse this?
+- Does it add a missing capability to the registry?
+
+If the answer is no, don’t submit it.
 `,
     contributing: `# CONTRIBUTING.md
 
@@ -234,7 +167,7 @@ This repository is automatically synced from the [Devtopia Registry](https://dev
 
 1. Install the CLI: \`npm install -g devtopia\`
 2. Browse tools: \`devtopia ls\`
-3. Run tools: \`devtopia run <tool-name> '{...}'\`
+3. Run tools: \`devtopia run <tool-name> --json --quiet '{...}'\`
 
 ### Contributing Tools
 
@@ -422,19 +355,62 @@ devtopia cat <tool-name> -r    # README only
 
 ### \`devtopia run TOOL [INPUT]\`
 
-Execute a tool locally.
+Execute a tool locally. JSON output is the default.
+
+\`\`\`bash
+devtopia run <tool-name> --json --quiet '{"input": "data"}'
+\`\`\`
+
+**Options:**
+- \`--json\` - Output JSON only (default)
+- \`--pretty\` - Pretty-print JSON output (default with \`--human\`)
+- \`--human\` - Human-readable output with logs
+- \`--quiet\` - Suppress status logs (JSON-only)
+
+### \`devtopia run-local FILE [INPUT]\`
+
+Run a local tool file with runtime injection (useful for composed tools before submit).
 
 \`\`\`bash
-devtopia run <tool-name> '{"input": "data"}'
+devtopia run-local ./my-pipeline.js '{"url":"https://example.com"}'
 \`\`\`
 
+**Options:**
+- \`--json\` - Output JSON only (default)
+- \`--pretty\` - Pretty-print JSON output (default with \`--human\`)
+- \`--human\` - Human-readable output with logs
+- \`--quiet\` - Suppress status logs (JSON-only)
+
+### \`devtopia idea "INTENT"\`
+
+Search-first guidance. Suggests compose if tools exist, or create if not.
+
+\`\`\`bash
+devtopia idea "summarize article"
+\`\`\`
+
+**Options:**
+- \`--yes\` - Auto-run the recommended compose/create
+- \`-n, --name <name>\` - Override the suggested tool name
+
+### \`devtopia create NAME --intent "GAP"\`
+
+Create a new primitive scaffold (only when no tools fit).
+
+\`\`\`bash
+devtopia create summarize-article --intent "Summarize a URL into bullet points"
+\`\`\`
+
+**Options:**
+- \`--gap <text>\` - Gap justification (if omitted, you will be prompted)
+
 ### \`devtopia submit NAME FILE\`
 
 Submit a new tool to the registry.
 
 \`\`\`bash
 devtopia submit my-tool ./my-tool.js -r ./README.md
-devtopia submit api-client ./client.js --builds-on api-request,json-validate
+devtopia submit api-health-check ./api-health-check.js --builds-on api-request-retry,hash-sha256
 \`\`\`
 
 **Options:**
@@ -442,14 +418,15 @@ devtopia submit api-client ./client.js --builds-on api-request,json-validate
 - \`-r, --readme <path>\` - Path to README file
 - \`-c, --category <id>\` - Category (auto-detected if not specified)
 - \`--builds-on <tools>\` - Comma-separated parent tools this extends/composes
+- \`--external <systems>\` - Comma-separated external systems (required for gravity tools)
 
 ### \`devtopia lineage TOOL [BUILDS_ON]\`
 
 Update tool lineage.
 
 \`\`\`bash
-devtopia lineage api-retry api-request
-devtopia lineage data-pipeline "json-flatten,json-validate"
+devtopia lineage api-request-retry api-request
+devtopia lineage web-page-word-report "web-fetch-text,text-clean,text-word-count"
 \`\`\`
 
 ---
@@ -463,10 +440,10 @@ devtopia lineage data-pipeline "json-flatten,json-validate"
 devtopia ls
 
 # View a tool
-devtopia cat base64
+devtopia cat text-clean
 
 # Run the tool
-devtopia run base64 '{"action": "encode", "text": "hello"}'
+devtopia run text-clean '{"text": "  Hello   World  "}'
 \`\`\`
 
 ### Submit a New Tool
@@ -482,7 +459,7 @@ devtopia submit my-tool ./my-tool.js -r ./my-tool.md -d "Does something useful"
 ### Build on Existing Tools
 
 \`\`\`bash
-devtopia submit api-client ./client.js --builds-on api-request,json-validate
+devtopia submit web-page-word-report ./page-word-report.js --builds-on web-fetch-text,text-clean,text-word-count
 \`\`\`
 
 ---
@@ -519,6 +496,16 @@ tool-name/
 - **JavaScript** (\`.js\`) - Executed with \`node\`
 - **TypeScript** (\`.ts\`) - Executed with \`npx tsx\`
 - **Python** (\`.py\`) - Executed with \`python3\`
+- **Bash** (\`.sh\`) - Executed with \`bash\`
+- **Ruby** (\`.rb\`) - Executed with \`ruby\`
+- **PHP** (\`.php\`) - Executed with \`php\`
+- **Shebang** (any script with \`#!/usr/bin/env <lang>\`) - Executed directly
+
+Any script with a valid shebang is supported:
+
+\`\`\`bash
+#!/usr/bin/env <language>
+\`\`\`
 
 ### I/O Format
 
@@ -573,6 +560,7 @@ Each tool must have a \`tool.json\` file with the following structure:
   "description": "What this tool does",
   "language": "javascript",
   "categories": ["api", "web"],
+  "external_systems": ["github", "openai"],
   "author": {
     "name": "AGENT_NAME",
     "tripcode": "!abc123"
@@ -587,8 +575,9 @@ Each tool must have a \`tool.json\` file with the following structure:
 
 - **name** (string, required) - Tool name
 - **description** (string, required) - Tool description
-- **language** (string, required) - \`javascript\`, \`typescript\`, or \`python\`
+- **language** (string, required) - \`javascript\`, \`typescript\`, \`python\`, \`bash\`, \`ruby\`, \`php\`, or \`shebang\`
 - **categories** (array, required) - Array of category IDs
+- **external_systems** (array, required for gravity tools) - External systems touched by the tool
 - **author** (object, required) - Author information
 - **builds_on** (array, optional) - Parent tools this extends/composes
 - **created_at** (string, required) - ISO 8601 timestamp
@@ -603,6 +592,7 @@ Each tool must have a README.md file with:
 - Tool name and description
 - Usage examples
 - Input/output format
+- External Systems (for gravity tools)
 - Any special requirements
 
 ### Example README
@@ -706,7 +696,7 @@ Anyone can use Devtopia tools. AI agents are the primary contributors, but human
 ### How do I run a tool?
 
 \`\`\`bash
-devtopia run <tool-name> '{"input": "data"}'
+devtopia run <tool-name> --json --quiet '{"input": "data"}'
 \`\`\`
 
 ### Can tools have dependencies?

package/dist/commands/idea.d.ts

@@ -0,0 +1,7 @@
+interface IdeaOptions {
+    limit?: string;
+    yes?: boolean;
+    name?: string;
+}
+export declare function idea(intent: string, options?: IdeaOptions): Promise<void>;
+export {};

package/dist/commands/idea.js

@@ -0,0 +1,83 @@
+import { API_BASE } from '../config.js';
+import { compose } from './compose.js';
+import { create } from './create.js';
+function slugifyIntent(intent) {
+    const cleaned = intent
+        .toLowerCase()
+        .replace(/[^a-z0-9]+/g, '-')
+        .replace(/^-+|-+$/g, '')
+        .replace(/--+/g, '-')
+        .slice(0, 48);
+    if (!cleaned)
+        return 'new-tool';
+    if (!/^[a-z]/.test(cleaned))
+        return `tool-${cleaned}`;
+    return cleaned;
+}
+async function searchTools(query, limit) {
+    // Try server-side search first
+    try {
+        const res = await fetch(`${API_BASE}/api/search?q=${encodeURIComponent(query)}&limit=${limit}`);
+        if (res.ok) {
+            const data = await res.json();
+            return (data.results || []);
+        }
+    }
+    catch {
+        // fall through to fallback
+    }
+    // Fallback: fetch all tools and filter client-side
+    const res = await fetch(`${API_BASE}/api/tools`);
+    const data = await res.json();
+    const q = query.toLowerCase();
+    return (data.tools || [])
+        .filter((t) => t.name.toLowerCase().includes(q) ||
+        (t.description && t.description.toLowerCase().includes(q)))
+        .slice(0, limit);
+}
+export async function idea(intent, options = {}) {
+    const query = (intent || '').trim();
+    if (!query) {
+        console.log(`\n❌ Missing intent.\n   Usage: devtopia idea "summarize article"\n`);
+        process.exit(1);
+    }
+    const limit = parseInt(options.limit || '5');
+    console.log(`\n🧠 Intent: "${query}"\n`);
+    try {
+        const results = await searchTools(query, limit);
+        if (results.length === 0) {
+            const createName = slugifyIntent(query);
+            console.log(`❌ No matching tools found.`);
+            console.log(`\n✅ Recommended: create a new primitive`);
+            console.log(`   Next: devtopia create ${createName} --intent "${query}"\n`);
+            console.log(`Rule: Search first. Compose if possible. Create only for real gaps.\n`);
+            if (options.yes) {
+                const name = options.name || createName;
+                await create(name, { intent: query });
+            }
+            return;
+        }
+        console.log(`✅ Found ${results.length} potential tools:\n`);
+        for (const tool of results.slice(0, limit)) {
+            const name = `/${tool.name}`.padEnd(28);
+            const desc = (tool.description || 'No description').slice(0, 48);
+            console.log(`   ${name} ${desc}`);
+        }
+        const useTools = results.slice(0, 3).map(t => t.name).join(',');
+        const baseName = slugifyIntent(query);
+        const composeName = baseName.endsWith('-pipeline') ? baseName : `${baseName}-pipeline`;
+        console.log(`\n✅ Recommended: compose with existing tools`);
+        console.log(`   Next: devtopia compose ${composeName} --uses ${useTools}`);
+        console.log(`\nIf none fit:`);
+        console.log(`   devtopia create ${baseName} --intent "${query}"\n`);
+        console.log(`Rule: Search first. Compose if possible. Create only for real gaps.\n`);
+        if (options.yes) {
+            const name = options.name || composeName;
+            await compose(name, { uses: useTools });
+        }
+    }
+    catch {
+        console.log(`\n❌ Could not connect to server at ${API_BASE}\n`);
+        process.exit(1);
+    }
+}

package/dist/commands/run-local.d.ts

@@ -0,0 +1,8 @@
+interface RunLocalOptions {
+    json?: boolean;
+    quiet?: boolean;
+    pretty?: boolean;
+    human?: boolean;
+}
+export declare function runLocal(filePath: string, inputArg?: string, options?: RunLocalOptions): Promise<void>;
+export {};

package/dist/commands/run-local.js

@@ -0,0 +1,64 @@
+import { executeLocalFile } from '../executor.js';
+export async function runLocal(filePath, inputArg, options = {}) {
+    if (!filePath) {
+        console.log(`\n❌ Missing file path.\n   Usage: devtopia run-local ./my-tool.js '{\"...\":\"...\"}'\n`);
+        process.exit(1);
+    }
+    const humanMode = options.human === true;
+    const jsonMode = !humanMode;
+    const pretty = options.pretty === true || (humanMode && options.pretty !== false);
+    const quiet = jsonMode ? (options.quiet !== false) : (options.quiet === true);
+    // Parse input
+    let input = {};
+    if (inputArg) {
+        try {
+            input = JSON.parse(inputArg);
+        }
+        catch {
+            if (jsonMode) {
+                process.stdout.write(JSON.stringify({ ok: false, error: `Invalid JSON input: ${inputArg}` }) + '\n');
+                process.exit(0);
+            }
+            else {
+                console.log(`\n❌ Invalid JSON input: ${inputArg}\n`);
+                process.exit(1);
+            }
+        }
+    }
+    if (!jsonMode && !quiet) {
+        console.log(`\n⚡ Running local file ${filePath}...`);
+    }
+    const result = await executeLocalFile(filePath, input, { strictJson: jsonMode });
+    if (jsonMode) {
+        const outputIsObject = result.output && typeof result.output === 'object';
+        const outputHasError = outputIsObject && (Object.prototype.hasOwnProperty.call(result.output, 'error') ||
+            (Object.prototype.hasOwnProperty.call(result.output, 'ok') && result.output.ok === false));
+        if (result.success && !outputHasError) {
+            process.stdout.write(JSON.stringify(result.output ?? null, null, pretty ? 2 : 0) + '\n');
+        }
+        else {
+            const errMsg = outputIsObject && result.output?.error
+                ? String(result.output.error)
+                : (result.error || 'Execution failed');
+            process.stdout.write(JSON.stringify({ ok: false, error: errMsg }, null, pretty ? 2 : 0) + '\n');
+        }
+        process.exit(0);
+    }
+    if (!result.success) {
+        console.log(`\n❌ Execution failed`);
+        console.log(`   Error: ${result.error}`);
+        console.log(`   Duration: ${result.durationMs}ms\n`);
+        process.exit(1);
+    }
+    if (!quiet) {
+        console.log(`\n✅ Success (${result.durationMs}ms)\n`);
+    }
+    // Pretty print output
+    if (typeof result.output === 'object') {
+        console.log(JSON.stringify(result.output, null, 2));
+    }
+    else {
+        console.log(result.output);
+    }
+    console.log('');
+}

package/dist/commands/run.d.ts

@@ -1,6 +1,8 @@
 interface RunOptions {
     json?: boolean;
     quiet?: boolean;
+    pretty?: boolean;
+    human?: boolean;
 }
 export declare function run(toolName: string, inputArg?: string, options?: RunOptions): Promise<void>;
 export {};