@meshxdata/fops 0.0.1 → 0.0.4

package/src/skills/foundation/SKILL.md

@@ -1,107 +1,241 @@
 ---
 name: foundation-stack
-description: Managing the Foundation data mesh stack with fops
+description: Complete Foundation data mesh platform knowledge — domain model, API, operations, and troubleshooting
 ---
-## Foundation Stack Management
 
-### Lifecycle
+## What is Foundation
+
+Foundation is a data mesh platform. Users create **meshes** (domains), register **data systems** and **data sources**, define **data products** from **data objects**, and connect downstream **applications**. Data flows through compute pipelines (Spark/Ray) that transform raw data into queryable products via Trino SQL.
+
+## Domain Model
 
-Start the full stack:
-```bash
-fops up
+```
+Mesh (data domain)
+├─ Data Product (SADP — source/atomic)
+│  ├─ Data Object (file/table resource)
+│  │  └─ Data Source (connector)
+│  │     └─ Data System (external platform)
+│  ├─ Compute (builder + transformations)
+│  └─ Schema (column definitions)
+├─ Data Product (CADP — composite/aggregated)
+│  └─ depends on other Data Products
+├─ Application (downstream consumer)
+├─ Data System
+└─ Data Source
 ```
 
-Stop all services:
-```bash
-fops down
+### Entity Types
+
+- **Mesh**: top-level domain container. Has name, label, description, purpose, assignees, security policies.
+- **Data Product**: unit of data ownership. Two types:
+  - **SADP** (Source-Aligned): wraps raw data objects from external sources.
+  - **CADP** (Consumer-Aligned): derived from other data products via transformations.
+- **Data Object**: atomic data resource (CSV, Parquet, table). Has path, delimiter, schema.
+- **Data Source**: connection to external storage (S3, databases). Holds credentials via Vault.
+- **Data System**: external platform (e.g. Procore, SAP, Salesforce).
+- **Application**: downstream consumer of data products (dashboards, APIs, ML models).
+- **Compute**: processing config — builder code (PySpark/Ray), transformations, scheduling.
+
+## API Reference
+
+Base URL: `http://localhost:9001/api`
+Auth: `POST /api/iam/login` with `{"username":"compose@meshx.io","password":""}` → returns bearer token.
+Headers: `Authorization: Bearer <token>`, `x-org: root`
+
+### Mesh
+
+```
+POST   /api/data/mesh                    — Create mesh
+GET    /api/data/mesh?identifier=<id>    — Get mesh
+GET    /api/data/mesh/list               — List meshes (paginated)
+PUT    /api/data/mesh                    — Update mesh
+DELETE /api/data/mesh/cascade            — Delete mesh + all contents
+GET    /api/data/mesh/landscape          — Get mesh DAG (visual graph)
 ```
 
-Stop and remove all volumes (clean slate):
-```bash
-fops down --clean
+Create body:
+```json
+{
+  "entity": {"name":"My Mesh","entity_type":"mesh","label":"MM","description":"...","purpose":"...","assignees":[{"email":"user@meshx.io","role":""}]},
+  "entity_info": {"owner":"user@meshx.io","contact_ids":[],"links":[]}
+}
 ```
 
-Check what's running:
-```bash
-fops status
+### Data System
+
+```
+POST   /api/data/data_system             — Create
+GET    /api/data/data_system/list        — List
+DELETE /api/data/data_system             — Delete
 ```
 
-### Debugging
+### Data Source
 
-Run full environment diagnostics:
-```bash
-fops doctor
+```
+POST   /api/data/data_source             — Create
+PUT    /api/data/data_source/connection  — Set connection (S3, etc.)
+POST   /api/data/data_source/secret      — Set secrets (access keys)
+GET    /api/data/data_source/list        — List
 ```
 
-Auto-fix detected issues:
-```bash
-fops doctor --fix
+Connection body (S3):
+```json
+{"connection":{"connection_type":"s3","url":"http://foundation-storage-engine:8080","access_key":{"env_key":"S3_KEY"},"access_secret":{"env_key":"S3_SECRET"}}}
 ```
 
-Tail logs for all services:
-```bash
-fops logs
+### Data Object
+
+```
+POST   /api/data/data_object             — Create
+PUT    /api/data/data_object/config      — Configure (type, path, delimiter)
+GET    /api/data/data_object/schema      — Get inferred schema
+GET    /api/data/data_object/list        — List
 ```
 
-Tail logs for a specific service:
-```bash
-fops logs backend
-fops logs frontend
-fops logs postgres
+Config body:
+```json
+{"configuration":{"data_object_type":"csv","path":"/spark/samples/file.csv","has_header":true,"delimiter":","}}
 ```
 
-### Suggesting Commands
+### Data Product
+
+```
+POST   /api/data/data_product                    — Create (SADP or CADP)
+PUT    /api/data/data_product/schema             — Set schema
+PUT    /api/data/data_product/compute/builder    — Set builder + transformations
+POST   /api/data/data_product/compute/run        — Trigger compute
+GET    /api/data/data_product/data               — Query product data
+POST   /api/data/data_product/data/query         — Trino SQL query
+GET    /api/data/data_product/list               — List
+```
+
+### Linking Entities
 
-Always suggest **2–3 commands** in separate fenced blocks so the user can choose. Pair the primary action with a useful follow-up:
+All links follow the pattern:
+```
+POST   /api/data/link/{parent_type}/{child_type}?identifier={parent_id}&child_identifier={child_id}
+DELETE /api/data/link/{parent_type}/{child_type}?identifier={parent_id}&child_identifier={child_id}
+```
 
-- Restart → then logs: `fops restart kafka` + `fops logs kafka`
-- Debug → then fix: `fops doctor` + `fops doctor --fix`
-- Status → then logs: `fops status` + `fops logs`
-- Setup → then verify: `fops init` + `fops doctor`
+Common links:
+- `data_system → data_source` (source belongs to system)
+- `data_source → data_object` (object comes from source)
+- `data_object → data_product` (SADP wraps object)
+- `data_product → data_product` (CADP depends on SADP)
+- `data_product → application` (app consumes product)
 
-Never suggest only 1 command when a follow-up would be useful.
+### Applications
 
-### Stale Images
+```
+POST   /api/data/application             — Create
+GET    /api/data/application/list        — List
+```
+
+### Search
 
-If an image is more than 7 days old, it's likely stale — dependencies may have changed. When you see errors about missing packages, commands not found, or broken virtualenvs, **always check image ages first** and suggest a rebuild:
+```
+GET    /api/data/search?q=<term>         — Full-text search across all entities
+```
+
+### Pagination
+
+All list endpoints accept `page` and `per_page` (default 20, max 100). Response includes `total`, `page`, `per_page`, `total_pages`.
+
+## Workflow: Creating a Data Mesh from Scratch
+
+1. **Login**: `POST /api/iam/login` → get token
+2. **Create Mesh**: `POST /api/data/mesh`
+3. **Create Data Systems**: `POST /api/data/data_system` (one per external platform)
+4. **Create Data Sources**: `POST /api/data/data_source` + configure connection + set secrets
+5. **Link sources to systems**: `POST /api/data/link/data_system/data_source`
+6. **Create Data Objects**: `POST /api/data/data_object` + configure (path, format)
+7. **Link objects to sources**: `POST /api/data/link/data_source/data_object`
+8. **Create SADPs**: `POST /api/data/data_product` (type: SADP)
+9. **Link objects to products**: `POST /api/data/link/data_object/data_product`
+10. **Set schemas + builders**: configure compute pipelines
+11. **Create CADPs**: `POST /api/data/data_product` (type: CADP), link to SADPs
+12. **Run compute**: `POST /api/data/data_product/compute/run`
+13. **Create Applications**: link apps to products they consume
+14. **Query data**: via Trino SQL or the data endpoint
+
+## Services & Ports
+
+| Service | Port | Purpose |
+|---------|------|---------|
+| Frontend | 3002 | Web UI (Next.js) |
+| Backend | 9001 | REST API (Python/FastAPI) |
+| Storage Engine | 9002 | S3-compatible storage (MinIO) |
+| Trino | 8081 | Distributed SQL engine |
+| OPA | 8181 | Policy/authorization engine |
+| Kafka/Redpanda | 9092 | Event streaming |
+| Postgres | 5432 | Metadata database |
+| Hive Metastore | 9083 | Table metadata catalog |
+| Vault | 18201 | Secrets management |
+| NATS | 4222 | Message queue |
+| Redis | 6379 | Caching |
+| Watcher | 8000 | Compute orchestration |
+| Scheduler | — | Job scheduling (cron) |
+| Profiler | — | Data quality profiling |
+
+### Access Points
+- **Web UI**: http://localhost:3002
+- **API**: http://localhost:9001/api
+- **SQL Workbench**: http://localhost:3002/home/sql-workbench
+- **Storage (MinIO)**: http://localhost:9002
+
+## Stack Operations
+
+### Lifecycle
 ```bash
-docker compose build --pull
+fops up          # start all services
+fops down        # stop services
+fops down --clean # stop + remove volumes (clean slate)
+fops status      # show running containers
+fops restart     # restart all services
 ```
+
+### Debugging
 ```bash
-fops doctor
+fops doctor      # full environment diagnostics
+fops doctor --fix # auto-fix detected issues
+fops logs        # tail all logs
+fops logs backend # tail specific service
 ```
 
-### Common Issues
+### First-Time Setup
+```bash
+npm install -g @meshxdata/fops
+fops init        # clone repo, set up .env, init submodules
+fops up          # boot the stack
+fops doctor      # verify health
+```
+
+If the context shows no containers running or .env not configured, **always lead with `fops init` (then `fops up`)** before anything else — don't assume the environment is ready.
 
-**Containers stuck in "unhealthy"**: Check logs for the specific service, then restart. Often caused by a dependency not being ready yet — `fops down` then `fops up` usually resolves it.
+### Suggesting Commands
+Always suggest **2–3 commands** in separate fenced blocks. Pair primary action with follow-up:
+- Restart → logs: `fops restart kafka` + `fops logs kafka`
+- Debug → fix: `fops doctor` + `fops doctor --fix`
+- Status → logs: `fops status` + `fops logs`
+
+### Stale Images
+Images older than 7 days are likely stale. For errors about missing packages or broken deps:
+```bash
+docker compose build --pull
+```
 
-**Port conflicts**: Run `fops doctor` to see which ports are in use. Kill the conflicting process or change the port mapping in `.env`.
+## Common Issues
 
-**ECR auth expired**: Run `fops login` or `fops doctor --fix` to re-authenticate with AWS ECR.
+**"how do I create a foundation/mesh?"**: Use the Web UI at http://localhost:3002 or the API. Create a mesh first, then add systems, sources, objects, and products. See "Workflow" section above.
 
-**Missing .env**: Run `fops setup` to regenerate from `.env.example`.
+**Containers stuck unhealthy**: dependency not ready. `fops down` then `fops up`.
 
-**Submodules out of date**: Run `fops setup` to re-init and pull submodules.
+**Port conflicts**: `fops doctor` shows which ports are in use.
 
-### Services & Ports
+**ECR auth expired**: `fops doctor --fix` to re-authenticate.
 
-| Service         | Port  | Purpose                    |
-|-----------------|-------|----------------------------|
-| Backend         | 9001  | Core API server            |
-| Frontend        | 3002  | Web UI                     |
-| Storage Engine  | 9002  | Object storage layer       |
-| Trino           | 8081  | Distributed SQL engine     |
-| OPA             | 8181  | Policy engine              |
-| Kafka           | 9092  | Event streaming            |
-| Postgres        | 5432  | Metadata database          |
-| Hive Metastore  | 9083  | Table metadata catalog     |
-| Vault           | 18201 | Secrets management         |
+**Missing .env**: `fops setup` regenerates from `.env.example`.
 
-### First-Time Setup
+**Backend migrations failed**: check `fops logs backend` — usually a Postgres connection issue. Ensure Postgres is healthy first, then restart: `docker compose restart foundation-backend-migrations`.
 
-```bash
-npm install -g @meshxdata/fops
-fops init
-fops up
-fops doctor
-```
+**Compute jobs failing**: check watcher logs `fops logs watcher`, verify storage engine is healthy, check Spark/K3s resources.

package/src/ui/confirm.js

@@ -50,8 +50,9 @@ export function SelectPrompt({ message, options, onResult }) {
     ),
     ...options.map((opt, i) =>
       h(Box, { key: i },
-        h(Text, { color: i === cursor ? "cyan" : "gray" },
-          `  ${i === cursor ? "❯" : " "} ${opt.label}`
+        h(Text, { color: "cyan" }, i === cursor ? "  ❯ " : "    "),
+        h(Text, { color: i === cursor ? "white" : undefined, dimColor: i !== cursor },
+          opt.label
         )
       )
     )

package/src/ui/input.js

@@ -4,19 +4,24 @@ import { render, Box, Text, useInput, useApp } from "ink";
 const h = React.createElement;
 
 /**
- * Input box with history support
+ * Full-width horizontal separator line
  */
-export function InputBox({ onSubmit, onExit, history = [], placeholder = "Type a message..." }) {
+function Separator() {
+  const cols = process.stdout.columns || 80;
+  return h(Text, { dimColor: true }, "\u2500".repeat(cols));
+}
+
+/**
+ * Input with history support — Claude Code style (lines + status bar)
+ */
+export function InputBox({ onSubmit, onExit, history = [], statusText }) {
   const [input, setInput] = useState("");
   const [historyIndex, setHistoryIndex] = useState(-1);
   const [cursorVisible, setCursorVisible] = useState(true);
   const { exit } = useApp();
 
-  // Blinking cursor effect
   useEffect(() => {
-    const interval = setInterval(() => {
-      setCursorVisible(v => !v);
-    }, 500);
+    const interval = setInterval(() => setCursorVisible(v => !v), 500);
     return () => clearInterval(interval);
   }, []);
 
@@ -69,29 +74,24 @@ export function InputBox({ onSubmit, onExit, history = [], placeholder = "Type a
     }
   });
 
-  const cursor = cursorVisible ? "▋" : " ";
-  const displayText = input || "";
+  const cursor = cursorVisible ? "\u258b" : " ";
 
-  return h(Box, {
-    flexDirection: "column",
-    borderStyle: "round",
-    borderColor: "cyan",
-    paddingX: 1,
-    marginTop: 1,
-  },
+  return h(Box, { flexDirection: "column" },
+    h(Separator),
     h(Box, null,
-      h(Text, { color: "cyan", bold: true }, "❯ "),
-      h(Text, null, displayText),
+      h(Text, { color: "cyan", bold: true }, "\u276f "),
+      h(Text, null, input),
       h(Text, { color: "cyan" }, cursor)
     ),
-    !input && h(Text, { dimColor: true }, placeholder)
+    h(Separator),
+    statusText && h(Text, { color: "#888888" }, "  " + statusText)
   );
 }
 
 // State for standalone input
 let inputState = { resolve: null, history: [] };
 
-export function StandaloneInput({ placeholder, onResult }) {
+export function StandaloneInput({ onResult, statusText }) {
   const [input, setInput] = useState("");
   const [historyIndex, setHistoryIndex] = useState(-1);
   const [cursorVisible, setCursorVisible] = useState(true);
@@ -151,43 +151,40 @@ export function StandaloneInput({ placeholder, onResult }) {
     }
   });
 
-  return h(Box, {
-    flexDirection: "column",
-    borderStyle: "round",
-    borderColor: "cyan",
-    paddingX: 1,
-  },
+  const cursor = cursorVisible ? "\u258b" : " ";
+
+  return h(Box, { flexDirection: "column" },
+    h(Separator),
     h(Box, null,
-      h(Text, { color: "cyan", bold: true }, "❯ "),
+      h(Text, { color: "cyan", bold: true }, "\u276f "),
       h(Text, null, input),
-      h(Text, { color: "cyan" }, cursorVisible ? "▋" : " ")
+      h(Text, { color: "cyan" }, cursor)
     ),
-    !input && placeholder && h(Text, { dimColor: true }, placeholder)
+    h(Separator),
+    statusText && h(Text, { color: "#888888" }, "  " + statusText)
   );
 }
 
 /**
- * Prompt for input with history support
+ * Prompt for input with history support — Claude Code style
  * Returns the input string or null if cancelled
  */
-export async function promptInput(placeholder = "Type a message... (↑↓ history, esc to exit)") {
+export async function promptInput(statusText = "/exit to quit \u00b7 \u2191\u2193 history \u00b7 esc to cancel") {
   return new Promise((resolve) => {
     let resolved = false;
-    let userInput = null;
     const onResult = (result) => {
       if (resolved) return;
       resolved = true;
-      userInput = result;
       clear();
       unmount();
       // Echo what user typed and reset terminal
       process.stdout.write("\x1b[0m\n");
       if (result) {
-        console.log("\x1b[36m❯\x1b[0m " + result + "\n");
+        console.log("\x1b[36m\u276f\x1b[0m " + result + "\n");
       }
       setTimeout(() => resolve(result), 50);
     };
-    const { unmount, clear } = render(h(StandaloneInput, { placeholder, onResult }));
+    const { unmount, clear } = render(h(StandaloneInput, { onResult, statusText }));
   });
 }
 

package/src/ui/spinner.js

@@ -6,7 +6,7 @@ const h = React.createElement;
 
 const SPARKLE_FRAMES = ["✻", "✼", "✻", "✦"];
 const SPARKLE_INTERVAL = 120;
-const INTENT_LINE = chalk.cyan("⏺") + chalk.gray(" Thinking...");
+const INTENT_LINE = chalk.cyan("⏺") + chalk.dim(" Thinking...");
 
 // Claude-style verbs for the spinner
 export const VERBS = [
@@ -55,11 +55,11 @@ export function ThinkingSpinner({ message }) {
   return h(Box, { flexDirection: "column" },
     h(Box, null,
       h(Text, { color: "cyan" }, "⏺"),
-      h(Text, { color: "gray" }, " Thinking...")
+      h(Text, { dimColor: true }, " Thinking...")
     ),
     h(Box, null,
       h(Text, { color: "magenta" }, SPARKLE_FRAMES[frame]),
-      h(Text, { color: "gray" }, ` ${message || `${verb}…`} `),
+      h(Text, { dimColor: true }, ` ${message || `${verb}…`} `),
       h(Text, { dimColor: true }, "(esc to interrupt)")
     )
   );
@@ -89,8 +89,9 @@ export function renderSpinner(message) {
 
 /**
  * Thinking display component - shows what the agent is doing
+ * Displays reasoning text (thinking) and response preview separately
  */
-function ThinkingDisplay({ status, detail, content }) {
+function ThinkingDisplay({ status, detail, thinking, content }) {
   const [verb, setVerb] = useState(getRandomVerb());
   const [frame, setFrame] = useState(0);
 
@@ -106,6 +107,20 @@ function ThinkingDisplay({ status, detail, content }) {
     return () => clearInterval(interval);
   }, []);
 
+  // Show last 4 non-empty lines of thinking text
+  const thinkingPreview = (() => {
+    if (!thinking) return null;
+    const lines = thinking.split("\n").filter(l => l.trim());
+    return lines.slice(-4).join("\n");
+  })();
+
+  // Show last 4 non-empty lines of response content
+  const contentPreview = (() => {
+    if (!content) return null;
+    const lines = content.split("\n").filter(l => l.trim());
+    return lines.slice(-4).join("\n");
+  })();
+
   return h(Box, { flexDirection: "column" },
     // Status line with spinner
     h(Box, null,
@@ -113,37 +128,40 @@ function ThinkingDisplay({ status, detail, content }) {
       h(Text, { color: "yellow" }, ` ${status || verb}... `),
       detail && h(Text, { dimColor: true }, detail)
     ),
-    // Content preview (truncated)
-    content && h(Box, { marginTop: 1, marginLeft: 2 },
-      h(Text, { dimColor: true },
-        content.length > 100 ? content.slice(0, 100) + "..." : content
-      )
+    // Thinking preview (dimmed, italic)
+    thinkingPreview && h(Box, { marginLeft: 2 },
+      h(Text, { dimColor: true }, thinkingPreview)
+    ),
+    // Response content preview
+    contentPreview && h(Box, { marginLeft: 2 },
+      h(Text, { dimColor: true }, contentPreview)
     )
   );
 }
 
 // State for thinking display
-let thinkingState = { status: "", detail: "", content: "", rerender: null };
+let thinkingState = { status: "", detail: "", thinking: "", content: "", rerender: null };
 
 /**
  * Render thinking display
- * Returns controls to update status and content
+ * Returns controls to update status, thinking text, and content
  */
 export function renderThinking() {
-  thinkingState = { status: "", detail: "", content: "" };
+  thinkingState = { status: "", detail: "", thinking: "", content: "" };
 
   const update = () => {
     if (thinkingState.rerender) {
       thinkingState.rerender(h(ThinkingDisplay, {
         status: thinkingState.status,
         detail: thinkingState.detail,
+        thinking: thinkingState.thinking,
         content: thinkingState.content,
       }));
     }
   };
 
   const { rerender, unmount, clear } = render(
-    h(ThinkingDisplay, { status: "", detail: "", content: "" })
+    h(ThinkingDisplay, { status: "", detail: "", thinking: "", content: "" })
   );
   thinkingState.rerender = rerender;
 
@@ -153,6 +171,14 @@ export function renderThinking() {
       thinkingState.detail = detail;
       update();
     },
+    setThinking: (text) => {
+      thinkingState.thinking = text;
+      update();
+    },
+    appendThinking: (text) => {
+      thinkingState.thinking += text;
+      update();
+    },
     setContent: (content) => {
       thinkingState.content = content;
       update();

package/src/ui/streaming.js

@@ -11,7 +11,7 @@ export function ResponseBox({ content, title = "Claude" }) {
   return h(Box, {
     flexDirection: "column",
     borderStyle: "round",
-    borderColor: "gray",
+    borderColor: "white",
     paddingX: 1,
     marginY: 1,
   },
@@ -64,7 +64,7 @@ export function StreamingResponse({ title = "Claude" }) {
   return h(Box, {
     flexDirection: "column",
     borderStyle: "round",
-    borderColor: "gray",
+    borderColor: "white",
     paddingX: 1,
     marginTop: 1,
   },

package/STRUCTURE.md

@@ -1,43 +0,0 @@
-# Foundation CLI — Project structure
-
-Long-term layout for a single package inside the foundation-compose repo. No git submodules required; the CLI lives in `foundation-cli/` and is versioned with the repo.
-
-## Layout
-
-```
-foundation-cli/
-├── foundation.mjs          # Entry point: parses argv, registers commands, runs
-├── package.json
-├── README.md
-├── STRUCTURE.md            # This file
-├── install.sh              # Optional curl | bash installer
-└── src/
-    ├── config.js           # PKG, CLI_BRAND, printFoundationBanner
-    ├── project.js          # rootDir, requireRoot, isFoundationRoot, findComposeRootUp
-    ├── shell.js            # make(), dockerCompose()
-    ├── auth.js             # resolveAnthropicApiKey, resolveOpenAiApiKey, authHelp, offerClaudeLogin
-    ├── setup.js            # runSetup, runInitWizard, checkEcrRepos
-    ├── agent.js            # gatherStackContext, runAgentSingleTurn, runAgentInteractive, streaming
-    ├── doctor.js           # runDoctor (checks + optional --fix)
-    └── commands/
-        └── index.js       # registerCommands(program) — wires all CLI commands
-```
-
-## Conventions
-
-- **Single entry**: `foundation.mjs` is the only file executed; it imports `src/commands/index.js` and calls `registerCommands(program)`.
-- **No circular imports**: Flow is entry → commands → lib (config, project, shell, auth, setup, agent, doctor). Lib modules do not import commands.
-- **ESM only**: `"type": "module"` in package.json; all sources use `import`/`export`.
-- **Shared helpers**: Project root detection and `make`/docker are in `project.js` and `shell.js` so every command reuses the same semantics.
-
-## Adding a command
-
-1. Implement logic in the appropriate `src/*.js` (or a new one if it’s a new domain).
-2. In `src/commands/index.js`, add a `program.command(...).action(...)` that calls your function and uses `requireRoot(program)` or `rootDir()` as needed.
-
-## Optional: separate repo (submodule)
-
-If you later move the CLI to its own repo and add it as a submodule under `foundation-compose/foundation-cli`:
-
-- Keep this same layout inside the submodule.
-- Root `setup.sh` and docs can still say “run `node foundation-cli/foundation.mjs`” or “`npx foundation`” if the package is published.