@meshxdata/fops 0.1.32 → 0.1.34

package/CHANGELOG.md

@@ -1,3 +1,187 @@
+## [0.1.34] - 2026-03-05
+
+- electron app (59ad0bb)
+- compose and fops file plugin (1cf0e81)
+- bump (346ffc1)
+- localhost replaced by 127.0.0.1 (82b9f30)
+- .29 (587b0e1)
+- improve up down and bootstrap script (b79ebaf)
+- checksum (22c8086)
+- checksum (96b434f)
+- checksum (15ed3c0)
+- checksum (8a6543a)
+- bump embed trino linksg (8440504)
+- bump data (765ffd9)
+- bump (cb8b232)
+- broken tests (c532229)
+- release 0.1.18, preflight checks (d902249)
+- fix compute display bug (d10f5d9)
+- cleanup packer files (6330f18)
+- plan mode (cb36a8a)
+- bump to 0.1.16 - agent ui (41ac1a2)
+- bump to 0.1.15 - agent ui (4ebe2e1)
+- bump to 0.1.14 (6c3a7fa)
+- bump to 0.1.13 (8db570f)
+- release 0.1.12 (c1c79e5)
+- bump (11aa3b0)
+- git keep and bump tui (be1678e)
+- skills, index, rrf, compacted context (100k > 10k) (7b2fffd)
+- cloudflare and token consumption, graphs indexing (0ad9eec)
+- bump storage default (22c83ba)
+- storage fix (68a22a0)
+- skills update (7f56500)
+- v9 bump (3864446)
+- bump (c95eedc)
+- rrf (dbf8c95)
+- feat: warning when running predictions (95e8c52)
+- feat: support for local predictions (45cf26b)
+- feat: wip support for predictions + mlflow (3457052)
+- add Reciprocal Rank Fusion (RRF) to knowledge and skill retrieval (61549bc)
+- validate CSV headers in compute_run readiness check (a8c7a43)
+- fix corrupted Iceberg metadata: probe tables + force cleanup on re-apply (50578af)
+- enforce: never use foundation_apply to fix broken products (2e049bf)
+- update SKILL.md with complete tool reference for knowledge retrieval (30b1924)
+- add storage read, input DP table probe, and compute_run improvements (34e6c4c)
+- skills update (1220385)
+- skills update (bb66958)
+- some tui improvement andd tools apply overwrite (e90c35c)
+- skills update (e9227a1)
+- skills update (669c4b3)
+- fix plugin pre-flight checks (f741743)
+- increase agent context (6479aaa)
+- skills and init sql fixes (5fce35e)
+- checksum (3518b56)
+- penging job limit (a139861)
+- checksum (575d28c)
+- bump (92049ba)
+- fix bug per tab status (0a33657)
+- fix bug per tab status (50457c6)
+- checksumming (0ad842e)
+- shot af mardkwon overlapping (51f63b9)
+- add spark dockerfile for multiarch builds (95abbd1)
+- fix plugin initialization (16b9782)
+- split index.js (50902a2)
+- cloudflare cidr (cc4e021)
+- cloduflare restrictions (2f6ba2d)
+- sequential start (86b496e)
+- sequential start (4930fe1)
+- sequential start (353f014)
+- qa tests (2dc6a1a)
+- bump sha for .85 (dc2edfe)
+- preserve env on sudo (7831227)
+- bump sha for .84 (6c052f9)
+- non interactive for azure vms (0aa8a2f)
+- keep .env if present (d072450)
+- bump (7a8e732)
+- ensure opa is on compose if not set (f4a5228)
+- checksum bump (a2ccc20)
+- netrc defensive checks (a0b0ccc)
+- netrc defensive checks (ae37403)
+- checksum (ec45d11)
+- update sync and fix up (7f9af72)
+- expand test for azure and add new per app tag support (388a168)
+- checksum on update (44005fc)
+- cleanup for later (15e5313)
+- cleanup for later (11c9597)
+- switch branch feature (822fecc)
+- add pull (d1c19ab)
+- Bump hono from 4.11.9 to 4.12.0 in /operator-cli (ad25144)
+- tests (f180a9a)
+- cleanup (39c49a3)
+- registry (7b7126a)
+- reconcile kafka (832d0db)
+- gh login bug (025886c)
+- cleanup (bb96cab)
+- strip envs from process (2421180)
+- force use of gh creds not tokens in envs var (fff7787)
+- resolve import between npm installs and npm link (79522e1)
+- fix gh scope and azure states (afd846c)
+- refactoring (da50352)
+- split fops repo (d447638)
+- aks (b791f8f)
+- refactor azure (67d3bad)
+- wildcard (391f023)
+- azure plugin (c074074)
+- zap (d7e6e7f)
+- fix knock (cf89c05)
+- azure (4adec98)
+- Bump tar from 7.5.7 to 7.5.9 in /operator-cli (e41e98e)
+- azure stack index.js split (de12272)
+- Bump ajv from 8.17.1 to 8.18.0 in /operator-cli (76da21f)
+- packer (9665fbc)
+- remove stack api (db0fd4d)
+- packer cleanup (fe1bf14)
+- force refresh token (3a3d7e2)
+- provision shell (2ad505f)
+- azure vm management (91dcb31)
+- azure specific (2b0cca8)
+- azure packer (12175b8)
+- init hashed pwd (db8523c)
+- packer (5b5c7c4)
+- doctor for azure vm (ed524fa)
+- packer and 1pwd (c6d053e)
+- split big index.js (dc85a1b)
+- kafka volume update (21815ec)
+- fix openai azure tools confirmation and flow (0118cd1)
+- nighly fixx, test fix (5e0d04f)
+- open ai training (cdc494a)
+- openai integration in azure (1ca1475)
+- ci (672cea9)
+- refresh ghcr creds (4220c48)
+- cleaned up version (1a0074f)
+- traefik on ghcr and templates (8e31a05)
+- apply fcl (e78911f)
+- demo landscape (dd205fe)
+- smarter login and schema (1af514f)
+- no down before up unless something broke (56b1132)
+- dai, reconcile failed containers (12907fa)
+- reconcile dead container (7da75e4)
+- defensive around storage buckets dir (b98871d)
+- defensive around storage buckets dir (e86e132)
+- gear in for multiarch (bf3fa3e)
+- up autofix (99c7f89)
+- autofix stale containers on up (43c7d0f)
+- shared sessions fix (5de1359)
+- share sessions between ui and tui (8321391)
+- fix chat view display details (e263996)
+- fix chat view display details (9babdda)
+- tui up fixes (86e9f17)
+- fix commands init (442538b)
+- enable k3s profile (b2dcfc8)
+- test up till job creation (656d388)
+- tui fixes (0599779)
+- cleanup (27731f0)
+- train (90bf559)
+- training (f809bf6)
+- training (ba2b836)
+- training (6fc5267)
+- training (4af8ac9)
+- fix build script (bd82836)
+- infra test (5b79815)
+- infra test (3a0ac05)
+- infra test (e5c67b5)
+- tests (ae7b621)
+- tests (c09ae6a)
+- update tui (4784153)
+- training (0a5a330)
+- tui (df4dd4a)
+- pkg builds (4dc9993)
+- also source env for creds (9a17d8f)
+- fcl support (e8a5743)
+- fcl support (8d6b6cd)
+- fcl support (cb76a4a)
+- bump package (df2ee85)
+- add iam mgmt (2d3c294)
+- fix k3s (976ae77)
+- fix trino, add storage plugin (75cb1f4)
+- add project root as config (a2863c6)
+- failure learnings (637ef5c)
+- Apple signed binaries (63a610e)
+- send build info to apple for notary service (300c220)
+- migration failure fixes (c7f0b2f)
+- release to wipe clean pg on duplicate key error (a38bf4d)
+- small fix (a26a674)
+
 # Changelog
 
 All notable changes to @meshxdata/fops (Foundation Operator CLI) are documented here.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@meshxdata/fops",
-  "version": "0.1.32",
+  "version": "0.1.34",
   "description": "CLI to install and manage data mesh platforms",
   "keywords": [
     "fops",
@@ -46,7 +46,6 @@
     "chalk": "^5.3.0",
     "commander": "^12.0.0",
     "dataloader": "^2.2.3",
-    "electron": "^34.5.8",
     "execa": "^9.5.2",
     "graphql": "^16.13.0",
     "hcl2-parser": "^1.0.3",

package/src/commands/lifecycle.js

@@ -1316,6 +1316,11 @@ async function runUp(program, registry, opts) {
   const hasTraefik = profileArgs.includes("traefik");
   const needsSequentialUp = hasK3s && hasTraefik && !componentOnlyUp;
 
+  // Stop the outer pre-flight spinner before runUpInner starts its own ticker.
+  // Without this, two setInterval calls both write \r\x1b[K to stderr simultaneously,
+  // causing the status line to flicker between the outer and inner spinner text.
+  clearSpinner();
+
   let result;
   if (componentOnlyUp) {
     const serviceList = opts.frontendDev && opts.component === "frontend"
@@ -1521,6 +1526,17 @@ async function runUp(program, registry, opts) {
     console.log(chalk.dim(`\n  Foundation is running at ${browseUrl}`));
     console.log(chalk.dim("  Run `fops agent` to start the AI assistant.\n"));
   }
+
+  // Launch the system tray (macOS menu bar or Windows taskbar, full-stack up only)
+  if ((process.platform === "darwin" || process.platform === "win32") && !opts.component) {
+    const { spawn } = await import("node:child_process");
+    const tray = spawn(process.argv[0], [process.argv[1], "foundation", "tray"], {
+      stdio: "ignore",
+      detached: true,
+      env: { ...process.env },
+    });
+    tray.unref();
+  }
 }
 
 // ── Restart backend after grant-admin ────────────────────────────────────────

package/src/plugins/bundled/fops-plugin-embeddings/index.js

@@ -264,7 +264,9 @@ export default {
         if (!texts || texts.length === 0) return [];
         if (!embeddingClient.isModelCached()) return [];
         try {
-          return await embeddingClient.embed(texts);
+          await embeddingClient.isReady();
+          markOnnxLoaded();
+          return await embeddingClient.embedBatch(texts);
         } catch { return []; }
       },
     });

package/src/plugins/bundled/fops-plugin-embeddings/lib/indexer.js

@@ -1047,7 +1047,7 @@ export async function runIndex({ source, sources: explicitSources, force, onProg
     // If the stored model differs from current NL_MODEL, force full re-index
     const currentStoreModel = nlStore.load().model;
     if (currentStoreModel && currentStoreModel !== NL_MODEL) {
-      log(WARN(`  Model changed (${currentStoreModel} → ${NL_MODEL}), forcing full re-index`));
+      log(WARN(`  Model changed (${currentStoreModel} → ${NL_MODEL}) — one-time re-index to rebuild vectors with the new model`));
       force = true;
       nlStore.clear();
     }

package/src/plugins/bundled/fops-plugin-file/demo/landscape.yaml

@@ -0,0 +1,67 @@
+mesh:
+  ecommerce:
+    label: EC
+    description: E-commerce analytics mesh (file plugin demo)
+    purpose: Demonstrate fops embed file smart matching
+
+    data_system:
+      datalake:
+        label: DL
+        description: Central data lake
+
+    data_source:
+      orders_bucket:
+        label: ORD
+        description: Orders data in S3
+        system: data_system.datalake
+        connection:
+          type: s3
+          url: http://foundation-storage-engine:8080
+          access_key: S3_ACCESS_KEY
+          access_secret: S3_SECRET_KEY
+        secret:
+          S3_ACCESS_KEY: minio
+          S3_SECRET_KEY: minio123
+
+    data_object:
+      orders:
+        label: ORD
+        description: Raw orders dataset
+        source: data_source.orders_bucket
+        config:
+          data_object_type: csv
+          path: /raw/orders/orders.csv
+          has_header: true
+          delimiter: ","
+
+    data_product:
+      orders_sadp:
+        label: ORD
+        description: Source-aligned orders data product
+        object: data_object.orders
+        template: sadp_passthrough
+        select_columns: [order_id, customer_id, amount, currency, order_date, status, region]
+        cast_changes:
+          - column: amount
+            data_type: decimal
+            kwargs:
+              precision: 10
+              scale: 2
+        schema:
+          - name: order_id
+            type: integer
+            primary: true
+          - name: customer_id
+            type: integer
+          - name: amount
+            type: decimal
+            precision: 10
+            scale: 2
+          - name: currency
+            type: varchar
+          - name: order_date
+            type: date
+          - name: status
+            type: varchar
+          - name: region
+            type: varchar

package/src/plugins/bundled/fops-plugin-file/demo/orders_bad.csv

@@ -0,0 +1,6 @@
+order_id,customer_id,total_amount,currency,order_date,region
+3001,7001,120.00,USD,2024-03-01,north_america
+3002,7002,not_a_number,EUR,2024-03-02,europe
+3003,7003,88.50,USD,bad-date,asia_pacific
+3004,7004,-50.00,GBP,2024-03-04,europe
+3005,7005,405.00,USD,2024-03-05,north_america

package/src/plugins/bundled/fops-plugin-file/demo/orders_good.csv

@@ -0,0 +1,7 @@
+order_id,customer_id,amount,currency,order_date,status,region
+2001,6001,99.00,USD,2024-03-01,completed,north_america
+2002,6002,175.50,EUR,2024-03-02,pending,europe
+2003,6003,420.00,USD,2024-03-03,completed,asia_pacific
+2004,6004,58.25,GBP,2024-03-04,completed,europe
+2005,6005,200.00,USD,2024-03-05,pending,north_america
+2006,6006,315.00,USD,2024-03-06,completed,north_america

package/src/plugins/bundled/fops-plugin-file/demo/orders_reference.csv

@@ -0,0 +1,6 @@
+order_id,customer_id,amount,currency,order_date,status,region
+1001,5001,149.99,USD,2024-01-03,completed,north_america
+1002,5002,89.50,EUR,2024-01-04,pending,europe
+1003,5003,230.00,USD,2024-01-05,completed,north_america
+1004,5004,45.00,GBP,2024-01-06,cancelled,europe
+1005,5005,310.75,USD,2024-01-07,completed,asia_pacific

package/src/plugins/bundled/fops-plugin-file/demo/orders_renamed.aligned.csv

@@ -0,0 +1,6 @@
+id,customer_id,price,ccy,date,order_status,geo_region
+4001,8001,99.00,USD,2024-03-01,completed,north_america
+4002,8002,145.75,EUR,2024-03-02,pending,europe
+4003,8003,280.00,USD,2024-03-03,completed,asia_pacific
+4004,8004,62.50,GBP,2024-03-04,completed,europe
+4005,8005,195.00,USD,2024-03-05,pending,north_america

package/src/plugins/bundled/fops-plugin-file/demo/orders_renamed.csv

@@ -0,0 +1,6 @@
+id,cust_id,price,ccy,date,order_status,geo_region
+4001,8001,99.00,USD,2024-03-01,completed,north_america
+4002,8002,145.75,EUR,2024-03-02,pending,europe
+4003,8003,280.00,USD,2024-03-03,completed,asia_pacific
+4004,8004,62.50,GBP,2024-03-04,completed,europe
+4005,8005,195.00,USD,2024-03-05,pending,north_america

package/src/plugins/bundled/fops-plugin-file/demo/rules.json

@@ -0,0 +1,8 @@
+{
+  "columns": {
+    "amount": { "min": 0, "type": "decimal" },
+    "order_id": { "type": "integer", "unique": true },
+    "currency": { "allowed": ["USD", "EUR", "GBP"] },
+    "status": { "allowed": ["pending", "completed", "cancelled"] }
+  }
+}

package/src/plugins/bundled/fops-plugin-file/demo/run.sh

@@ -0,0 +1,110 @@
+#!/usr/bin/env bash
+# fops embed file — demo script
+#
+# Showcases the full pipeline:
+#   Steps 1-5: pure schema / type comparison (no index needed)
+#   Step 6:    MiniLM + RRF smart matching (requires Foundation + `fops embed file index`)
+#   Step 7:    Fix CSV — rename columns to match reference (fops foundation align)
+#              In interactive mode, step 6 prompts to fix CSV or update the Data Product automatically.
+#
+# Files:
+#   orders_reference.csv  — expected schema (7 cols: order_id, customer_id, amount, currency, order_date, status, region)
+#   orders_good.csv       — clean file, matches reference exactly      → PASS
+#   orders_bad.csv        — renamed column (total_amount), missing status → FAIL
+#   orders_renamed.csv    — same data, all column names abbreviated (id, cust_id, price...) → no Jaccard match → MiniLM matches
+
+set -e
+DEMO_DIR="$(cd "$(dirname "$0")" && pwd)"
+
+section() {
+  echo
+  echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+  echo "  $1"
+  echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+}
+
+# ── 1. Preview: inspect schema + inferred types ───────────────────────────────
+section "1 · Preview — inspect schema + inferred types"
+fops embed file preview "$DEMO_DIR/orders_good.csv"
+
+# ── 2. Validate: clean file against reference → PASS ─────────────────────────
+section "2 · Validate — clean file (expect PASS)"
+fops embed file validate "$DEMO_DIR/orders_good.csv" \
+  --reference "$DEMO_DIR/orders_reference.csv"
+
+# ── 3. Validate: bad file → FAIL (missing column, renamed column) ─────────────
+section "3 · Validate — bad file (expect FAIL)"
+fops embed file validate "$DEMO_DIR/orders_bad.csv" \
+  --reference "$DEMO_DIR/orders_reference.csv" || true
+
+# ── 4. Preview with reference diff ────────────────────────────────────────────
+section "4 · Preview with diff — visualise column drift vs reference"
+fops embed file preview "$DEMO_DIR/orders_bad.csv" \
+  --reference "$DEMO_DIR/orders_reference.csv"
+
+# ── 5. Smart match (Jaccard only, no index) ───────────────────────────────────
+# orders_renamed.csv has fully abbreviated column names — zero Jaccard overlap.
+# --no-semantic forces Jaccard-only mode (skips MiniLM, no ONNX load).
+section "5 · Smart match — Jaccard only (no index): expect no strong match"
+fops embed file "$DEMO_DIR/orders_renamed.csv" --local --no-semantic || true
+
+# ── 6. Smart match (MiniLM + RRF, requires index) ────────────────────────────
+# When the SQLite schema index exists (built by `fops embed file index`), the
+# matching pipeline upgrades to:
+#   - Signal 1: column-level MiniLM cosine + greedy bipartite matching
+#   - Signal 2: Jaccard overlap on exact column names
+#   - Signal 3: type compatibility score on matched column pairs
+#   - Signal 4: MiniLM cosine on table/entity names
+#   - Signal 5: fraction of candidate columns with sim > 0.75
+#   All fused via Reciprocal Rank Fusion (RRF k=60).
+#
+# With MiniLM, "id"→"order_id", "price"→"amount", "ccy"→"currency" etc. all
+# match semantically even with zero Jaccard overlap.
+section "6 · Smart match — MiniLM + RRF (requires Foundation + index)"
+
+SCHEMA_DB="$HOME/.fops/file/schema-index.db"
+if [ -f "$SCHEMA_DB" ]; then
+  echo "  Index found — running full semantic match..."
+  fops embed file "$DEMO_DIR/orders_renamed.csv"
+else
+  echo "  No schema index found. To enable MiniLM + RRF matching, run:"
+  echo
+  echo "    # 1. Start Foundation"
+  echo "    fops up"
+  echo
+  echo "    # 2. Register your data objects (or use the demo landscape)"
+  echo "    fops apply $DEMO_DIR/landscape.yaml"
+  echo
+  echo "    # 3. Build the SQLite schema index (embeds column names with MiniLM)"
+  echo "    fops embed file index"
+  echo
+  echo "    # 4. Re-run this step — orders_renamed.csv will now match via"
+  echo "    #    semantic similarity (id→order_id, price→amount, ccy→currency)"
+  echo "    fops embed file $DEMO_DIR/orders_renamed.csv"
+  echo
+  echo "  Matching signals when index is present:"
+  echo "    [1] MiniLM column cosine  — greedy bipartite match per column pair"
+  echo "    [2] Jaccard overlap       — exact column name set overlap"
+  echo "    [3] Type compatibility    — integer/decimal/date match on mapped cols"
+  echo "    [4] Name semantic         — MiniLM cosine on table/entity names"
+  echo "    [5] Mapping confidence    — fraction of cols with sim > 0.75"
+  echo "        ↳ all fused via Reciprocal Rank Fusion (RRF k=60)"
+fi
+
+# ── 7. Fix CSV — rename abbreviated columns to reference names ────────────────
+# After a smart match, fops embed file (interactive) prompts:
+#   ▸ Fix CSV — rename columns to match reference (→ orders_renamed.aligned.csv)
+#     Fix Data Product — update schema to match this file
+#     Nothing — done
+#
+# This step demonstrates the "Fix CSV" path explicitly via fops foundation align,
+# which uses the same MiniLM + Levenshtein column alignment under the hood.
+section "7 · Fix CSV — rename columns to match reference"
+# Uses MiniLM semantic matching to map abbreviated column names to reference schema.
+# The interactive Fix CSV prompt inside `fops embed file` (step 6) runs this automatically.
+# || true: ONNX native handles trigger a SIGKILL on exit — this is expected and harmless.
+fops foundation align "$DEMO_DIR/orders_renamed.csv" \
+  "order_id,customer_id,amount,currency,order_date,status,region" \
+  --output "$DEMO_DIR/orders_renamed.aligned.csv" || true
+
+echo