@fangorn-network/westmarch 2026.10.21-dev → 2026.10.22-dev
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/docs/app-to-agent.md +25 -39
- package/package.json +3 -1
- package/site/worker.js +6 -6
- package/src/publish/init.js +74 -0
- package/src/publish/ship.js +34 -73
- package/src/publish/view-cli.js +6 -3
- package/src/publish/view.js +27 -43
- package/template/.github/workflows/ship.yml +33 -0
- package/template/README.md +38 -0
- package/template/app.json +27 -0
- package/template/ci/ship.sh +21 -0
- package/template/eval/goals.md +7 -0
- package/template/eval/golden.jsonl +0 -0
- package/template/gitignore +3 -0
- package/template/package.json +8 -0
package/docs/app-to-agent.md
CHANGED
|
@@ -19,9 +19,9 @@ which anything can produce: a scraper, a database export, a converted spreadshee
|
|
|
19
19
|
|
|
20
20
|
```
|
|
21
21
|
your data ─► data/graph.json ─► fangorn commit/push ─┐
|
|
22
|
-
schema/graph.json ─► fangorn commit/push ────────────┤ (namespace fangorn.schema)
|
|
23
22
|
▼
|
|
24
23
|
chain ─► westmarch-view ─► site/view/
|
|
24
|
+
app.json types ──────────────────────────────────────────────┘
|
|
25
25
|
(embeds only what changed, applies the schema)
|
|
26
26
|
app/ (page) ─► vite build ─► site/ ◄── the page loads site/view
|
|
27
27
|
pipeline/agent-card.mjs ─► site/.well-known/agent-card.json (views: [site/view]) ─► deploy
|
|
@@ -69,7 +69,7 @@ update: each step checks the chain (or Cloudflare) first and acts only on a diff
|
|
|
69
69
|
```
|
|
70
70
|
|
|
71
71
|
```sh
|
|
72
|
-
npx westmarch-ship app.json # claim, join,
|
|
72
|
+
npx westmarch-ship app.json # claim, join, crawl + publish, view, site, card, deploy, agent
|
|
73
73
|
npx westmarch-ship app.json --dry-run # every step, printed, none done
|
|
74
74
|
npx westmarch-ship app.json --no-crawl # rebuild view, site and card from what is on chain
|
|
75
75
|
npx westmarch-ship app.json --replace # each source's crawl becomes its namespace's whole state
|
|
@@ -81,21 +81,20 @@ Workers AI), the view, `/mcp`, and the page. What `ship` does, in order:
|
|
|
81
81
|
|
|
82
82
|
1. **claim** the app on the first run, and record the claim block (the card's `fromBlock`);
|
|
83
83
|
the claimer is the app's first member;
|
|
84
|
-
2. **
|
|
85
|
-
3. **run each source**: a command that publishes into `--namespace` when given `--publish`.
|
|
84
|
+
2. **run each source**: a command that publishes into `--namespace` when given `--publish`.
|
|
86
85
|
Every quickbeam scraper Source does. A source with `"each": "rows.json"` is a template,
|
|
87
86
|
one source per row (`{field}`, `{field|slug}`, `{args...}`), so adding a town, a feed or
|
|
88
87
|
an account is adding a row;
|
|
89
|
-
|
|
88
|
+
3. **the site** in `.ship/site`: the stock page (search, plus a feed of what is coming up and
|
|
90
89
|
what just happened, by the `temporal` role and the first `facets` field; for rows that are
|
|
91
90
|
not meetings, an explore page: `site.examples` as searches to try, the latest records by
|
|
92
91
|
`temporal`, and each declared facet's values to browse), the agent card,
|
|
93
92
|
`/openapi.json` and `/.well-known/x402` when anything is for sale, and `_headers`;
|
|
94
|
-
|
|
95
|
-
the cron (`site.cron`, default every 5 minutes) and its CONFIG. With `paid`, it sells each
|
|
93
|
+
4. **deploy the Worker** (`wrangler deploy`): its R2 bucket (created if missing), Workers AI,
|
|
94
|
+
the cron (`site.cron`, default every 5 minutes) and its CONFIG, which carries the schema. With `paid`, it sells each
|
|
96
95
|
record over x402; only records whose sha256 a live row carries are for sale. If the app
|
|
97
96
|
has no view yet, ship builds the first one here (`--index`) rather than wait for the cron;
|
|
98
|
-
|
|
97
|
+
5. **register** the ERC-8004 agent and bind the card, if the chain points elsewhere. An
|
|
99
98
|
agent is minted once; later runs only rebind.
|
|
100
99
|
|
|
101
100
|
State lives in `.ship/` next to `app.json`. In CI, cache `.ship/cache` and `.ship/stage`
|
|
@@ -304,40 +303,30 @@ What makes records work well:
|
|
|
304
303
|
|
|
305
304
|
## 5. Declare the schema
|
|
306
305
|
|
|
307
|
-
The schema says what your data is.
|
|
308
|
-
|
|
309
|
-
-
|
|
310
|
-
-
|
|
306
|
+
The schema says what your data is. It lives in `app.json` and nowhere else; nothing is
|
|
307
|
+
committed on chain:
|
|
308
|
+
- **`types`:** one entry per type tag: `description`, `role_map`, `presentation`.
|
|
309
|
+
- **`relations`:** the references allowed between types, `{ "rel", "from", "to" }`.
|
|
311
310
|
|
|
312
|
-
`westmarch-
|
|
311
|
+
`westmarch-ship` hands it to the Worker in its CONFIG, and `westmarch-view` reads it with
|
|
312
|
+
`--config app.json`. It is applied to every record they embed, so it decides:
|
|
313
313
|
- **what gets embedded:** the `text` role, plus title, subtitle and tags;
|
|
314
314
|
- **what readers see:** titles, links, labels;
|
|
315
315
|
- **what agents can do:** filters, counts, and choosing among views by their description.
|
|
316
316
|
|
|
317
|
-
`
|
|
317
|
+
In `app.json`:
|
|
318
318
|
|
|
319
319
|
```json
|
|
320
|
-
{
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
|
|
328
|
-
],
|
|
329
|
-
"edges": []
|
|
330
|
-
}
|
|
331
|
-
```
|
|
332
|
-
|
|
333
|
-
```sh
|
|
334
|
-
cd schema && fangorn repo init fangorn.schema \
|
|
335
|
-
&& fangorn commit graph.json -m "schema v1" --replace && fangorn push && cd ..
|
|
320
|
+
"types": {
|
|
321
|
+
"my-app.tree.v1": {
|
|
322
|
+
"description": "Tree species: what they are, where they grow, and what they are good for.",
|
|
323
|
+
"role_map": { "identity": "id", "title": "name", "subtitle": "species",
|
|
324
|
+
"tags": ["family", "region"], "text": ["description"] },
|
|
325
|
+
"presentation": { "externalUrl": { "my-app.tree.v1": "https://en.wikipedia.org/wiki/{species}" } } }
|
|
326
|
+
},
|
|
327
|
+
"relations": []
|
|
336
328
|
```
|
|
337
329
|
|
|
338
|
-
`fangorn repo init` tracks one namespace per directory (in `.fangorn/`), which is why the
|
|
339
|
-
schema is committed from its own `schema/` folder.
|
|
340
|
-
|
|
341
330
|
**`role_map`** says which field is what:
|
|
342
331
|
|
|
343
332
|
| role | means | used for |
|
|
@@ -354,13 +343,10 @@ schema is committed from its own `schema/` folder.
|
|
|
354
343
|
map inferred from its fields. That works for display and often fails for search: a
|
|
355
344
|
field called `desc` or `nct` isn't recognized, and a `price` can end up as the date.
|
|
356
345
|
Declare every type you publish.
|
|
357
|
-
- **
|
|
358
|
-
|
|
359
|
-
- **You can change it at any time** with another commit. A view takes the new role map and
|
|
360
|
-
description on its next `westmarch-view` run. **Records already embedded keep the text they were embedded
|
|
346
|
+
- **You can change it at any time**: edit `app.json` and ship again. The view takes the new
|
|
347
|
+
role map and description on the Worker's next cron run (or the next `westmarch-view`). **Records already embedded keep the text they were embedded
|
|
361
348
|
with** until they're re-embedded, so change the `text` role early.
|
|
362
|
-
- **Relations** (
|
|
363
|
-
"to": "region" }`) declare which edges the app allows. Nothing enforces them yet; they
|
|
349
|
+
- **Relations** (e.g. `{ "rel": "grows_in", "from": "my-app.tree.v1", "to": "my-app.region.v1" }`) declare which edges the app allows. Nothing enforces them yet; they
|
|
364
350
|
are what publisher graphs will be validated against.
|
|
365
351
|
|
|
366
352
|
## 6. Commit and push the data
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@fangorn-network/westmarch",
|
|
3
|
-
"version": "2026.10.
|
|
3
|
+
"version": "2026.10.22-dev",
|
|
4
4
|
"description": "all-in-one builder for fangorn apps",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|
|
@@ -68,6 +68,7 @@
|
|
|
68
68
|
"viem": "^2.56.3"
|
|
69
69
|
},
|
|
70
70
|
"bin": {
|
|
71
|
+
"westmarch": "./src/publish/init.js",
|
|
71
72
|
"westmarch-publish": "./src/publish/cli.js",
|
|
72
73
|
"westmarch-view": "./src/publish/view-cli.js",
|
|
73
74
|
"westmarch-ship": "./src/publish/ship.js",
|
|
@@ -78,6 +79,7 @@
|
|
|
78
79
|
},
|
|
79
80
|
"files": [
|
|
80
81
|
"src",
|
|
82
|
+
"template",
|
|
81
83
|
"docs/app-to-agent.md",
|
|
82
84
|
"site/agent.js",
|
|
83
85
|
"site/dist",
|
package/site/worker.js
CHANGED
|
@@ -27,7 +27,7 @@
|
|
|
27
27
|
// paying, so nobody pays for a record that does not exist, and decrypts and checks it after.
|
|
28
28
|
//
|
|
29
29
|
// Bindings: VIEW (R2), AI (Workers AI), ASSETS (the site), CONFIG (a JSON var: app, name,
|
|
30
|
-
// description, fromBlock, repos, paid, deployment), and optionally RPC_URL and IPFS_GATEWAY
|
|
30
|
+
// description, fromBlock, schema, repos, paid, deployment), and optionally RPC_URL and IPFS_GATEWAY
|
|
31
31
|
// (vars) and PAID_SECRET (a secret).
|
|
32
32
|
|
|
33
33
|
import { Fangorn, FangornConfig, SupportedNetworks, repoExcluded } from "@fangorn-network/sdk";
|
|
@@ -105,7 +105,7 @@ export async function index(env, { log = console.log } = {}) {
|
|
|
105
105
|
// Imported here: x402.js's self-check imports this module, and a cycle at load would deadlock it.
|
|
106
106
|
const { openRow } = await import("../src/agent/x402.js");
|
|
107
107
|
const report = await publishView({
|
|
108
|
-
fangorn: r.fangorn, app: CONFIG.app, repos: CONFIG.repos, store: r2Store(env.VIEW),
|
|
108
|
+
fangorn: r.fangorn, app: CONFIG.app, schema: CONFIG.schema, repos: CONFIG.repos, store: r2Store(env.VIEW),
|
|
109
109
|
presented: await presented(r, state), embedBatch: encode(env, DOCUMENT), embedder: await embedderOf(env), log,
|
|
110
110
|
...(CONFIG.private ? { open: (p) => openRow(env.PAID_SECRET, p) } : {}),
|
|
111
111
|
});
|
|
@@ -488,7 +488,7 @@ export default worker;
|
|
|
488
488
|
|
|
489
489
|
// ── self-check: `node site/worker.js` — a fake bucket, a fake AI binding, a fake registry ──
|
|
490
490
|
if (typeof process !== "undefined" && import.meta.url === `file://${process.argv[1]}` && process.argv[1]?.endsWith("/worker.js")) {
|
|
491
|
-
const { localFangorn
|
|
491
|
+
const { localFangorn } = await import("../src/publish/view.js");
|
|
492
492
|
const eq = (a, b, m) => { if (JSON.stringify(a) !== JSON.stringify(b)) throw new Error(`${m}: ${JSON.stringify(a)} != ${JSON.stringify(b)}`); };
|
|
493
493
|
|
|
494
494
|
const objs = new Map();
|
|
@@ -517,14 +517,14 @@ if (typeof process !== "undefined" && import.meta.url === `file://${process.argv
|
|
|
517
517
|
|
|
518
518
|
const APP = `0x${"ab".repeat(32)}`, OWNER = `0x${"11".repeat(20)}`, PUB = `0x${"22".repeat(20)}`, TAG = "shop.item.v1";
|
|
519
519
|
const rec = (cid, type, payload) => ({ id: cid, cid, type, payload });
|
|
520
|
+
CONFIG.schema = { types: { [TAG]: { role_map: { identity: "sku", title: "name", text: "about", media: "doc" } } } };
|
|
520
521
|
const chain = {
|
|
521
|
-
[`${OWNER}/${SCHEMA_REPO}`]: { records: [rec("t", TYPE_TAG, { tag: TAG, role_map: { identity: "sku", title: "name", text: "about", media: "doc" } })] },
|
|
522
522
|
[`${PUB}/shop`]: { records: [
|
|
523
523
|
rec("c1", TAG, { sku: "a1", name: "Oak table", about: "a solid oak dining table", doc: "https://docs.example/oak.pdf" }),
|
|
524
524
|
rec("c2", TAG, { sku: "b2", name: "Brass lamp", about: "a brass reading lamp" })] },
|
|
525
525
|
};
|
|
526
526
|
const fangorn = localFangorn(chain, { appId: APP, owner: OWNER });
|
|
527
|
-
await publishView({ fangorn, app: APP, store: r2Store(VIEW), embedBatch: encode(env, DOCUMENT), embedder: WORKER_EMBEDDER, log: () => {} });
|
|
527
|
+
await publishView({ fangorn, app: APP, schema: CONFIG.schema, store: r2Store(VIEW), embedBatch: encode(env, DOCUMENT), embedder: WORKER_EMBEDDER, log: () => {} });
|
|
528
528
|
eq([calls[0].model, calls[0].text[0].startsWith(DOCUMENT)], [AI_MODEL, true], "documents go to the AI binding with the document prompt");
|
|
529
529
|
|
|
530
530
|
const call = async (method, params) => (await (await worker.fetch(new Request("https://shop.test/mcp",
|
|
@@ -551,7 +551,7 @@ if (typeof process !== "undefined" && import.meta.url === `file://${process.argv
|
|
|
551
551
|
const sealedAgain = await sealRow(SECRET, { sku: "v1", name: "Velvet chair", about: "a green velvet armchair" });
|
|
552
552
|
eq(sealedAgain, chain[`${PUB}/vault`].records[0].payload, "an unchanged row seals to the same bytes (the same CID: not embedded again)");
|
|
553
553
|
if (JSON.stringify(sealedAgain).includes("Velvet")) throw new Error("a sealed row must not carry its plaintext");
|
|
554
|
-
await publishView({ fangorn, app: APP, store: r2Store(VIEW), embedBatch: encode(env, DOCUMENT), embedder: WORKER_EMBEDDER, log: () => {},
|
|
554
|
+
await publishView({ fangorn, app: APP, schema: CONFIG.schema, store: r2Store(VIEW), embedBatch: encode(env, DOCUMENT), embedder: WORKER_EMBEDDER, log: () => {},
|
|
555
555
|
open: (p) => openRow(SECRET, p) });
|
|
556
556
|
env.PAID_SECRET = SECRET;
|
|
557
557
|
CONFIG = env.CONFIG = { ...env.CONFIG, private: { network: "eip155:84532", asset: `0x${"aa".repeat(20)}`, payTo: `0x${"5e".repeat(20)}`,
|
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// A new app from the template: its recipe (app.json, eval/, ci/), no data.
|
|
3
|
+
// npx @fangorn-network/westmarch init <app> [dir] [--no-install]
|
|
4
|
+
// The app's data is what its members commit; the repo this makes can be public.
|
|
5
|
+
import { cpSync, existsSync, mkdtempSync, readdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
|
|
6
|
+
import { spawnSync } from "node:child_process";
|
|
7
|
+
import { tmpdir } from "node:os";
|
|
8
|
+
import { dirname, join, resolve } from "node:path";
|
|
9
|
+
import { fileURLToPath } from "node:url";
|
|
10
|
+
|
|
11
|
+
const ROOT = join(dirname(fileURLToPath(import.meta.url)), "../..");
|
|
12
|
+
const TEMPLATE = join(ROOT, "template");
|
|
13
|
+
// What fangorn.network accepts: a name a shell and a URL need no quoting for.
|
|
14
|
+
const APP_NAME = /^[a-z0-9][a-z0-9._-]{1,62}$/;
|
|
15
|
+
|
|
16
|
+
export function init(app, dir = app, { install = true, log = console.log } = {}) {
|
|
17
|
+
if (!APP_NAME.test(app ?? "")) throw new Error(`"${app}": 2-63 characters, lowercase letters, digits, dot, dash or underscore`);
|
|
18
|
+
dir = resolve(dir);
|
|
19
|
+
if (existsSync(dir) && readdirSync(dir).length) throw new Error(`${dir} is not empty`);
|
|
20
|
+
cpSync(TEMPLATE, dir, { recursive: true });
|
|
21
|
+
renameSync(join(dir, "gitignore"), join(dir, ".gitignore")); // npm drops a published .gitignore
|
|
22
|
+
|
|
23
|
+
const cfg = JSON.parse(readFileSync(join(dir, "app.json"), "utf8"));
|
|
24
|
+
const tag = `${app}.item.v1`;
|
|
25
|
+
const type = JSON.parse(JSON.stringify(cfg.types["my-app.item.v1"]).replaceAll("my-app.item.v1", tag));
|
|
26
|
+
Object.assign(cfg, { app, name: app, types: { [tag]: type } });
|
|
27
|
+
cfg.site.project = app.replace(/[._]/g, "-"); // a Worker name: letters, digits, dashes
|
|
28
|
+
writeFileSync(join(dir, "app.json"), JSON.stringify(cfg, null, 1) + "\n");
|
|
29
|
+
writeFileSync(join(dir, "README.md"), readFileSync(join(dir, "README.md"), "utf8").replaceAll("my-app", app));
|
|
30
|
+
const pkg = JSON.parse(readFileSync(join(dir, "package.json"), "utf8"));
|
|
31
|
+
pkg.dependencies["@fangorn-network/westmarch"] = `^${JSON.parse(readFileSync(join(ROOT, "package.json"), "utf8")).version}`;
|
|
32
|
+
writeFileSync(join(dir, "package.json"), JSON.stringify(pkg, null, 2) + "\n");
|
|
33
|
+
|
|
34
|
+
if (install) {
|
|
35
|
+
const sh = (cmd, ...args) => { if (spawnSync(cmd, args, { cwd: dir, stdio: "inherit" }).status !== 0) throw new Error(`${cmd} ${args[0]} failed`); };
|
|
36
|
+
sh("npm", "install"); // the lock CI installs from
|
|
37
|
+
sh("git", "init", "-q", "-b", "main");
|
|
38
|
+
sh("git", "add", "-A");
|
|
39
|
+
sh("git", "commit", "-q", "-m", `${app}: from the westmarch template`);
|
|
40
|
+
}
|
|
41
|
+
log(`${app} is in ${dir}. Next:
|
|
42
|
+
1. app.json: the description, and the type's fields (role_map)
|
|
43
|
+
2. gh repo create ${app} --private --source ${dir} --push
|
|
44
|
+
3. gh secret set ETH_PRIVATE_KEY # the wallet that claimed ${app}; read from the prompt, never echoed
|
|
45
|
+
gh secret set PINATA_JWT
|
|
46
|
+
gh variable set PINATA_GATEWAY --body <your-gateway>.mypinata.cloud
|
|
47
|
+
gh secret set CLOUDFLARE_API_TOKEN
|
|
48
|
+
gh secret set CLOUDFLARE_ACCOUNT_ID
|
|
49
|
+
4. gh workflow run ship # deploys the Worker; then publish records (README.md)`);
|
|
50
|
+
return dir;
|
|
51
|
+
}
|
|
52
|
+
|
|
53
|
+
if (process.argv[2] === "--selfcheck") {
|
|
54
|
+
const dir = join(mkdtempSync(join(tmpdir(), "init-")), "app");
|
|
55
|
+
try {
|
|
56
|
+
init("my.app_1", dir, { install: false, log: () => {} });
|
|
57
|
+
const cfg = JSON.parse(readFileSync(join(dir, "app.json"), "utf8"));
|
|
58
|
+
if (cfg.app !== "my.app_1" || cfg.site.project !== "my-app-1" || !cfg.types["my.app_1.item.v1"]?.presentation.types["my.app_1.item.v1"])
|
|
59
|
+
throw new Error(`app.json not filled: ${JSON.stringify(cfg)}`);
|
|
60
|
+
if (!existsSync(join(dir, ".gitignore")) || !existsSync(join(dir, ".github/workflows/ship.yml"))) throw new Error("template files missing");
|
|
61
|
+
if (readFileSync(join(dir, "README.md"), "utf8").includes("my-app")) throw new Error("README still names my-app");
|
|
62
|
+
let threw = false;
|
|
63
|
+
try { init("x", join(dir, "y"), { install: false, log: () => {} }); } catch { threw = true; }
|
|
64
|
+
if (!threw) throw new Error("a one-letter name must be refused");
|
|
65
|
+
console.log("init: ok");
|
|
66
|
+
} finally { rmSync(dirname(dir), { recursive: true, force: true }); }
|
|
67
|
+
} else if (process.argv[2] === "init") {
|
|
68
|
+
const [app, dir] = process.argv.slice(3).filter((a) => !a.startsWith("--"));
|
|
69
|
+
try { init(app, dir, { install: !process.argv.includes("--no-install") }); }
|
|
70
|
+
catch (e) { console.error(e.message); process.exit(1); }
|
|
71
|
+
} else {
|
|
72
|
+
console.error("usage: westmarch init <app> [dir] [--no-install]");
|
|
73
|
+
process.exit(2);
|
|
74
|
+
}
|
package/src/publish/ship.js
CHANGED
|
@@ -14,16 +14,15 @@
|
|
|
14
14
|
//
|
|
15
15
|
// 1. claim the app (first run only; the claim block becomes the card's fromBlock).
|
|
16
16
|
// The claimer is the app's first member, so its own updates are accepted.
|
|
17
|
-
// 2.
|
|
18
|
-
// 3.
|
|
19
|
-
// 4. the
|
|
20
|
-
// 5.
|
|
21
|
-
// 6. register the ERC-8004 agent and bind the card, if the chain points elsewhere
|
|
17
|
+
// 2. run each source, which crawls and publishes straight into the app
|
|
18
|
+
// 3. the stock page, the card and _headers → .ship/site
|
|
19
|
+
// 4. deploy the Worker: its R2 bucket, Workers AI, the cron, CONFIG (app.json's schema rides in it)
|
|
20
|
+
// 5. register the ERC-8004 agent and bind the card, if the chain points elsewhere
|
|
22
21
|
//
|
|
23
|
-
// The Worker picks up what 2
|
|
22
|
+
// The Worker picks up what 2 published on its next cron run (every few minutes),
|
|
24
23
|
// so the view follows the chain with nobody running anything.
|
|
25
24
|
//
|
|
26
|
-
// --register is the fast path, run before there is any data: 1,
|
|
25
|
+
// --register is the fast path, run before there is any data: 1, 3, 4 and 5 binding the card
|
|
27
26
|
// without minting. The mint copies the card's skills into its IPFS registration file and is
|
|
28
27
|
// never redone, so it waits for the first full ship, when the card lists what the app does.
|
|
29
28
|
//
|
|
@@ -38,7 +37,6 @@ import { createRequire } from "node:module";
|
|
|
38
37
|
import { dirname, join, resolve } from "node:path";
|
|
39
38
|
import { homedir } from "node:os";
|
|
40
39
|
import { fileURLToPath, pathToFileURL } from "node:url";
|
|
41
|
-
import { RELATION_TAG, TYPE_TAG } from "./view.js";
|
|
42
40
|
|
|
43
41
|
const HERE = dirname(fileURLToPath(import.meta.url));
|
|
44
42
|
// USDC per x402 network (CAIP-2): its address and EIP-712 domain, as each contract reports them.
|
|
@@ -151,16 +149,6 @@ export function loadConfig(path) {
|
|
|
151
149
|
return { relations: [], tags: [], types: {}, ...c };
|
|
152
150
|
}
|
|
153
151
|
|
|
154
|
-
/** The schema commit for `types` + `relations`: one fangorn.type.v1 record per type, one
|
|
155
|
-
* fangorn.relation.v1 record per allowed reference. A file for `fangorn commit --replace`. */
|
|
156
|
-
export function schemaRecords(types, relations = []) {
|
|
157
|
-
return { records: [
|
|
158
|
-
...Object.entries(types).map(([tag, t]) => ({ id: `type:${tag}`, type: TYPE_TAG, payload: {
|
|
159
|
-
tag, description: t.description ?? "", role_map: t.role_map ?? {}, presentation: t.presentation ?? {} } })),
|
|
160
|
-
...relations.map((r) => ({ id: `rel:${r.rel}:${r.from}:${r.to}`, type: RELATION_TAG, payload: { rel: r.rel, from: r.from, to: r.to } })),
|
|
161
|
-
] };
|
|
162
|
-
}
|
|
163
|
-
|
|
164
152
|
/** The Worker's wrangler config: the stock worker, the site as its assets, its bucket, the AI
|
|
165
153
|
* binding, the cron and CONFIG. `worker` is site/worker.js unless a test points elsewhere. */
|
|
166
154
|
export function wranglerConfig({ name, site, account, cron = "*/5 * * * *", config, url, ipfsGateway, worker = join(HERE, "../../site/worker.js") }) {
|
|
@@ -186,13 +174,6 @@ export function wranglerConfig({ name, site, account, cron = "*/5 * * * *", conf
|
|
|
186
174
|
const stable = (v) => Array.isArray(v) ? v.map(stable)
|
|
187
175
|
: v && typeof v === "object" ? Object.fromEntries(Object.keys(v).sort().map((k) => [k, stable(v[k])])) : v;
|
|
188
176
|
|
|
189
|
-
/** Same shape parseSchema returns, so a config and the chain compare directly. */
|
|
190
|
-
const canonical = (types, relations = []) => JSON.stringify(stable({
|
|
191
|
-
types: Object.fromEntries(Object.entries(types).sort(([a], [b]) => a.localeCompare(b))
|
|
192
|
-
.map(([tag, t]) => [tag, { description: t.description ?? "", role_map: t.role_map ?? {}, presentation: t.presentation ?? {} }])),
|
|
193
|
-
relations: [...relations].sort((a, b) => JSON.stringify(a).localeCompare(JSON.stringify(b))),
|
|
194
|
-
}));
|
|
195
|
-
|
|
196
177
|
// ── processes ─────────────────────────────────────────────────────────────────
|
|
197
178
|
|
|
198
179
|
/** `${VAR}` and `${VAR:-default}` from the environment, so one app.json runs on a laptop
|
|
@@ -240,8 +221,9 @@ function assertSameDeployment(cli) {
|
|
|
240
221
|
|
|
241
222
|
export async function ship(configPath, { crawl = true, deploy = true, dry = false, replace = false, only = null, register = false } = {}) {
|
|
242
223
|
const cfg = loadConfig(configPath);
|
|
243
|
-
|
|
244
|
-
|
|
224
|
+
// No sources is fine: the app then presents whatever its members commit (repos: [] = all of them).
|
|
225
|
+
if (!register && !Object.keys(cfg.types).length)
|
|
226
|
+
throw new Error(`${configPath}: no types yet, so nothing to build; --register sets up the app alone`);
|
|
245
227
|
// No default: one that changed would claim every app that leaned on it again, elsewhere.
|
|
246
228
|
if (!cfg.network) throw new Error(`${configPath}: "network" is required: "baseSepolia" for a new app, or the network the app was claimed on (before 2026-10-20 the default was "arbitrumSepolia")`);
|
|
247
229
|
CLI = cliArgv(cfg);
|
|
@@ -263,7 +245,7 @@ export async function ship(configPath, { crawl = true, deploy = true, dry = fals
|
|
|
263
245
|
|
|
264
246
|
// The SDK's network name: view.js reads with it, and the CLI and the sources spawned below sign on it.
|
|
265
247
|
process.env.FANGORN_NETWORK = process.env.CHAIN_NAME = cfg.network;
|
|
266
|
-
const [{ createPublicClient, http }, view] = await Promise.all([import("viem"), import("./view.js")]);
|
|
248
|
+
const [{ createPublicClient, http, parseAbiItem }, view] = await Promise.all([import("viem"), import("./view.js")]);
|
|
267
249
|
const FangornConfig = await view.network(cfg.network);
|
|
268
250
|
const fangorn = await view.readOnlyFangorn();
|
|
269
251
|
fangorn.setAppId(cfg.app);
|
|
@@ -276,8 +258,9 @@ export async function ship(configPath, { crawl = true, deploy = true, dry = fals
|
|
|
276
258
|
const me = (fangornCli(cfg.app, ["wallet"]).match(/Address:\s+(0x[0-9a-fA-F]{40})/) ?? [])[1]?.toLowerCase();
|
|
277
259
|
if (!me) throw new Error("`fangorn wallet` shows no address; run `fangorn init`");
|
|
278
260
|
let owner = (await registry.getAppOwner()).toLowerCase();
|
|
261
|
+
const chain = createPublicClient({ transport: http(FangornConfig.rpcUrl) });
|
|
279
262
|
if (/^0x0+$/.test(owner)) {
|
|
280
|
-
const block = await
|
|
263
|
+
const block = await chain.getBlockNumber();
|
|
281
264
|
state.fromBlock = String(block);
|
|
282
265
|
log(`${cfg.app} is unclaimed; claiming it for ${me} (fromBlock ${block})`);
|
|
283
266
|
fangornCli(cfg.app, ["app", "claim"], { dry });
|
|
@@ -286,25 +269,20 @@ export async function ship(configPath, { crawl = true, deploy = true, dry = fals
|
|
|
286
269
|
} else if (owner !== me) {
|
|
287
270
|
throw new Error(`${cfg.app} belongs to ${owner}, not this wallet (${me}); pick another name`);
|
|
288
271
|
}
|
|
272
|
+
// Claimed elsewhere (fangorn.network, the CLI): the view starts at the claim, found once.
|
|
273
|
+
if (state.fromBlock == null && cfg.fromBlock == null) {
|
|
274
|
+
const from = BigInt(FangornConfig.appRegistryFromBlock), to = await chain.getBlockNumber(), step = BigInt(FangornConfig.logWindow ?? 10000);
|
|
275
|
+
const event = parseAbiItem("event AppRegistered(bytes32 indexed app_id, address indexed owner)");
|
|
276
|
+
for (let a = from; a <= to && state.fromBlock == null; a += step) {
|
|
277
|
+
const [claimed] = await chain.getLogs({ address: FangornConfig.appRegistryContractAddress, event, args: { app_id: appId },
|
|
278
|
+
fromBlock: a, toBlock: a + step - 1n < to ? a + step - 1n : to });
|
|
279
|
+
if (claimed) state.fromBlock = String(claimed.blockNumber);
|
|
280
|
+
}
|
|
281
|
+
}
|
|
289
282
|
state.fromBlock ??= String(cfg.fromBlock ?? FangornConfig.appRegistryFromBlock);
|
|
290
283
|
log(`app ${cfg.app} = ${appId}, owner ${owner}, fromBlock ${state.fromBlock}`);
|
|
291
284
|
|
|
292
|
-
// 2.
|
|
293
|
-
if (!register) {
|
|
294
|
-
let onChain = null;
|
|
295
|
-
try { onChain = view.parseSchema((await fangorn.readRepo(owner, view.SCHEMA_REPO)).contents); } catch { /* none yet */ }
|
|
296
|
-
if (!onChain || canonical(onChain.types, onChain.relations) !== canonical(cfg.types, cfg.relations)) {
|
|
297
|
-
const sdir = join(dir, "schema");
|
|
298
|
-
mkdirSync(sdir, { recursive: true });
|
|
299
|
-
writeFileSync(join(sdir, "schema.json"), JSON.stringify(schemaRecords(cfg.types, cfg.relations), null, 1));
|
|
300
|
-
log(`schema ${onChain ? "changed" : "is new"}: committing ${Object.keys(cfg.types).join(", ")}`);
|
|
301
|
-
fangornCli(cfg.app, ["repo", "init", view.SCHEMA_REPO], { cwd: sdir, dry });
|
|
302
|
-
fangornCli(cfg.app, ["commit", "schema.json", "-m", "schema", "--replace"], { cwd: sdir, dry });
|
|
303
|
-
fangornCli(cfg.app, ["push"], { cwd: sdir, dry });
|
|
304
|
-
} else log("schema unchanged");
|
|
305
|
-
}
|
|
306
|
-
|
|
307
|
-
// 3. sources: crawl + publish straight into the app. One source failing (a site that
|
|
285
|
+
// 2. sources: crawl + publish straight into the app. One source failing (a site that
|
|
308
286
|
// blocks us, a feed that is down) keeps its last published data and does not stop the
|
|
309
287
|
// others; the ship still exits non-zero at the end, naming it.
|
|
310
288
|
const failed = [];
|
|
@@ -322,7 +300,7 @@ export async function ship(configPath, { crawl = true, deploy = true, dry = fals
|
|
|
322
300
|
}
|
|
323
301
|
}
|
|
324
302
|
|
|
325
|
-
//
|
|
303
|
+
// 3. the page, the card, the headers
|
|
326
304
|
const { site, writeCard } = await buildSite(cfg, base);
|
|
327
305
|
// site.url: the app's own domain, routed to the Worker. Otherwise its workers.dev URL,
|
|
328
306
|
// which is only known once it is deployed: the first deploy learns it and the card follows.
|
|
@@ -350,13 +328,13 @@ export async function ship(configPath, { crawl = true, deploy = true, dry = fals
|
|
|
350
328
|
const writeSite = (u) => writeCard(u, { fromBlock: state.fromBlock, repos, paid: sale(u).paid, access: access(u).card });
|
|
351
329
|
writeSite(url ?? `https://${cfg.site.project}.workers.dev`);
|
|
352
330
|
|
|
353
|
-
//
|
|
331
|
+
// 4. the Worker
|
|
354
332
|
const workerConfig = (u) => {
|
|
355
333
|
const path = join(dir, "wrangler.json");
|
|
356
334
|
writeFileSync(path, JSON.stringify(wranglerConfig({
|
|
357
335
|
name: cfg.site.project, site, account: cfg.site.account, cron: cfg.site.cron, url: cfg.site.url, ipfsGateway: view.ipfsGateway(),
|
|
358
336
|
config: { app: cfg.app, name: cfg.name, description: cfg.description, fromBlock: state.fromBlock, network: cfg.network,
|
|
359
|
-
repos: register ? [] : repos, paid: sale(u).sale, private: access(u).config },
|
|
337
|
+
schema: { types: cfg.types, relations: cfg.relations }, repos: register ? [] : repos, paid: sale(u).sale, private: access(u).config },
|
|
360
338
|
}), null, 1));
|
|
361
339
|
return path;
|
|
362
340
|
};
|
|
@@ -380,7 +358,7 @@ export async function ship(configPath, { crawl = true, deploy = true, dry = fals
|
|
|
380
358
|
else if (cfg.paid) log("paid: set the key secret once with `npx wrangler secret put PAID_SECRET --config .ship/wrangler.json`");
|
|
381
359
|
} else workerConfig(url ?? `https://${cfg.site.project}.workers.dev`);
|
|
382
360
|
|
|
383
|
-
//
|
|
361
|
+
// 5. ERC-8004 + the binding
|
|
384
362
|
const cardUrl = `${url}/.well-known/agent-card.json`;
|
|
385
363
|
const bound = await registry.appAgentUri(appId).catch(() => "");
|
|
386
364
|
const mint = () => {
|
|
@@ -403,7 +381,7 @@ export async function ship(configPath, { crawl = true, deploy = true, dry = fals
|
|
|
403
381
|
} else if (deploy) log("card already bound");
|
|
404
382
|
|
|
405
383
|
// A new app has no view until the cron runs, which may be hours off: build the first one now.
|
|
406
|
-
if (deploy && !dry && !register && !await fetch(`${url}/view/cdn/catalog`).then((r) => r.ok, () => false)) {
|
|
384
|
+
if (deploy && !dry && !register && cfg.sources.length && !await fetch(`${url}/view/cdn/catalog`).then((r) => r.ok, () => false)) {
|
|
407
385
|
log("no view yet: indexing it here rather than waiting for the cron");
|
|
408
386
|
await indexHere(configPath).catch((e) => log(`✗ first index: ${e.message}; the cron will build it`));
|
|
409
387
|
}
|
|
@@ -579,21 +557,18 @@ export function readStage(cfg, base, { crawl = false, only = null, log: say = lo
|
|
|
579
557
|
}
|
|
580
558
|
|
|
581
559
|
/**
|
|
582
|
-
* The whole app on this machine (devnet.js): a local chain and IPFS, the app claimed and its
|
|
583
|
-
* schema committed there, what the sources staged published into it, and its Worker under
|
|
560
|
+
* The whole app on this machine (devnet.js): a local chain and IPFS, the app claimed there, what the sources staged published into it, and its Worker under
|
|
584
561
|
* `wrangler dev` with a local R2 and an encoder on localhost (or Workers AI with ai=true).
|
|
585
562
|
* Runs the indexer once, says where everything is, and stays up until interrupted.
|
|
586
563
|
*/
|
|
587
564
|
export async function devnetShip(configPath, { crawl = false, only = null, port = 8787, ai = false } = {}) {
|
|
588
565
|
const cfg = loadConfig(configPath);
|
|
589
566
|
const base = dirname(resolve(configPath));
|
|
590
|
-
const
|
|
567
|
+
const { devnet } = await import("./devnet.js");
|
|
591
568
|
const staged = readStage(cfg, base, { crawl, only });
|
|
592
569
|
const net = await devnet({ log });
|
|
593
570
|
const owner = await net.client(cfg.app);
|
|
594
571
|
await owner.getAppRegistry().registerApp();
|
|
595
|
-
await owner.initRepo(SCHEMA_REPO);
|
|
596
|
-
await owner.uploadBatch(SCHEMA_REPO, schemaRecords(cfg.types, cfg.relations).records);
|
|
597
572
|
// The claimer publishes the sources, as `fangorn` does in a ship.
|
|
598
573
|
for (const [ns, records] of Object.entries(staged)) {
|
|
599
574
|
if (!records.length) continue;
|
|
@@ -607,7 +582,7 @@ export async function devnetShip(configPath, { crawl = false, only = null, port
|
|
|
607
582
|
const { site, writeCard } = await buildSite(cfg, base);
|
|
608
583
|
writeCard(url, { fromBlock: "0", repos });
|
|
609
584
|
const wcfg = wranglerConfig({ name: cfg.site.project, site, config: { app: cfg.app, name: cfg.name, description: cfg.description,
|
|
610
|
-
fromBlock: "0", repos, paid: null, deployment: net.deployment } });
|
|
585
|
+
fromBlock: "0", schema: { types: cfg.types, relations: cfg.relations }, repos, paid: null, deployment: net.deployment } });
|
|
611
586
|
// Its own R2 and state, apart from a deployed app's.
|
|
612
587
|
wcfg.vars.RPC_URL = net.rpcUrl;
|
|
613
588
|
wcfg.vars.IPFS_GATEWAY = net.gatewayUrl;
|
|
@@ -701,8 +676,7 @@ export async function buildLocal(configPath, out, { crawl = false, only = null,
|
|
|
701
676
|
env.cacheDir = `${homedir()}/.cache/fangorn-mcp/models`;
|
|
702
677
|
// Not the app's real id or owner: nothing here is compared with the chain.
|
|
703
678
|
const appId = `0x${createHash("sha256").update(cfg.app).digest("hex")}`, owner = `0x${"10ca1".padStart(40, "0")}`;
|
|
704
|
-
const chain = {
|
|
705
|
-
records: schemaRecords(cfg.types, cfg.relations).records.map((r) => ({ ...r, cid: r.id })) } };
|
|
679
|
+
const chain = {};
|
|
706
680
|
// Storage addresses a record by its content, and so does this.
|
|
707
681
|
for (const [ns, records] of Object.entries(readStage(cfg, base, { crawl, only, log: say })))
|
|
708
682
|
chain[`${owner}/${ns}`] = { records: records.map((r) => ({ ...r, cid: createHash("sha256").update(JSON.stringify(stable(r.payload))).digest("hex") })) };
|
|
@@ -711,7 +685,7 @@ export async function buildLocal(configPath, out, { crawl = false, only = null,
|
|
|
711
685
|
const embed = view.cachedEmbed(join(base, ".ship/vectors.ndjson"));
|
|
712
686
|
const repos = [...new Set(cfg.sources.map((s) => s.namespace))].filter((n) => !only || only.includes(n));
|
|
713
687
|
const report = await view.publishView({ fangorn: view.localFangorn(chain, { appId, owner }), app: cfg.app,
|
|
714
|
-
repos, out, embed, log: say });
|
|
688
|
+
schema: { types: cfg.types, relations: cfg.relations }, repos, out, embed, log: say });
|
|
715
689
|
const rows = Object.values(report).reduce((n, r) => n + r.added, 0);
|
|
716
690
|
say(`local view: ${out} · ${rows} records, ${embed.misses} embedded, ${rows - embed.misses} from the vector cache`);
|
|
717
691
|
return report;
|
|
@@ -720,19 +694,6 @@ export async function buildLocal(configPath, out, { crawl = false, only = null,
|
|
|
720
694
|
// ── cli ───────────────────────────────────────────────────────────────────────
|
|
721
695
|
if (process.argv[1]?.endsWith("ship.js") || process.argv[1]?.endsWith("westmarch-ship")) {
|
|
722
696
|
if (process.argv[2] === "--selfcheck") {
|
|
723
|
-
// The schema as committed, read back the way the chain returns it, must compare
|
|
724
|
-
// equal to the config, or every run re-commits an unchanged schema.
|
|
725
|
-
const { parseSchema } = await import("./view.js");
|
|
726
|
-
const types = { "t.v1": { description: "d", role_map: { title: "name", text: ["body"] }, presentation: { externalUrl: { "t.v1": "{url}" } } },
|
|
727
|
-
"u.v1": { description: "e", role_map: { title: "label" } } };
|
|
728
|
-
const relations = [{ rel: "cites", from: "t.v1", to: "u.v1" }];
|
|
729
|
-
const g = schemaRecords(types, relations);
|
|
730
|
-
// Keys reversed at every depth, as a canonical (sorted) encoding hands them back.
|
|
731
|
-
const reverse = (v) => Array.isArray(v) ? v.map(reverse)
|
|
732
|
-
: v && typeof v === "object" ? Object.fromEntries(Object.keys(v).reverse().map((k) => [k, reverse(v[k])])) : v;
|
|
733
|
-
const readBack = parseSchema({ records: g.records.map((r) => ({ ...r, cid: `cid-${r.id}`, payload: reverse(r.payload) })) });
|
|
734
|
-
if (canonical(readBack.types, readBack.relations) !== canonical(types, relations)) throw new Error("schema does not round-trip");
|
|
735
|
-
if (canonical(readBack.types, readBack.relations) === canonical({ ...types, "v.v1": {} }, relations)) throw new Error("a new type must count as a change");
|
|
736
697
|
const row = { city: "Wisconsin Rapids", state: "WI", site: "https://x", args: ["--archive-page", "/569/A"] };
|
|
737
698
|
const cmd = fill(["py", "--site", "{site}", "--city", "{city}", "{args...}", "--ns", "us-{state|lower}-{city|slug}"], row);
|
|
738
699
|
if (JSON.stringify(cmd) !== JSON.stringify(["py", "--site", "https://x", "--city", "Wisconsin Rapids", "--archive-page", "/569/A", "--ns", "us-wi-wisconsin-rapids"]))
|
|
@@ -750,7 +711,7 @@ if (process.argv[1]?.endsWith("ship.js") || process.argv[1]?.endsWith("westmarch
|
|
|
750
711
|
throw new Error(`private discovery: ${JSON.stringify(dp.x402)}`);
|
|
751
712
|
if (units("0.01") !== "10000" || units("5") !== "5000000" || units("1.5", 2) !== "150") throw new Error("units");
|
|
752
713
|
if (USDC["eip155:84532"].name !== "USDC" || USDC["eip155:8453"].name !== "USD Coin") throw new Error("USDC's EIP-712 name differs on Base Sepolia and Base");
|
|
753
|
-
console.log("ship.js self-check ok — a
|
|
714
|
+
console.log("ship.js self-check ok — a source template fills per row, the Worker config binds its bucket, AI and cron, and the paid route is priced per id in /openapi.json and named in /.well-known/x402");
|
|
754
715
|
process.exit(0);
|
|
755
716
|
}
|
|
756
717
|
const [path, ...flags] = process.argv.slice(2);
|
package/src/publish/view-cli.js
CHANGED
|
@@ -1,12 +1,14 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
2
|
// westmarch-view: publish what an app presents as a view (see view.js).
|
|
3
3
|
//
|
|
4
|
-
// westmarch-view --app my-app --out site/view [--repo catalog …] [--from-block N] [--rebake]
|
|
4
|
+
// westmarch-view --app my-app --out site/view [--config app.json] [--repo catalog …] [--from-block N] [--rebake]
|
|
5
5
|
//
|
|
6
|
+
// The schema (types, relations) is read from --config; without it every type is guessed.
|
|
6
7
|
// Reads only; needs no key. Deploy <out> with the site, and put its URL in the card's
|
|
7
8
|
// `views`. An app on the free tier needs none of this: its Worker runs the same code on a
|
|
8
9
|
// cron (site/worker.js).
|
|
9
10
|
|
|
11
|
+
import { readFileSync } from "node:fs";
|
|
10
12
|
import { homedir } from "node:os";
|
|
11
13
|
|
|
12
14
|
const flags = { repo: [] };
|
|
@@ -18,7 +20,7 @@ for (let i = 2; i < process.argv.length; i++) {
|
|
|
18
20
|
if (m[1] === "repo") flags.repo.push(v); else flags[m[1]] = v;
|
|
19
21
|
}
|
|
20
22
|
if (flags.help || !flags.app || !flags.out) {
|
|
21
|
-
console.error("usage: westmarch-view --app <name|0xid> --out <dir> [--repo <name> …] [--from-block N] [--rebake]");
|
|
23
|
+
console.error("usage: westmarch-view --app <name|0xid> --out <dir> [--config app.json] [--repo <name> …] [--from-block N] [--rebake]");
|
|
22
24
|
process.exit(flags.help ? 0 : 2);
|
|
23
25
|
}
|
|
24
26
|
|
|
@@ -31,7 +33,8 @@ const fangorn = await readOnlyFangorn();
|
|
|
31
33
|
|
|
32
34
|
const t0 = Date.now();
|
|
33
35
|
const report = await publishView({
|
|
34
|
-
fangorn, app: flags.app, repos: flags.repo,
|
|
36
|
+
fangorn, app: flags.app, repos: flags.repo,
|
|
37
|
+
schema: flags.config ? JSON.parse(readFileSync(flags.config, "utf8")) : undefined, out: flags.out,
|
|
35
38
|
fromBlock: flags["from-block"], rebake: !!flags.rebake,
|
|
36
39
|
log: (s) => console.error(s),
|
|
37
40
|
});
|
package/src/publish/view.js
CHANGED
|
@@ -5,8 +5,8 @@
|
|
|
5
5
|
// site/worker.js, on a cron (the free tier)
|
|
6
6
|
//
|
|
7
7
|
// Reads the registry and storage and nothing else: every repo the app presents (its
|
|
8
|
-
// members' repos, less its exclusions, `appRepos`), at their latest heads
|
|
9
|
-
//
|
|
8
|
+
// members' repos, less its exclusions, `appRepos`), at their latest heads; the schema
|
|
9
|
+
// (app.json's `types` + `relations`) comes from the caller. Embeds only records the view does not hold
|
|
10
10
|
// yet, drops removed ones, and writes, through a store (a directory or an R2 bucket):
|
|
11
11
|
//
|
|
12
12
|
// cdn/catalog
|
|
@@ -40,15 +40,9 @@ import { dirname } from "node:path";
|
|
|
40
40
|
import { EMBED_DIM, EMBED_DTYPE, EMBED_MODEL, embedDocumentDirect, packVec, unpackVec } from "../core/embed.js";
|
|
41
41
|
import { coverage as fitCoverage } from "../market/reactions.js";
|
|
42
42
|
|
|
43
|
-
/** The app owner's repo that declares the app's types: one TYPE_TAG record per type, one
|
|
44
|
-
* RELATION_TAG record per allowed reference between two of them. */
|
|
45
|
-
export const SCHEMA_REPO = "fangorn.schema";
|
|
46
|
-
export const TYPE_TAG = "fangorn.type.v1";
|
|
47
|
-
export const RELATION_TAG = "fangorn.relation.v1";
|
|
48
43
|
export const EMBEDDER = { runtime: "transformers.js", model: EMBED_MODEL, dtype: EMBED_DTYPE };
|
|
49
44
|
const SINGULAR = ["identity", "title", "subtitle", "temporal", "spatial", "media", "thread"];
|
|
50
45
|
const MULTI = ["tags", "measures", "relations", "text"];
|
|
51
|
-
const ZERO = "0x0000000000000000000000000000000000000000";
|
|
52
46
|
// Refit coverage once a domain has grown this many times over what it was fit on.
|
|
53
47
|
const COVERAGE_REFIT = 2;
|
|
54
48
|
|
|
@@ -75,17 +69,13 @@ export function composeText(fields, rm) {
|
|
|
75
69
|
return s.slice(0, 1000 - "search_document: ".length);
|
|
76
70
|
}
|
|
77
71
|
|
|
78
|
-
/**
|
|
79
|
-
export function
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
else if (r.type === RELATION_TAG && p.rel && p.from && p.to)
|
|
86
|
-
relations.push({ rel: String(p.rel), from: String(p.from), to: String(p.to) });
|
|
87
|
-
}
|
|
88
|
-
return { types, relations };
|
|
72
|
+
/** app.json's `types` + `relations` → { types: {tag: {description, role_map, presentation}}, relations }. */
|
|
73
|
+
export function schemaOf({ types = {}, relations = [] } = {}) {
|
|
74
|
+
return {
|
|
75
|
+
types: Object.fromEntries(Object.entries(types).map(([tag, t]) =>
|
|
76
|
+
[tag, { description: String(t.description ?? ""), role_map: t.role_map ?? {}, presentation: t.presentation ?? {} }])),
|
|
77
|
+
relations: relations.filter((r) => r.rel && r.from && r.to).map((r) => ({ rel: String(r.rel), from: String(r.from), to: String(r.to) })),
|
|
78
|
+
};
|
|
89
79
|
}
|
|
90
80
|
|
|
91
81
|
/** Declared means declared: every role present, left-out ones empty. */
|
|
@@ -281,7 +271,7 @@ const sha256 = (s) => createHash("sha256").update(s).digest("hex");
|
|
|
281
271
|
|
|
282
272
|
/**
|
|
283
273
|
* Bring the view up to date with what the app presents. `fangorn` is an SDK client (reads
|
|
284
|
-
* only); the view goes to `store`, or the directory `out`. `repos` narrows it to those repo
|
|
274
|
+
* only); `schema` is app.json's { types, relations } (a type it leaves out is guessed); the view goes to `store`, or the directory `out`. `repos` narrows it to those repo
|
|
285
275
|
* names (every publisher's); without it, every repo the app presents. `fromBlock` must be
|
|
286
276
|
* at or before the first update to a repo the app presents, or a repo the scan misses is left out;
|
|
287
277
|
* left out, the SDK's deployment start. `embedBatch(texts)` embeds many at once where the
|
|
@@ -290,25 +280,20 @@ const sha256 = (s) => createHash("sha256").update(s).digest("hex");
|
|
|
290
280
|
* ([{owner, repo, root}], exclusions already applied), as a Worker does across runs.
|
|
291
281
|
* Returns per-domain counts of what changed.
|
|
292
282
|
*/
|
|
293
|
-
export async function publishView({ fangorn, app, repos, out, store = fsStore(out), fromBlock, presented, rebake = false,
|
|
283
|
+
export async function publishView({ fangorn, app, schema, repos, out, store = fsStore(out), fromBlock, presented, rebake = false,
|
|
294
284
|
embed = embedDocumentDirect, embedBatch = (ts) => Promise.all(ts.map(embed)),
|
|
295
285
|
embedder = EMBEDDER, log = console.log, shardRows = 20000, concurrency = 4, open = (p) => p }) {
|
|
296
286
|
fangorn.setAppId(app);
|
|
297
287
|
const appId = fangorn.getAppId();
|
|
298
|
-
|
|
299
|
-
let schema = { types: {}, relations: [] };
|
|
300
|
-
if (owner === ZERO) log(`app ${app} is unclaimed, so it has no schema; every type will be guessed`);
|
|
301
|
-
else {
|
|
302
|
-
try { schema = parseSchema((await retry(() => fangorn.readRepo(owner, SCHEMA_REPO))).contents); }
|
|
303
|
-
catch (e) { log(`schema: none read (${e.message}); every type will be guessed`); }
|
|
304
|
-
}
|
|
288
|
+
schema = schemaOf(schema);
|
|
305
289
|
const declared = Object.keys(schema.types);
|
|
306
290
|
log(`schema: ${declared.length ? declared.join(", ") : "no declared types"}`);
|
|
307
291
|
const schemaHash = sha256(JSON.stringify(schema));
|
|
308
292
|
|
|
309
293
|
// Every repo the app presents, at its accepted head: one log scan, exclusions applied.
|
|
310
294
|
const listed = (presented ?? await retry(() => fangorn.appRepos(fromBlock == null ? {} : { fromBlock: BigInt(fromBlock) })))
|
|
311
|
-
|
|
295
|
+
// ponytail: "fangorn.schema" is where apps shipped before 2026-10-22 committed their schema; never data.
|
|
296
|
+
.filter((t) => t.repo !== "fangorn.schema" && (!repos?.length || repos.includes(t.repo)));
|
|
312
297
|
log(`${listed.length} repo(s) presented${repos?.length ? ` of ${repos.join(", ")}` : ""}`);
|
|
313
298
|
|
|
314
299
|
const report = {};
|
|
@@ -535,10 +520,9 @@ if (typeof process !== "undefined" && import.meta.url === `file://${process.argv
|
|
|
535
520
|
|
|
536
521
|
const APP = `0x${"ab".repeat(32)}`, OWNER = `0x${"11".repeat(20)}`, PUB = `0x${"22".repeat(20)}`;
|
|
537
522
|
const TAG = "shop.item.v1";
|
|
523
|
+
const schema = { types: { [TAG]: { description: "Things for a room.", role_map: { identity: "sku", title: "name", text: "about" },
|
|
524
|
+
presentation: { externalUrl: { [TAG]: "https://shop.example/{sku}" } } } } };
|
|
538
525
|
const chain = {
|
|
539
|
-
[`${OWNER}/${SCHEMA_REPO}`]: { records: [rec("t1", TYPE_TAG, {
|
|
540
|
-
tag: TAG, description: "Things for a room.", role_map: { identity: "sku", title: "name", text: "about" },
|
|
541
|
-
presentation: { externalUrl: { [TAG]: "https://shop.example/{sku}" } } })] },
|
|
542
526
|
[`${PUB}/shop`]: { records: [
|
|
543
527
|
rec("c1", TAG, { sku: "a1", name: "Oak table", about: "a solid oak dining table", price: 3 }),
|
|
544
528
|
rec("c2", TAG, { sku: "b2", name: "Lamp", about: "brass reading lamp", price: 1 }),
|
|
@@ -557,11 +541,11 @@ if (typeof process !== "undefined" && import.meta.url === `file://${process.argv
|
|
|
557
541
|
appendFileSync(`${out}/vectors.ndjson`, '["cut sho');
|
|
558
542
|
eq(packVec(await cachedEmbed(`${out}/vectors.ndjson`, embed)("Title: Oak")), packVec(v1), "a line cut short by a kill is skipped, not fatal");
|
|
559
543
|
embedded.length = 0; rmSync(`${out}/vectors.ndjson`);
|
|
560
|
-
const run = () => publishView({ fangorn, app: APP, out, embed, log: () => {} });
|
|
544
|
+
const run = () => publishView({ fangorn, app: APP, schema, out, embed, log: () => {} });
|
|
561
545
|
|
|
562
546
|
const d = domainFor(APP, PUB, "shop");
|
|
563
547
|
const split = mkdtempSync(`${tmpdir()}/view-`);
|
|
564
|
-
await publishView({ fangorn, app: APP, out: split, embed, log: () => {}, shardRows: 2 });
|
|
548
|
+
await publishView({ fangorn, app: APP, schema, out: split, embed, log: () => {}, shardRows: 2 });
|
|
565
549
|
eq(manifestOf(split, d).shards.map((s) => s.count), [2, 1], "rows past shardRows go to the next shard");
|
|
566
550
|
embedded.length = 0;
|
|
567
551
|
eq((await run())[d], { added: 3, removed: 0 }, "first run embeds everything");
|
|
@@ -626,16 +610,16 @@ if (typeof process !== "undefined" && import.meta.url === `file://${process.argv
|
|
|
626
610
|
let budget = 40;
|
|
627
611
|
const capped = (ts) => (budget -= ts.length) < 0 ? Promise.reject(new Error("daily allowance used")) : Promise.all(ts.map(embed));
|
|
628
612
|
const bd = domainFor(APP, PUB, "big");
|
|
629
|
-
eq((await publishView({ fangorn, app: APP, out: big, repos: ["big"], embedBatch: capped, log: () => {} }))[bd],
|
|
613
|
+
eq((await publishView({ fangorn, app: APP, schema, out: big, repos: ["big"], embedBatch: capped, log: () => {} }))[bd],
|
|
630
614
|
{ added: 32, removed: 0, partial: true }, "an encoder that gives out keeps what it embedded");
|
|
631
615
|
eq([manifestOf(big, bd).count, manifestOf(big, bd).head], [32, null], "served, and the head not taken");
|
|
632
616
|
budget = 1e9;
|
|
633
|
-
eq((await publishView({ fangorn, app: APP, out: big, repos: ["big"], embedBatch: capped, log: () => {} }))[bd],
|
|
617
|
+
eq((await publishView({ fangorn, app: APP, schema, out: big, repos: ["big"], embedBatch: capped, log: () => {} }))[bd],
|
|
634
618
|
{ added: 48, removed: 0 }, "the next run embeds only the rest");
|
|
635
619
|
eq(manifestOf(big, bd).count, 80, "and serves all of it");
|
|
636
620
|
delete chain[BIG];
|
|
637
621
|
rmSync(big, { recursive: true, force: true });
|
|
638
|
-
eq((await publishView({ fangorn, app: APP, out, embed, rebake: true, log: () => {} }))[d],
|
|
622
|
+
eq((await publishView({ fangorn, app: APP, schema, out, embed, rebake: true, log: () => {} }))[d],
|
|
639
623
|
{ added: 2, removed: 0 }, "--rebake starts the domain over");
|
|
640
624
|
eq(readdirSync(`${out}/cdn/domains/${d}/shards`).length, 1, "and leaves no orphaned shards");
|
|
641
625
|
rmSync(out, { recursive: true, force: true });
|
|
@@ -644,16 +628,16 @@ if (typeof process !== "undefined" && import.meta.url === `file://${process.argv
|
|
|
644
628
|
// undeclared one is left alone; a target whose identity moves relinks the row without embedding it.
|
|
645
629
|
{
|
|
646
630
|
const T = "civic.item.v1", M = "civic.matter.v1";
|
|
647
|
-
const
|
|
648
|
-
|
|
649
|
-
|
|
650
|
-
|
|
631
|
+
const relSchema = { types: { [T]: { role_map: { identity: "item_id", title: "heading", thread: "part_of" } },
|
|
632
|
+
[M]: { role_map: { identity: "matter_id", title: "title" } } },
|
|
633
|
+
relations: [{ rel: "part_of", from: T, to: M }] };
|
|
634
|
+
const rel = {
|
|
651
635
|
[`${PUB}/civic`]: { records: [
|
|
652
636
|
rec("i1", T, { item_id: "a#1", heading: "Rezone, committee", part_of: { local: "m" } }),
|
|
653
637
|
rec("i2", T, { item_id: "b#4", heading: "Rezone, council", cites: { local: "m" } }),
|
|
654
638
|
rec("m1", M, { matter_id: "plover:7", title: "Rezone" }, "m")] } };
|
|
655
639
|
const o = mkdtempSync(`${tmpdir()}/view-`), dr = domainFor(APP, PUB, "civic");
|
|
656
|
-
const go = () => publishView({ fangorn: localFangorn(rel, { appId: APP, owner: OWNER }), app: APP, out: o, embed, log: () => {} });
|
|
640
|
+
const go = () => publishView({ fangorn: localFangorn(rel, { appId: APP, owner: OWNER }), app: APP, schema: relSchema, out: o, embed, log: () => {} });
|
|
657
641
|
const rows = async () => Object.fromEntries([...(await loadDomain(fsStore(o), `cdn/domains/${dr}`)).rows.values()]
|
|
658
642
|
.map((r) => [r.fields.heading ?? r.fields.title, r.fields.part_of ?? r.fields.cites ?? null]));
|
|
659
643
|
await go();
|
|
@@ -682,7 +666,7 @@ if (typeof process !== "undefined" && import.meta.url === `file://${process.argv
|
|
|
682
666
|
},
|
|
683
667
|
};
|
|
684
668
|
const r2 = r2Store(bucket);
|
|
685
|
-
const via = await publishView({ fangorn, app: APP, store: r2, embed, log: () => {} });
|
|
669
|
+
const via = await publishView({ fangorn, app: APP, schema, store: r2, embed, log: () => {} });
|
|
686
670
|
eq(Object.keys(via), [d], "a view builds into a bucket");
|
|
687
671
|
eq(await r2.list("cdn/domains"), [d], "list names a folder's children");
|
|
688
672
|
eq((await liveFields(r2)).length, 2, "and reads back");
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# Ship when app.json changes (or by hand); grade the live view daily. Data never comes
|
|
2
|
+
# through here: publishers commit it to the app, and the Worker indexes it every 5 minutes.
|
|
3
|
+
name: ship
|
|
4
|
+
on:
|
|
5
|
+
push: { branches: [main], paths: [app.json] }
|
|
6
|
+
schedule: [{ cron: "17 13 * * *" }]
|
|
7
|
+
workflow_dispatch:
|
|
8
|
+
concurrency: { group: ship, cancel-in-progress: false }
|
|
9
|
+
permissions: { contents: write } # to commit .ship/state.json back
|
|
10
|
+
jobs:
|
|
11
|
+
ship:
|
|
12
|
+
runs-on: ubuntu-latest
|
|
13
|
+
timeout-minutes: 30
|
|
14
|
+
env:
|
|
15
|
+
ETH_PRIVATE_KEY: ${{ secrets.ETH_PRIVATE_KEY }}
|
|
16
|
+
PINATA_JWT: ${{ secrets.PINATA_JWT }}
|
|
17
|
+
PINATA_GATEWAY: ${{ vars.PINATA_GATEWAY }}
|
|
18
|
+
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
|
|
19
|
+
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
|
|
20
|
+
steps:
|
|
21
|
+
- uses: actions/checkout@v4
|
|
22
|
+
- uses: actions/setup-node@v4
|
|
23
|
+
with: { node-version: 22, cache: npm }
|
|
24
|
+
- if: github.event_name != 'schedule'
|
|
25
|
+
run: ci/ship.sh ship
|
|
26
|
+
- if: github.event_name != 'schedule'
|
|
27
|
+
name: keep the claim block, URL and agent id
|
|
28
|
+
run: |
|
|
29
|
+
git add .ship/state.json
|
|
30
|
+
git diff --cached --quiet || { git -c user.name=github-actions -c user.email=41898282+github-actions[bot]@users.noreply.github.com commit -m "ship: state" && git push; }
|
|
31
|
+
- uses: actions/cache@v4
|
|
32
|
+
with: { path: ~/.cache/fangorn-mcp/models, key: models-embeddinggemma }
|
|
33
|
+
- run: ci/ship.sh eval
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
# A Fangorn app
|
|
2
|
+
|
|
3
|
+
This repo holds the app's recipe, not its data: what a record is (`app.json`), what a good
|
|
4
|
+
answer looks like (`eval/`), and how to ship it (`ci/`). It can be public. The data lives
|
|
5
|
+
on chain and IPFS, committed by the app's publishers; the Worker indexes it every 5
|
|
6
|
+
minutes, and serves the site, the view, `/mcp` and the agent card.
|
|
7
|
+
|
|
8
|
+
## Set up (testnet, your own Cloudflare)
|
|
9
|
+
|
|
10
|
+
1. Edit `app.json`: `app` (the name you claim, globally unique), `name`, `description`,
|
|
11
|
+
`site.project` (the Worker's name), and the type: rename `my-app.item.v1` and map your
|
|
12
|
+
fields in `role_map`.
|
|
13
|
+
2. Add repo secrets (Settings → Secrets and variables → Actions):
|
|
14
|
+
- `ETH_PRIVATE_KEY`: a Base Sepolia wallet with test ETH; it claims and owns the app
|
|
15
|
+
- `PINATA_JWT`, and the variable `PINATA_GATEWAY`
|
|
16
|
+
- `CLOUDFLARE_API_TOKEN` (template "Edit Cloudflare Workers", plus R2 and Workers AI) and `CLOUDFLARE_ACCOUNT_ID`
|
|
17
|
+
3. Push `app.json` to `main`. The `ship` workflow claims the app, deploys the Worker and
|
|
18
|
+
binds its agent, then commits `.ship/state.json` back. Ship again whenever `app.json` changes.
|
|
19
|
+
|
|
20
|
+
## Publish data
|
|
21
|
+
|
|
22
|
+
Any member of the app commits records of its type, from anywhere:
|
|
23
|
+
|
|
24
|
+
```sh
|
|
25
|
+
echo '{"records":[{"id":"first","type":"my-app.item.v1","payload":{"id":"first","title":"Hello","text":"The first item.","tags":["demo"]}}]}' > items.json
|
|
26
|
+
fangorn --app my-app repo init items
|
|
27
|
+
fangorn --app my-app commit items.json -m "first items"
|
|
28
|
+
fangorn --app my-app push
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
The view picks it up within 5 minutes at `<url>/view`. A source that crawls something
|
|
32
|
+
on a schedule goes in `app.json` `sources`; see westmarch's docs.
|
|
33
|
+
|
|
34
|
+
## Keep it good
|
|
35
|
+
|
|
36
|
+
Write `eval/goals.md` (who it serves) and `eval/golden.jsonl` (one check per line:
|
|
37
|
+
`{"id","q","expect","min","goal"}`). The workflow grades the live view daily, and the run
|
|
38
|
+
goes red when a check fails. `ci/ship.sh` runs the same steps on any CI or a laptop.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
{
|
|
2
|
+
"app": "my-app",
|
|
3
|
+
"name": "My App",
|
|
4
|
+
"description": "One sentence: what is in here, and who it is for.",
|
|
5
|
+
"network": "baseSepolia",
|
|
6
|
+
"site": {
|
|
7
|
+
"project": "my-app",
|
|
8
|
+
"examples": []
|
|
9
|
+
},
|
|
10
|
+
"tags": [],
|
|
11
|
+
"types": {
|
|
12
|
+
"my-app.item.v1": {
|
|
13
|
+
"description": "One item: what a record is, in the words a searcher would use.",
|
|
14
|
+
"role_map": {
|
|
15
|
+
"identity": "id",
|
|
16
|
+
"title": "title",
|
|
17
|
+
"tags": ["tags"],
|
|
18
|
+
"text": ["text"]
|
|
19
|
+
},
|
|
20
|
+
"presentation": {
|
|
21
|
+
"types": { "my-app.item.v1": { "singular": "item", "plural": "items" } },
|
|
22
|
+
"facets": ["tags"]
|
|
23
|
+
}
|
|
24
|
+
}
|
|
25
|
+
},
|
|
26
|
+
"sources": []
|
|
27
|
+
}
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
# Portable CI: any runner (or a laptop) with node 22 and jq.
|
|
3
|
+
# ci/ship.sh ship claim on first run, commit the schema, build the site, deploy the Worker, bind the agent
|
|
4
|
+
# ci/ship.sh eval grade the live view against eval/golden.jsonl
|
|
5
|
+
# Env: ETH_PRIVATE_KEY, PINATA_JWT, PINATA_GATEWAY, CLOUDFLARE_API_TOKEN, CLOUDFLARE_ACCOUNT_ID
|
|
6
|
+
set -euo pipefail
|
|
7
|
+
cd "$(dirname "$0")/.."
|
|
8
|
+
export FANGORN_IPFS_GATEWAY="${FANGORN_IPFS_GATEWAY:-${PINATA_GATEWAY:-}}" # westmarch reads this name, not PINATA_GATEWAY
|
|
9
|
+
|
|
10
|
+
grep -q '"app": "my-app"' app.json && { echo "edit app.json first: app, name, description, site.project, the type" >&2; exit 1; }
|
|
11
|
+
[ -d node_modules ] || npm ci
|
|
12
|
+
# ship signs with the fangorn CLI, which must be the SDK version westmarch reads with
|
|
13
|
+
command -v fangorn >/dev/null || npm i -g "@fangorn-network/sdk@$(node -p 'require("@fangorn-network/sdk/package.json").version')"
|
|
14
|
+
|
|
15
|
+
case "${1:-}" in
|
|
16
|
+
ship) npx westmarch-ship app.json ;;
|
|
17
|
+
eval)
|
|
18
|
+
[ -s eval/golden.jsonl ] || { echo "no golden queries in eval/golden.jsonl yet: nothing to grade"; exit 0; }
|
|
19
|
+
npx westmarch-eval "$(jq -r .url .ship/state.json)/view" ;;
|
|
20
|
+
*) echo "usage: ci/ship.sh ship|eval" >&2; exit 2 ;;
|
|
21
|
+
esac
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
# Goals
|
|
2
|
+
|
|
3
|
+
One section per kind of person this app serves. Each check in golden.jsonl names one with `goal`.
|
|
4
|
+
|
|
5
|
+
## <goal-id>: who finds what
|
|
6
|
+
- **They ask:** two or three real questions, in their words.
|
|
7
|
+
- **Failure looks like:** what a wrong or missing answer would be.
|
|
File without changes
|