@jimhoyd/urlcode 0.4.0-alpha.1 → 0.4.0-alpha.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/urlcode-authoring/SKILL.md +36 -22
- package/.claude/skills/urlcode-operations/SKILL.md +16 -22
- package/.claude-plugin/marketplace.json +1 -1
- package/CONTRIBUTING.md +20 -16
- package/README.md +83 -80
- package/ROADMAP.md +61 -331
- package/SECURITY.md +23 -8
- package/dist/BUILD-MANIFEST.json +42 -47
- package/dist/agents-guide.js +18 -18
- package/dist/build-static.js +136 -0
- package/dist/capabilities.js +85 -31
- package/dist/capability-query.js +0 -1
- package/dist/cli.js +24 -32
- package/dist/compliance-rules/baseline.js +2 -10
- package/dist/compliance-rules/privacy.js +5 -16
- package/dist/compliance-rules/shared.js +0 -2
- package/dist/compliance.js +6 -8
- package/dist/config.js +15 -8
- package/dist/context.js +9 -10
- package/dist/examples.js +2 -2
- package/dist/explain-cli.js +7 -5
- package/dist/explain.js +10 -5
- package/dist/extensions.js +61 -2
- package/dist/function-sources.js +34 -2
- package/dist/function-worker.js +3 -1
- package/dist/functions.js +84 -13
- package/dist/guest-api.js +29 -3
- package/dist/index.js +3 -6
- package/dist/manifest.js +11 -7
- package/dist/mcp-authoring.js +2 -2
- package/dist/mcp.js +1 -1
- package/dist/observability.js +1 -21
- package/dist/policies/cache.js +8 -3
- package/dist/policies.js +3 -1
- package/dist/policy.js +28 -9
- package/dist/prerender.js +4 -0
- package/dist/project-tests.js +3 -3
- package/dist/readiness.js +35 -9
- package/dist/route-diff.js +12 -5
- package/dist/router.js +5 -7
- package/dist/runtime.js +77 -58
- package/dist/sandbox.js +48 -0
- package/dist/scaffold.js +0 -0
- package/dist/scripts/operational-drills.js +12 -54
- package/dist/server.js +3 -29
- package/dist/tooling.js +1 -1
- package/dist/trusted-functions.js +210 -0
- package/dist/types/build-static.d.ts +43 -0
- package/dist/types/capabilities.d.ts +14 -5
- package/dist/types/compliance-rules/shared.d.ts +0 -2
- package/dist/types/compliance.d.ts +0 -3
- package/dist/types/config.d.ts +2 -1
- package/dist/types/context.d.ts +2 -1
- package/dist/types/examples.d.ts +1 -1
- package/dist/types/explain.d.ts +6 -0
- package/dist/types/extensions.d.ts +55 -0
- package/dist/types/function-sources.d.ts +4 -0
- package/dist/types/functions.d.ts +48 -5
- package/dist/types/guest-api.d.ts +1 -0
- package/dist/types/index.d.ts +3 -6
- package/dist/types/manifest.d.ts +5 -3
- package/dist/types/observability.d.ts +1 -14
- package/dist/types/project-tests.d.ts +1 -2
- package/dist/types/readiness.d.ts +12 -3
- package/dist/types/router.d.ts +2 -1
- package/dist/types/runtime.d.ts +0 -27
- package/dist/types/sandbox.d.ts +12 -0
- package/dist/types/scaffold.d.ts +0 -2
- package/dist/types/server.d.ts +1 -4
- package/dist/types/tooling.d.ts +3 -3
- package/dist/types/trusted-functions.d.ts +29 -0
- package/dist/types/types.d.ts +22 -8
- package/dist/types/verify-deployment.d.ts +2 -2
- package/dist/types.js +25 -5
- package/dist/typescript-authoring.js +67 -18
- package/dist/verify-deployment.js +3 -3
- package/docs/AI-AUTHORING.md +109 -10
- package/docs/ASSETS.md +2 -1
- package/docs/AUTH-BACKUP.md +32 -0
- package/docs/AWS.md +4 -4
- package/docs/BEST-PRACTICES.md +17 -8
- package/docs/CAPABILITIES.md +30 -17
- package/docs/CAPACITY.md +128 -31
- package/docs/CI-FOLLOWUP-2026-09-19.md +97 -0
- package/docs/CI-RELEASE-AUDIT-2026-09-19.md +322 -0
- package/docs/CI.md +8 -3
- package/docs/CLOUDFLARE.md +1 -2
- package/docs/CODEBASE-AUDIT-2026-09-20.md +278 -0
- package/docs/COMPLIANCE.md +6 -9
- package/docs/DEPLOYMENT-CHECKS.md +1 -1
- package/docs/DEVELOPMENT-PIPELINE.md +181 -0
- package/docs/EXTENSIONS.md +193 -11
- package/docs/FRAMEWORK.md +53 -33
- package/docs/FUNCTION-SECURITY.md +173 -32
- package/docs/INSTALL.md +0 -5
- package/docs/LOAD-TESTING.md +4 -4
- package/docs/LOCAL-DEVELOPMENT.md +3 -0
- package/docs/MIDDLEWARE.md +27 -16
- package/docs/MONITORING.md +2 -19
- package/docs/OBSERVABILITY.md +7 -16
- package/docs/OPEN-DECISIONS.md +184 -0
- package/docs/OPERATIONAL-PROOF.md +26 -30
- package/docs/OPERATIONS.md +23 -32
- package/docs/POLICIES.md +27 -8
- package/docs/PRERENDER.md +25 -13
- package/docs/PROJECT-DIRECTION.md +35 -10
- package/docs/READINESS.md +6 -2
- package/docs/README.md +20 -41
- package/docs/RELEASE-0.4.0-alpha.3.md +50 -0
- package/docs/RELEASE-READINESS.md +50 -30
- package/docs/RELEASE-SECURITY.md +72 -86
- package/docs/RESILIENCE.md +16 -15
- package/docs/ROUTING.md +7 -10
- package/docs/SANDBOX-REVIEW.md +19 -6
- package/docs/SCAFFOLDING.md +0 -2
- package/docs/SECURITY-AUDIT.md +42 -2
- package/docs/SPECIFICATION.md +63 -27
- package/docs/SPIKE-AI-FRAMEWORK-BENCHMARK.md +288 -0
- package/docs/SPIKE-BUSINESS-SUITE.md +16 -0
- package/docs/SPIKE-CORE-LAYERING.md +368 -0
- package/docs/SPIKE-DEFAULT-TRUST-MODEL.md +211 -0
- package/docs/SPIKE-LAMBDA-COMPILE.md +12 -2
- package/docs/STARTERS.md +1 -1
- package/docs/STATIC.md +105 -0
- package/docs/TOOLING.md +17 -12
- package/docs/TUNNELS.md +0 -3
- package/docs/TYPESCRIPT-AUTHORING.md +32 -12
- package/docs/TYPESCRIPT.md +25 -4
- package/docs/VERCEL.md +4 -5
- package/docs/VERSION-ALIGNMENT.md +46 -0
- package/docs/YAML-GUIDE.md +0 -3
- package/docs/YAML-REFERENCE.md +16 -6
- package/docs/archive/2026-09-19/EXTENSION-IMPLEMENTATION.md +68 -0
- package/docs/{MANAGEMENT-SECURITY.md → archive/2026-09-19/MANAGEMENT-SECURITY.md} +20 -0
- package/docs/{NEXT-PHASE-PLAN.md → archive/2026-09-19/NEXT-PHASE-PLAN.md} +26 -8
- package/docs/{NEXT-STEPS.md → archive/2026-09-19/NEXT-STEPS.md} +274 -43
- package/docs/archive/2026-09-19/OPEN-DECISIONS.md +277 -0
- package/docs/archive/2026-09-19/RELEASE-SECURITY.md +186 -0
- package/docs/archive/2026-09-19/ROADMAP.md +386 -0
- package/docs/{SPIKE-EXTENSION-MODEL.md → archive/2026-09-19/SPIKE-EXTENSION-MODEL.md} +12 -2
- package/docs/{SPIKE-EXTENSIONS.md → archive/2026-09-19/SPIKE-EXTENSIONS.md} +28 -12
- package/docs/archive/2026-09-19/SPIKE-MONOREPO.md +776 -0
- package/docs/{USABILITY-REVIEW.md → archive/2026-09-19/USABILITY-REVIEW.md} +18 -2
- package/docs/archive/README.md +27 -0
- package/docs/policies/agents.md +1 -1
- package/docs/policies/cache.md +13 -0
- package/docs/policies/compression.md +3 -2
- package/docs/policies/security.md +3 -2
- package/examples/assets/example.yaml +2 -2
- package/examples/aws/example.yaml +1 -1
- package/examples/cloudflare/example.yaml +1 -1
- package/examples/conditions/example.yaml +1 -1
- package/examples/cookbook/example.yaml +1 -1
- package/examples/cookbook/middleware/bucket.mjs +12 -2
- package/examples/cookbook/middleware/locale.mjs +7 -3
- package/examples/cookbook/route-index.json +1 -1
- package/examples/cookbook/routes/middleware.yaml +1 -1
- package/examples/egress/example.yaml +1 -1
- package/examples/extensions/example.yaml +1 -1
- package/examples/prerender/README.md +14 -6
- package/examples/prerender/example.yaml +1 -1
- package/examples/prerender/functions/page.mjs +4 -2
- package/examples/prerender/middleware/template.mjs +1 -1
- package/examples/prerender/prerender.mjs +1 -1
- package/examples/prerender/urlcode.yaml +8 -4
- package/examples/provider-conformance/example.yaml +1 -1
- package/examples/vercel/example.yaml +1 -1
- package/llms-full.txt +686 -239
- package/llms.txt +27 -15
- package/package.json +32 -5
- package/packaging/claude-plugin/.claude-plugin/plugin.json +2 -2
- package/packaging/claude-plugin/skills/urlcode-authoring/SKILL.md +36 -22
- package/packaging/claude-plugin/skills/urlcode-operations/SKILL.md +16 -22
- package/recipes/authenticated-json-api/README.md +4 -3
- package/recipes/authenticated-json-api/functions/profile.mjs +2 -1
- package/recipes/authenticated-json-api/recipe.yaml +2 -2
- package/recipes/contact-form/functions/contact.mjs +2 -1
- package/recipes/contact-form/recipe.yaml +3 -3
- package/recipes/cors-api/README.md +2 -2
- package/recipes/cors-api/recipe.yaml +2 -2
- package/recipes/health-page/README.md +1 -1
- package/recipes/health-page/recipe.yaml +1 -1
- package/recipes/json-api/README.md +1 -1
- package/recipes/json-api/recipe.yaml +4 -4
- package/recipes/middleware/README.md +8 -4
- package/recipes/middleware/middleware/bucket.mjs +12 -2
- package/recipes/middleware/middleware/locale.mjs +7 -3
- package/recipes/middleware/recipe.yaml +1 -1
- package/recipes/protected-download/README.md +1 -1
- package/recipes/protected-download/recipe.yaml +2 -2
- package/recipes/redirect/recipe.yaml +1 -1
- package/recipes/static-plus-api/README.md +2 -2
- package/recipes/static-plus-api/public/index.html +1 -1
- package/recipes/static-plus-api/recipe.yaml +2 -2
- package/recipes/static-plus-api/urlcode.yaml +1 -1
- package/recipes/typescript/README.md +2 -1
- package/recipes/typescript/recipe.yaml +5 -5
- package/recipes/webhook-receiver/README.md +5 -1
- package/recipes/webhook-receiver/recipe.yaml +2 -1
- package/recipes/webhook-receiver/urlcode.yaml +7 -0
- package/schemas/recipe.schema.json +4 -3
- package/schemas/urlcode.schema.json +14 -41
- package/skills/urlcode/SKILL.md +32 -11
- package/starters/default/AGENTS.md +18 -18
- package/starters/default/urlcode.yaml +0 -1
- package/dist/link-api.js +0 -136
- package/dist/link-cli.js +0 -141
- package/dist/link-events.js +0 -76
- package/dist/link-records.js +0 -31
- package/dist/link-store-worker.js +0 -150
- package/dist/link-store.js +0 -250
- package/dist/management-policy.js +0 -40
- package/dist/sqlite-version.js +0 -6
- package/dist/types/link-api.d.ts +0 -30
- package/dist/types/link-cli.d.ts +0 -37
- package/dist/types/link-events.d.ts +0 -27
- package/dist/types/link-records.d.ts +0 -11
- package/dist/types/link-store-worker.d.ts +0 -1
- package/dist/types/link-store.d.ts +0 -130
- package/dist/types/management-policy.d.ts +0 -8
- package/dist/types/sqlite-version.d.ts +0 -1
- package/docs/DYNAMIC-LINKS.md +0 -61
- package/docs/links/cli.md +0 -110
- package/docs/links/limits.md +0 -175
- package/docs/links/management-api.md +0 -80
- package/docs/links/pools.md +0 -75
- package/docs/links/setup.md +0 -135
- package/docs/yaml/links.md +0 -30
- package/examples/live-links/README.md +0 -11
- package/examples/live-links/example.yaml +0 -21
- package/examples/live-links/tests/requests.json +0 -6
- package/examples/live-links/urlcode.yaml +0 -16
package/docs/SANDBOX-REVIEW.md
CHANGED
|
@@ -1,9 +1,19 @@
|
|
|
1
1
|
# Independent sandbox review gate
|
|
2
2
|
|
|
3
|
+
Scope: this gate is about the `sandbox: true` execution path only —
|
|
4
|
+
`function`/`middleware` routes run trusted and unsandboxed by default
|
|
5
|
+
(docs/SPIKE-DEFAULT-TRUST-MODEL.md), and a trusted route's full Node access is
|
|
6
|
+
by design, not a gap this review closes. Nothing here claims to review
|
|
7
|
+
arbitrary trusted project code; that trust is the project's own call, made
|
|
8
|
+
per route.
|
|
9
|
+
|
|
3
10
|
Status: **external assessment not performed**. Internal source review, CI, CodeQL
|
|
4
11
|
and adversarial regressions are useful evidence, not an independent sign-off.
|
|
5
|
-
Do not host anonymous hostile multi-tenant code
|
|
6
|
-
|
|
12
|
+
Do not host anonymous hostile multi-tenant code in a `sandbox: true` route
|
|
13
|
+
before this gate is closed, and do not run untrusted/unreviewed code in a
|
|
14
|
+
trusted (non-`sandbox`) route at all — that path was never sandboxed and this
|
|
15
|
+
gate does not apply to it. No unrestricted Node execution fallback is
|
|
16
|
+
permitted *within a route that declared `sandbox: true`*.
|
|
7
17
|
|
|
8
18
|
## Review package
|
|
9
19
|
|
|
@@ -15,14 +25,17 @@ resource settings. Give a reviewer independent of the implementation access to:
|
|
|
15
25
|
module graph, import denial, export validation, binding grants and message bridge.
|
|
16
26
|
- `src/config.ts`, `src/config-worker.ts`, router and assets: parser/schema limits,
|
|
17
27
|
file containment, activation, memory amplification and host-side compilation.
|
|
18
|
-
- HTTP server
|
|
19
|
-
|
|
20
|
-
- `test/sandbox.test.ts`, middleware/config/links/logging/reload tests, Dockerfile,
|
|
28
|
+
- HTTP server and policy: request smuggling, admission, body/response framing.
|
|
29
|
+
- `test/sandbox.test.ts`, middleware/config/logging/reload tests, Dockerfile,
|
|
21
30
|
protected workflows and `docs/FUNCTION-SECURITY.md`.
|
|
31
|
+
- `src/runtime.ts`'s dispatch decision (`route.sandbox ? pool : trusted`) and
|
|
32
|
+
`src/trusted-functions.ts`: confirm a route that declares `sandbox: true`
|
|
33
|
+
can never be dispatched through the trusted, in-process path by any code
|
|
34
|
+
path, and that `sandbox: false`/absent never reaches `FunctionPool`.
|
|
22
35
|
|
|
23
36
|
Run `npm ci --ignore-scripts`, `npm run verify`, `npm run test:package`, and
|
|
24
37
|
`node scripts/operational-drills.ts`. Record the exact commands and result files.
|
|
25
|
-
CI adds constrained-container
|
|
38
|
+
CI adds a constrained-container test.
|
|
26
39
|
Use only disposable local/staging systems with synthetic data.
|
|
27
40
|
|
|
28
41
|
## Threat model and required probes
|
package/docs/SCAFFOLDING.md
CHANGED
|
@@ -68,5 +68,3 @@ Scaffolding does not recursively invent dependencies imported by existing code,
|
|
|
68
68
|
crawl HTML/CSS links, generate binary content, or implement your business logic.
|
|
69
69
|
See [readiness checks](READINESS.md) and [asset behavior](ASSETS.md).
|
|
70
70
|
|
|
71
|
-
Scaffolding enforces the entry-level `dynamicLinks` opt-in for `link` handlers
|
|
72
|
-
and reports the effective boolean. It never enables this capability for you.
|
package/docs/SECURITY-AUDIT.md
CHANGED
|
@@ -1,3 +1,38 @@
|
|
|
1
|
+
# Security review — 2026-09-18
|
|
2
|
+
|
|
3
|
+
Scope: trust-model change (docs/SPIKE-DEFAULT-TRUST-MODEL.md). `function`/
|
|
4
|
+
`middleware` routes now run trusted and unsandboxed by default, with
|
|
5
|
+
`sandbox: true` as an explicit per-route opt-in to the isolation this document's
|
|
6
|
+
earlier entries describe. This is a maintainer-decided policy reversal, not a
|
|
7
|
+
finding; it is recorded here because it changes what every earlier entry's
|
|
8
|
+
"guest"/"sandbox" language means going forward.
|
|
9
|
+
|
|
10
|
+
**What did not change:** the `sandbox: true` execution path — QuickJS/WASM
|
|
11
|
+
isolation, module-graph restriction, fresh heap per call, worker deadline —
|
|
12
|
+
is byte-for-byte the same as every earlier entry describes; nothing in this
|
|
13
|
+
change touched `src/functions.ts`, `src/function-worker.ts` or `src/guest-api.ts`.
|
|
14
|
+
Binding grants are unaffected either way: `env`/`secrets` still reach only a
|
|
15
|
+
route that explicitly declared them and an operator policy pinned to the
|
|
16
|
+
project revision explicitly granted, whether that route is trusted or
|
|
17
|
+
sandboxed.
|
|
18
|
+
|
|
19
|
+
**What did change, and the resulting residual risk:** a `function`/
|
|
20
|
+
`middleware` route with no `sandbox` field (the common case going forward, and
|
|
21
|
+
every existing project's routes after an upgrade with no YAML change) now runs
|
|
22
|
+
in the host process with full Node access — the isolation earlier entries'
|
|
23
|
+
"remaining gaps" language assumed for *all* guest code no longer applies to
|
|
24
|
+
it. The residual-risk framing in earlier entries ("before exposing hostile
|
|
25
|
+
multi-tenant workloads, obtain independent review of WASM/host boundaries")
|
|
26
|
+
is scoped to `sandbox: true` routes specifically; it was never a claim that
|
|
27
|
+
covered a route that opts out of the sandbox, and after this change most
|
|
28
|
+
routes do exactly that by default. Whether a given project's own function/
|
|
29
|
+
middleware code is safe to trust with full host access is now the project's
|
|
30
|
+
judgment call, not something this runtime's isolation reviews (past or
|
|
31
|
+
future) can speak to. New source review of the dispatch decision itself
|
|
32
|
+
(`src/runtime.ts`, `src/trusted-functions.ts`, `src/policy.ts`'s split
|
|
33
|
+
grant-hashing) accompanies the change; see the pull request that introduced
|
|
34
|
+
`sandbox` for its own description of what was and was not verified.
|
|
35
|
+
|
|
1
36
|
# Security review — 2026-09-17
|
|
2
37
|
|
|
3
38
|
Scope: follow-up source review of worker/connection replacement, probe admission,
|
|
@@ -5,6 +40,11 @@ request correlation, operational logging, dependency/release supply chain and th
|
|
|
5
40
|
live-link Node requirement. Regression tests accompany the fixes. Internal review,
|
|
6
41
|
not an independent penetration test.
|
|
7
42
|
|
|
43
|
+
This is a dated record, not current guidance. The `link` handler, its SQLite
|
|
44
|
+
store and the link-store/management findings below were removed from core in
|
|
45
|
+
a later change; they describe the runtime as it existed on this date, not the
|
|
46
|
+
current one.
|
|
47
|
+
|
|
8
48
|
## Findings fixed in this revision
|
|
9
49
|
|
|
10
50
|
| Finding | Impact and evidence | Fix / regression |
|
|
@@ -45,7 +85,7 @@ Management event status 0 means no response headers were sent before disconnect.
|
|
|
45
85
|
An aborted request may have committed a mutation: reconcile record/version before
|
|
46
86
|
retrying. These logs are best effort, may be dropped under pressure, identify a configured credential ID (or a legacy shared token), and are not a
|
|
47
87
|
tamper-evident journal. Successful store mutations now have separate transactional
|
|
48
|
-
audit records; see [management security](MANAGEMENT-SECURITY.md). A failed sink needs collector/operator recovery;
|
|
88
|
+
audit records; see [management security](archive/2026-09-19/MANAGEMENT-SECURITY.md). A failed sink needs collector/operator recovery;
|
|
49
89
|
URLCode cannot report failures reliably through the same broken output stream.
|
|
50
90
|
|
|
51
91
|
## Boundaries checked
|
|
@@ -82,7 +122,7 @@ limits. No anonymous code/config upload service is approved by these changes.
|
|
|
82
122
|
collection/action scopes, expiry and hot revocation, plus atomic SQLite mutation
|
|
83
123
|
audits are implemented. Legacy shared tokens remain for compatibility. There is
|
|
84
124
|
no public user-account system, MFA/SSO, built-in rate limiter, credential issuance
|
|
85
|
-
service or external tamper-evident archive. See [management security](MANAGEMENT-SECURITY.md).
|
|
125
|
+
service or external tamper-evident archive. See [management security](archive/2026-09-19/MANAGEMENT-SECURITY.md).
|
|
86
126
|
|
|
87
127
|
**Before claiming operational readiness:** execute sustained mixed-workload soak,
|
|
88
128
|
backup restoration, disk-full, process-kill, proxy timeout and rollback drills on
|
package/docs/SPECIFICATION.md
CHANGED
|
@@ -35,7 +35,7 @@ support a terminal `/*` wildcard with an otherwise literal path. Route keys cann
|
|
|
35
35
|
percent encoding, spaces, backslashes or query strings. Path length is limited
|
|
36
36
|
to 2,048 characters and 32 segments. `/_urlcode` is reserved.
|
|
37
37
|
|
|
38
|
-
One handler per route: `function`, `redirect`, `page`, `static`, `download`, `respond`, `
|
|
38
|
+
One handler per route: `function`, `redirect`, `page`, `static`, `download`, `respond`, `conditional`, `proxy` or an `extension` mount.
|
|
39
39
|
See [asset configuration](ASSETS.md) for file handlers. Optional properties:
|
|
40
40
|
|
|
41
41
|
- `methods`: unique HTTP methods; default GET and HEAD. Explicit lists are exact;
|
|
@@ -56,8 +56,8 @@ Requests decode the path once; invalid UTF-8/percent encoding, encoded slashes o
|
|
|
56
56
|
backslashes, control characters and dot segments return 400. Query values decode
|
|
57
57
|
once. Incoming query data is not automatically forwarded.
|
|
58
58
|
|
|
59
|
-
See [route matching
|
|
60
|
-
limits
|
|
59
|
+
See [route matching](ROUTING.md) for examples, precedence, wildcard
|
|
60
|
+
limits and reload behavior.
|
|
61
61
|
|
|
62
62
|
The optional top-level `site` block (entry file only) generates native routes
|
|
63
63
|
for site conventions: `robots` → `/robots.txt`, `sitemap` → `/sitemap.xml`,
|
|
@@ -157,15 +157,11 @@ keys preserve repeated values. Headers are forwarded only through explicit maps.
|
|
|
157
157
|
|
|
158
158
|
## Stored links
|
|
159
159
|
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
`link
|
|
165
|
-
path input against an operator-bound store. GET/HEAD only. The logical collection
|
|
166
|
-
is portable; file paths and store credentials are external deployment bindings.
|
|
167
|
-
No general storage capability is exposed to guest code. See [dynamic links](DYNAMIC-LINKS.md)
|
|
168
|
-
for validation, persistence, mutation, expiry and read-after-write behavior.
|
|
160
|
+
Core has no native `link` handler or `dynamicLinks` project flag; both were
|
|
161
|
+
removed. The `urlcode-dynamic-link` extension package that replaced them has
|
|
162
|
+
been retired and unpublished, so a project needing stored short links owns that
|
|
163
|
+
storage itself; there is no in-core replacement or deprecation shim for
|
|
164
|
+
`link`/`dynamicLinks`.
|
|
169
165
|
|
|
170
166
|
## Functions
|
|
171
167
|
|
|
@@ -211,8 +207,34 @@ ES modules only (`.mjs` or `.js`, independent of Node package settings).
|
|
|
211
207
|
JavaScript modules in a separate output project; serving does not transpile them.
|
|
212
208
|
The build never imports application code into Node, uses fixed compiler settings,
|
|
213
209
|
and does not perform semantic type checking. Grants must target the built
|
|
214
|
-
configuration/source revision. `export` defaults to `default`.
|
|
215
|
-
|
|
210
|
+
configuration/source revision. `export` defaults to `default`.
|
|
211
|
+
|
|
212
|
+
### Trust: unsandboxed by default, `sandbox: true` opt-in
|
|
213
|
+
|
|
214
|
+
A route's `function`/`middleware` chain runs one of two ways, chosen by the
|
|
215
|
+
route's `sandbox` field (false or absent — the default — versus `true`; see
|
|
216
|
+
[docs/SPIKE-DEFAULT-TRUST-MODEL.md](SPIKE-DEFAULT-TRUST-MODEL.md)):
|
|
217
|
+
|
|
218
|
+
- **`sandbox` false/absent (trusted, the default):** the module is imported
|
|
219
|
+
directly into the host process with ordinary Node module resolution — bare
|
|
220
|
+
specifiers, `node:` builtins, npm packages and dynamic `import()` all work,
|
|
221
|
+
none of the module-graph/source-size limits below apply, and there is no
|
|
222
|
+
fresh-heap-per-call reset (module-level state persists across requests like
|
|
223
|
+
any other Node server). The handler/middleware signature and `context`
|
|
224
|
+
shape are unchanged from the sandboxed contract below; a trusted function
|
|
225
|
+
additionally has the full Fetch API, Node built-ins and the filesystem
|
|
226
|
+
available to it, not just the guest text/JSON subset. `args`/`env`/
|
|
227
|
+
`secrets` are exactly what the route declares and an operator grants,
|
|
228
|
+
identically to the sandboxed path.
|
|
229
|
+
- **`sandbox: true`:** the rest of this section, unchanged from every earlier
|
|
230
|
+
release. A sandboxed route's functions and middleware execute inside
|
|
231
|
+
QuickJS/WASM, never through Node imports.
|
|
232
|
+
|
|
233
|
+
The remainder of this section (module restrictions, the guest API subset,
|
|
234
|
+
guest limits and deadlines) describes the `sandbox: true` path specifically,
|
|
235
|
+
unless stated otherwise.
|
|
236
|
+
|
|
237
|
+
Only relative `.js`/`.mjs`
|
|
216
238
|
project imports are supported, with a snapshotted dependency graph. No bare/npm,
|
|
217
239
|
Node built-in, remote, dynamic source imports or `import.meta`. Runtime-created
|
|
218
240
|
imports remain restricted to the route's middleware and handler dependency graphs; there is no fallback.
|
|
@@ -225,7 +247,8 @@ delete/get/has/entries/getSetCookie; Response constructor with string/null body,
|
|
|
225
247
|
Requests decode body bytes as UTF-8. Binary/streaming bodies, URL helpers,
|
|
226
248
|
fetch/WebSocket, crypto and filesystem are not exposed. Promise/async and
|
|
227
249
|
bounded timers (128 pending per invocation) work inside the guest. Unsupported
|
|
228
|
-
APIs fail; they never execute on the host. Do not claim full browser/Node API
|
|
250
|
+
APIs fail; they never execute on the host. Do not claim full browser/Node API
|
|
251
|
+
parity for a `sandbox: true` route; a trusted route has no such restriction.
|
|
229
252
|
|
|
230
253
|
Context contains `inputs.path/query/header`, `args`, `env`, `secrets`. Arguments
|
|
231
254
|
may be scalar literals, input references, `{env: alias}` or `{secret: alias}`.
|
|
@@ -241,12 +264,17 @@ Dotenv supports single-line NAME=value, paired single/double quotes, blank lines
|
|
|
241
264
|
and full-line comments, without expansion/escapes/shell execution. Loading a
|
|
242
265
|
value does not authorize exposing it to a function; the policy still applies.
|
|
243
266
|
|
|
244
|
-
|
|
245
|
-
counters, cached secrets or prototype mutation. QuickJS
|
|
246
|
-
stack limit 512 KiB; outer worker and deployment limits
|
|
247
|
-
not a claim that total process RSS is capped at 32
|
|
248
|
-
saturation returns 503. The independent 5-second
|
|
249
|
-
and returns 504. Generic failures return 502;
|
|
267
|
+
A `sandbox: true` invocation has a fresh guest heap and module state every
|
|
268
|
+
time. No cross-request counters, cached secrets or prototype mutation. QuickJS
|
|
269
|
+
heap limit is 32 MiB, stack limit 512 KiB; outer worker and deployment limits
|
|
270
|
+
are additional defenses, not a claim that total process RSS is capped at 32
|
|
271
|
+
MiB. Two workers, no queue; saturation returns 503. The independent 5-second
|
|
272
|
+
deadline terminates a worker and returns 504. Generic failures return 502;
|
|
273
|
+
worker replacement is bounded. A trusted route has none of this: no fresh
|
|
274
|
+
heap/module reset, no fixed worker-pool ceiling (bounded instead by ordinary
|
|
275
|
+
Node concurrency and the HTTP admission cap), and its deadline races the
|
|
276
|
+
call's promise rather than force-terminating a worker — see
|
|
277
|
+
[capacity](CAPACITY.md) for both models side by side.
|
|
250
278
|
|
|
251
279
|
HEAD invokes the handler as HEAD and suppresses body output. Code must guard
|
|
252
280
|
its own application side effects when future brokered integrations are enabled.
|
|
@@ -255,12 +283,18 @@ not arbitrary Host/forwarded headers. Request/response bodies default to 1 MiB;
|
|
|
255
283
|
response headers 16 KiB, maximum 256 pairs. Hop-by-hop headers are stripped;
|
|
256
284
|
cookies are preserved individually. Default response cache policy is `no-store`.
|
|
257
285
|
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
|
|
261
|
-
|
|
286
|
+
A trusted (non-`sandbox`) route already has unrestricted host execution by
|
|
287
|
+
design — that is the point of the default described above. For a `sandbox:
|
|
288
|
+
true` route, there is no unrestricted host-execution fallback: declarative
|
|
289
|
+
proxy and webhook signals use the separately granted host broker described in
|
|
290
|
+
[egress](EGRESS.md), and that route's guests still have no fetch API or
|
|
291
|
+
general persistent state capability. Approved secrets can be returned by code
|
|
292
|
+
that receives them, in either mode; isolation does not automatically enforce
|
|
262
293
|
information-flow rules on authorized inputs. Keep grants narrow and review the
|
|
263
|
-
exact pinned revision. The sandbox still needs independent
|
|
294
|
+
exact pinned revision. The `sandbox: true` path still needs independent
|
|
295
|
+
security review before hostile multi-tenant use of that specific mode; that
|
|
296
|
+
review's scope was never a claim about a route that opts out of the sandbox
|
|
297
|
+
(see [docs/SANDBOX-REVIEW.md](SANDBOX-REVIEW.md)).
|
|
264
298
|
|
|
265
299
|
## Reload and status
|
|
266
300
|
|
|
@@ -295,7 +329,9 @@ are local Git-owned examples; they grant no capabilities. [Build-time TypeScript
|
|
|
295
329
|
is separate from runtime execution.
|
|
296
330
|
|
|
297
331
|
The [tooling SDK and optional local MCP](TOOLING.md) inspect and validate without
|
|
298
|
-
executing handlers or reading binding values.
|
|
332
|
+
executing handlers or reading binding values. MCP defaults to inspection; the
|
|
333
|
+
explicit `--allow-authoring` option enables bounded project writes, separately
|
|
334
|
+
from host-file loading and grants. [Provider conformance](PROVIDER-VERIFICATION.md)
|
|
299
335
|
distinguishes local adapter replay from actual deployment observations; no real
|
|
300
336
|
provider deployment is implied by CI. [Proxy and signal egress](EGRESS.md) requires
|
|
301
337
|
external revision-pinned operator grants and bounded host-owned transport;
|
|
@@ -0,0 +1,288 @@
|
|
|
1
|
+
# Spike: AI-first URLCode framework benchmark
|
|
2
|
+
|
|
3
|
+
Status: proposed research and execution plan. This document adds no benchmark
|
|
4
|
+
implementation or measured results. It preserves the agreed independent-agent
|
|
5
|
+
experiment and improvement loop; executing the phases below is follow-up work.
|
|
6
|
+
|
|
7
|
+
## Objective and core principle
|
|
8
|
+
|
|
9
|
+
Measure whether autonomous agents can build a correct, realistic application
|
|
10
|
+
with URLCode, how much application intent its abstractions express, how easily
|
|
11
|
+
agents discover those abstractions, and how the resulting application performs.
|
|
12
|
+
Report AI effectiveness, framework expressiveness, AI discoverability and runtime
|
|
13
|
+
performance separately. Do not collapse them into a single score or assume that
|
|
14
|
+
URLCode will outperform another framework.
|
|
15
|
+
|
|
16
|
+
The canonical implementation rule for the explicitly guided run is:
|
|
17
|
+
|
|
18
|
+
> Use URLCode's highest-level declarative features whenever possible. Generate custom JavaScript only when the framework cannot express the requirement.
|
|
19
|
+
|
|
20
|
+
Before implementing functionality manually, check for a declarative primitive,
|
|
21
|
+
YAML configuration, built-in capability, extension, reusable template or documented
|
|
22
|
+
pattern. Custom JavaScript is an escape hatch. The experiment must also establish
|
|
23
|
+
whether an unfamiliar agent discovers this approach without explicit coaching.
|
|
24
|
+
|
|
25
|
+
## Phase 0 — Research existing work first
|
|
26
|
+
|
|
27
|
+
Before building the application or a new harness, investigate:
|
|
28
|
+
|
|
29
|
+
- RealWorld / Conduit and its reusable application and acceptance specification.
|
|
30
|
+
- TechEmpower Framework Benchmarks and their runtime methodology.
|
|
31
|
+
- SWE-bench and other coding-agent benchmarks.
|
|
32
|
+
- Agent productivity studies, framework comparison applications and standard
|
|
33
|
+
CRUD/full-stack benchmark applications.
|
|
34
|
+
- URLCode's existing [agent harness](../benchmarks/agent/README.md), authoring
|
|
35
|
+
evals, [next-steps plan](archive/2026-09-19/NEXT-STEPS.md), and runtime benchmarks in
|
|
36
|
+
`benchmarks/routing.ts`, `benchmarks/bulk.ts` and
|
|
37
|
+
`benchmarks/sandbox-vs-trusted.ts`.
|
|
38
|
+
|
|
39
|
+
Write a dated research report with primary-source links, versions/revisions,
|
|
40
|
+
what each candidate measures, what can be reused, what is missing, and the
|
|
41
|
+
reuse/adapt/build decision. Verify current maintenance status during research;
|
|
42
|
+
do not assume this proposal is novel or that an additional suite is necessary.
|
|
43
|
+
|
|
44
|
+
The existing agent harness already records shared acceptance results, generated
|
|
45
|
+
lines, prompts and provider-reported usage. Its README explicitly limits claims:
|
|
46
|
+
its tasks do not test persistence or runtime performance, its URLCode arm receives
|
|
47
|
+
curated guidance, and stub records are pipeline tests rather than model evidence.
|
|
48
|
+
Reuse suitable accounting and acceptance infrastructure after auditing it against
|
|
49
|
+
the current runtime. Do not treat its guided arm as the unprompted baseline or
|
|
50
|
+
silently reinterpret historical measurements under a new counting rule.
|
|
51
|
+
|
|
52
|
+
Exit criterion: record the selected specification and harness approach before
|
|
53
|
+
implementation begins, including the evidence for any departure from existing work.
|
|
54
|
+
|
|
55
|
+
## Phase 1 — Freeze a realistic, framework-neutral specification
|
|
56
|
+
|
|
57
|
+
Prefer RealWorld when practical. Otherwise justify a smaller RealWorld-inspired
|
|
58
|
+
application, such as TaskFlow: users own projects, projects contain tasks, and
|
|
59
|
+
users cannot access another user's private records. A basic Todo application alone
|
|
60
|
+
is insufficient. Freeze observable API behavior and shared acceptance tests before
|
|
61
|
+
agent runs; use exactly the same requirements for URLCode and Fastify.
|
|
62
|
+
|
|
63
|
+
Exercise routing, CRUD, authentication, authorization, persistent relational data,
|
|
64
|
+
relationships, validation, structured errors, filtering, sorting, pagination,
|
|
65
|
+
environment configuration, secrets, middleware, logging and automated tests.
|
|
66
|
+
Include registration/login, current-user lookup, ownership checks, restart
|
|
67
|
+
persistence, invalid inputs and unauthenticated/unauthorized requests in acceptance
|
|
68
|
+
coverage. Define logout semantics if applicable to the selected authentication
|
|
69
|
+
model. Specify a health endpoint and reproducible schema/migrations and seed data.
|
|
70
|
+
|
|
71
|
+
Also define isolated plaintext, JSON serialization, single-record database read
|
|
72
|
+
and database write endpoints, borrowing established runtime benchmark semantics
|
|
73
|
+
where practical. Freeze response bodies, status codes, headers, data size, database
|
|
74
|
+
work and cache behavior; database reads must actually hit the database, and writes
|
|
75
|
+
must persist. Keep these separate from realistic application workloads.
|
|
76
|
+
|
|
77
|
+
Use the same database engine/version and equivalent data in all comparable runs.
|
|
78
|
+
Choose it before implementation, not separately for each framework. Use synthetic
|
|
79
|
+
data and environment-supplied secrets. Keep benchmark applications isolated from
|
|
80
|
+
production runtime source. Record permitted extension packages and versions;
|
|
81
|
+
extension functionality must not be presented as built into core.
|
|
82
|
+
|
|
83
|
+
Correctness is the first gate: report passed/total acceptance cases, incomplete
|
|
84
|
+
requirements and failures. Do not compare an incomplete application's apparent
|
|
85
|
+
code savings or throughput as if it delivered the same functionality.
|
|
86
|
+
|
|
87
|
+
## Phase 2 — Reproducible harness and measurement
|
|
88
|
+
|
|
89
|
+
Capture exact prompts, model identifiers, agent/version, framework and extension
|
|
90
|
+
commits, lockfiles, documentation snapshots, harness version, runtime, OS, hardware,
|
|
91
|
+
database, tools and configuration. Record the initial workspace and available
|
|
92
|
+
instructions/tools. Fix time, turn and token budgets, stopping rules and permitted
|
|
93
|
+
human assistance before running; log interventions and failed or capped runs.
|
|
94
|
+
|
|
95
|
+
| Area | Measurements |
|
|
96
|
+
|---|---|
|
|
97
|
+
| Agent work | Input/output/cached/total tokens, elapsed time, turns, tool calls, shell commands, documentation searches, test/fix cycles, completion rate |
|
|
98
|
+
| Implementation | Total/source/configuration/JS/TS files, application LOC, configuration LOC, test LOC, direct/transitive dependencies, custom handlers and custom JavaScript |
|
|
99
|
+
| URLCode expression | Declarative routes, routes needing custom code, declarative versus imperative LOC, required extensions, JavaScript escape hatches, functionality manually duplicated despite an existing capability |
|
|
100
|
+
| Runtime | Startup time, idle memory, CPU, memory under load, requests/sec, p50/p95/p99 latency, response errors and timeouts |
|
|
101
|
+
|
|
102
|
+
Never estimate unavailable measurements: mark them unavailable and explain why.
|
|
103
|
+
Preserve provider usage fields and document whether cached tokens are included in
|
|
104
|
+
input totals so totals do not double-count them. Separate measured values from
|
|
105
|
+
derived calculations. Define LOC counting and exclusions in advance; separately
|
|
106
|
+
report configuration, tests, generated output, dependencies and copied templates.
|
|
107
|
+
Existing code-ratio metrics are not automatically declarative-coverage metrics.
|
|
108
|
+
|
|
109
|
+
For runtime comparisons hold hardware/resource limits, runtime, database, data,
|
|
110
|
+
concurrency, warm-up, duration and load generator constant. Specify startup start/
|
|
111
|
+
ready events and memory/CPU sampling. Record trust/sandbox mode, worker counts,
|
|
112
|
+
policies and extension overhead; compare equivalent behavior and disclose any
|
|
113
|
+
unavoidable differences. Do not disable correctness or security requirements to
|
|
114
|
+
improve scores. Reset data between trials and verify benchmark response semantics.
|
|
115
|
+
|
|
116
|
+
Predeclare repeated fresh-agent runs and repeated runtime trials, retain every
|
|
117
|
+
trial and report variability, sample counts and failures rather than selecting
|
|
118
|
+
best runs. Keep the load generator from becoming the bottleneck. Version changed
|
|
119
|
+
prompts, fixtures and counting rules so incompatible results are not pooled.
|
|
120
|
+
|
|
121
|
+
## Phase 3 — Independent URLCode agents
|
|
122
|
+
|
|
123
|
+
Each run starts with a fresh context and isolated worktree/environment. Agents
|
|
124
|
+
must not see another agent's implementation, logs or findings. The coordinator
|
|
125
|
+
collects evidence after each run; shared prompts contain the neutral specification
|
|
126
|
+
and operational rules, not the hypothesis or a desired token/LOC outcome.
|
|
127
|
+
|
|
128
|
+
### Protect the discovery baseline
|
|
129
|
+
|
|
130
|
+
This spike itself contains the instruction being tested. Do not expose it, its
|
|
131
|
+
index entry, benchmark prompts, prepared answers or prior findings to Agent A.
|
|
132
|
+
Use a pinned source/documentation snapshot preceding this spike, or a documented
|
|
133
|
+
filtered workspace with no access to excluded artifacts or repository history.
|
|
134
|
+
Apply the same underlying snapshot and access controls to the comparison runs,
|
|
135
|
+
with only the intended prompt treatment different. Record the exact manifest and
|
|
136
|
+
restrictions; if an agent reads excluded material, mark the run contaminated and
|
|
137
|
+
repeat it with a fresh agent rather than counting it as unprompted evidence.
|
|
138
|
+
|
|
139
|
+
Existing product documentation, skills and ordinary repository instructions are
|
|
140
|
+
part of what Agent A may discover; do not remove existing declarative guidance to
|
|
141
|
+
manufacture a worse baseline. Preserve the baseline before changing those sources.
|
|
142
|
+
|
|
143
|
+
### Agent A — Unprompted discovery
|
|
144
|
+
|
|
145
|
+
Supply the application specification, URLCode source and existing documentation.
|
|
146
|
+
The framework-specific instruction is only:
|
|
147
|
+
|
|
148
|
+
> Build this application using URLCode following the framework's documented conventions and recommended practices.
|
|
149
|
+
|
|
150
|
+
Do not supply the declarative-first rule or curated hints about where to find it.
|
|
151
|
+
Record whether the agent independently finds and uses it, including evidence of
|
|
152
|
+
unnecessary JavaScript, duplicated capabilities, missed primitives, misunderstood
|
|
153
|
+
abstractions, repeated searches, incorrect assumptions, unclear errors and stalls.
|
|
154
|
+
Treat these as potential framework/documentation problems, not automatically as
|
|
155
|
+
agent failures.
|
|
156
|
+
|
|
157
|
+
### Agent B — Explicitly URLCode-native
|
|
158
|
+
|
|
159
|
+
Run the identical application from scratch with a fresh independent agent. Add:
|
|
160
|
+
|
|
161
|
+
> Use URLCode's highest-level declarative features whenever possible. Generate custom JavaScript only when the framework cannot express the requirement.
|
|
162
|
+
|
|
163
|
+
> Before implementing functionality manually, determine whether URLCode already provides a declarative primitive, YAML configuration, built-in capability, extension, reusable template, or documented pattern.
|
|
164
|
+
|
|
165
|
+
Compare A and B on correctness, tokens, custom JavaScript, YAML, files, LOC,
|
|
166
|
+
implementation time, debugging cycles and documentation searches. A repeatable
|
|
167
|
+
advantage for B is evidence to investigate discoverability, not proof that every
|
|
168
|
+
individual mistake is a documentation defect.
|
|
169
|
+
|
|
170
|
+
### Agent C — Gap finder
|
|
171
|
+
|
|
172
|
+
Use another fresh agent, independent of A/B implementations and findings during
|
|
173
|
+
its attempt. Give it the same requirements and this instruction:
|
|
174
|
+
|
|
175
|
+
> Attempt to implement every requirement using URLCode's intended abstractions. Whenever URLCode cannot express something cleanly, document the limitation rather than hiding it behind substantial custom code.
|
|
176
|
+
|
|
177
|
+
Classify findings as missing capability, documentation gap, AI discoverability
|
|
178
|
+
problem, confusing API/schema, poor error message, unnecessary boilerplate,
|
|
179
|
+
extension-system limitation, performance problem, possible framework bug or agent
|
|
180
|
+
misunderstanding. Record the attempted declaration, missing requirement, relevant
|
|
181
|
+
documentation and any necessary workaround. A requirement that cannot be expressed
|
|
182
|
+
stays visibly incomplete; substantial custom code must not hide the gap.
|
|
183
|
+
|
|
184
|
+
## Phase 4 — Verify findings and prepare/file GitHub issues
|
|
185
|
+
|
|
186
|
+
After independent attempts are preserved, reproduce each candidate problem against
|
|
187
|
+
the pinned revision, inspect the supported contract and search existing issues and
|
|
188
|
+
PRs. Distinguish missing functionality from functionality the agent failed to find.
|
|
189
|
+
Do not file an issue merely because one agent made a mistake. Note corroborating
|
|
190
|
+
encounters by multiple fresh agents, while avoiding unsupported causal claims.
|
|
191
|
+
|
|
192
|
+
For each verified, actionable issue include:
|
|
193
|
+
|
|
194
|
+
- Problem and benchmark scenario, affected repository/revision and environment.
|
|
195
|
+
- Expected versus actual behavior and minimal synthetic reproduction.
|
|
196
|
+
- Relevant YAML/code, commands, errors and links to run evidence.
|
|
197
|
+
- Classification, proposed improvement and impact on human developers and agents.
|
|
198
|
+
- Acceptance criteria and related/duplicate issue links.
|
|
199
|
+
|
|
200
|
+
Use the owning repository's issue template and the ownership map in
|
|
201
|
+
[AGENTS.md](../AGENTS.md): core/runtime issues belong here, documentation
|
|
202
|
+
included -- `urlcode-docs` is deleted and cannot accept issues -- and extension
|
|
203
|
+
defects in the corresponding extension repository.
|
|
204
|
+
Update existing issues with new evidence rather than duplicating them. Track
|
|
205
|
+
prepared, filed, duplicate and unverified dispositions in the findings report;
|
|
206
|
+
file verified issues and retain unresolved hypotheses as explicitly unverified
|
|
207
|
+
observations. Follow [SECURITY.md](../SECURITY.md) for private vulnerability reports.
|
|
208
|
+
|
|
209
|
+
## Phase 5 — Audit AI discoverability
|
|
210
|
+
|
|
211
|
+
Audit README, documentation, examples, schemas, CLI help, package metadata,
|
|
212
|
+
`llms.txt`, agent instructions/skills, scaffolding and errors. Determine whether an
|
|
213
|
+
unfamiliar agent can find the framework's purpose, declarative-first philosophy,
|
|
214
|
+
YAML capabilities, built-in primitives, extensions, authentication, database access,
|
|
215
|
+
validation, errors, custom functionality, testing and debugging guidance.
|
|
216
|
+
|
|
217
|
+
Specifically search for guidance equivalent to the canonical rule above; record
|
|
218
|
+
exact locations, prominence and the navigation/search path used to discover it.
|
|
219
|
+
Do not assume it is absent just because the exact sentence is missing. Check the
|
|
220
|
+
current contract when documents and historical benchmark assumptions differ.
|
|
221
|
+
|
|
222
|
+
Preserve Agent A's baseline before editing guidance. Then propose the smallest
|
|
223
|
+
authoritative set of changes that makes the principle clear to people and agents,
|
|
224
|
+
including when custom JavaScript is appropriate. Public authoring guidance and
|
|
225
|
+
contributor records both belong here, now that `urlcode-docs` is deleted. Do not copy the rule
|
|
226
|
+
everywhere or expose the experimental prompt as product guidance by accident.
|
|
227
|
+
|
|
228
|
+
## Phase 6 — Fastify comparison
|
|
229
|
+
|
|
230
|
+
Once the URLCode methodology works, give a fresh independent agent the same frozen
|
|
231
|
+
specification, budgets, tool access and acceptance criteria, using:
|
|
232
|
+
|
|
233
|
+
> Build this application using Fastify following the framework's documented conventions and recommended practices.
|
|
234
|
+
|
|
235
|
+
Do not mention URLCode or expose its implementations/findings. Use Fastify
|
|
236
|
+
idiomatically; do not force it to imitate URLCode's architecture. Compare correctness,
|
|
237
|
+
agent work, files/LOC/dependencies, configuration, debugging/searches and runtime
|
|
238
|
+
measurements separately. Document framework-specific dependencies and setup work.
|
|
239
|
+
Later candidates include Hono, Express, NestJS and Elysia; they are not prerequisites
|
|
240
|
+
for the first comparison.
|
|
241
|
+
|
|
242
|
+
## Phase 7 — Findings and deliverables
|
|
243
|
+
|
|
244
|
+
Publish the contributor findings in `docs/benchmarks/AI_FRAMEWORK_BENCHMARK.md` when
|
|
245
|
+
runs exist. Keep implementation and raw evidence paths versioned and linked from
|
|
246
|
+
that report; do not create a results document implying measurements already exist.
|
|
247
|
+
Deliver:
|
|
248
|
+
|
|
249
|
+
1. Dated benchmark research and reuse decision.
|
|
250
|
+
2. Frozen application specification and shared acceptance suite.
|
|
251
|
+
3. Reproducible harness, commands and independent agent prompts.
|
|
252
|
+
4. URLCode A/B/C and Fastify implementations or explicit incomplete outcomes.
|
|
253
|
+
5. Raw agent logs/usage, implementation counts and runtime trial data.
|
|
254
|
+
6. Methodology, provenance, comparison tables, uncertainty and evidence limitations.
|
|
255
|
+
7. Agent behavior/search/failure analysis and classified framework/discovery gaps.
|
|
256
|
+
8. Verified issue ledger, proposed/filed issue links and prioritized improvements.
|
|
257
|
+
9. Before/after findings following the fresh-agent regression phase.
|
|
258
|
+
|
|
259
|
+
Remove secrets from published logs while retaining measurement provenance. Keep
|
|
260
|
+
stub/harness validation records distinct from real-agent evidence. Report every
|
|
261
|
+
failed attempt, not just successful applications, and avoid subjective scores or
|
|
262
|
+
claims beyond the measured cases.
|
|
263
|
+
|
|
264
|
+
## Phases 8–9 — Improve and rerun with fresh agents
|
|
265
|
+
|
|
266
|
+
Prioritize demonstrated friction: missing declarative primitives, defaults, YAML,
|
|
267
|
+
schemas, errors, examples, extension discovery, README/AI documentation, CLI and
|
|
268
|
+
scaffolding. Make legitimate application-development improvements, not special
|
|
269
|
+
cases that game the benchmark. Track each change to its verified issue/evidence.
|
|
270
|
+
|
|
271
|
+
After improvements, rerun with entirely fresh agents and isolated environments.
|
|
272
|
+
The new discovery agent must again receive no explicit declarative-first rule or
|
|
273
|
+
prior findings; it may discover the improved ordinary documentation naturally.
|
|
274
|
+
Keep this spike and benchmark answers excluded. Repeat B, C and the comparison
|
|
275
|
+
where needed to distinguish framework changes from model/harness drift. Hold model,
|
|
276
|
+
budgets, requirements and runtime conditions fixed where possible; disclose changes
|
|
277
|
+
and do not attribute their effects solely to URLCode.
|
|
278
|
+
|
|
279
|
+
Compare before/after correctness, tokens, elapsed time, generated JavaScript, YAML,
|
|
280
|
+
LOC, files, dependencies, tool calls, searches, failed attempts, test/fix cycles and
|
|
281
|
+
runtime performance. Keep dimensions separate and preserve the original baseline.
|
|
282
|
+
|
|
283
|
+
The repeatable loop is: independent agent encounters friction → evidence captures
|
|
284
|
+
it → verified issue → URLCode improvement → fresh-agent rerun → measured outcome.
|
|
285
|
+
Use it for significant releases to detect regressions in AI usability, declarative
|
|
286
|
+
coverage, documentation, capability and runtime performance. Completion means
|
|
287
|
+
reproducible evidence and an actionable issue/improvement trail, not a claim that
|
|
288
|
+
URLCode won the comparison.
|
|
@@ -1,10 +1,26 @@
|
|
|
1
1
|
# URLCode business suite spike
|
|
2
2
|
|
|
3
|
+
> Review update, 2026-09-19: Current status: an unapproved candidate list, gated on observed repetition
|
|
4
|
+
> and benchmark evidence under PROJECT-DIRECTION.md. The short-link products
|
|
5
|
+
> are retired; the seven-product recommendation below is historical, not agreed
|
|
6
|
+
> current scope. No business-suite implementation is implied.
|
|
7
|
+
|
|
8
|
+
|
|
3
9
|
Date: 2026-09-18. Status: proposal, not an implemented contract or production claim.
|
|
4
10
|
Core inspected at `50790d3` (0.4.0-alpha.1), plus local auth, admin, UI and
|
|
5
11
|
shortener source/status files. Competitor research below is a documentation
|
|
6
12
|
review, not hands-on benchmarking. Features and commercial packaging can change.
|
|
7
13
|
|
|
14
|
+
> **Update:** this proposal was written when `link`/`LinkStore` was still a
|
|
15
|
+
> native core feature. Core no longer has that API — it was extracted to a
|
|
16
|
+
> separate `urlcode-dynamic-link` package, which has since been retired,
|
|
17
|
+
> unpublished and deleted (September 2026), along with the `urlcode-short`
|
|
18
|
+
> shortener this document proposes. Those two products are no longer planned.
|
|
19
|
+
> References below to core owning link storage (e.g. "Reuse core's LinkStore",
|
|
20
|
+
> "Retain core link semantics", the `core LinkStore -> short` dependency line)
|
|
21
|
+
> describe the pre-extraction state this spike was proposing against, and the
|
|
22
|
+
> `urlcode-short` migration sections record an abandoned plan.
|
|
23
|
+
|
|
8
24
|
## Recommendation
|
|
9
25
|
|
|
10
26
|
Build seven independently released Apache-2.0 applications on URLCode:
|