@intent-systems/nexus 2026.1.5-3

package/skills/onboarding/docs/SKILLS_HUB_SPEC.md

@@ -0,0 +1,477 @@
+# Skills Hub Spec (Draft)
+
+This document scopes the Skills Hub and how it fits into the Nexus CLI,
+the Nexus website, and nexus-cloud storage/auth. It also highlights the
+gaps between the current CLI implementation and the new CLI grammar plan.
+
+------------------------------------------------------------------------------
+Goals and positioning
+------------------------------------------------------------------------------
+
+Goals:
+- Central registry for skills with higher quality and security bar.
+- Open-source only for tools/connectors, AI-audited and sandbox-tested.
+- One-command publish and install from the Nexus CLI.
+- Fast, trustworthy discovery with capability-aware search.
+
+Non-goals (v1):
+- Paid marketplace (defer until volume).
+- Fully automated behavioral verification (basic tripwires only).
+- Multi-agent skill execution semantics (handled elsewhere).
+
+------------------------------------------------------------------------------
+Terminology (aligned with onboarding docs)
+------------------------------------------------------------------------------
+
+Skill types:
+- guide (pure): prompt-only, zero config.
+- tool: prompt + open-source binary.
+- connector: prompt + external auth/config.
+
+Capabilities:
+- Abstract goals (email-read, chat-send).
+- Many providers per capability.
+
+------------------------------------------------------------------------------
+Current state vs. desired CLI grammar
+------------------------------------------------------------------------------
+
+New CLI grammar plan (from onboarding docs):
+- `nexus skill use <name>` (primary guide fetch)
+- `nexus skill info <name>`
+- `nexus skill list`
+- `nexus tool verify`, `nexus connector verify`
+- `nexus status`, `nexus capabilities`, `nexus quest`
+
+Current Nexus CLI implementation (repo):
+- `nexus skills list|status|refresh|path|cursor-rules`
+- `nexus skill show|verify|run|stats`
+- Skills loaded from multiple dirs and tracked in `~/nexus/state/skills.json`
+- Managed skills installed to `~/nexus/skills` (ClawdHub CLI)
+
+Gap summary:
+- "use/info/list" vs "show/status/list" naming mismatch.
+- "skills" plural used for local management; "skill" singular used for execution.
+- Docs show only bundled/home/project skill dirs; code includes managed dir.
+
+Recommendation:
+- Keep `nexus skill` for local usage (use/show/info/verify/run/stats).
+- Reserve `nexus skills` for hub/registry actions (search/install/publish/update).
+- Add aliases: `use` -> `show`, `info` -> `show --meta`, `list` -> `skills list`.
+
+------------------------------------------------------------------------------
+Surface area and ownership
+------------------------------------------------------------------------------
+
+Nexus CLI (primary interface):
+- Installation, updates, local listing, usage stats.
+- Always renders SKILL.md for agents (no execution gateway).
+
+Nexus website (registry + UI):
+- Auth, publish, search, details, author profiles, reviews.
+- Quality/security status and audit reports.
+
+nexus-cloud (artifact storage + distribution):
+- R2-backed artifact storage.
+- Refs-based registry pointers (skills/{author}/{name}).
+- Cloud token issuance already supported via website.
+
+nexus-cloud CLI:
+- No direct hub UX required in v1.
+- Optional: reuse cloud auth/token helpers if needed by nexus CLI.
+
+Note:
+- Nexus core is in flux; the hub should ship in website + nexus-cloud first.
+- CLI integration can be delivered via nexus-cloud CLI initially, and mirrored
+  in nexus later.
+
+------------------------------------------------------------------------------
+Website hub scope (v1)
+------------------------------------------------------------------------------
+
+Build in website now:
+- Registry and metadata store (authoritative source of truth).
+- Publish flow (upload or repo link) + versioning.
+- Security/audit pipeline with visible status.
+- Search, browse, and install UX (public).
+- Authenticated dashboards for authors.
+
+Defer:
+- Nexus core CLI changes.
+- Marketplace monetization.
+
+------------------------------------------------------------------------------
+Registry and storage (website + nexus-cloud)
+------------------------------------------------------------------------------
+
+Source of truth:
+- Website DB owns metadata, audit status, and discovery rankings.
+- nexus-cloud stores artifacts and version manifests (R2).
+
+Artifact flow (publish):
+1) Client POSTs metadata to website (session or bearer token).
+2) Website validates + creates draft skill_version row.
+3) Website requests presigned upload URLs from nexus-cloud.
+4) Client uploads artifact (and optional source snapshot) directly to R2.
+5) Website verifies checksum + size, then marks version as pending audit.
+6) Website updates `skills.latest_version_id` when audits pass.
+7) Optional: update refs in nexus-cloud (`skills/{slug}` -> manifest hash).
+
+Download flow:
+1) Website records download event (and optional user id).
+2) Website returns a signed R2 URL (or a nexus-cloud download token).
+3) Client verifies checksum after download.
+
+Caching:
+- Provide a public registry snapshot (ETag + CDN) for fast index hydration.
+- Do not cache audit or safety state longer than a short TTL.
+
+------------------------------------------------------------------------------
+Local skill lifecycle (CLI state)
+------------------------------------------------------------------------------
+
+Local tracking goals:
+- Track installed hub skills with version + manifest hash.
+- Detect newer hub versions and prompt for updates.
+- Preserve local edits and help users merge safely.
+
+Suggested local state:
+- Store per-skill records in `~/.nexus/state/skills.json` (or a dedicated db).
+- Fields: slug, version, manifestHash, installedAt, source (hub/url),
+  managed (bool), overrides (paths or patch refs), lastUpdateCheckAt.
+
+Update UX:
+- `nexus skills status` shows available updates.
+- `nexus skills update <slug>` pulls new version, runs an agent diff/merge,
+  and offers a dry-run before applying.
+- `nexus skills lock` pins versions when users want stability.
+
+------------------------------------------------------------------------------
+Skill packaging and storage
+------------------------------------------------------------------------------
+
+Skill bundle (artifact):
+- SKILL.md
+- assets/
+- references/
+- manifest.json (normalized metadata, derived from frontmatter)
+- tool binary (optional, for tool skills)
+- source snapshot (optional; required if repo is unavailable)
+
+Recommended packaging:
+- .tar.zst or .zip
+- Signed checksum (sha256) stored in metadata
+
+Storage layout (cloud/R2):
+- `skills/{author}/{name}/{version}/artifact.tar.zst`
+- `skills/{author}/{name}/{version}/manifest.json`
+- `skills/{author}/{name}/{version}/source.tar.zst` (optional)
+
+Registry pointer (refs):
+- `refs: skills/{author}/{name} -> {latestVersionHash}`
+
+------------------------------------------------------------------------------
+Metadata schema (extend existing SKILL.md frontmatter)
+------------------------------------------------------------------------------
+
+Frontmatter:
+- name, description (existing)
+- metadata.nexus (single-line JSON)
+
+Additions:
+- type: guide | tool | connector
+- capabilities: [email-read, chat-send, ...]
+- license: SPDX identifier
+- source:
+  - repo: https://github.com/user/repo
+  - commit: abc123
+- security:
+  - openSource: true
+  - audit: required
+- install: (existing) installers for tools
+- requires: (existing) bins/env/config
+
+Normalization (website):
+- Parse SKILL.md frontmatter into a normalized manifest JSON object.
+- Store normalized fields for fast filtering and search.
+- Preserve the original SKILL.md in the artifact bundle.
+
+Recommended manifest fields (JSON, versioned):
+- name, slug, description, version, type
+- capabilities[], tags[], category
+- license, homepage, repository, sourceCommit
+- os[], arch[] (optional)
+- requires { bins?, anyBins?, env?, config? }
+- install[] (brew/node/go/uv installers)
+- security { openSource, auditRequired }
+- files { skillMdHash, artifactHash, sourceHash? }
+- createdAt, publishedBy
+
+------------------------------------------------------------------------------
+Website DB schema (detailed)
+------------------------------------------------------------------------------
+
+Notes:
+- The website uses Drizzle schema definitions but raw SQL for queries.
+- Existing auth tables remain unchanged.
+
+skills
+- id (uuid)
+- slug (unique, "author/name")
+- name, description
+- type (guide|tool|connector)
+- author_id (users.id)
+- status (pending|published|verified|delisted)
+- latest_version_id (fk skill_versions.id)
+- latest_version (string, for quick display)
+- license, homepage, repository, source_commit
+- created_at, updated_at, published_at, verified_at
+- downloads_count, stars_count, rating_avg, rating_count
+- search_document (tsvector, optional)
+- flags_count (moderation)
+- visibility (public|unlisted|private)
+
+skill_versions
+- id (uuid)
+- skill_id (fk skills.id)
+- version (semver)
+- manifest_json (jsonb)
+- artifact_url (R2 object path)
+- artifact_sha256, artifact_bytes
+- source_url (optional), source_sha256 (optional)
+- readme_excerpt (optional for search)
+- created_at, published_by
+- status (draft|pending_audit|published|rejected)
+
+skill_audits
+- id (uuid)
+- skill_version_id (fk skill_versions.id)
+- kind (metadata|license|dependency|static|sandbox|ai)
+- status (pending|running|pass|warn|fail)
+- risk_level (low|medium|high)
+- report_json (jsonb)
+- started_at, completed_at, error
+
+skill_reviews
+- id (uuid)
+- skill_id (fk skills.id)
+- user_id (fk users.id)
+- rating (1-5)
+- comment
+- created_at, updated_at
+- unique (skill_id, user_id)
+
+skill_flags
+- id (uuid)
+- skill_id (fk skills.id)
+- user_id (fk users.id)
+- reason, details
+- created_at
+
+skill_downloads
+- id (uuid)
+- skill_id (fk skills.id)
+- version_id (fk skill_versions.id)
+- user_id (nullable)
+- ip_hash (optional), user_agent
+- created_at
+
+Optional for scale:
+- skill_dependencies (skill_id, dependency_slug, kind)
+- skill_capabilities (skill_id, capability)
+- skill_stats_daily (skill_id, date, downloads, installs, views)
+- skill_jobs (async audit queue)
+
+Indexes:
+- skills.slug (unique)
+- skills.type, skills.status
+- skill_versions.skill_id + version (unique)
+- skill_reviews.skill_id + user_id (unique)
+- skills.search_document (GIN, if using FTS)
+- skill_stats_daily.skill_id + date
+
+------------------------------------------------------------------------------
+Website API (detailed)
+------------------------------------------------------------------------------
+
+Public:
+- GET /api/skills?q=&type=&capability=&author=&license=&verified=&sort=
+- GET /api/skills/:slug
+- GET /api/skills/:slug/versions
+- GET /api/skills/:slug/manifest?version=1.2.3
+- GET /api/skills/registry.json (cached snapshot for index hydration)
+- GET /api/skills/capabilities (list + counts)
+- GET /api/users/:id/profile
+
+Authenticated (bearer token):
+- POST /api/skills (create skill + version)
+- POST /api/skills/:slug/versions (publish new version)
+- POST /api/skills/:slug/download (track + return signed URL)
+- POST /api/skills/:slug/review (rating/comment)
+- POST /api/skills/:slug/flag (report)
+- POST /api/skills/:slug/star (favorite)
+- DELETE /api/skills/:slug (unpublish)
+- GET /api/skills/mine (author dashboard)
+- POST /api/skills/:slug/audits/rescan (author/admin)
+
+Skill search should support:
+- full-text on name/description/readme
+- faceted filters on type/capability/os/license/verified
+
+Auth model:
+- Session auth for dashboard UX (existing Auth.js).
+- Bearer token auth for CLI/publish APIs (existing api_tokens).
+- Optional: add token scopes for least-privilege in v2.
+
+------------------------------------------------------------------------------
+------------------------------------------------------------------------------
+Publish pipeline (open-source + audit)
+------------------------------------------------------------------------------
+
+Synchronous checks (reject on failure):
+- Valid skill format + metadata schema
+- Declared type and capabilities
+- License present (OSI-approved)
+- Source repo link reachable
+- Tool binaries open source (repo required)
+
+Asynchronous checks (pending until pass):
+- Dependency audit (lockfile scan, known vulns)
+- Static scan (network exfiltration, obfuscation, post-install scripts)
+- Sandbox execution (timeouts, filesystem boundaries)
+- AI review report (summarize risks, compliance)
+
+Outcomes:
+- PASS -> published (or verified after manual spot-check)
+- FAIL -> rejected with report
+- WARN -> published but unverified (visible warnings)
+
+Security checks (detail)
+- Open-source enforcement:
+  - Require repo URL + license.
+  - For tool skills, require source for any included binary.
+  - Store source snapshot hash for audit reproducibility.
+- Dependency scan:
+  - Parse lockfiles when provided.
+  - Query OSV or Snyk for known vulns.
+- Static analysis:
+  - Detect obfuscated code, eval/Function usage, encrypted blobs.
+  - Flag outbound network calls to undeclared hosts.
+  - Flag file system access outside declared scopes.
+- Sandbox run:
+  - Install in ephemeral container with strict timeouts.
+  - Log network/file access; compare to declared requirements.
+- AI audit:
+  - Generate short report for display on the skill page.
+  - Label as pass/warn/fail, store rationale in report_json.
+
+Moderation:
+- Any flag increments `flags_count`.
+- Auto-delist on repeated flags or audit fail.
+- Manual review pathway for verified badge.
+
+------------------------------------------------------------------------------
+Search, discovery, and install UX (website)
+------------------------------------------------------------------------------
+
+Search:
+- Postgres full-text search (tsvector) on name + description + readme excerpt.
+- Facet filters (type, capability, OS, license, verified).
+- Ranking options: trending (last 7d downloads), newest, top-rated.
+
+Discovery:
+- Capability pages ("/skills?capability=email-read").
+- Collections (editorial lists in DB or static config).
+- Author profiles with totals and verified status.
+
+Install:
+- Provide a single "Install" command on skill pages:
+  - `nexus-cloud skills install author/name`
+  - (later) `nexus skills install author/name`
+- Provide a "Download artifact" option for manual install.
+- Always show checksum + version next to the install command.
+
+Registry snapshot:
+- Serve `registry.json` for fast index hydration (CDN + ETag).
+- Include minimal fields: slug, name, version, type, capabilities, checksum.
+
+------------------------------------------------------------------------------
+CLI integration (nexus)
+------------------------------------------------------------------------------
+
+Local usage (existing):
+- `nexus skill use <name>` -> show SKILL.md (alias of show)
+- `nexus skill verify <name>`
+- `nexus skill run <name>`
+- `nexus skill stats [name]`
+
+Hub operations (new/extend `nexus skills`):
+- `nexus skills search <query>` + filters
+- `nexus skills install <slug>` (stores to `~/nexus/skills`)
+- `nexus skills update <slug|--all>`
+- `nexus skills publish <path>`
+- `nexus skills info <slug>` (hub metadata)
+- `nexus skills list --installed` (managed + local)
+
+Install behavior:
+- Downloads artifact from hub, verifies checksum
+- Expands into `~/nexus/skills/<name>`
+- Triggers `nexus skills refresh` to update manifest
+
+------------------------------------------------------------------------------
+Website integration (UX)
+------------------------------------------------------------------------------
+
+Public:
+- Browse/search page with filters (type, capability, OS, verified)
+- Skill detail page with install command and audit summary
+- Author profiles with badges and download counts
+
+Authenticated:
+- Publish flow (upload or link repo)
+- Version management and audit status
+- Delist/unpublish controls
+
+------------------------------------------------------------------------------
+nexus-cloud integration
+------------------------------------------------------------------------------
+
+Reuse cloud storage and refs for artifacts:
+- CLI gets cloud token from website, then fetches artifact via R2.
+- Store artifact blobs and version manifests in R2.
+- Use refs (`skills/{slug}`) as registry pointers (CAS updates).
+
+Optional:
+- Use `GET /api/v1/refs?prefix=skills/` for index hydration.
+- Keep hub metadata in website DB; use cloud only for artifacts.
+
+------------------------------------------------------------------------------
+Migration from ClawdHub
+------------------------------------------------------------------------------
+
+Options:
+1) Replace ClawdHub registry with Nexus Hub:
+   - Keep CLI install path `~/nexus/skills`.
+   - Repoint registry URL and publish workflows.
+2) Bridge registry compatibility:
+   - Accept ClawdHub metadata format in hub.
+   - Provide a compatibility proxy for `clawdhub` CLI.
+
+------------------------------------------------------------------------------
+Open questions
+------------------------------------------------------------------------------
+
+- Exact metadata schema for capabilities and connectors.
+- Whether to allow private skills (team-only).
+- How long to block publish while audits run (sync vs async).
+- OSS license policy (permissive only vs any OSI).
+
+------------------------------------------------------------------------------
+Decision checklist
+------------------------------------------------------------------------------
+
+- Final CLI grammar and aliases
+- Registry base URL
+- Audit pipeline v1 scope
+- Required metadata fields for publish
+- Search backend (Postgres FTS vs Meilisearch/Algolia)