loadout-ai 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/MASTER_PLAN.md +1711 -0
- package/README.md +453 -0
- package/SECURITY.md +18 -0
- package/SIMPLE_PLAN.md +44 -0
- package/catalog/discovered.json +10469 -0
- package/catalog/packages.json +1150 -0
- package/dashboard/app.js +607 -0
- package/dashboard/index.html +249 -0
- package/dashboard/styles.css +384 -0
- package/dist/src/cli.js +2735 -0
- package/dist/src/core/active-policy.js +233 -0
- package/dist/src/core/active-set.js +382 -0
- package/dist/src/core/adapters.js +254 -0
- package/dist/src/core/adopt.js +92 -0
- package/dist/src/core/agent-health-score.js +475 -0
- package/dist/src/core/agent-inspection.js +152 -0
- package/dist/src/core/agent-versions.js +131 -0
- package/dist/src/core/api.js +90 -0
- package/dist/src/core/atomic-file.js +22 -0
- package/dist/src/core/audit.js +174 -0
- package/dist/src/core/benchmark-campaign.js +496 -0
- package/dist/src/core/benchmark-evidence.js +458 -0
- package/dist/src/core/benchmark-fixtures.js +1157 -0
- package/dist/src/core/benchmark-runner.js +301 -0
- package/dist/src/core/benchmark-trust.js +795 -0
- package/dist/src/core/canary.js +77 -0
- package/dist/src/core/candidate-intelligence.js +661 -0
- package/dist/src/core/catalog-coverage.js +85 -0
- package/dist/src/core/catalog-install.js +186 -0
- package/dist/src/core/catalog-release.js +286 -0
- package/dist/src/core/catalog.js +327 -0
- package/dist/src/core/codex-mcp.js +122 -0
- package/dist/src/core/community.js +86 -0
- package/dist/src/core/compatibility-intelligence.js +465 -0
- package/dist/src/core/completion.js +160 -0
- package/dist/src/core/components.js +138 -0
- package/dist/src/core/conversion.js +56 -0
- package/dist/src/core/credentials.js +377 -0
- package/dist/src/core/demo.js +136 -0
- package/dist/src/core/diff.js +83 -0
- package/dist/src/core/discovery-connector.js +165 -0
- package/dist/src/core/doctor.js +95 -0
- package/dist/src/core/ecosystem-import.js +1072 -0
- package/dist/src/core/evaluate.js +72 -0
- package/dist/src/core/file-lock.js +140 -0
- package/dist/src/core/freshness-alerts.js +230 -0
- package/dist/src/core/github-discovery.js +97 -0
- package/dist/src/core/github.js +148 -0
- package/dist/src/core/head-to-head.js +437 -0
- package/dist/src/core/health-score-evidence.js +169 -0
- package/dist/src/core/health.js +129 -0
- package/dist/src/core/improve.js +223 -0
- package/dist/src/core/install.js +233 -0
- package/dist/src/core/intelligence-feed-build.js +59 -0
- package/dist/src/core/intelligence-feed.js +353 -0
- package/dist/src/core/loadout-badge.js +55 -0
- package/dist/src/core/loadout-card.js +87 -0
- package/dist/src/core/manifest.js +307 -0
- package/dist/src/core/mcp-recipes.js +419 -0
- package/dist/src/core/mcp-registry-discovery.js +455 -0
- package/dist/src/core/mcp.js +287 -0
- package/dist/src/core/model-config.js +102 -0
- package/dist/src/core/observations.js +87 -0
- package/dist/src/core/outcomes.js +164 -0
- package/dist/src/core/package.js +225 -0
- package/dist/src/core/paths.js +214 -0
- package/dist/src/core/portable.js +130 -0
- package/dist/src/core/private-discovery.js +45 -0
- package/dist/src/core/profiles.js +255 -0
- package/dist/src/core/provenance.js +278 -0
- package/dist/src/core/ranking.js +104 -0
- package/dist/src/core/recommend.js +181 -0
- package/dist/src/core/registry-api.js +97 -0
- package/dist/src/core/registry.js +316 -0
- package/dist/src/core/release-claims.js +175 -0
- package/dist/src/core/remove.js +122 -0
- package/dist/src/core/review-queue.js +200 -0
- package/dist/src/core/runtime-tool-recipe.js +315 -0
- package/dist/src/core/runtime-tools.js +453 -0
- package/dist/src/core/safety.js +234 -0
- package/dist/src/core/sandbox.js +78 -0
- package/dist/src/core/scheduler.js +264 -0
- package/dist/src/core/share-report.js +85 -0
- package/dist/src/core/signing.js +115 -0
- package/dist/src/core/skill-compare.js +239 -0
- package/dist/src/core/skill-inventory.js +193 -0
- package/dist/src/core/skill-security.js +340 -0
- package/dist/src/core/skills-sh-discovery.js +336 -0
- package/dist/src/core/skills.js +208 -0
- package/dist/src/core/snapshot.js +210 -0
- package/dist/src/core/source.js +305 -0
- package/dist/src/core/state.js +300 -0
- package/dist/src/core/sync.js +348 -0
- package/dist/src/core/terminal.js +53 -0
- package/dist/src/core/transaction.js +211 -0
- package/dist/src/core/update-watch.js +39 -0
- package/dist/src/core/update.js +216 -0
- package/dist/src/core/upgrade.js +207 -0
- package/dist/src/dashboard.js +418 -0
- package/dist/src/shared/schemas.js +336 -0
- package/dist/src/shared/types.js +1 -0
- package/docs/ACTIVE_SET.md +53 -0
- package/docs/ACTIVE_SET_POLICY.md +45 -0
- package/docs/CANDIDATE_INTELLIGENCE.md +139 -0
- package/docs/CATALOG.md +78 -0
- package/docs/CATALOG_POLICY.md +41 -0
- package/docs/COMMUNITY_DISCOVERY.md +14 -0
- package/docs/COMPATIBILITY_POLICY.md +22 -0
- package/docs/CONVERSION_AND_SANDBOX.md +27 -0
- package/docs/CREDENTIAL_AND_UPDATE_POLICY.md +31 -0
- package/docs/DISCOVERED.md +280 -0
- package/docs/EVALUATION_PROTOCOL.md +40 -0
- package/docs/EVALUATION_PROTOCOL_V1.md +300 -0
- package/docs/FEATURE_TEST_MATRIX.md +802 -0
- package/docs/GITHUB_AUTHORIZATION.md +38 -0
- package/docs/HEAD_TO_HEAD_EVALUATION.md +79 -0
- package/docs/PROVENANCE_AND_COMPARISON.md +72 -0
- package/docs/PROVIDER_CONFIGURATION.md +45 -0
- package/docs/RELEASE_REVIEW.md +117 -0
- package/docs/SAFE_UPDATE_DEMO.md +25 -0
- package/docs/SCHEMA_DECISIONS.md +25 -0
- package/docs/TEAM_POLICY.md +18 -0
- package/docs/TESTING.md +190 -0
- package/package.json +82 -0
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
# Optional GitHub authorization
|
|
2
|
+
|
|
3
|
+
Public discovery remains anonymous. Private discovery is opt-in and begins only
|
|
4
|
+
when the user explicitly connects GitHub from the local CLI or loopback dashboard.
|
|
5
|
+
|
|
6
|
+
## Authorization model
|
|
7
|
+
|
|
8
|
+
Use a GitHub App installation, not a broad OAuth `repo` token. The app asks for
|
|
9
|
+
read-only **Contents** access, is installed on repositories selected by the user,
|
|
10
|
+
and uses the user-to-server OAuth flow solely to identify the user's installation.
|
|
11
|
+
The resulting short-lived installation token is scoped by GitHub to that selected
|
|
12
|
+
repository set. Metadata is read-only. No organization administration, workflow,
|
|
13
|
+
issues, pull-request write, or repository-write permission is requested.
|
|
14
|
+
|
|
15
|
+
The app client ID, redirect URI, and application slug may be committed as public
|
|
16
|
+
configuration. The private key and client secret cannot: they belong to a hosted
|
|
17
|
+
broker or a user-controlled local credential manager. Loadout never asks the user to
|
|
18
|
+
paste a token into a manifest, catalog, log, or dashboard URL.
|
|
19
|
+
|
|
20
|
+
## Local flow and failure modes
|
|
21
|
+
|
|
22
|
+
1. `loadout connect github` opens a browser using PKCE and a loopback callback.
|
|
23
|
+
2. The local process verifies `state`, PKCE verifier, expiration, and callback host.
|
|
24
|
+
3. It stores only an OS-keychain reference to the refresh/session material.
|
|
25
|
+
4. `loadout discover --private` obtains a short-lived installation token in memory,
|
|
26
|
+
lists only selected repositories, and discards it after the request.
|
|
27
|
+
5. `loadout disconnect github` deletes the keychain item and local connection record.
|
|
28
|
+
|
|
29
|
+
If no registered GitHub App/client and secure credential backend are configured,
|
|
30
|
+
private discovery stays unavailable and public discovery keeps working. A token that
|
|
31
|
+
cannot be refreshed or no longer has access is reported as disconnected; it is never
|
|
32
|
+
silently retried with a broader scope.
|
|
33
|
+
|
|
34
|
+
## Non-goals
|
|
35
|
+
|
|
36
|
+
This does not clone private repositories, execute private source, or expose private
|
|
37
|
+
repository names in shared exports. A user must still explicitly add and plan a
|
|
38
|
+
repository before Loadout fetches it.
|
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
# Head-to-head capability evaluation design
|
|
2
|
+
|
|
3
|
+
This protocol compares two reviewed skill instruction sets on one declared task
|
|
4
|
+
family. It does not rank repositories globally, execute candidate code, or silently
|
|
5
|
+
change an active set.
|
|
6
|
+
|
|
7
|
+
## Evidence envelope
|
|
8
|
+
|
|
9
|
+
Every run stores:
|
|
10
|
+
|
|
11
|
+
- evaluation schema and harness version;
|
|
12
|
+
- category and fixture ids plus the fixture/rubric SHA-256;
|
|
13
|
+
- candidate package, skill path, reviewed commit, and instruction SHA-256;
|
|
14
|
+
- agent surface, model provider/model/version, sampling settings, and tool policy;
|
|
15
|
+
- per-trial rubric dimensions, grader rationale, raw aggregate, variance, and trial
|
|
16
|
+
count;
|
|
17
|
+
- input/output token counts, reported provider cost or 'unknown', duration, failures,
|
|
18
|
+
and retries;
|
|
19
|
+
- deterministic grader version, human audit status, uncertainty statement, timestamp,
|
|
20
|
+
and machine-verifiable signature.
|
|
21
|
+
|
|
22
|
+
Candidate instructions are data. The harness never runs their scripts, shell commands,
|
|
23
|
+
hooks, MCP servers, network requests, or tool calls. Fixtures use disposable synthetic
|
|
24
|
+
repositories and redact secrets. A model request is permitted only through an explicit
|
|
25
|
+
provider selection and cost ceiling.
|
|
26
|
+
|
|
27
|
+
## Initial categories
|
|
28
|
+
|
|
29
|
+
### Workflow adherence
|
|
30
|
+
|
|
31
|
+
Fixtures ask for a change under explicit constraints: inspect before editing, preserve
|
|
32
|
+
unrelated work, write a focused test, verify, and report uncertainty. Rubric:
|
|
33
|
+
constraint recall (25), plan-to-action consistency (20), safe scope (20), verification
|
|
34
|
+
quality (20), and honest completion report (15). A hidden forbidden action produces a
|
|
35
|
+
blocking safety failure rather than a negative quality point.
|
|
36
|
+
|
|
37
|
+
### Code-review coverage
|
|
38
|
+
|
|
39
|
+
Fixtures contain synthetic diffs with seeded correctness, security, concurrency,
|
|
40
|
+
compatibility, and test-gap defects plus harmless distractors. Rubric: weighted seeded
|
|
41
|
+
defect recall (45), precision (20), severity calibration (15), actionable file/line
|
|
42
|
+
evidence (10), and regression-test advice (10). Findings not grounded in the fixture
|
|
43
|
+
count against precision.
|
|
44
|
+
|
|
45
|
+
### Documentation retrieval
|
|
46
|
+
|
|
47
|
+
Fixtures provide versioned local documentation and deliberately stale alternatives.
|
|
48
|
+
Rubric: correct source/version choice (30), supported API facts (30), citation
|
|
49
|
+
traceability (20), uncertainty handling (10), and absence of invented APIs (10).
|
|
50
|
+
Network access is disabled so retrieval quality is not confused with internet access.
|
|
51
|
+
|
|
52
|
+
### Browser-test planning
|
|
53
|
+
|
|
54
|
+
Fixtures provide an HTML/accessibility snapshot, user journey, and failure evidence.
|
|
55
|
+
Rubric: critical-path coverage (25), stable locator strategy (20), accessibility and
|
|
56
|
+
responsive cases (20), isolation/setup (15), failure diagnostics (10), and avoidance
|
|
57
|
+
of brittle timing (10). The first harness plans tests only; browser execution is a
|
|
58
|
+
separate sandboxed verification stage.
|
|
59
|
+
|
|
60
|
+
## Trial and decision rules
|
|
61
|
+
|
|
62
|
+
- Minimum five trials per candidate/fixture/model; alternate candidate order.
|
|
63
|
+
- Identical prompts, context budget, model version, temperature, tool policy, and
|
|
64
|
+
timeout within a comparison.
|
|
65
|
+
- Report mean, median, standard deviation, failures, and 95% bootstrap interval. Fewer
|
|
66
|
+
than five successful trials is 'insufficient-evidence'.
|
|
67
|
+
- A result is only 'preferred-for-this-category' when the interval clears the declared
|
|
68
|
+
practical-effect threshold and no safety failure exists. Otherwise it is 'tie',
|
|
69
|
+
'mixed', or 'insufficient-evidence'.
|
|
70
|
+
- One fixture, model, or judge cannot promote a global default. Active-set replacement
|
|
71
|
+
always requires a preview and explicit user approval.
|
|
72
|
+
|
|
73
|
+
## Signed snapshot boundary
|
|
74
|
+
|
|
75
|
+
The harness writes canonical JSON, hashes fixtures and candidate inputs, and signs the
|
|
76
|
+
evidence envelope with Loadout's existing Ed25519 catalog-signing primitive. Private
|
|
77
|
+
keys stay outside the repository. Verification must fail on any changed score,
|
|
78
|
+
metadata, candidate hash, or rubric. Unsigned local experiments may be displayed but
|
|
79
|
+
cannot influence shared ranking evidence.
|
|
@@ -0,0 +1,72 @@
|
|
|
1
|
+
# Provenance and comparison contract
|
|
2
|
+
|
|
3
|
+
Loadout must answer “where did this skill come from?” and “is there a better reviewed
|
|
4
|
+
alternative?” without turning weak signals into facts. Both workflows are read-only.
|
|
5
|
+
|
|
6
|
+
## Provenance confidence
|
|
7
|
+
|
|
8
|
+
From strongest to weakest:
|
|
9
|
+
|
|
10
|
+
1. `catalog-exact` / `exact`: the installed `SKILL.md` SHA-256 is byte-identical to a
|
|
11
|
+
skill indexed from a catalog repository at its reviewed immutable commit.
|
|
12
|
+
2. `loadout-managed` / `high`: Loadout state owns the installed path. The file may
|
|
13
|
+
still be reported separately as drifted if the user edited it after installation.
|
|
14
|
+
3. `embedded-source` / `medium`: the skill text contains a GitHub repository that is in
|
|
15
|
+
the reviewed index, but the instruction fingerprint differs. A documentation link
|
|
16
|
+
or fork can create this signal, so it is evidence rather than proof.
|
|
17
|
+
4. `catalog-name-candidate` / `low`: only the normalized skill name matches. Names are
|
|
18
|
+
not unique and cannot establish authorship, equivalence, or quality.
|
|
19
|
+
5. `unknown` / `unknown`: no supported evidence matched. Loadout preserves this result
|
|
20
|
+
rather than guessing from a directory name.
|
|
21
|
+
|
|
22
|
+
`loadout scan` uses only an existing local reviewed index. It never performs network
|
|
23
|
+
access unless `--refresh-provenance` is passed. Refresh fetches only catalog repositories
|
|
24
|
+
with reviewed skill evidence, checks out their exact 40-character commit, fingerprints
|
|
25
|
+
their `SKILL.md` files, and writes a local index below `LOADOUT_HOME/provenance`.
|
|
26
|
+
|
|
27
|
+
## Relationship classification
|
|
28
|
+
|
|
29
|
+
`loadout compare` uses deterministic relationships:
|
|
30
|
+
|
|
31
|
+
- `exact-copy`: identical instruction fingerprint;
|
|
32
|
+
- `divergent-same-name`: same normalized name, different instructions;
|
|
33
|
+
- `overlapping-capability`: at least 30% Jaccard overlap after conservative
|
|
34
|
+
name/description token normalization; and
|
|
35
|
+
- `same-category-candidate`: reviewed catalog category matches but textual overlap is
|
|
36
|
+
weak.
|
|
37
|
+
|
|
38
|
+
Two additional capability-family states are deliberately not inferred from text:
|
|
39
|
+
|
|
40
|
+
- `complementary-capability` requires an explicit reviewed catalog relationship showing
|
|
41
|
+
that the tools solve different parts of a workflow; and
|
|
42
|
+
- `verified-hard-conflict` requires a catalog conflict-family record with hard severity
|
|
43
|
+
and blocks installation until resolved.
|
|
44
|
+
|
|
45
|
+
If neither reviewed relationship exists, Loadout leaves the relationship unclassified
|
|
46
|
+
rather than calling two vaguely related tools complementary or incompatible.
|
|
47
|
+
|
|
48
|
+
Cross-agent installations with the same name and fingerprint are mirrors, not duplicate
|
|
49
|
+
problems. Divergent same-name installations require `--agent` disambiguation. Existing
|
|
50
|
+
catalog conflict families remain the source of verified soft/hard package conflicts.
|
|
51
|
+
|
|
52
|
+
## Comparison evidence and guardrails
|
|
53
|
+
|
|
54
|
+
Alternatives show repository, reviewed commit, tier, license, deterministic relationship,
|
|
55
|
+
text similarity, and the existing catalog evidence score. Adoption, momentum,
|
|
56
|
+
maintenance, compatibility, permission, and evaluation evidence are printed separately;
|
|
57
|
+
missing evidence is stated instead of being silently treated as good. The score orders
|
|
58
|
+
available catalog evidence; it does not measure model-output quality.
|
|
59
|
+
|
|
60
|
+
The comparison recommendation follows these rules:
|
|
61
|
+
|
|
62
|
+
- strongly attributable installed skills remain in place by default;
|
|
63
|
+
- same-name or overlapping candidates require head-to-head evaluation or explicit user
|
|
64
|
+
experience before replacement;
|
|
65
|
+
- weak same-category candidates are exploration only; and
|
|
66
|
+
- no command changes agent files, activates a skill, or removes an unknown skill.
|
|
67
|
+
|
|
68
|
+
Limitations are explicit: `SKILL.md` fingerprinting does not prove that referenced assets
|
|
69
|
+
are identical, GitHub URLs embedded in instructions do not prove provenance, semantic
|
|
70
|
+
token overlap is not an embedding or behavioral evaluation, and current comparison does
|
|
71
|
+
not claim universal “best.” Phase 12 task evaluations and local human outcomes are
|
|
72
|
+
required before guided optimization may apply a replacement.
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
# Provider-neutral model configuration
|
|
2
|
+
|
|
3
|
+
Loadout can validate and share a model-selection document before it has a
|
|
4
|
+
provider adapter. The schema is provider-neutral: a selection identifies a
|
|
5
|
+
provider, model, HTTPS endpoint, optional credential _reference_, and optional
|
|
6
|
+
target agent IDs. `openrouter` is an ordinary provider identifier, so an
|
|
7
|
+
OpenRouter selection needs no special schema or stored integration.
|
|
8
|
+
|
|
9
|
+
```json
|
|
10
|
+
{
|
|
11
|
+
"schemaVersion": 1,
|
|
12
|
+
"selections": [
|
|
13
|
+
{
|
|
14
|
+
"id": "coding",
|
|
15
|
+
"provider": "openrouter",
|
|
16
|
+
"model": "anthropic/claude-sonnet-4",
|
|
17
|
+
"endpoint": "https://openrouter.ai/api/v1",
|
|
18
|
+
"credential": { "kind": "environment", "name": "OPENROUTER_API_KEY" },
|
|
19
|
+
"targetAgents": ["codex", "claude-code"]
|
|
20
|
+
}
|
|
21
|
+
]
|
|
22
|
+
}
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
## Credential boundary
|
|
26
|
+
|
|
27
|
+
The document contains only a credential location, never a credential value.
|
|
28
|
+
The accepted references are an environment-variable **name** or an OS-keychain
|
|
29
|
+
service/account reference. It has no `apiKey`, `token`, `authorization`, or
|
|
30
|
+
arbitrary-header fields; strict runtime validation rejects them. Endpoints must
|
|
31
|
+
be credential-free HTTPS URLs (no userinfo, query string, or fragment), and
|
|
32
|
+
secret-looking values are rejected in model/identifier fields.
|
|
33
|
+
|
|
34
|
+
Loadout does not resolve the reference, write it to state/lockfiles, log it,
|
|
35
|
+
or transmit it in this design. A future provider adapter may read the named
|
|
36
|
+
environment variable or keychain entry only at execution time, and must redact
|
|
37
|
+
it from errors, plans, snapshots, logs, exports, and telemetry. That adapter is
|
|
38
|
+
explicitly outside P11-05.
|
|
39
|
+
|
|
40
|
+
## Deliberate scope
|
|
41
|
+
|
|
42
|
+
This schema declares intent; it does not configure Codex, Claude, Cursor, or
|
|
43
|
+
any provider, call OpenRouter, select a model automatically, or store provider
|
|
44
|
+
headers. Keeping those side effects out of the schema layer makes the document
|
|
45
|
+
portable and safe to review in version control.
|
|
@@ -0,0 +1,117 @@
|
|
|
1
|
+
# Release review — 2026-07-15
|
|
2
|
+
|
|
3
|
+
This review covers the current Loadout implementation, not an aspirational
|
|
4
|
+
roadmap. It was performed after the transaction, source-fetch, dashboard, and
|
|
5
|
+
adapter test suites passed locally.
|
|
6
|
+
|
|
7
|
+
## P4-08: atomic-commit review — accepted with explicit durability boundary
|
|
8
|
+
|
|
9
|
+
Mutation metadata that can make a completed install unrecoverable now uses a
|
|
10
|
+
temporary sibling file followed by `rename`:
|
|
11
|
+
|
|
12
|
+
- `~/.loadout/state.json`
|
|
13
|
+
- `loadout.lock`
|
|
14
|
+
- edits to an existing `loadout.json`
|
|
15
|
+
- MCP JSON/TOML config writers already used the same-directory temporary-file
|
|
16
|
+
pattern.
|
|
17
|
+
|
|
18
|
+
The replacement prevents readers from observing a partially written JSON file
|
|
19
|
+
on local filesystems with atomic same-directory rename support. Snapshots are
|
|
20
|
+
created before mutation and restoration is exercised by tests. CI run
|
|
21
|
+
`29401149042` executed the atomic-file and transaction suites on Node 20 and
|
|
22
|
+
22 for Windows, macOS, and Linux. The decision is therefore accepted for the
|
|
23
|
+
supported local-filesystem scope.
|
|
24
|
+
|
|
25
|
+
This is not a claim of durable, power-loss-safe multi-file transactions: a
|
|
26
|
+
process or system failure can still leave either the old or new version of an
|
|
27
|
+
individual file. Transaction journals recover interrupted multi-file work before
|
|
28
|
+
the next synchronization; users needing database-grade durability should not rely
|
|
29
|
+
on filesystem rename alone.
|
|
30
|
+
|
|
31
|
+
## P7-15: product and security review — pass with stated boundaries
|
|
32
|
+
|
|
33
|
+
The reviewed release flow is appropriate for a hackathon demo and local use:
|
|
34
|
+
|
|
35
|
+
- Plans are read-only until an explicit apply command or dashboard action.
|
|
36
|
+
- Install/update safety requires approval for scripts, hooks, binaries, new
|
|
37
|
+
domains, environment references, suspicious instructions, and MCP changes.
|
|
38
|
+
- Snapshots are taken before managed mutations; failures restore them.
|
|
39
|
+
- Repository cloning does not run package lifecycle scripts. Generic Git URLs
|
|
40
|
+
with embedded credentials, query strings, or fragments are rejected so a
|
|
41
|
+
failed Git command cannot echo a token.
|
|
42
|
+
- The API is loopback-only. The dashboard also validates the actual socket peer
|
|
43
|
+
instead of trusting a spoofable `Host` header; mutation endpoints require a
|
|
44
|
+
random per-process session token and reject cross-origin requests.
|
|
45
|
+
- The dashboard and CLI summaries avoid emitting MCP environment values.
|
|
46
|
+
|
|
47
|
+
Known boundaries remain intentional: Loadout does not execute third-party
|
|
48
|
+
install scripts, silently replace existing Codex TOML MCP tables, or support
|
|
49
|
+
private repository credentials. It should not be marketed as a full arbitrary
|
|
50
|
+
plugin executor.
|
|
51
|
+
|
|
52
|
+
## P9-07: cross-platform go/no-go — bounded go for native skills
|
|
53
|
+
|
|
54
|
+
CI run `29401149042` planed, installed, byte-verified, and removed a real
|
|
55
|
+
`SKILL.md` through every declared agent-owned skill layout on Windows, macOS,
|
|
56
|
+
and Linux, using disposable native home and state directories. **Current
|
|
57
|
+
decision:** go for the bounded native-skill-directory claim. The no-go remains
|
|
58
|
+
for plugins, hooks, executables, and arbitrary MCP runtimes.
|
|
59
|
+
|
|
60
|
+
## Local verification
|
|
61
|
+
|
|
62
|
+
On 2026-07-15 the corrective audit also verified:
|
|
63
|
+
|
|
64
|
+
```text
|
|
65
|
+
npm run format:check
|
|
66
|
+
npm run lint
|
|
67
|
+
npm run typecheck
|
|
68
|
+
npm test -- --run # 55 files, 169 tests
|
|
69
|
+
npm run build
|
|
70
|
+
npm run test:e2e # Chromium first-run preview/apply flow
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
The audit fixed these release blockers and added regressions for them:
|
|
74
|
+
|
|
75
|
+
- Vitest now assigns every test file disposable Loadout and user homes; a CRLF
|
|
76
|
+
fixture can no longer write into the developer's real `~/.loadout`.
|
|
77
|
+
- A stale refresh cache overlays mutable metadata onto the bundled catalog instead
|
|
78
|
+
of hiding newer bundled records or retaining deleted cache-only packages.
|
|
79
|
+
- The Maximum profile references only catalog packages that actually exist.
|
|
80
|
+
- Dashboard Installed and Updates routes have unique container/list IDs and are
|
|
81
|
+
exercised through navigation in Playwright.
|
|
82
|
+
- Demo output distinguishes planned skill directories from tracked files instead of
|
|
83
|
+
printing an impossible installed/total fraction.
|
|
84
|
+
- Multi-file synchronization now writes a durable journal and recovers interrupted
|
|
85
|
+
work even when the next synchronization is otherwise empty.
|
|
86
|
+
- The npm tarball exposes the `loadout` executable, carries runtime code/assets but
|
|
87
|
+
not compiled tests, and resolves its catalog/dashboard independently of the current
|
|
88
|
+
working directory.
|
|
89
|
+
|
|
90
|
+
An outside-checkout 72-file tarball smoke test launched version `0.1.0`, read all 20
|
|
91
|
+
catalog records, and served the packaged dashboard. The live isolated demo fetched
|
|
92
|
+
`obra/superpowers`, planned 14 skill directories and 48 files, then verified rollback
|
|
93
|
+
and removed its temporary profile. The npm package is named `loadout-ai`; the shorter
|
|
94
|
+
`loadout` registry name is owned by an unrelated package.
|
|
95
|
+
|
|
96
|
+
## CLI-first product correction
|
|
97
|
+
|
|
98
|
+
The dashboard is now explicitly secondary. Running `loadout` in an interactive terminal
|
|
99
|
+
starts Maximum/Stable/Custom onboarding, while `loadout setup --mode maximum` provides
|
|
100
|
+
a scriptable read-only preview and `--yes --approve-risk` applies the reviewed result.
|
|
101
|
+
|
|
102
|
+
Catalog setup now:
|
|
103
|
+
|
|
104
|
+
- filters out MCP/executable-only records before cloning;
|
|
105
|
+
- fetches up to four repositories concurrently with visible progress;
|
|
106
|
+
- fetches the catalog's exact reviewed commit rather than mutable default-branch HEAD;
|
|
107
|
+
- reuses only a clean cache whose Git HEAD matches that commit;
|
|
108
|
+
- resolves duplicate skill targets by retaining the higher-ranked reviewed source and
|
|
109
|
+
reporting every lower-ranked duplicate;
|
|
110
|
+
- installs all remaining packages through one transaction and one restore point;
|
|
111
|
+
- keeps local health checks network-free unless `--updates` is explicitly requested.
|
|
112
|
+
|
|
113
|
+
A disposable real Maximum Boost run prepared all 31 skill-bearing repositories from the
|
|
114
|
+
50-record catalog, deferred 19 explicit MCP-only setups, exposed 1,219 skill directories,
|
|
115
|
+
and resolved 48 duplicate target directories. The same isolated flow exercised library
|
|
116
|
+
inspection, project optimization, apply, and rollback successfully before removing its
|
|
117
|
+
test home. No real user agent directory was used.
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
# Safe update demonstration
|
|
2
|
+
|
|
3
|
+
Loadout treats an update as a reviewable filesystem transaction; it never runs
|
|
4
|
+
repository scripts, hooks, package-manager lifecycle commands, or MCP servers.
|
|
5
|
+
|
|
6
|
+
The regression demo uses only the inert local repositories in
|
|
7
|
+
`tests/fixtures/update-safety`:
|
|
8
|
+
|
|
9
|
+
1. `benign-v1` to `benign-v2` is a documentation-only skill update. Loadout
|
|
10
|
+
copies it, validates the copied `SKILL.md`, and records the new commit.
|
|
11
|
+
2. `risky-v2` adds a hook-shaped shell file and an external domain. Static
|
|
12
|
+
inspection marks it as requiring explicit approval. Without that approval,
|
|
13
|
+
Loadout writes review metadata to its quarantine directory and leaves the
|
|
14
|
+
installed files and state untouched.
|
|
15
|
+
3. A simulated post-copy static verification failure restores the exact
|
|
16
|
+
transaction snapshot, including the previous skill bytes and install state.
|
|
17
|
+
|
|
18
|
+
Run the evidence locally:
|
|
19
|
+
|
|
20
|
+
```bash
|
|
21
|
+
npm test -- --reporter=verbose tests/update-safety-demo.test.ts
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
The hook fixture contains a marker command solely to prove that inspection does
|
|
25
|
+
not execute it. The test asserts that the marker is absent.
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
# Runtime schema decision
|
|
2
|
+
|
|
3
|
+
Loadout treats catalog JSON, shared manifests and lockfiles, persisted install
|
|
4
|
+
state, and inter-module install plans as untrusted data. `src/shared/schemas.ts`
|
|
5
|
+
is the single runtime-validation boundary for those structures. It uses Zod so
|
|
6
|
+
the parsed output has inferred TypeScript types as well as a path-aware runtime
|
|
7
|
+
error for the CLI and tests.
|
|
8
|
+
|
|
9
|
+
The schemas intentionally model identifiers, source provenance, agent IDs,
|
|
10
|
+
component kinds, hashes, and safety levels. They do not define fields for MCP
|
|
11
|
+
environment values, API keys, tokens, or other secret values. MCP environment
|
|
12
|
+
values remain in their agent-owned configuration and are only summarized using
|
|
13
|
+
counts/names elsewhere in the product.
|
|
14
|
+
|
|
15
|
+
Compatibility and failure handling:
|
|
16
|
+
|
|
17
|
+
- New manifest, lockfile, and state writers always emit schema version 1.
|
|
18
|
+
- Lockfiles produced before `generatedAt` existed remain readable as the
|
|
19
|
+
explicit legacy value `unknown`; newly written lockfiles always include the
|
|
20
|
+
timestamp.
|
|
21
|
+
- Unknown catalog and policy fields are retained where required for existing
|
|
22
|
+
portable-export secret scanning. They are never interpreted as executable
|
|
23
|
+
configuration by the schema layer.
|
|
24
|
+
- Invalid data fails before install, audit, or update processing with the exact
|
|
25
|
+
field path (for example, `installs.0.targetAgents.0`).
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
# Team policy and audit schema
|
|
2
|
+
|
|
3
|
+
A shared Loadout is a versioned declarative manifest plus a signed catalog reference;
|
|
4
|
+
it never contains credentials or private repository names by default. Team policy has
|
|
5
|
+
four independent controls:
|
|
6
|
+
|
|
7
|
+
- allowlist: package IDs/repository identities eligible for planning;
|
|
8
|
+
- denylist: identities, domains, component types, or risk findings that block;
|
|
9
|
+
- required approvals: roles required for risky changes; and
|
|
10
|
+
- audit events: timestamp, actor label, manifest/catalog hashes, decision, and reason.
|
|
11
|
+
|
|
12
|
+
The serverless/local MVP stores policy in a project-controlled JSON file and validates
|
|
13
|
+
it before a plan. There is no hidden admin override. A later hosted product can add
|
|
14
|
+
identity and retention without changing the portable policy semantics.
|
|
15
|
+
|
|
16
|
+
The current manifest implementation enforces package/repository allowlists and
|
|
17
|
+
denylists before synchronization, alongside the existing blocked-domain and
|
|
18
|
+
blocked-command rules. Violations are included in the dry-run plan and stop apply.
|
package/docs/TESTING.md
ADDED
|
@@ -0,0 +1,190 @@
|
|
|
1
|
+
# Testing Loadout as a product
|
|
2
|
+
|
|
3
|
+
The primary product is the CLI. Use this walkthrough before installing into your real
|
|
4
|
+
Codex, Claude Code, or other agent directories.
|
|
5
|
+
|
|
6
|
+
The automated product journey runs the built CLI against disposable user, state, and
|
|
7
|
+
project directories. It performs a real scan and offline provenance comparison, then
|
|
8
|
+
previews and applies project optimization, verifies the installed bytes, and rolls the
|
|
9
|
+
snapshot back:
|
|
10
|
+
|
|
11
|
+
```bash
|
|
12
|
+
npm run test:e2e:cli
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
This test does not use the dashboard, network, mock command output, or any real agent
|
|
16
|
+
profile. It is a required CI gate on Ubuntu; the manual cross-platform workflow runs
|
|
17
|
+
the broader native filesystem suite.
|
|
18
|
+
|
|
19
|
+
## 1. Build the exact npm package entry point
|
|
20
|
+
|
|
21
|
+
```bash
|
|
22
|
+
cd /path/to/loadout
|
|
23
|
+
npm ci
|
|
24
|
+
npm run build
|
|
25
|
+
npx . --help
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
`npx .` runs the same `loadout` executable that `npx loadout-ai` will run after npm
|
|
29
|
+
publication.
|
|
30
|
+
|
|
31
|
+
Before creating a disposable profile, the only recommended real-profile command is the
|
|
32
|
+
read-only inventory:
|
|
33
|
+
|
|
34
|
+
```bash
|
|
35
|
+
npx . scan
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
It reports actual `SKILL.md` capabilities, Loadout ownership, duplicates, and capacity
|
|
39
|
+
warnings. It does not treat unmanaged content as unsafe and does not change any agent.
|
|
40
|
+
|
|
41
|
+
To build the local reviewed provenance index from exact catalog commits and compare a
|
|
42
|
+
reported skill:
|
|
43
|
+
|
|
44
|
+
```bash
|
|
45
|
+
npx . scan --refresh-provenance
|
|
46
|
+
npx . compare <skill-name>
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
The first command writes only Loadout's local cache. `compare` is read-only and uses
|
|
50
|
+
`--offline` when a test must forbid network fallback. A same-name match is a candidate,
|
|
51
|
+
not proof of provenance or quality.
|
|
52
|
+
|
|
53
|
+
To inspect and exercise the managed active set after installing a package:
|
|
54
|
+
|
|
55
|
+
```bash
|
|
56
|
+
npx . library
|
|
57
|
+
npx . disable <managed-package> # dry-run
|
|
58
|
+
npx . disable <managed-package> --yes # cache, verify, deactivate
|
|
59
|
+
npx . enable <managed-package> # dry-run
|
|
60
|
+
npx . enable <managed-package> --yes # verify and reactivate
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
Copy the snapshot id printed by either applied command to test
|
|
64
|
+
`npx . rollback --snapshot <id>`. These commands refuse unmanaged packages, drifted
|
|
65
|
+
managed content, incomplete library copies, quarantined entries, and occupied enable
|
|
66
|
+
targets.
|
|
67
|
+
|
|
68
|
+
## 2. Create a completely disposable agent profile
|
|
69
|
+
|
|
70
|
+
macOS or Linux:
|
|
71
|
+
|
|
72
|
+
```bash
|
|
73
|
+
TEST_HOME="$(mktemp -d)"
|
|
74
|
+
export LOADOUT_USER_HOME="$TEST_HOME/user"
|
|
75
|
+
export LOADOUT_HOME="$TEST_HOME/state"
|
|
76
|
+
mkdir -p "$LOADOUT_USER_HOME/.codex" "$LOADOUT_USER_HOME/.claude"
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
PowerShell:
|
|
80
|
+
|
|
81
|
+
```powershell
|
|
82
|
+
$TestHome = Join-Path $env:TEMP ("loadout-test-" + [guid]::NewGuid())
|
|
83
|
+
$env:LOADOUT_USER_HOME = Join-Path $TestHome "user"
|
|
84
|
+
$env:LOADOUT_HOME = Join-Path $TestHome "state"
|
|
85
|
+
New-Item -ItemType Directory -Force (Join-Path $env:LOADOUT_USER_HOME ".codex") | Out-Null
|
|
86
|
+
New-Item -ItemType Directory -Force (Join-Path $env:LOADOUT_USER_HOME ".claude") | Out-Null
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
Those empty directories make Loadout detect virtual Codex and Claude Code profiles.
|
|
90
|
+
Every write stays below the disposable path.
|
|
91
|
+
|
|
92
|
+
## 3. Preview Power, then inspect Maximum Library
|
|
93
|
+
|
|
94
|
+
Power is the broad daily-use profile:
|
|
95
|
+
|
|
96
|
+
```bash
|
|
97
|
+
npx . setup --mode power
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
Expect a curated skill-level set across the strongest reviewed collections. Stable
|
|
101
|
+
remains the smallest Superpowers + Context7 option. Maximum prepares the full library:
|
|
102
|
+
|
|
103
|
+
```bash
|
|
104
|
+
npx . setup --mode maximum
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
This is read-only. Expect the CLI to show:
|
|
108
|
+
|
|
109
|
+
- detected agents;
|
|
110
|
+
- reviewed catalog repositories fetched at pinned commits;
|
|
111
|
+
- actual skill-directory targets, not only repository count;
|
|
112
|
+
- lower-ranked duplicate skills that were deferred;
|
|
113
|
+
- MCP-only repositories requiring explicit configuration;
|
|
114
|
+
- safety categories requiring a separate approval.
|
|
115
|
+
|
|
116
|
+
No agent skill directory or Loadout install state is created by preview.
|
|
117
|
+
|
|
118
|
+
## 4. Download the reviewed library
|
|
119
|
+
|
|
120
|
+
```bash
|
|
121
|
+
npx . setup --mode maximum --yes --approve-risk
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
`--approve-risk` acknowledges the displayed scripts, domains, environment names, or
|
|
125
|
+
instruction findings inside the reviewed skill content. Loadout copies skills into its
|
|
126
|
+
disabled library but does not execute repository installation or lifecycle scripts.
|
|
127
|
+
|
|
128
|
+
The command should finish with one snapshot identifier. The library download is one
|
|
129
|
+
transaction: a failure restores every library/state target rather than leaving a
|
|
130
|
+
half-installed profile.
|
|
131
|
+
|
|
132
|
+
## 5. Inspect the installed product
|
|
133
|
+
|
|
134
|
+
```bash
|
|
135
|
+
npx . list
|
|
136
|
+
npx . status
|
|
137
|
+
npx . health
|
|
138
|
+
npx . library
|
|
139
|
+
npx . optimize --project .
|
|
140
|
+
npx . optimize --project . --yes
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
The first optimize command is a dry run. The second activates only reviewed,
|
|
144
|
+
project-relevant skill units, verifies their hashes, and prints the exact snapshot
|
|
145
|
+
rollback command.
|
|
146
|
+
|
|
147
|
+
`health` is local and fast by default. Use `npx . health --updates` or
|
|
148
|
+
`npx . update` when you intentionally want live network update checks.
|
|
149
|
+
|
|
150
|
+
You can also inspect the disposable files directly:
|
|
151
|
+
|
|
152
|
+
```bash
|
|
153
|
+
find "$LOADOUT_USER_HOME/.agents/skills" -name SKILL.md | wc -l
|
|
154
|
+
find "$LOADOUT_USER_HOME/.claude/skills" -name SKILL.md | wc -l
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
## 6. Verify rollback
|
|
158
|
+
|
|
159
|
+
```bash
|
|
160
|
+
npx . rollback
|
|
161
|
+
npx . list
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
The second command should report that no Loadout-managed packages are installed. Files
|
|
165
|
+
that existed before setup must remain byte-identical.
|
|
166
|
+
|
|
167
|
+
## 7. Test the real interactive experience
|
|
168
|
+
|
|
169
|
+
After the disposable run succeeds, open a fresh terminal without `LOADOUT_HOME` or
|
|
170
|
+
`LOADOUT_USER_HOME` overrides and run:
|
|
171
|
+
|
|
172
|
+
```bash
|
|
173
|
+
npx .
|
|
174
|
+
```
|
|
175
|
+
|
|
176
|
+
Choose Maximum, Stable, or Custom, review the plan, and confirm only when you want to
|
|
177
|
+
write to the detected real agent profiles. After npm publication, replace `npx .` with
|
|
178
|
+
`npx loadout-ai`.
|
|
179
|
+
|
|
180
|
+
## Optional dashboard
|
|
181
|
+
|
|
182
|
+
The dashboard is a secondary inspection surface, not the onboarding requirement:
|
|
183
|
+
|
|
184
|
+
```bash
|
|
185
|
+
npx . dashboard
|
|
186
|
+
```
|
|
187
|
+
|
|
188
|
+
Open the printed loopback URL. CLI setup, updates, removal, discovery, and rollback all
|
|
189
|
+
work without it. Browser automation is also optional and runs only when manually
|
|
190
|
+
dispatched in CI; locally, use `npm run test:e2e:dashboard`.
|