@softspark/ai-toolkit 1.5.0 → 1.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +25 -0
- package/README.md +16 -3
- package/action.yml +2 -1
- package/app/.claude-plugin/plugin.json +1 -1
- package/app/ARCHITECTURE.md +3 -1
- package/app/agents/documenter.md +2 -2
- package/app/plugins/memory-pack/scripts/strip_private.py +1 -1
- package/app/skills/docs/SKILL.md +1 -1
- package/app/skills/documentation-standards/SKILL.md +1 -1
- package/app/skills/write-a-prd/scripts/frame-template.html +1 -25
- package/app/skills/write-a-prd/scripts/poll.js +23 -0
- package/app/skills/write-a-prd/scripts/visual-server.cjs +11 -1
- package/kb/planning/cloud-security-pack-plan.md +1166 -0
- package/kb/planning/enterprise-config-inheritance-plan.md +917 -0
- package/kb/planning/local-dashboard-plan.md +773 -0
- package/kb/planning/offline-slm-profile-plan.md +742 -0
- package/kb/procedures/maintenance-sop.md +10 -4
- package/llms-full.txt +3915 -4
- package/llms.txt +5 -0
- package/manifest.json +1 -1
- package/package.json +1 -1
- package/scripts/dir_rules_shared.py +96 -0
- package/scripts/generate_antigravity.py +16 -5
- package/scripts/generate_augment_rules.py +41 -7
- package/scripts/generate_cline_rules.py +13 -3
- package/scripts/generate_cursor_mdc.py +37 -8
- package/scripts/generate_roo_rules.py +13 -3
- package/scripts/generate_windsurf_rules.py +13 -3
- package/scripts/install_steps/ai_tools.py +16 -8
- package/scripts/validate.py +1 -0
package/llms-full.txt
CHANGED
|
@@ -14,7 +14,12 @@
|
|
|
14
14
|
- [Best Practices](kb/best-practices/README.md)
|
|
15
15
|
- [No Hardcoded Counts in Secondary Docs](kb/best-practices/no-hardcoded-counts.md)
|
|
16
16
|
- [How-To Guides](kb/howto/README.md)
|
|
17
|
+
- [Plan: Cloud Security Pack — Multi-Cloud Audit](kb/planning/cloud-security-pack-plan.md)
|
|
18
|
+
- [Plan: Enterprise Config Inheritance — Multi-Repo Governance with `extends`](kb/planning/enterprise-config-inheritance-plan.md)
|
|
19
|
+
- [Plan: Local Dashboard — `ai-toolkit ui`](kb/planning/local-dashboard-plan.md)
|
|
20
|
+
- [Plan: Offline-First SLM Profile — Lightweight Mode for Local Models](kb/planning/offline-slm-profile-plan.md)
|
|
17
21
|
- [SOP: Claude Toolkit Maintenance](kb/procedures/maintenance-sop.md)
|
|
22
|
+
- [SOP: Release Preparation](kb/procedures/release-preparation-sop.md)
|
|
18
23
|
- [SOP: Release Verification](kb/procedures/release-verification-sop.md)
|
|
19
24
|
- [Agents Catalog](kb/reference/agents-catalog.md)
|
|
20
25
|
- [Anti-Pattern Registry Format](kb/reference/anti-pattern-registry-format.md)
|
|
@@ -266,6 +271,3624 @@ Step-by-step guides for common tasks. Guides will be added here as they are crea
|
|
|
266
271
|
|
|
267
272
|
---
|
|
268
273
|
|
|
274
|
+
## kb/planning/cloud-security-pack-plan.md
|
|
275
|
+
|
|
276
|
+
---
|
|
277
|
+
title: "Plan: Cloud Security Pack — Multi-Cloud Audit (GCP/AWS/Azure)"
|
|
278
|
+
category: planning
|
|
279
|
+
service: ai-toolkit
|
|
280
|
+
tags:
|
|
281
|
+
- cloud-security
|
|
282
|
+
- gcp
|
|
283
|
+
- aws
|
|
284
|
+
- azure
|
|
285
|
+
- firebase
|
|
286
|
+
- plugin-pack
|
|
287
|
+
- security-audit
|
|
288
|
+
- credentials
|
|
289
|
+
doc_type: plan
|
|
290
|
+
status: proposed
|
|
291
|
+
created: "2026-04-10"
|
|
292
|
+
last_updated: "2026-04-10"
|
|
293
|
+
completion: "0%"
|
|
294
|
+
council_review: "2026-04-10 — conditional FOR, scope reduction recommended"
|
|
295
|
+
description: "Plugin pack for deterministic, read-only security auditing of GCP (Firebase/Cloud Functions), AWS (S3/Lambda/IAM), and Azure (NSG/Functions/CosmosDB). Includes CLI credential management, static+live modes, false positive resolution, SARIF output, incremental scanning, and CI integration."
|
|
296
|
+
---
|
|
297
|
+
|
|
298
|
+
# Plan: Cloud Security Pack — Multi-Cloud Audit
|
|
299
|
+
|
|
300
|
+
**Status:** Proposed
|
|
301
|
+
**Completion:** 0%
|
|
302
|
+
**Created:** 2026-04-10
|
|
303
|
+
**Origin:** Firebase RTDB/Firestore rules audit, Cloud Functions public exposure, false positive resolution for App Check/Gateway patterns
|
|
304
|
+
**Estimated Effort:** 5-6 weeks (council-revised from original 3-4 weeks)
|
|
305
|
+
|
|
306
|
+
---
|
|
307
|
+
|
|
308
|
+
## 1. Objective
|
|
309
|
+
|
|
310
|
+
Create `cloud-security-pack` plugin pack that provides deterministic, read-only security auditing for three major cloud providers. All scripts are stdlib-only Python with zero external dependencies. The pack includes a dedicated agent, multiple scan scripts, CLI credential management, and CI pipeline integration.
|
|
311
|
+
|
|
312
|
+
**Key design principles:**
|
|
313
|
+
- **Read-only** — never modifies cloud resources, only reads state
|
|
314
|
+
- **Deterministic** — reproducible results, no LLM-driven regex (same pattern as `hipaa_scan.py`)
|
|
315
|
+
- **False positive aware** — context graph resolves "public endpoint behind gateway/App Check/WAF"
|
|
316
|
+
- **CI-ready** — `--output json` + `--output sarif` (SARIF v2.1.0 for GitHub Advanced Security), exit code 1 on HIGH, 0 otherwise
|
|
317
|
+
- **Credential isolation** — keys stored in `~/.ai-toolkit/credentials/`, accessible only by this pack's scripts
|
|
318
|
+
- **Static-first** — static mode (no credentials) is the default, live mode is opt-in upgrade
|
|
319
|
+
- **Incremental** — `--changed` flag scans only files modified since last commit (PR workflow)
|
|
320
|
+
- **IaC via `terraform show -json`** — wraps Terraform's own JSON output instead of parsing HCL directly
|
|
321
|
+
|
|
322
|
+
---
|
|
323
|
+
|
|
324
|
+
## 2. Architecture Overview
|
|
325
|
+
|
|
326
|
+
```
|
|
327
|
+
ai-toolkit credentials add gcp --file ~/sa-viewer.json
|
|
328
|
+
ai-toolkit credentials add aws --profile my-audit-profile
|
|
329
|
+
ai-toolkit credentials add azure --subscription abc-123
|
|
330
|
+
|
|
331
|
+
┌──────────────────────────────────────────────────────┐
|
|
332
|
+
│ cloud-security-pack │
|
|
333
|
+
│ │
|
|
334
|
+
│ Agent: cloud-security-auditor │
|
|
335
|
+
│ Tools: Read, Grep, Glob, Bash (read-only commands) │
|
|
336
|
+
│ │
|
|
337
|
+
│ Skills: │
|
|
338
|
+
│ /cloud-security-audit (orchestrator) │
|
|
339
|
+
│ /firebase-rules-audit (GCP: rules) │
|
|
340
|
+
│ /cloud-functions-audit (GCP: CF + IAM) │
|
|
341
|
+
│ /aws-security-audit (AWS: S3/Lambda) │
|
|
342
|
+
│ /azure-security-audit (Azure: NSG/Fn) │
|
|
343
|
+
│ │
|
|
344
|
+
│ Scripts (stdlib Python, zero deps): │
|
|
345
|
+
│ gcp_auth.py (credential helper)│
|
|
346
|
+
│ firebase_rules_scan.py (static parser) │
|
|
347
|
+
│ cloud_functions_audit.py (CF IAM + context) │
|
|
348
|
+
│ aws_security_scan.py (S3/Lambda/IAM) │
|
|
349
|
+
│ azure_security_scan.py (NSG/Fn/RBAC) │
|
|
350
|
+
│ false_positive_resolver.py (context graph) │
|
|
351
|
+
│ sarif_formatter.py (SARIF v2.1.0) │
|
|
352
|
+
│ incremental.py (git diff filter) │
|
|
353
|
+
│ │
|
|
354
|
+
│ Modes: │
|
|
355
|
+
│ --static (no credentials, parse IaC/source) │
|
|
356
|
+
│ --live (credentials, deployed state) │
|
|
357
|
+
│ --output json|sarif (CI pipeline) │
|
|
358
|
+
│ --changed <ref> (incremental, static only) │
|
|
359
|
+
│ --explain <id> (remediation lookup) │
|
|
360
|
+
└──────────────────────────────────────────────────────┘
|
|
361
|
+
|
|
362
|
+
### YAML Parsing Constraint (BLOCKER)
|
|
363
|
+
|
|
364
|
+
Python stdlib has NO YAML parser. This affects AWS static mode:
|
|
365
|
+
- CloudFormation templates (`.yaml`) — YAML
|
|
366
|
+
- `serverless.yml` — YAML
|
|
367
|
+
- SAM templates (`template.yaml`) — YAML
|
|
368
|
+
|
|
369
|
+
**Decision: JSON-only for static IaC parsing.** Rationale:
|
|
370
|
+
1. CloudFormation supports both JSON and YAML — JSON variant parseable with `json` module
|
|
371
|
+
2. `terraform show -json` outputs JSON — the primary Terraform path
|
|
372
|
+
3. `serverless.yml` → recommend users run `sls print --format json` to convert
|
|
373
|
+
4. Adding `pyyaml` breaks the stdlib-only constraint for the entire toolkit
|
|
374
|
+
|
|
375
|
+
**Practical impact:** ~30% of CloudFormation users use JSON, ~70% YAML. For YAML users, live mode (`--mode live`) still works (queries APIs directly, no file parsing needed). The `--explain` output will suggest `sls print --format json` conversion.
|
|
376
|
+
|
|
377
|
+
**v2 option:** Ship a vendored minimal YAML subset parser (~150 LOC, handles flat key-value and simple nested maps — enough for security-relevant fields like `Principal`, `Effect`, `authLevel`).
|
|
378
|
+
```
|
|
379
|
+
|
|
380
|
+
---
|
|
381
|
+
|
|
382
|
+
## 3. Progress Tracking
|
|
383
|
+
|
|
384
|
+
| # | Feature | Priority | Status | Est. Time | Notes |
|
|
385
|
+
|---|---------|----------|--------|-----------|-------|
|
|
386
|
+
| 1.1 | CLI `credentials` command (add/list/remove/test) | P0 | Proposed | 2d | 0600 perms, allowlist wrapper |
|
|
387
|
+
| 1.1b | `credentials init` interactive wizard | P1 | Proposed | 1.5d | **Deferred to Milestone 2** (orchestration-review: saves 1.5d in M1 critical path) |
|
|
388
|
+
| 1.2 | `cloud-security-auditor` agent | P0 | Proposed | 1d | Agent definition |
|
|
389
|
+
| 1.3 | SARIF + incremental scan infrastructure | P0 | Proposed | 2d | `--output sarif`, `--changed` flag |
|
|
390
|
+
| 2.1 | `firebase-rules-audit` skill + script | P0 | Proposed | 4-5d | Recursive descent parser (orchestration-review: +1d vs regex) |
|
|
391
|
+
| 2.2 | `cloud-functions-audit` skill + script | P0 | Proposed | 3-4d | CF IAM + App Check context |
|
|
392
|
+
| 2.3 | False positive resolver (GCP context) | P0 | Proposed | 3-4d | Context graph engine — GCP only (~8 code paths). **+2d per provider** in later milestones (orchestration-review) |
|
|
393
|
+
| 3.1 | `aws-security-audit` skill + script | P1 | Proposed | 4-5d | S3/Lambda/IAM/SG, `terraform show -json` |
|
|
394
|
+
| 3.2 | `/cloud-security-audit` orchestrator + plugin.json | P1 | Proposed | 3d | Multi-provider orchestration + pack |
|
|
395
|
+
| 4.1 | `azure-security-audit` skill + script | P1 | Proposed | 4-5d | NSG/Functions/RBAC |
|
|
396
|
+
| 5.1 | Tests + CI integration docs | P1 | Proposed | 3d | Tests + SARIF + pipeline examples |
|
|
397
|
+
| 5.2 | Documentation (kb/) | P2 | Proposed | 1d | Checklists, patterns, KB |
|
|
398
|
+
|
|
399
|
+
**Phasing (full delivery, all 3 providers):**
|
|
400
|
+
- **Phase 1 (week 1-2):** Foundation + GCP — credentials CLI, agent, SARIF/incremental infra, firebase-rules, cloud-functions, false positive resolver (GCP context)
|
|
401
|
+
- **Phase 2 (week 3-4):** AWS + orchestrator — AWS security audit, `terraform show -json`, orchestrator skill, plugin pack, `credentials init`
|
|
402
|
+
- **Phase 3 (week 5-6):** Azure + polish — Azure security audit, false positive resolver (Azure context), full test suite, documentation
|
|
403
|
+
|
|
404
|
+
---
|
|
405
|
+
|
|
406
|
+
## 4. Dependency Graph
|
|
407
|
+
|
|
408
|
+
```
|
|
409
|
+
Phase 1: Foundation + GCP (week 1-2)
|
|
410
|
+
====================================
|
|
411
|
+
credentials CLI (1.1) ─────┐
|
|
412
|
+
├──► firebase-rules-audit (2.1)
|
|
413
|
+
SARIF + incremental (1.3) ──┤
|
|
414
|
+
├──► cloud-functions-audit (2.2) ──► false-positive-resolver (2.3)
|
|
415
|
+
agent definition (1.2) ─────┘
|
|
416
|
+
|
|
417
|
+
Phase 2: AWS + Orchestrator (week 3-4)
|
|
418
|
+
======================================
|
|
419
|
+
credentials init (1.1b) ───┐
|
|
420
|
+
├──► aws-security-audit (3.1) ─────► false-positive-resolver (+AWS context)
|
|
421
|
+
└──► orchestrator skill + plugin.json (3.2)
|
|
422
|
+
|
|
423
|
+
Phase 3: Azure + Polish (week 5-6)
|
|
424
|
+
==================================
|
|
425
|
+
├──► azure-security-audit (4.1) ───► false-positive-resolver (+Azure context)
|
|
426
|
+
└──► tests + docs (5.1, 5.2)
|
|
427
|
+
```
|
|
428
|
+
|
|
429
|
+
**All 3 providers ship.** No conditional gates — full delivery in 6 weeks.
|
|
430
|
+
|
|
431
|
+
---
|
|
432
|
+
|
|
433
|
+
## 5. Detailed Implementation
|
|
434
|
+
|
|
435
|
+
### Faza 1: Foundation (tydzien 1)
|
|
436
|
+
|
|
437
|
+
#### 1.1 CLI `credentials` Command
|
|
438
|
+
|
|
439
|
+
**Purpose:** Secure credential storage for cloud provider API access. Credentials live outside any project directory and are only accessible by this pack's scripts.
|
|
440
|
+
|
|
441
|
+
**CLI interface:**
|
|
442
|
+
```bash
|
|
443
|
+
# GCP — Service Account JSON
|
|
444
|
+
ai-toolkit credentials add gcp --file ~/sa-viewer.json
|
|
445
|
+
ai-toolkit credentials add gcp --file ~/sa-viewer.json --project my-project-id
|
|
446
|
+
|
|
447
|
+
# GCP — use existing gcloud session (no file needed)
|
|
448
|
+
ai-toolkit credentials add gcp --gcloud --project my-project-id
|
|
449
|
+
|
|
450
|
+
# AWS — named profile (reads from ~/.aws/credentials)
|
|
451
|
+
ai-toolkit credentials add aws --profile audit-readonly
|
|
452
|
+
ai-toolkit credentials add aws --profile audit-readonly --region eu-west-1
|
|
453
|
+
|
|
454
|
+
# AWS — explicit keys (interactive, never on CLI args)
|
|
455
|
+
ai-toolkit credentials add aws --interactive
|
|
456
|
+
|
|
457
|
+
# Azure — subscription
|
|
458
|
+
ai-toolkit credentials add azure --subscription abc-123-def
|
|
459
|
+
|
|
460
|
+
# Azure — use existing az login session
|
|
461
|
+
ai-toolkit credentials add azure --az-cli
|
|
462
|
+
|
|
463
|
+
# Interactive guided setup (reduces onboarding from 4 steps to 1)
|
|
464
|
+
ai-toolkit credentials init # auto-detect provider, interactive wizard
|
|
465
|
+
ai-toolkit credentials init --provider gcp # skip auto-detect, go straight to GCP setup
|
|
466
|
+
|
|
467
|
+
# Management
|
|
468
|
+
ai-toolkit credentials list
|
|
469
|
+
ai-toolkit credentials remove gcp
|
|
470
|
+
ai-toolkit credentials remove aws
|
|
471
|
+
ai-toolkit credentials remove azure
|
|
472
|
+
ai-toolkit credentials test gcp # verify read-only access works
|
|
473
|
+
ai-toolkit credentials test aws
|
|
474
|
+
```
|
|
475
|
+
|
|
476
|
+
**Storage structure:**
|
|
477
|
+
```
|
|
478
|
+
~/.ai-toolkit/
|
|
479
|
+
credentials/
|
|
480
|
+
gcp.json # SA key file (copied, chmod 0600)
|
|
481
|
+
gcp.meta.json # { project_id, added_at, method: "file"|"gcloud" }
|
|
482
|
+
aws.json # { profile, region, method: "profile"|"keys" }
|
|
483
|
+
azure.json # { subscription_id, method: "subscription"|"az-cli" }
|
|
484
|
+
```
|
|
485
|
+
|
|
486
|
+
**`credentials init` interactive flow:**
|
|
487
|
+
1. Auto-detect providers from project files (`firebase.json` → GCP, `*.tf` with `provider "aws"` → AWS, etc.)
|
|
488
|
+
2. If multiple detected → ask user: "Found GCP and AWS markers. Which provider to configure first? [gcp/aws/both]"
|
|
489
|
+
3. Per provider:
|
|
490
|
+
- GCP: "Do you have a Service Account JSON file? [y/n]" → if yes: ask path → if no: "Run `gcloud auth application-default login` and we'll use that"
|
|
491
|
+
- AWS: "Do you have a named profile in ~/.aws/credentials? [y/n]" → if yes: ask profile name → if no: "Run `aws configure` first"
|
|
492
|
+
- Azure: "Do you have an active `az login` session? [y/n]" → if yes: ask subscription ID → if no: "Run `az login` first"
|
|
493
|
+
4. Run `credentials test` automatically after setup
|
|
494
|
+
5. Generate `.cloud-security.json` scaffold with detected context
|
|
495
|
+
|
|
496
|
+
**Security requirements:**
|
|
497
|
+
- All credential files: `chmod 0600` (owner read/write only)
|
|
498
|
+
- Never log credential contents to stdout/stderr
|
|
499
|
+
- `credentials test` validates:
|
|
500
|
+
- Connection works (GCP: `gcloud auth list`, AWS: `aws sts get-caller-identity`, Azure: `az account show`)
|
|
501
|
+
- SA/role has **only read permissions** — **REFUSE to store** if write access detected (orchestration-review: warn-only is ignored by users). Override: `--force` flag with explicit acknowledgment
|
|
502
|
+
- Project/subscription exists
|
|
503
|
+
- `.gitignore`-proof — lives in `~/.ai-toolkit/`, never in project directory
|
|
504
|
+
- Scripts access credentials via `gcp_auth.py` helper — single entry point, no direct file reads
|
|
505
|
+
|
|
506
|
+
**Files to create/modify:**
|
|
507
|
+
|
|
508
|
+
| File | Action | Description |
|
|
509
|
+
|------|--------|-------------|
|
|
510
|
+
| `scripts/credentials_cli.py` | CREATE | CLI: add/list/remove/test credentials |
|
|
511
|
+
| `bin/ai-toolkit.js` | EDIT | Register `credentials` subcommand |
|
|
512
|
+
| `tests/test_credentials.bats` | CREATE | Tests: add, remove, permissions, test |
|
|
513
|
+
|
|
514
|
+
**Success Criteria:**
|
|
515
|
+
- [ ] `credentials add gcp --file` copies and secures SA key
|
|
516
|
+
- [ ] `credentials add aws --profile` stores profile reference
|
|
517
|
+
- [ ] `credentials add azure --subscription` stores subscription
|
|
518
|
+
- [ ] `credentials test` validates read access + warns on write perms
|
|
519
|
+
- [ ] `credentials list` shows providers without exposing secrets
|
|
520
|
+
- [ ] `credentials remove` cleans up securely
|
|
521
|
+
- [ ] All files created with 0600 permissions
|
|
522
|
+
- [ ] Tests: >= 8
|
|
523
|
+
|
|
524
|
+
---
|
|
525
|
+
|
|
526
|
+
#### 1.3 SARIF Output + Incremental Scanning (Council addition)
|
|
527
|
+
|
|
528
|
+
**SARIF v2.1.0 output** — industry standard consumed by GitHub Advanced Security (inline PR annotations), VS Code SARIF Viewer, Azure DevOps, and SonarQube.
|
|
529
|
+
|
|
530
|
+
```bash
|
|
531
|
+
/cloud-security-audit --output sarif > results.sarif
|
|
532
|
+
|
|
533
|
+
# GitHub Actions: upload SARIF for inline PR annotations
|
|
534
|
+
- uses: github/codeql-action/upload-sarif@v3
|
|
535
|
+
with:
|
|
536
|
+
sarif_file: results.sarif
|
|
537
|
+
```
|
|
538
|
+
|
|
539
|
+
**SARIF structure (per finding):**
|
|
540
|
+
```json
|
|
541
|
+
{
|
|
542
|
+
"$schema": "https://raw.githubusercontent.com/oasis-tcs/sarif-spec/main/sarif-2.1/schema/sarif-schema-2.1.0.json",
|
|
543
|
+
"version": "2.1.0",
|
|
544
|
+
"runs": [{
|
|
545
|
+
"tool": { "driver": {
|
|
546
|
+
"name": "cloud-security-audit", "version": "1.0.0",
|
|
547
|
+
"rules": [{ "id": "GCP-CF-001", "shortDescription": { "text": "Public Cloud Function invoker" }, "helpUri": "https://cloud.google.com/functions/docs/securing" }]
|
|
548
|
+
} },
|
|
549
|
+
"results": [{
|
|
550
|
+
"ruleId": "GCP-CF-001",
|
|
551
|
+
"level": "error",
|
|
552
|
+
"message": { "text": "Cloud Function 'adminEndpoint' has allUsers invoker with no protection layer" },
|
|
553
|
+
"locations": [{ "physicalLocation": { "artifactLocation": { "uri": "functions/src/admin.ts" }, "region": { "startLine": 42 } } }],
|
|
554
|
+
"properties": { "resolved_severity": "HIGH", "context_chain": [], "provider": "gcp" }
|
|
555
|
+
}]
|
|
556
|
+
}]
|
|
557
|
+
}
|
|
558
|
+
```
|
|
559
|
+
|
|
560
|
+
**Incremental scan mode** — scan only changed files since last commit:
|
|
561
|
+
|
|
562
|
+
```bash
|
|
563
|
+
/cloud-security-audit --changed # files changed vs HEAD~1
|
|
564
|
+
/cloud-security-audit --changed HEAD~5 # files changed in last 5 commits
|
|
565
|
+
/cloud-security-audit --changed main # files changed vs main branch (PR workflow)
|
|
566
|
+
```
|
|
567
|
+
|
|
568
|
+
Implementation: use `git diff --name-only <ref>` to get changed files, filter to relevant extensions (.rules, .tf, .json, .ts, .py, .bicep), scan only those. Falls back to full scan if no git repo detected.
|
|
569
|
+
|
|
570
|
+
**`--changed` applies to static mode ONLY.** Live mode always scans all deployed resources (it queries cloud APIs, not files). If user passes `--changed` with `--mode live`, emit warning: "Incremental scan applies to static analysis only. Live mode will scan all resources." and proceed with full live scan.
|
|
571
|
+
|
|
572
|
+
**`--explain` flag** — detailed remediation for a specific finding:
|
|
573
|
+
|
|
574
|
+
```bash
|
|
575
|
+
/cloud-security-audit --explain GCP-CF-001
|
|
576
|
+
# Output: what the finding means, why it matters, exact fix steps,
|
|
577
|
+
# links to GCP documentation, CIS Benchmark reference
|
|
578
|
+
```
|
|
579
|
+
|
|
580
|
+
**Files:**
|
|
581
|
+
|
|
582
|
+
| File | Action | Description |
|
|
583
|
+
|------|--------|-------------|
|
|
584
|
+
| `app/skills/cloud-security-audit/scripts/sarif_formatter.py` | CREATE | SARIF v2.1.0 output |
|
|
585
|
+
| `app/skills/cloud-security-audit/scripts/incremental.py` | CREATE | Git diff + file filter |
|
|
586
|
+
| `app/skills/cloud-security-audit/reference/rule-explanations.json` | CREATE | Per-rule remediation guides |
|
|
587
|
+
|
|
588
|
+
**Success Criteria:**
|
|
589
|
+
- [ ] `--output sarif` produces valid SARIF v2.1.0 JSON with `driver.rules[]` array (orchestration-review: GitHub silently drops annotations without rule metadata)
|
|
590
|
+
- [ ] SARIF upload to GitHub Advanced Security works (inline PR annotations)
|
|
591
|
+
- [ ] SARIF `level` mapping: HIGH→`error`, WARN→`warning`, INFO→`note`
|
|
592
|
+
- [ ] `--changed main` scans only PR-changed files
|
|
593
|
+
- [ ] `--explain <rule-id>` shows detailed remediation
|
|
594
|
+
- [ ] Tests: >= 6
|
|
595
|
+
|
|
596
|
+
---
|
|
597
|
+
|
|
598
|
+
#### 1.2 Agent Definition: `cloud-security-auditor`
|
|
599
|
+
|
|
600
|
+
**File:** `app/agents/cloud-security-auditor.md`
|
|
601
|
+
|
|
602
|
+
```markdown
|
|
603
|
+
---
|
|
604
|
+
name: cloud-security-auditor
|
|
605
|
+
description: "Multi-cloud security auditor (GCP/AWS/Azure). Read-only deterministic scans
|
|
606
|
+
for IAM, network, storage, serverless, and compliance. False positive resolution
|
|
607
|
+
via security context graph."
|
|
608
|
+
model: opus
|
|
609
|
+
color: red
|
|
610
|
+
tools: Read, Grep, Glob, Bash
|
|
611
|
+
skills: security-patterns, cloud-security-audit
|
|
612
|
+
---
|
|
613
|
+
|
|
614
|
+
# Cloud Security Auditor Agent
|
|
615
|
+
|
|
616
|
+
You are the **Cloud Security Auditor**. You perform read-only security assessments
|
|
617
|
+
across GCP, AWS, and Azure. You never modify cloud resources.
|
|
618
|
+
|
|
619
|
+
## Core Philosophy
|
|
620
|
+
**"Read everything, change nothing. Context before verdict."**
|
|
621
|
+
|
|
622
|
+
## Mandatory Protocol
|
|
623
|
+
Before any audit:
|
|
624
|
+
1. Check credentials: `ai-toolkit credentials test <provider>`
|
|
625
|
+
2. Determine mode: `--static` (IaC/source only) or `--live` (deployed state)
|
|
626
|
+
3. Run deterministic scripts first, then interpret results
|
|
627
|
+
|
|
628
|
+
## Responsibilities
|
|
629
|
+
|
|
630
|
+
### 1. Static Analysis (no credentials needed)
|
|
631
|
+
- Parse IaC: Terraform (.tf), CloudFormation (.yaml/.json), Bicep (.bicep)
|
|
632
|
+
- Parse Firebase rules: firestore.rules, database.rules.json
|
|
633
|
+
- Parse source: Cloud Functions (onCall vs onRequest), Lambda handlers, Azure Functions
|
|
634
|
+
- Parse configs: firebase.json, serverless.yml, sam-template.yaml
|
|
635
|
+
|
|
636
|
+
### 2. Live Analysis (credentials required, READ-ONLY)
|
|
637
|
+
- GCP: `gcloud` CLI commands (list, describe, get-iam-policy)
|
|
638
|
+
- AWS: `aws` CLI commands (s3api, lambda, iam, ec2 — get/list/describe only)
|
|
639
|
+
- Azure: `az` CLI commands (network nsg, functionapp, cosmosdb — list/show only)
|
|
640
|
+
|
|
641
|
+
### 3. False Positive Resolution
|
|
642
|
+
Build security context graph before rendering verdict:
|
|
643
|
+
- Public endpoint → check: API Gateway? WAF? App Check? CDN?
|
|
644
|
+
- Open port → check: behind Load Balancer? VPN? private subnet?
|
|
645
|
+
- Broad IAM role → check: scoped to specific resource? temporary?
|
|
646
|
+
|
|
647
|
+
## Allowed CLI Commands (WHITELIST — read-only only)
|
|
648
|
+
|
|
649
|
+
### GCP
|
|
650
|
+
- gcloud functions list / describe / get-iam-policy
|
|
651
|
+
- gcloud projects get-iam-policy
|
|
652
|
+
- gcloud app-check services list
|
|
653
|
+
- gcloud firestore indexes list
|
|
654
|
+
- gcloud compute firewall-rules list
|
|
655
|
+
- gcloud run services list / describe
|
|
656
|
+
- firebase apps:list
|
|
657
|
+
|
|
658
|
+
### AWS
|
|
659
|
+
- aws s3api get-bucket-policy / get-bucket-acl / get-public-access-block
|
|
660
|
+
- aws lambda get-policy / get-function-configuration / list-functions
|
|
661
|
+
- aws iam list-roles / list-policies / get-role / get-policy-version
|
|
662
|
+
- aws ec2 describe-security-groups / describe-network-acls
|
|
663
|
+
- aws apigateway get-rest-apis / get-resources
|
|
664
|
+
- aws elbv2 describe-load-balancers / describe-listeners
|
|
665
|
+
|
|
666
|
+
### Azure
|
|
667
|
+
- az network nsg list / show / rule list
|
|
668
|
+
- az functionapp list / show / config show
|
|
669
|
+
- az cosmosdb list / show / keys list
|
|
670
|
+
- az role assignment list
|
|
671
|
+
- az webapp show / config show
|
|
672
|
+
- az network application-gateway list
|
|
673
|
+
|
|
674
|
+
### NEVER ALLOWED
|
|
675
|
+
- Any create/update/delete/put/set/deploy/push command
|
|
676
|
+
- Any command that modifies state
|
|
677
|
+
- `gcloud auth activate-service-account` (credential pivot)
|
|
678
|
+
- `aws sts assume-role` (lateral movement)
|
|
679
|
+
- `az login` (session hijack)
|
|
680
|
+
- `terraform apply` / `terraform destroy`
|
|
681
|
+
|
|
682
|
+
## Output Format
|
|
683
|
+
(see skill SKILL.md for detailed format)
|
|
684
|
+
```
|
|
685
|
+
|
|
686
|
+
**SECURITY: Programmatic Bash Allowlist (orchestration-review P0 BLOCKER)**
|
|
687
|
+
|
|
688
|
+
The read-only whitelist above lives in the agent's system prompt. An LLM can be prompt-injected via malicious IaC files or source code comments. **The prompt-only approach is insufficient.**
|
|
689
|
+
|
|
690
|
+
**Required:** A shell wrapper script (`scripts/cloud_security_allowlist.sh`) that validates every CLI invocation against a compiled allowlist regex BEFORE execution, independent of the LLM:
|
|
691
|
+
|
|
692
|
+
```bash
|
|
693
|
+
#!/bin/bash
|
|
694
|
+
# cloud_security_allowlist.sh — wraps Bash calls from cloud-security-auditor
|
|
695
|
+
# Rejects any command not matching read-only patterns
|
|
696
|
+
|
|
697
|
+
ALLOWED_PATTERNS=(
|
|
698
|
+
'^gcloud (functions|projects|app-check|firestore|compute|run) (list|describe|get-iam-policy|indexes)'
|
|
699
|
+
'^gcloud auth list$'
|
|
700
|
+
'^firebase apps:list'
|
|
701
|
+
'^aws (s3api|lambda|iam|ec2|apigateway|elbv2|sts|cloudfront) (get-|list-|describe-|generate-credential-report)'
|
|
702
|
+
'^aws sts get-caller-identity$'
|
|
703
|
+
'^az (network|functionapp|cosmosdb|role|webapp|storage) (list|show|rule list|config show|assignment list|account show)'
|
|
704
|
+
'^az account show$'
|
|
705
|
+
'^terraform show -json'
|
|
706
|
+
'^git diff --name-only'
|
|
707
|
+
'^python3 .*/scripts/.*\.(py)$'
|
|
708
|
+
)
|
|
709
|
+
|
|
710
|
+
CMD="$*"
|
|
711
|
+
for pattern in "${ALLOWED_PATTERNS[@]}"; do
|
|
712
|
+
if [[ "$CMD" =~ $pattern ]]; then
|
|
713
|
+
exec $CMD
|
|
714
|
+
fi
|
|
715
|
+
done
|
|
716
|
+
|
|
717
|
+
echo "BLOCKED: Command not in read-only allowlist: $CMD" >&2
|
|
718
|
+
exit 1
|
|
719
|
+
```
|
|
720
|
+
|
|
721
|
+
The agent's `allowed-tools` in SKILL.md references this wrapper instead of raw Bash. All cloud CLI calls go through the allowlist.
|
|
722
|
+
|
|
723
|
+
**`terraform plan` risk (orchestration-review):** `terraform plan -json` executes providers and provisioners. A malicious `.tf` file with `local-exec` provisioner runs arbitrary code during plan. **Decision: use `terraform show -json` ONLY (reads existing state), NOT `terraform plan -json`.** Document this prominently.
|
|
724
|
+
|
|
725
|
+
**Files:**
|
|
726
|
+
|
|
727
|
+
| File | Action | Description |
|
|
728
|
+
|------|--------|-------------|
|
|
729
|
+
| `app/skills/cloud-security-audit/scripts/cloud_security_allowlist.sh` | CREATE | Bash allowlist wrapper |
|
|
730
|
+
|
|
731
|
+
**Success Criteria:**
|
|
732
|
+
- [ ] Agent file created in `app/agents/`
|
|
733
|
+
- [ ] Read-only command whitelist documented
|
|
734
|
+
- [ ] NEVER ALLOWED section explicit
|
|
735
|
+
- [ ] Programmatic Bash allowlist enforced (not just prompt)
|
|
736
|
+
- [ ] `terraform plan` excluded — only `terraform show -json` allowed
|
|
737
|
+
- [ ] Allowlist tested: blocked commands return exit 1
|
|
738
|
+
|
|
739
|
+
---
|
|
740
|
+
|
|
741
|
+
### Faza 2: GCP / Firebase (Phase 1, tydzien 1-2)
|
|
742
|
+
|
|
743
|
+
#### 2.1 Skill: `firebase-rules-audit`
|
|
744
|
+
|
|
745
|
+
**Purpose:** Static analysis of Firestore rules and RTDB rules. No credentials needed.
|
|
746
|
+
|
|
747
|
+
**What it scans:**
|
|
748
|
+
|
|
749
|
+
| Check | Severity | Description |
|
|
750
|
+
|-------|----------|-------------|
|
|
751
|
+
| `allow read, write: if true` | HIGH | World-readable/writable collection |
|
|
752
|
+
| `allow read: if true` without `write` guard | WARN | Public read — may be intentional |
|
|
753
|
+
| `allow write: if request.auth != null` without field validation | WARN | Authenticated but no field-level validation |
|
|
754
|
+
| Missing `request.resource.data` validation on writes | WARN | No schema enforcement |
|
|
755
|
+
| Wildcard collection `/{document=**}` with broad permissions | HIGH | Recursive wildcard + open access |
|
|
756
|
+
| RTDB `.read: true` or `.write: true` at root | HIGH | Entire database public |
|
|
757
|
+
| RTDB `.read: "auth != null"` without path scoping | WARN | All authenticated users can read everything |
|
|
758
|
+
| Timestamp/TTL rules missing for sensitive collections | WARN | No data lifecycle enforcement |
|
|
759
|
+
| `get()` / `exists()` cross-collection reads without auth check | WARN | Privilege escalation via rule chaining |
|
|
760
|
+
| Rules file > 256KB (approaching Firebase limit) | WARN | May hit deployment limit |
|
|
761
|
+
|
|
762
|
+
**Script:** `scripts/firebase_rules_scan.py`
|
|
763
|
+
- Parses `firestore.rules` via **recursive descent parser** (not regex — orchestration-review P1)
|
|
764
|
+
- Grammar: ~8 production rules (service, match, allow, function, condition)
|
|
765
|
+
- Tracks: brace depth, current match path, accumulated allow blocks
|
|
766
|
+
- Handles: nested `match` blocks, multi-line conditions with `&&`/`||`, custom `function` declarations
|
|
767
|
+
- Estimated: 550-650 LOC for parser + check logic
|
|
768
|
+
- Unsupported (documented): CEL ternary expressions, complex `get()`/`exists()` chains with computed paths
|
|
769
|
+
- Parses `database.rules.json` (JSON rules — stdlib `json` module)
|
|
770
|
+
- Outputs findings as JSON or text
|
|
771
|
+
- Exit code 1 on HIGH, 0 otherwise
|
|
772
|
+
- Supports `.cloud-security-ignore` for suppressions
|
|
773
|
+
|
|
774
|
+
**Reference file:** `reference/firebase-rules-patterns.md` — safe/unsafe patterns with examples
|
|
775
|
+
|
|
776
|
+
**Files:**
|
|
777
|
+
|
|
778
|
+
| File | Action | Description |
|
|
779
|
+
|------|--------|-------------|
|
|
780
|
+
| `app/skills/firebase-rules-audit/SKILL.md` | CREATE | Skill definition |
|
|
781
|
+
| `app/skills/firebase-rules-audit/scripts/firebase_rules_scan.py` | CREATE | Deterministic scanner |
|
|
782
|
+
| `app/skills/firebase-rules-audit/reference/firebase-rules-patterns.md` | CREATE | Safe/unsafe patterns |
|
|
783
|
+
|
|
784
|
+
**Success Criteria:**
|
|
785
|
+
- [ ] Parses `firestore.rules` — detects 10+ check patterns
|
|
786
|
+
- [ ] Parses `database.rules.json` — detects root-level open access
|
|
787
|
+
- [ ] `--output json` for CI
|
|
788
|
+
- [ ] `.cloud-security-ignore` support
|
|
789
|
+
- [ ] Tests: >= 10 (one per check pattern + edge cases)
|
|
790
|
+
|
|
791
|
+
---
|
|
792
|
+
|
|
793
|
+
#### 2.2 Skill: `cloud-functions-audit`
|
|
794
|
+
|
|
795
|
+
**Purpose:** Audit Cloud Functions permissions and detect false positives.
|
|
796
|
+
|
|
797
|
+
**Static mode (no credentials):**
|
|
798
|
+
|
|
799
|
+
| Check | Severity | Description |
|
|
800
|
+
|-------|----------|-------------|
|
|
801
|
+
| `onRequest` handler without auth middleware | WARN | Potentially public — needs context |
|
|
802
|
+
| `onCall` handler (inherently authenticated) | INFO | Informational — callable is auth'd |
|
|
803
|
+
| Hardcoded API keys / secrets in source | HIGH | Secrets in code |
|
|
804
|
+
| CORS `origin: '*'` in CF handler | WARN | Unrestricted CORS |
|
|
805
|
+
| Missing rate limiting patterns | WARN | No throttling on public endpoint |
|
|
806
|
+
| `functions.https.onRequest` + no `validateFirebaseIdToken` | WARN | HTTP function without Firebase Auth check |
|
|
807
|
+
|
|
808
|
+
**Live mode (credentials required):**
|
|
809
|
+
|
|
810
|
+
| Check | Severity | CLI Command | Description |
|
|
811
|
+
|-------|----------|-------------|-------------|
|
|
812
|
+
| `allUsers` invoker on CF | CONTEXT | `gcloud functions get-iam-policy` | Public — resolve with context graph |
|
|
813
|
+
| `allAuthenticatedUsers` invoker | WARN | `gcloud functions get-iam-policy` | Any Google account can invoke |
|
|
814
|
+
| App Check enforcement status | CONTEXT | `gcloud app-check services list` | Feeds into false positive resolution |
|
|
815
|
+
| Deployed rules vs local rules diff | WARN | `gcloud firestore indexes` + local | Rules drift detection |
|
|
816
|
+
| Cloud Run public ingress | CONTEXT | `gcloud run services describe` | Public — resolve with context graph |
|
|
817
|
+
| Overly broad IAM roles on SA | HIGH | `gcloud projects get-iam-policy` | CF service account with editor/owner |
|
|
818
|
+
|
|
819
|
+
**Script:** `scripts/cloud_functions_audit.py`
|
|
820
|
+
|
|
821
|
+
**Files:**
|
|
822
|
+
|
|
823
|
+
| File | Action | Description |
|
|
824
|
+
|------|--------|-------------|
|
|
825
|
+
| `app/skills/cloud-functions-audit/SKILL.md` | CREATE | Skill definition |
|
|
826
|
+
| `app/skills/cloud-functions-audit/scripts/cloud_functions_audit.py` | CREATE | Scanner |
|
|
827
|
+
| `app/skills/cloud-functions-audit/scripts/gcp_auth.py` | CREATE | Credential loader |
|
|
828
|
+
| `app/skills/cloud-functions-audit/reference/false-positives-gcp.md` | CREATE | False positive patterns |
|
|
829
|
+
|
|
830
|
+
**Success Criteria:**
|
|
831
|
+
- [ ] Static: parses CF source for auth patterns
|
|
832
|
+
- [ ] Live: checks IAM bindings via `gcloud`
|
|
833
|
+
- [ ] False positive resolution for App Check + Gateway patterns
|
|
834
|
+
- [ ] Tests: >= 8
|
|
835
|
+
|
|
836
|
+
---
|
|
837
|
+
|
|
838
|
+
#### 2.3 False Positive Resolver
|
|
839
|
+
|
|
840
|
+
**Purpose:** Central engine that resolves "is this actually a problem?" by building a security context graph.
|
|
841
|
+
|
|
842
|
+
**How it works:**
|
|
843
|
+
```
|
|
844
|
+
Input: Finding { resource, severity, type }
|
|
845
|
+
|
|
846
|
+
Step 1: Gather context
|
|
847
|
+
├── Check API Gateway routes (firebase.json rewrites, API Gateway configs)
|
|
848
|
+
├── Check WAF/CDN (Cloudflare, CloudFront, Azure Front Door)
|
|
849
|
+
├── Check App Check / AppArmor / Shield
|
|
850
|
+
├── Check callable vs HTTP function type
|
|
851
|
+
├── Check VPC / private subnet placement
|
|
852
|
+
└── Check Load Balancer + auth middleware
|
|
853
|
+
|
|
854
|
+
Step 2: Apply resolution rules
|
|
855
|
+
├── Public CF + App Check ENFORCED → SUPPRESSED (protected)
|
|
856
|
+
├── Public CF + API Gateway route → SUPPRESSED (gateway handles auth)
|
|
857
|
+
├── Public CF + onCall() → SUPPRESSED (callable is auth'd by SDK)
|
|
858
|
+
├── Public S3 + CloudFront OAI → SUPPRESSED (not directly accessible)
|
|
859
|
+
├── Open SG port + ALB → SUPPRESSED (ALB handles TLS + auth)
|
|
860
|
+
├── Open NSG + Application Gateway → SUPPRESSED (WAF handles filtering)
|
|
861
|
+
└── No context found → KEEP ORIGINAL SEVERITY
|
|
862
|
+
|
|
863
|
+
Step 3: Output
|
|
864
|
+
├── Original severity
|
|
865
|
+
├── Resolved severity (SUPPRESSED / DOWNGRADED / CONFIRMED)
|
|
866
|
+
├── Context chain (what protections were found)
|
|
867
|
+
└── Confidence (high if multiple protections, low if single)
|
|
868
|
+
```
|
|
869
|
+
|
|
870
|
+
**Output example:**
|
|
871
|
+
```json
|
|
872
|
+
{
|
|
873
|
+
"resource": "processPayment",
|
|
874
|
+
"provider": "gcp",
|
|
875
|
+
"type": "cloud-function-public-invoker",
|
|
876
|
+
"original_severity": "HIGH",
|
|
877
|
+
"resolved_severity": "SUPPRESSED",
|
|
878
|
+
"confidence": "high",
|
|
879
|
+
"context_chain": [
|
|
880
|
+
{ "layer": "app_check", "status": "ENFORCED", "source": "gcloud app-check services list" },
|
|
881
|
+
{ "layer": "function_type", "status": "onCall", "source": "source:index.ts:42" },
|
|
882
|
+
{ "layer": "api_gateway", "status": "ROUTED", "source": "firebase.json:rewrites" }
|
|
883
|
+
],
|
|
884
|
+
"verdict": "3/3 protection layers active. Suppressing finding."
|
|
885
|
+
}
|
|
886
|
+
```
|
|
887
|
+
|
|
888
|
+
**Script:** `scripts/false_positive_resolver.py`
|
|
889
|
+
|
|
890
|
+
**Resolution rules stored in:** `reference/resolution-rules.json`
|
|
891
|
+
```json
|
|
892
|
+
{
|
|
893
|
+
"rules": [
|
|
894
|
+
{
|
|
895
|
+
"id": "gcp-cf-appcheck",
|
|
896
|
+
"finding_type": "cloud-function-public-invoker",
|
|
897
|
+
"provider": "gcp",
|
|
898
|
+
"context_required": ["app_check:ENFORCED"],
|
|
899
|
+
"action": "SUPPRESS",
|
|
900
|
+
"reason": "App Check enforced — only verified app instances can invoke"
|
|
901
|
+
},
|
|
902
|
+
{
|
|
903
|
+
"id": "gcp-cf-callable",
|
|
904
|
+
"finding_type": "cloud-function-public-invoker",
|
|
905
|
+
"provider": "gcp",
|
|
906
|
+
"context_required": ["function_type:onCall"],
|
|
907
|
+
"action": "SUPPRESS",
|
|
908
|
+
"reason": "onCall functions require Firebase Auth token from client SDK"
|
|
909
|
+
},
|
|
910
|
+
{
|
|
911
|
+
"id": "aws-s3-cloudfront-oai",
|
|
912
|
+
"finding_type": "s3-bucket-public-access",
|
|
913
|
+
"provider": "aws",
|
|
914
|
+
"context_required": ["cloudfront_oai:ACTIVE"],
|
|
915
|
+
"action": "SUPPRESS",
|
|
916
|
+
"reason": "Bucket accessed only via CloudFront Origin Access Identity"
|
|
917
|
+
},
|
|
918
|
+
{
|
|
919
|
+
"id": "azure-func-apigw",
|
|
920
|
+
"finding_type": "function-app-public",
|
|
921
|
+
"provider": "azure",
|
|
922
|
+
"context_required": ["application_gateway:ACTIVE"],
|
|
923
|
+
"action": "SUPPRESS",
|
|
924
|
+
"reason": "Function behind Application Gateway with WAF"
|
|
925
|
+
}
|
|
926
|
+
]
|
|
927
|
+
}
|
|
928
|
+
```
|
|
929
|
+
|
|
930
|
+
**Files:**
|
|
931
|
+
|
|
932
|
+
| File | Action | Description |
|
|
933
|
+
|------|--------|-------------|
|
|
934
|
+
| `app/skills/cloud-security-audit/scripts/false_positive_resolver.py` | CREATE | Context graph engine |
|
|
935
|
+
| `app/skills/cloud-security-audit/reference/resolution-rules.json` | CREATE | Configurable rules |
|
|
936
|
+
|
|
937
|
+
**Success Criteria:**
|
|
938
|
+
- [ ] Resolves GCP: App Check, Gateway, callable patterns
|
|
939
|
+
- [ ] Resolves AWS: CloudFront OAI, ALB, WAF patterns
|
|
940
|
+
- [ ] Resolves Azure: App Gateway, Front Door, VNET patterns
|
|
941
|
+
- [ ] JSON output with context chain
|
|
942
|
+
- [ ] User can add custom rules to `.cloud-security.json` `context` section
|
|
943
|
+
- [ ] Tests: >= 12 (4 per provider)
|
|
944
|
+
|
|
945
|
+
---
|
|
946
|
+
|
|
947
|
+
### Faza 3: AWS (Phase 2, tydzien 3-4)
|
|
948
|
+
|
|
949
|
+
#### 3.1 Skill: `aws-security-audit`
|
|
950
|
+
|
|
951
|
+
**Static mode (IaC parsing):**
|
|
952
|
+
|
|
953
|
+
| Check | Severity | Source | Description |
|
|
954
|
+
|-------|----------|--------|-------------|
|
|
955
|
+
| S3 bucket `"Effect": "Allow", "Principal": "*"` | HIGH | .tf / .json | Public bucket policy |
|
|
956
|
+
| S3 `BlockPublicAccess` all false | HIGH | .tf / .json | Public access not blocked |
|
|
957
|
+
| Lambda `resource-based policy` with `Principal: "*"` | HIGH | .tf / .json | Public Lambda |
|
|
958
|
+
| Security Group `0.0.0.0/0` ingress on non-80/443 | HIGH | .tf / .json | Open port to world |
|
|
959
|
+
| IAM policy with `Action: "*"` | HIGH | .tf / .json | God-mode IAM |
|
|
960
|
+
| IAM policy with `Resource: "*"` + sensitive actions | WARN | .tf / .json | Broad resource scope |
|
|
961
|
+
| Unencrypted RDS/DynamoDB | WARN | .tf / .json | Missing encryption at rest |
|
|
962
|
+
| CloudTrail disabled | HIGH | .tf / .json | No audit logging |
|
|
963
|
+
| Missing VPC Flow Logs | WARN | .tf / .json | No network monitoring |
|
|
964
|
+
|
|
965
|
+
**Live mode:**
|
|
966
|
+
|
|
967
|
+
| Check | CLI Command | Description |
|
|
968
|
+
|-------|-------------|-------------|
|
|
969
|
+
| S3 public buckets | `aws s3api get-public-access-block` | Per-bucket public access |
|
|
970
|
+
| Lambda public policies | `aws lambda get-policy` | Resource-based policies |
|
|
971
|
+
| Open Security Groups | `aws ec2 describe-security-groups` | Ingress from 0.0.0.0/0 |
|
|
972
|
+
| Overly permissive IAM | `aws iam list-roles` + `get-role` | Roles with admin/broad access |
|
|
973
|
+
| Unused IAM credentials | `aws iam generate-credential-report` | Stale access keys |
|
|
974
|
+
| API Gateway without auth | `aws apigateway get-rest-apis` | Endpoints without authorizer |
|
|
975
|
+
|
|
976
|
+
**Script:** `scripts/aws_security_scan.py`
|
|
977
|
+
|
|
978
|
+
**Files:**
|
|
979
|
+
|
|
980
|
+
| File | Action | Description |
|
|
981
|
+
|------|--------|-------------|
|
|
982
|
+
| `app/skills/aws-security-audit/SKILL.md` | CREATE | Skill definition |
|
|
983
|
+
| `app/skills/aws-security-audit/scripts/aws_security_scan.py` | CREATE | Scanner |
|
|
984
|
+
| `app/skills/aws-security-audit/scripts/aws_auth.py` | CREATE | Credential loader |
|
|
985
|
+
| `app/skills/aws-security-audit/reference/aws-security-checklist.md` | CREATE | CIS Benchmark mapping |
|
|
986
|
+
| `app/skills/aws-security-audit/reference/false-positives-aws.md` | CREATE | ALB/CloudFront/WAF patterns |
|
|
987
|
+
|
|
988
|
+
**Terraform approach (council revision):** Do NOT parse HCL directly (stdlib-only Python cannot handle heredocs, variable interpolation, modules, `for_each`). Instead wrap `terraform show -json` / `terraform plan -json` which outputs clean JSON. Fallback to flat-resource regex for projects without `terraform` CLI.
|
|
989
|
+
|
|
990
|
+
**Success Criteria:**
|
|
991
|
+
- [ ] Static: `terraform show -json` wrapper + CloudFormation/SAM JSON parsing
|
|
992
|
+
- [ ] Live: checks S3, Lambda, IAM, SG via `aws` CLI
|
|
993
|
+
- [ ] False positive resolution for CloudFront/ALB/WAF
|
|
994
|
+
- [ ] CIS Benchmark mapping in reference
|
|
995
|
+
- [ ] Tests: >= 10
|
|
996
|
+
|
|
997
|
+
---
|
|
998
|
+
|
|
999
|
+
### Faza 4: Azure (Phase 3, tydzien 5-6)
|
|
1000
|
+
|
|
1001
|
+
#### 4.1 Skill: `azure-security-audit`
|
|
1002
|
+
|
|
1003
|
+
**Static mode (IaC parsing):**
|
|
1004
|
+
|
|
1005
|
+
| Check | Severity | Source | Description |
|
|
1006
|
+
|-------|----------|--------|-------------|
|
|
1007
|
+
| NSG rule `0.0.0.0/0` source on management ports | HIGH | .tf / .bicep | Open RDP/SSH to world |
|
|
1008
|
+
| Function App `authLevel: "anonymous"` | WARN | source / .tf | Public Azure Function |
|
|
1009
|
+
| Cosmos DB `publicNetworkAccess: enabled` | WARN | .tf / .bicep | Public database access |
|
|
1010
|
+
| Storage Account `allowBlobPublicAccess: true` | HIGH | .tf / .bicep | Public blob storage |
|
|
1011
|
+
| Missing Key Vault references (hardcoded secrets) | HIGH | source | Secrets not in Key Vault |
|
|
1012
|
+
| Missing RBAC (classic co-admin model) | WARN | .tf | Legacy access model |
|
|
1013
|
+
|
|
1014
|
+
**Live mode:**
|
|
1015
|
+
|
|
1016
|
+
| Check | CLI Command | Description |
|
|
1017
|
+
|-------|-------------|-------------|
|
|
1018
|
+
| Open NSG rules | `az network nsg rule list` | Broad inbound rules |
|
|
1019
|
+
| Function App auth | `az functionapp show` + `config` | Auth level and provider |
|
|
1020
|
+
| Cosmos DB access | `az cosmosdb show` | Network access settings |
|
|
1021
|
+
| RBAC assignments | `az role assignment list` | Owner/Contributor sprawl |
|
|
1022
|
+
| Storage public access | `az storage account show` | Blob public access |
|
|
1023
|
+
| App Service auth | `az webapp auth show` | Auth configuration |
|
|
1024
|
+
|
|
1025
|
+
**Script:** `scripts/azure_security_scan.py`
|
|
1026
|
+
|
|
1027
|
+
**Files:**
|
|
1028
|
+
|
|
1029
|
+
| File | Action | Description |
|
|
1030
|
+
|------|--------|-------------|
|
|
1031
|
+
| `app/skills/azure-security-audit/SKILL.md` | CREATE | Skill definition |
|
|
1032
|
+
| `app/skills/azure-security-audit/scripts/azure_security_scan.py` | CREATE | Scanner |
|
|
1033
|
+
| `app/skills/azure-security-audit/scripts/azure_auth.py` | CREATE | Credential loader |
|
|
1034
|
+
| `app/skills/azure-security-audit/reference/azure-security-checklist.md` | CREATE | CIS Benchmark mapping |
|
|
1035
|
+
| `app/skills/azure-security-audit/reference/false-positives-azure.md` | CREATE | App Gateway/Front Door patterns |
|
|
1036
|
+
|
|
1037
|
+
**Success Criteria:**
|
|
1038
|
+
- [ ] Static: parses Terraform, Bicep, ARM templates
|
|
1039
|
+
- [ ] Live: checks NSG, Functions, CosmosDB, RBAC via `az` CLI
|
|
1040
|
+
- [ ] False positive resolution for App Gateway/Front Door/VNET
|
|
1041
|
+
- [ ] Tests: >= 10
|
|
1042
|
+
|
|
1043
|
+
---
|
|
1044
|
+
|
|
1045
|
+
### Faza 3 (cont.): Orchestration + Pack Integration (Phase 2, tydzien 3-4)
|
|
1046
|
+
|
|
1047
|
+
#### 3.2 Orchestrator Skill: `/cloud-security-audit`
|
|
1048
|
+
|
|
1049
|
+
**Purpose:** Single entry point that runs all provider audits detected in the project.
|
|
1050
|
+
|
|
1051
|
+
**Behavior:**
|
|
1052
|
+
1. Auto-detect providers from project files:
|
|
1053
|
+
- `firebase.json` / `firestore.rules` / `.firebaserc` → GCP
|
|
1054
|
+
- `serverless.yml` / `template.yaml` / `*.tf` with `provider "aws"` → AWS
|
|
1055
|
+
- `*.bicep` / `*.tf` with `provider "azurerm"` / `azure-pipelines.yml` → Azure
|
|
1056
|
+
2. Check available credentials: `ai-toolkit credentials list`
|
|
1057
|
+
3. Run detected provider scans in parallel
|
|
1058
|
+
4. Merge results through false positive resolver
|
|
1059
|
+
5. Output unified report
|
|
1060
|
+
|
|
1061
|
+
**Skill frontmatter:**
|
|
1062
|
+
```yaml
|
|
1063
|
+
---
|
|
1064
|
+
name: cloud-security-audit
|
|
1065
|
+
description: "Multi-cloud security audit — auto-detects GCP/AWS/Azure and runs
|
|
1066
|
+
deterministic scans with false positive resolution"
|
|
1067
|
+
user-invocable: true
|
|
1068
|
+
effort: high
|
|
1069
|
+
disable-model-invocation: true
|
|
1070
|
+
context: fork
|
|
1071
|
+
agent: cloud-security-auditor
|
|
1072
|
+
argument-hint: "[path] [--provider gcp|aws|azure|auto] [--mode static|live] [--severity high|warn] [--output json]"
|
|
1073
|
+
allowed-tools: Read, Grep, Glob, Bash
|
|
1074
|
+
---
|
|
1075
|
+
```
|
|
1076
|
+
|
|
1077
|
+
**CLI usage:**
|
|
1078
|
+
```bash
|
|
1079
|
+
/cloud-security-audit # auto-detect providers, static mode (default)
|
|
1080
|
+
/cloud-security-audit --provider gcp # GCP only
|
|
1081
|
+
/cloud-security-audit --provider aws,azure # AWS + Azure
|
|
1082
|
+
/cloud-security-audit --mode static # no credentials, IaC/source only (DEFAULT)
|
|
1083
|
+
/cloud-security-audit --mode live # deployed state (requires credentials)
|
|
1084
|
+
/cloud-security-audit --severity high # HIGH findings only
|
|
1085
|
+
/cloud-security-audit --output json # CI pipeline output
|
|
1086
|
+
/cloud-security-audit --output sarif # SARIF v2.1.0 for GitHub Advanced Security
|
|
1087
|
+
/cloud-security-audit --changed main # incremental: only files changed vs main
|
|
1088
|
+
/cloud-security-audit --explain GCP-CF-001 # detailed remediation for specific rule
|
|
1089
|
+
/cloud-security-audit src/functions/ # scan specific path
|
|
1090
|
+
```
|
|
1091
|
+
|
|
1092
|
+
**Unified report format:**
|
|
1093
|
+
```markdown
|
|
1094
|
+
## Cloud Security Audit Report
|
|
1095
|
+
|
|
1096
|
+
### Summary
|
|
1097
|
+
| Metric | GCP | AWS | Azure | Total |
|
|
1098
|
+
|--------|-----|-----|-------|-------|
|
|
1099
|
+
| Mode | live | static | n/a | — |
|
|
1100
|
+
| Resources scanned | 12 | 8 | 0 | 20 |
|
|
1101
|
+
| HIGH | 2 | 1 | 0 | 3 |
|
|
1102
|
+
| WARN | 4 | 3 | 0 | 7 |
|
|
1103
|
+
| SUPPRESSED (false positive) | 3 | 1 | 0 | 4 |
|
|
1104
|
+
|
|
1105
|
+
### Findings (sorted by severity)
|
|
1106
|
+
|
|
1107
|
+
#### [HIGH] GCP: Cloud Function "adminEndpoint" — public invoker, no protection
|
|
1108
|
+
...
|
|
1109
|
+
|
|
1110
|
+
#### [SUPPRESSED] GCP: Cloud Function "processPayment" — public invoker
|
|
1111
|
+
Context: App Check ENFORCED + onCall + API Gateway routed (3/3 layers)
|
|
1112
|
+
...
|
|
1113
|
+
```
|
|
1114
|
+
|
|
1115
|
+
---
|
|
1116
|
+
|
|
1117
|
+
#### 3.2.1 Plugin Pack Manifest
|
|
1118
|
+
|
|
1119
|
+
**File:** `app/plugins/cloud-security-pack/plugin.json`
|
|
1120
|
+
|
|
1121
|
+
```json
|
|
1122
|
+
{
|
|
1123
|
+
"name": "cloud-security-pack",
|
|
1124
|
+
"description": "Multi-cloud security auditing for GCP, AWS, and Azure",
|
|
1125
|
+
"version": "1.0.0",
|
|
1126
|
+
"domain": "cloud-security",
|
|
1127
|
+
"type": "plugin-pack",
|
|
1128
|
+
"status": "experimental",
|
|
1129
|
+
"requires": [],
|
|
1130
|
+
"includes": {
|
|
1131
|
+
"agents": ["cloud-security-auditor"],
|
|
1132
|
+
"skills": [
|
|
1133
|
+
"cloud-security-audit",
|
|
1134
|
+
"firebase-rules-audit",
|
|
1135
|
+
"cloud-functions-audit",
|
|
1136
|
+
"aws-security-audit",
|
|
1137
|
+
"azure-security-audit"
|
|
1138
|
+
],
|
|
1139
|
+
"rules": [],
|
|
1140
|
+
"hooks": []
|
|
1141
|
+
},
|
|
1142
|
+
"credentials": {
|
|
1143
|
+
"supported_providers": ["gcp", "aws", "azure"],
|
|
1144
|
+
"setup_command": "ai-toolkit credentials add <provider>"
|
|
1145
|
+
}
|
|
1146
|
+
}
|
|
1147
|
+
```
|
|
1148
|
+
|
|
1149
|
+
**Directory structure:**
|
|
1150
|
+
```
|
|
1151
|
+
app/plugins/cloud-security-pack/
|
|
1152
|
+
├── plugin.json
|
|
1153
|
+
├── README.md
|
|
1154
|
+
└── (skills and agent live in core app/ dirs, referenced by name)
|
|
1155
|
+
```
|
|
1156
|
+
|
|
1157
|
+
---
|
|
1158
|
+
|
|
1159
|
+
### Faza 5: Tests + Documentation (ongoing, ships with each milestone)
|
|
1160
|
+
|
|
1161
|
+
#### 5.1 Tests
|
|
1162
|
+
|
|
1163
|
+
| Test file | Count | Description |
|
|
1164
|
+
|-----------|-------|-------------|
|
|
1165
|
+
| `tests/test_credentials.bats` | 8+ | CLI credentials management |
|
|
1166
|
+
| `tests/test_firebase_rules_scan.py` | 10+ | Firestore/RTDB rules patterns |
|
|
1167
|
+
| `tests/test_cloud_functions_audit.py` | 8+ | CF static + live checks |
|
|
1168
|
+
| `tests/test_aws_security_scan.py` | 10+ | S3/Lambda/IAM/SG checks |
|
|
1169
|
+
| `tests/test_azure_security_scan.py` | 10+ | NSG/Functions/CosmosDB checks |
|
|
1170
|
+
| `tests/test_false_positive_resolver.py` | 12+ | Context graph resolution |
|
|
1171
|
+
| `tests/test_sarif_formatter.py` | 4+ | SARIF v2.1.0 output validation |
|
|
1172
|
+
| `tests/test_incremental.py` | 4+ | Git diff filtering, fallback |
|
|
1173
|
+
| `tests/test_credentials_init.py` | 4+ | Interactive wizard, auto-detect |
|
|
1174
|
+
| **Total** | **70+** | |
|
|
1175
|
+
|
|
1176
|
+
#### 5.2 Documentation
|
|
1177
|
+
|
|
1178
|
+
| File | Description |
|
|
1179
|
+
|------|-------------|
|
|
1180
|
+
| `kb/planning/cloud-security-pack-plan.md` | This document |
|
|
1181
|
+
| `kb/reference/cloud-security-checklist.md` | Unified multi-cloud checklist (created when Milestone 1 ships) |
|
|
1182
|
+
| Skills `reference/` dirs | Per-provider patterns and false positives |
|
|
1183
|
+
|
|
1184
|
+
---
|
|
1185
|
+
|
|
1186
|
+
## 6. Configuration & Suppression (Unified)
|
|
1187
|
+
|
|
1188
|
+
**Council revision:** merged `.cloud-security-ignore` + `.cloud-security-config` into a single `.cloud-security.json`. Two config files is one too many — developers expect one file.
|
|
1189
|
+
|
|
1190
|
+
Scaffold interactively: `ai-toolkit credentials init` generates this file.
|
|
1191
|
+
|
|
1192
|
+
**Suppression Governance (orchestration-review P0):**
|
|
1193
|
+
- **Wildcard ignores** (e.g., `GCP-CF-*`) REQUIRE a `justification` field — scanner refuses to suppress without one
|
|
1194
|
+
- **All ignore entries** require `justification` — enforced by schema validation
|
|
1195
|
+
- **CI diff detection:** When `.cloud-security.json` is modified in a PR, scanner emits a `SUPPRESSION_CHANGED` finding (severity: WARN) listing added/removed/modified ignore rules. This prevents silent suppression of vulnerabilities via committed config changes.
|
|
1196
|
+
- **In live mode:** context claims (e.g., `app_check_enforced: true`) are verified against actual cloud state. If claim doesn't match reality, emit `CONTEXT_MISMATCH` finding (severity: HIGH)
|
|
1197
|
+
|
|
1198
|
+
```json
|
|
1199
|
+
{
|
|
1200
|
+
"$schema": "https://softspark.github.io/ai-toolkit/schemas/cloud-security.json",
|
|
1201
|
+
|
|
1202
|
+
"ignore": [
|
|
1203
|
+
{ "rule": "GCP-CF-001:processPayment", "justification": "Behind App Check + API Gateway, verified 2026-04-10" },
|
|
1204
|
+
{ "rule": "AWS-S3-001:static-assets", "justification": "Intentionally public static bucket, CloudFront OAI active" },
|
|
1205
|
+
{ "rule": "GCP-CF-*", "justification": "REQUIRED for wildcard suppression — reviewed by @jacek in PR #142" }
|
|
1206
|
+
],
|
|
1207
|
+
|
|
1208
|
+
"context": {
|
|
1209
|
+
"gcp": {
|
|
1210
|
+
"app_check_enforced": true,
|
|
1211
|
+
"api_gateway": "projects/my-proj/locations/us-central1/gateways/main",
|
|
1212
|
+
"known_public_functions": ["healthCheck", "webhookReceiver"]
|
|
1213
|
+
},
|
|
1214
|
+
"aws": {
|
|
1215
|
+
"waf_enabled": true,
|
|
1216
|
+
"cloudfront_distributions": ["E1234567890"],
|
|
1217
|
+
"known_public_buckets": ["static-assets-prod"]
|
|
1218
|
+
},
|
|
1219
|
+
"azure": {
|
|
1220
|
+
"front_door_enabled": true,
|
|
1221
|
+
"application_gateway": "my-app-gw",
|
|
1222
|
+
"known_public_functions": ["webhookHandler"]
|
|
1223
|
+
}
|
|
1224
|
+
}
|
|
1225
|
+
}
|
|
1226
|
+
```
|
|
1227
|
+
|
|
1228
|
+
---
|
|
1229
|
+
|
|
1230
|
+
## 7. CI Pipeline Integration
|
|
1231
|
+
|
|
1232
|
+
### Basic: JSON output + fail on HIGH
|
|
1233
|
+
|
|
1234
|
+
```yaml
|
|
1235
|
+
- name: Cloud Security Audit
|
|
1236
|
+
run: |
|
|
1237
|
+
python3 scripts/cloud_security_audit.py \
|
|
1238
|
+
--mode static --output json --severity high \
|
|
1239
|
+
> security-report.json
|
|
1240
|
+
# Exit code 1 = HIGH findings → fail pipeline
|
|
1241
|
+
```
|
|
1242
|
+
|
|
1243
|
+
### Recommended: SARIF + GitHub Advanced Security (inline PR annotations)
|
|
1244
|
+
|
|
1245
|
+
```yaml
|
|
1246
|
+
- name: Cloud Security Audit
|
|
1247
|
+
run: |
|
|
1248
|
+
python3 scripts/cloud_security_audit.py \
|
|
1249
|
+
--mode static --output sarif --changed ${{ github.event.pull_request.base.sha }} \
|
|
1250
|
+
> results.sarif
|
|
1251
|
+
continue-on-error: true
|
|
1252
|
+
|
|
1253
|
+
- name: Upload SARIF
|
|
1254
|
+
uses: github/codeql-action/upload-sarif@v3
|
|
1255
|
+
with:
|
|
1256
|
+
sarif_file: results.sarif
|
|
1257
|
+
```
|
|
1258
|
+
|
|
1259
|
+
### Live mode with credentials (hardened — orchestration-review P0)
|
|
1260
|
+
|
|
1261
|
+
```yaml
|
|
1262
|
+
- name: Cloud Security Audit (Live)
|
|
1263
|
+
env:
|
|
1264
|
+
GCP_SA_KEY: ${{ secrets.GCP_SA_KEY }}
|
|
1265
|
+
run: |
|
|
1266
|
+
TMPFILE=$(mktemp -m 0600)
|
|
1267
|
+
trap 'rm -f "$TMPFILE"' EXIT
|
|
1268
|
+
echo "$GCP_SA_KEY" > "$TMPFILE"
|
|
1269
|
+
ai-toolkit credentials add gcp --file "$TMPFILE"
|
|
1270
|
+
python3 scripts/cloud_security_audit.py \
|
|
1271
|
+
--mode live --output sarif --changed "${{ github.event.pull_request.base.sha }}" \
|
|
1272
|
+
> results.sarif
|
|
1273
|
+
|
|
1274
|
+
- name: Validate & Upload SARIF
|
|
1275
|
+
if: always()
|
|
1276
|
+
run: python3 -c "import json; d=json.load(open('results.sarif')); assert d.get('version')=='2.1.0', 'Invalid SARIF'"
|
|
1277
|
+
continue-on-error: false
|
|
1278
|
+
|
|
1279
|
+
- name: Upload SARIF
|
|
1280
|
+
uses: github/codeql-action/upload-sarif@v3
|
|
1281
|
+
with:
|
|
1282
|
+
sarif_file: results.sarif
|
|
1283
|
+
```
|
|
1284
|
+
|
|
1285
|
+
**CI Security Notes (orchestration-review):**
|
|
1286
|
+
- `mktemp -m 0600` creates file with owner-only permissions (not world-readable `/tmp/sa.json`)
|
|
1287
|
+
- `trap 'rm -f' EXIT` ensures cleanup even on script failure
|
|
1288
|
+
- SARIF schema validated before upload to prevent injected/corrupted annotations
|
|
1289
|
+
- `${{ github.event.pull_request.base.sha }}` quoted to prevent shell injection via crafted refs
|
|
1290
|
+
- Consider GitHub OIDC workload identity federation instead of long-lived SA keys for production
|
|
1291
|
+
|
|
1292
|
+
---
|
|
1293
|
+
|
|
1294
|
+
## 8. Success Criteria (Overall)
|
|
1295
|
+
|
|
1296
|
+
| Metric | Target |
|
|
1297
|
+
|--------|--------|
|
|
1298
|
+
| Providers supported | 3 (GCP, AWS, Azure) |
|
|
1299
|
+
| Check patterns (total) | 40+ (GCP: 16, AWS: 15, Azure: 12) |
|
|
1300
|
+
| False positive rules | 10+ |
|
|
1301
|
+
| Output formats | 3 (text, JSON, SARIF v2.1.0) |
|
|
1302
|
+
| CLI commands | 7 (add, list, remove, test, init per provider) |
|
|
1303
|
+
| Scripts (stdlib Python) | 9 (auth + scanners + resolver + sarif + incremental) |
|
|
1304
|
+
| Tests | 70+ |
|
|
1305
|
+
| External dependencies | 0 (stdlib only, CLI tools: gcloud/aws/az/terraform) |
|
|
1306
|
+
| CI exit codes | 0=clean, 1=HIGH findings, 2=credential error |
|
|
1307
|
+
| Incremental scan | `--changed` flag works with git refs |
|
|
1308
|
+
| GitHub integration | SARIF upload → inline PR annotations |
|
|
1309
|
+
|
|
1310
|
+
---
|
|
1311
|
+
|
|
1312
|
+
## 9. Fix Strategy
|
|
1313
|
+
|
|
1314
|
+
**Same approach as HIPAA scanner v1: No auto-fix. Agent interprets and suggests.**
|
|
1315
|
+
|
|
1316
|
+
The deterministic scripts produce findings. The `cloud-security-auditor` agent then:
|
|
1317
|
+
1. **Reads** the flagged file/resource to understand actual context
|
|
1318
|
+
2. **Suggests** a specific fix (not generic advice — concrete code/config change)
|
|
1319
|
+
3. **Never** auto-applies changes — the user reviews and applies manually
|
|
1320
|
+
|
|
1321
|
+
Examples:
|
|
1322
|
+
- Firestore rules finding → agent suggests the exact `allow read: if request.auth != null` rule change
|
|
1323
|
+
- Public CF finding → agent suggests adding `validateFirebaseIdToken` middleware with code snippet
|
|
1324
|
+
- Open S3 bucket → agent suggests the exact bucket policy JSON to add `BlockPublicAccess`
|
|
1325
|
+
- Broad IAM role → agent suggests the minimal policy document with only required permissions
|
|
1326
|
+
|
|
1327
|
+
**Why no auto-fix in v1:**
|
|
1328
|
+
- Cloud security fixes require project-specific knowledge (which SA, which bucket, which auth flow)
|
|
1329
|
+
- Wrong auto-fix on IAM can lock out real users
|
|
1330
|
+
- Firestore rules changes can break client apps
|
|
1331
|
+
- The agent's context-aware suggestion is more valuable than a blind auto-fix
|
|
1332
|
+
|
|
1333
|
+
**v2 option:** `--fix-mode suggest` generates a `.cloud-security-fixes.patch` file that users can review and apply with `git apply`.
|
|
1334
|
+
|
|
1335
|
+
---
|
|
1336
|
+
|
|
1337
|
+
## 10. Risks and Mitigation (updated)
|
|
1338
|
+
|
|
1339
|
+
| Risk | Probability | Impact | Mitigation |
|
|
1340
|
+
|------|-------------|--------|------------|
|
|
1341
|
+
| CLI tools (gcloud/aws/az) not installed | Medium | Medium | Graceful fallback to static-only mode, clear error message |
|
|
1342
|
+
| Cloud APIs change breaking audit commands | Low | Medium | Version-pin CLI output format parsing, test with CI |
|
|
1343
|
+
| False positive rules too aggressive (suppress real issues) | Low | High | Default to WARN not SUPPRESS, require `.cloud-security-config` for suppression |
|
|
1344
|
+
| Credential leakage in logs | Low | Critical | Never log credentials, 0600 perms, `/tmp` cleanup in CI |
|
|
1345
|
+
| Scope creep (too many checks) | Medium | Medium | Start with top-10 per provider, expand based on feedback |
|
|
1346
|
+
| YAML IaC not parseable (stdlib-only) | High | Medium | JSON-only for static IaC; live mode unaffected; v2: vendored YAML subset parser |
|
|
1347
|
+
| `--changed` confused with live mode | Low | Low | Explicit warning: incremental applies to static only |
|
|
1348
|
+
|
|
1349
|
+
---
|
|
1350
|
+
|
|
1351
|
+
## 11. Pre-Mortem
|
|
1352
|
+
|
|
1353
|
+
1. **Firebase rules parser too simplistic** — Firestore rules use CEL-like syntax with nested `match` blocks. Parser needs proper recursive descent, not just regex. Mitigation: build minimal CEL parser or use line-by-line pattern matching with scope tracking.
|
|
1354
|
+
2. **False positive resolver gives false confidence** — Users may trust SUPPRESSED status and miss real issues. Mitigation: always show context chain, require explicit `.cloud-security-config` for auto-suppression, default to WARN.
|
|
1355
|
+
3. **AWS credential scope too broad** — User provides admin-level AWS profile. Mitigation: `credentials test` checks actual permissions, WARN if write access detected, suggest read-only IAM policy in docs.
|
|
1356
|
+
4. **Three providers = 3x maintenance** — Each provider's CLI evolves independently. Mitigation: abstract provider interface, single test matrix, version tracking per provider.
|
|
1357
|
+
5. **Terraform parsing incomplete** — HCL syntax is complex (modules, variables, conditionals). Mitigation: wrap `terraform show -json` instead of parsing HCL. Fallback to flat regex for projects without `terraform` CLI.
|
|
1358
|
+
6. **SARIF adoption low** — Users may not know how to use SARIF with GitHub. Mitigation: provide copy-paste GitHub Actions workflow in docs and `--explain` for onboarding.
|
|
1359
|
+
|
|
1360
|
+
---
|
|
1361
|
+
|
|
1362
|
+
## 12. Council Review Summary (2026-04-10)
|
|
1363
|
+
|
|
1364
|
+
**Verdict:** CONDITIONAL FOR — implement with scope reduction.
|
|
1365
|
+
**Confidence:** MEDIUM (weighted score: FOR 3.1 vs AGAINST 2.9)
|
|
1366
|
+
|
|
1367
|
+
**Key insights applied to this plan:**
|
|
1368
|
+
- [x] Timeline revised from 3-4 → 5-6 weeks
|
|
1369
|
+
- [x] ~~Azure deferred to Milestone 3~~ → **reinstated: full 3-provider delivery**
|
|
1370
|
+
- [x] SARIF v2.1.0 output added — essential for GitHub Advanced Security integration
|
|
1371
|
+
- [x] Incremental scan mode added (`--changed`) — how developers actually use security tools
|
|
1372
|
+
- [x] `terraform show -json` wrapper instead of HCL parsing — realistic path
|
|
1373
|
+
- [x] Single config file `.cloud-security.json` (merged ignore + context)
|
|
1374
|
+
- [x] `credentials init` interactive wizard — reduce onboarding friction
|
|
1375
|
+
- [x] `--explain <rule-id>` for on-demand remediation guidance
|
|
1376
|
+
- [x] Static mode as default — zero-setup first experience
|
|
1377
|
+
|
|
1378
|
+
**Deferred to v2:**
|
|
1379
|
+
- Kubernetes/container security (separate pack candidate)
|
|
1380
|
+
- Secret scanning with entropy detection
|
|
1381
|
+
- Compliance framework mapping (SOC2, PCI-DSS, NIST 800-53)
|
|
1382
|
+
- Visual security dashboard in browser
|
|
1383
|
+
- GitHub PR comment integration beyond SARIF
|
|
1384
|
+
- Vendored YAML subset parser for CloudFormation YAML static scanning
|
|
1385
|
+
|
|
1386
|
+
**Council strongest agreement:** False positive resolver is the killer feature and primary differentiator vs Checkov/Trivy/Prowler. No existing tool combines deterministic scanning with context-aware resolution.
|
|
1387
|
+
|
|
1388
|
+
---
|
|
1389
|
+
|
|
1390
|
+
## 13. Orchestration Review Summary (2026-04-10)
|
|
1391
|
+
|
|
1392
|
+
**Agents:** tech-lead, security-architect, product-manager, code-reviewer (4 parallel)
|
|
1393
|
+
|
|
1394
|
+
**Verdict:** Plan structurally complete (14/14 elements). Three P0 security blockers identified and resolved.
|
|
1395
|
+
|
|
1396
|
+
**Applied changes:**
|
|
1397
|
+
|
|
1398
|
+
| # | Action | Source | Priority | Applied? |
|
|
1399
|
+
|---|--------|--------|----------|----------|
|
|
1400
|
+
| 1 | Programmatic Bash allowlist | security-architect | P0 | Yes — allowlist.sh + agent integration |
|
|
1401
|
+
| 2 | CI hardening (mktemp/trap/SARIF validation) | security-architect | P0 | Yes — CI examples rewritten |
|
|
1402
|
+
| 3 | Suppression governance (justification + diff detection) | security-architect | P0 | Yes — schema + SUPPRESSION_CHANGED finding |
|
|
1403
|
+
| 4 | Recursive descent parser for Firestore | code-reviewer | P1 | Yes — replaced regex approach, +1d estimate |
|
|
1404
|
+
| 5 | SARIF `driver.rules[]` for GitHub annotations | code-reviewer | P1 | Yes — schema + success criteria |
|
|
1405
|
+
| 6 | Task 4.1→3.2 numbering fix | tech-lead | P1 | Yes — renumbered |
|
|
1406
|
+
| 7 | `terraform plan` execution risk documented | security-architect | P1 | Yes — only `show -json` allowed |
|
|
1407
|
+
| 8 | `credentials init` deferred to M2 | code-reviewer | P2 | Yes — saves 1.5d in M1 |
|
|
1408
|
+
| 9 | False positive resolver budgeted +2d/provider | code-reviewer | P2 | Yes — estimate updated |
|
|
1409
|
+
| 10 | `SUPPRESSION_CHANGED` finding type | security-architect | P2 | Yes — in governance section |
|
|
1410
|
+
|
|
1411
|
+
**Market positioning (product-manager):**
|
|
1412
|
+
- Not competing with Checkov on check count (40 vs 3000)
|
|
1413
|
+
- Competing on: zero-noise (false positive resolver), zero-setup (static-first), IDE-native (10 platforms), AI interpretation
|
|
1414
|
+
- Target: developers in ai-toolkit ecosystem, not enterprise security teams
|
|
1415
|
+
- Value as ecosystem feature, not standalone product
|
|
1416
|
+
|
|
1417
|
+
**Timeline revision (code-reviewer):**
|
|
1418
|
+
- 1 person: 6-7 weeks realistic (was 5-6)
|
|
1419
|
+
- 2 people: 4-5 weeks (parallel GCP + AWS tracks)
|
|
1420
|
+
- All 3 providers ship in 6 weeks — no conditional gates
|
|
1421
|
+
|
|
1422
|
+
---
|
|
1423
|
+
|
|
1424
|
+
## 14. Next Actions
|
|
1425
|
+
|
|
1426
|
+
1. [ ] Approve plan
|
|
1427
|
+
2. [ ] Implement `credentials` CLI command (1.1) + Bash allowlist
|
|
1428
|
+
3. [ ] Create `cloud-security-auditor` agent (1.2)
|
|
1429
|
+
4. [ ] Implement SARIF formatter + incremental scan (1.3)
|
|
1430
|
+
5. [ ] Implement `firebase-rules-audit` — recursive descent parser (2.1)
|
|
1431
|
+
6. [ ] Implement `cloud-functions-audit` + false positive resolver GCP (2.2, 2.3)
|
|
1432
|
+
7. [ ] Implement `aws-security-audit` + `terraform show -json` wrapper (3.1)
|
|
1433
|
+
8. [ ] Implement orchestrator + plugin pack + `credentials init` (3.2, 1.1b)
|
|
1434
|
+
9. [ ] Implement `azure-security-audit` + false positive resolver Azure (4.1)
|
|
1435
|
+
10. [ ] Full test suite (70+) + documentation + release
|
|
1436
|
+
|
|
1437
|
+
---
|
|
1438
|
+
|
|
1439
|
+
**Last Updated:** 2026-04-10
|
|
1440
|
+
**Council Reviewed:** 2026-04-10
|
|
1441
|
+
**Orchestration Reviewed:** 2026-04-10 (4 agents: tech-lead, security-architect, product-manager, code-reviewer)
|
|
1442
|
+
|
|
1443
|
+
---
|
|
1444
|
+
|
|
1445
|
+
## kb/planning/enterprise-config-inheritance-plan.md
|
|
1446
|
+
|
|
1447
|
+
---
|
|
1448
|
+
title: "Plan: Enterprise Config Inheritance — Multi-Repo Governance with extends"
|
|
1449
|
+
category: planning
|
|
1450
|
+
service: ai-toolkit
|
|
1451
|
+
tags:
|
|
1452
|
+
- enterprise
|
|
1453
|
+
- multi-repo
|
|
1454
|
+
- config-inheritance
|
|
1455
|
+
- extends
|
|
1456
|
+
- governance
|
|
1457
|
+
- team-management
|
|
1458
|
+
- monorepo
|
|
1459
|
+
doc_type: plan
|
|
1460
|
+
status: proposed
|
|
1461
|
+
created: "2026-04-10"
|
|
1462
|
+
last_updated: "2026-04-10"
|
|
1463
|
+
completion: "0%"
|
|
1464
|
+
description: "Configuration inheritance system for ai-toolkit. Enables organizations to define a shared base config (agents, rules, hooks, profiles, constitution overrides) published as an npm package or local path, which individual projects extend via an `extends` field. Changes to the base config propagate automatically on `ai-toolkit update`. Targets enterprises managing 10-100+ repositories with uniform AI governance."
|
|
1465
|
+
---
|
|
1466
|
+
|
|
1467
|
+
# Plan: Enterprise Config Inheritance — Multi-Repo Governance with `extends`
|
|
1468
|
+
|
|
1469
|
+
**Status:** Proposed
|
|
1470
|
+
**Completion:** 0%
|
|
1471
|
+
**Created:** 2026-04-10
|
|
1472
|
+
**Origin:** Organizations adopting ai-toolkit across 10-100+ repositories face a config synchronization problem — updating a rule or policy requires touching every repository individually. The `extends` pattern (popularized by ESLint, TypeScript, Prettier) solves this by establishing a single source of truth that projects inherit from.
|
|
1473
|
+
**Estimated Effort:** 5-7 weeks (1 person) — MVP (core engine + install integration) shippable in ~3.5 weeks
|
|
1474
|
+
|
|
1475
|
+
---
|
|
1476
|
+
|
|
1477
|
+
## 1. Objective
|
|
1478
|
+
|
|
1479
|
+
Create a configuration inheritance system where projects can extend a shared base config published as an npm package, a Git URL, or a local path. The base config defines organizational defaults (which agents to enable, which rules to enforce, which hooks to require, persona presets, and constitution amendments). Individual projects can override or supplement the base, creating a layered governance model.
|
|
1480
|
+
|
|
1481
|
+
**Key design principles:**
|
|
1482
|
+
- **Familiar pattern** — mirrors ESLint's `extends`, TypeScript's `extends`, and Prettier's shared configs
|
|
1483
|
+
- **npm-first distribution** — base configs are regular npm packages (e.g., `@mycompany/ai-toolkit-config`). Resolver shells out to `npm pack` CLI (respects `.npmrc` auth) — no hand-rolled npm client, preserves stdlib-only constraint
|
|
1484
|
+
- **Single extends in v1** — `"extends": "string"` only. Multi-base merge (`"extends": [...]`) deferred to v2 to avoid merge-ordering complexity (ESLint's multi-extends is a known source of confusion)
|
|
1485
|
+
- **Layered merge** — base → project, with explicit override semantics (`override: true` required for safety-critical overrides)
|
|
1486
|
+
- **Constitution immutable** — base constitution articles cannot be modified by projects, period. Projects can only ADD new articles (article 6+). No weakening detection heuristics — absolute immutability is simpler and safer
|
|
1487
|
+
- **Offline-capable** — resolved at `install`/`update` time, not at runtime
|
|
1488
|
+
- **Backward-compatible** — projects without `extends` work exactly as today (no breaking changes)
|
|
1489
|
+
- **Audit trail** — `state.json` records which base config was resolved and what was overridden
|
|
1490
|
+
|
|
1491
|
+
---
|
|
1492
|
+
|
|
1493
|
+
## 1a. Functional Requirements
|
|
1494
|
+
|
|
1495
|
+
| ID | Requirement | Priority | Success Metric |
|
|
1496
|
+
|----|-------------|----------|----------------|
|
|
1497
|
+
| FR1 | Resolve `extends` from npm package, git URL, local path | Must | 4 source types work |
|
|
1498
|
+
| FR2 | Deep merge base → project config with layered semantics | Must | Merge engine handles dict, list, scalar types |
|
|
1499
|
+
| FR3 | Constitution immutability — Articles I-V cannot be modified | Must | 100% block rate on modification attempts |
|
|
1500
|
+
| FR4 | Override validation with `override: true` + `justification` | Must | Missing justification → error |
|
|
1501
|
+
| FR5 | `enforce` block constraints (minHookProfile, requiredPlugins, forbidOverride, requiredAgents) | Must | All 4 constraint types enforced |
|
|
1502
|
+
| FR6 | Install/update integration — resolve extends during install | Must | `install --local` detects `.ai-toolkit.json` |
|
|
1503
|
+
| FR7 | `config diff` command — show project vs base differences | Must | All merge layers visible |
|
|
1504
|
+
| FR8 | `config validate` command — schema + enforcement validation | Must | Exit 0/1 for pass/fail |
|
|
1505
|
+
| FR9 | `config init` — interactive project config setup | Should | Guided flow produces valid `.ai-toolkit.json` |
|
|
1506
|
+
| FR10 | `config create-base` — scaffold npm base config package | Should | Ready-to-publish package with `package.json` |
|
|
1507
|
+
| FR11 | Lock file for reproducible installs | Should | Identical resolved config across team members |
|
|
1508
|
+
| FR12 | Audit trail in `state.json` | Should | Resolved version + overrides recorded |
|
|
1509
|
+
| FR13 | CI enforcement command (`config check`) | Could | Exit 0/1 for governance compliance |
|
|
1510
|
+
| FR14 | Multi-base extends (`"extends": [...]`) | Won't (v2) | Deferred — merge ordering complexity |
|
|
1511
|
+
|
|
1512
|
+
---
|
|
1513
|
+
|
|
1514
|
+
## 2. Architecture Overview
|
|
1515
|
+
|
|
1516
|
+
```
|
|
1517
|
+
Organization Level (published once, consumed by all repos):
|
|
1518
|
+
═══════════════════════════════════════════════════════════
|
|
1519
|
+
|
|
1520
|
+
@mycompany/ai-toolkit-config (npm package)
|
|
1521
|
+
├── ai-toolkit.config.json ← base configuration
|
|
1522
|
+
├── rules/
|
|
1523
|
+
│ ├── code-review-policy.md ← company-specific rules
|
|
1524
|
+
│ └── deployment-checklist.md
|
|
1525
|
+
├── agents/
|
|
1526
|
+
│ └── compliance-auditor.md ← company-specific agent
|
|
1527
|
+
└── package.json
|
|
1528
|
+
|
|
1529
|
+
Project Level (per-repository):
|
|
1530
|
+
══════════════════════════════
|
|
1531
|
+
|
|
1532
|
+
my-service/
|
|
1533
|
+
├── .ai-toolkit.json ← project config with "extends"
|
|
1534
|
+
├── .claude/
|
|
1535
|
+
│ ├── CLAUDE.md ← generated (base + project merged)
|
|
1536
|
+
│ └── settings.json ← generated (base hooks + project hooks merged)
|
|
1537
|
+
└── ...
|
|
1538
|
+
|
|
1539
|
+
Merge Pipeline:
|
|
1540
|
+
══════════════
|
|
1541
|
+
|
|
1542
|
+
@mycompany/ai-toolkit-config ← Layer 0: organizational defaults
|
|
1543
|
+
│
|
|
1544
|
+
▼
|
|
1545
|
+
ai-toolkit defaults (manifest.json) ← Layer 1: toolkit defaults
|
|
1546
|
+
│
|
|
1547
|
+
▼
|
|
1548
|
+
.ai-toolkit.json ← Layer 2: project overrides
|
|
1549
|
+
│
|
|
1550
|
+
▼
|
|
1551
|
+
Resolved Configuration ← Final: CLAUDE.md, settings.json, etc.
|
|
1552
|
+
```
|
|
1553
|
+
|
|
1554
|
+
### Config Resolution Order
|
|
1555
|
+
|
|
1556
|
+
```
|
|
1557
|
+
1. Load base config from "extends" (npm package, git URL, or local path)
|
|
1558
|
+
2. Merge with ai-toolkit defaults (manifest.json profiles)
|
|
1559
|
+
3. Apply project-level overrides from .ai-toolkit.json
|
|
1560
|
+
4. Validate merged config (constitution immutability, schema validation)
|
|
1561
|
+
5. Generate output files (CLAUDE.md, settings.json, agent symlinks, etc.)
|
|
1562
|
+
```
|
|
1563
|
+
|
|
1564
|
+
---
|
|
1565
|
+
|
|
1566
|
+
## 3. Progress Tracking
|
|
1567
|
+
|
|
1568
|
+
| # | Feature | Priority | Status | Est. Time | Notes |
|
|
1569
|
+
|---|---------|----------|--------|-----------|-------|
|
|
1570
|
+
| 1.1 | `.ai-toolkit.json` schema definition | P0 | Proposed | 1d | JSON Schema with `extends` field |
|
|
1571
|
+
| 1.2 | Config resolver (npm, git, local path) | P0 | Proposed | 3d | Fetch + cache + validate base configs |
|
|
1572
|
+
| 1.3 | Merge engine (layered merge with override semantics) | P0 | Proposed | 3d | Deep merge with `override: true` gates |
|
|
1573
|
+
| 1.4 | Constitution immutability guard | P0 | Proposed | 1d | Block weakening of safety articles |
|
|
1574
|
+
| 2.1 | Install/update integration | P0 | Proposed | 2d | Resolve extends during install/update |
|
|
1575
|
+
| 2.2 | `ai-toolkit config diff` command | P0 | Proposed | 1.5d | Show project vs base differences — primary debugging tool |
|
|
1576
|
+
| 2.3 | `ai-toolkit config validate` command | P0 | Proposed | 1d | Validate .ai-toolkit.json schema + extends resolution |
|
|
1577
|
+
| 2.4 | `ai-toolkit config init` command | P1 | Proposed | 1.5d | Interactive project config setup |
|
|
1578
|
+
| 2.5 | `ai-toolkit config create-base` command | P1 | Proposed | 2d | Scaffold base config package |
|
|
1579
|
+
| 3.1 | Audit trail in state.json | P1 | Proposed | 1d | Record resolved config provenance |
|
|
1580
|
+
| 3.2 | Lock file (`.ai-toolkit.lock.json`) | P1 | Proposed | 1.5d | Pin resolved versions for reproducibility |
|
|
1581
|
+
| 3.3 | Base config scaffolder (npm package template) | P1 | Proposed | 1.5d | Ready-to-publish template |
|
|
1582
|
+
| 3.4 | CI enforcement (`ai-toolkit config check`) | P2 | Proposed | 1d | Verify project adheres to base + no unapproved overrides |
|
|
1583
|
+
| 4.1 | Tests | P1 | Proposed | 3d | Unit: resolution, merge, immutability, override, CLI commands. Integration: `install --local` with `.ai-toolkit.json` containing `extends`, verify resolved `CLAUDE.md` has base + project rules merged end-to-end |
|
|
1584
|
+
| 4.2 | Documentation | P1 | Proposed | 3d | Enterprise setup guide + all 9 docs per CLAUDE.md: README, CLAUDE.md, ARCHITECTURE.md, package.json, llms.txt, llms-full.txt, AGENTS.md, skills-catalog.md, architecture-overview.md |
|
|
1585
|
+
|
|
1586
|
+
**Phasing (MVP-first):**
|
|
1587
|
+
- **MVP Phase 1 (week 1-2):** Core engine — schema (1.1), resolver (1.2), merge engine (1.3), constitution guard (1.4)
|
|
1588
|
+
- **MVP Phase 2 (week 2-3):** Integration + diff — install integration (2.1), `config diff` (2.2), `config validate` (2.3), tests for above (~3.5 weeks = shippable MVP)
|
|
1589
|
+
- **Phase 3 (week 4-5):** CLI polish — `config init` (2.4), `config create-base` (2.5), scaffolder (3.3)
|
|
1590
|
+
- **Phase 4 (week 5-6):** Enterprise — audit trail (3.1), lock file (3.2), CI enforcement (3.4) (**gate behind real enterprise feedback**)
|
|
1591
|
+
- **Phase 5 (week 6-7):** Tests + documentation (4.1, 4.2) (3d docs — all 9 docs per CLAUDE.md rules)
|
|
1592
|
+
|
|
1593
|
+
> **Demand validation gate:** Ship MVP (Phases 1-2), announce, measure adoption. Only build Phase 4 (lock file, CI enforcement, audit trail) in response to confirmed enterprise demand.
|
|
1594
|
+
|
|
1595
|
+
---
|
|
1596
|
+
|
|
1597
|
+
## 4. Dependency Graph
|
|
1598
|
+
|
|
1599
|
+
```
|
|
1600
|
+
MVP Phase 1: Core Engine (week 1-2)
|
|
1601
|
+
====================================
|
|
1602
|
+
Schema definition (1.1) ──────┐
|
|
1603
|
+
├──► Merge engine (1.3)
|
|
1604
|
+
Config resolver (1.2) ────────┤
|
|
1605
|
+
└──► Constitution guard (1.4)
|
|
1606
|
+
|
|
1607
|
+
MVP Phase 2: Integration + Diff (week 2-3)
|
|
1608
|
+
============================================
|
|
1609
|
+
Install integration (2.1) ──┐
|
|
1610
|
+
├──► config diff (2.2)
|
|
1611
|
+
└──► config validate (2.3)
|
|
1612
|
+
└──► MVP tests → SHIP
|
|
1613
|
+
|
|
1614
|
+
═══ DEMAND VALIDATION GATE ═══
|
|
1615
|
+
|
|
1616
|
+
Phase 3: CLI Polish (week 4-5)
|
|
1617
|
+
===============================
|
|
1618
|
+
├──► config init (2.4)
|
|
1619
|
+
└──► create-base (2.5) + scaffolder (3.3)
|
|
1620
|
+
|
|
1621
|
+
Phase 4: Enterprise (week 5-6)
|
|
1622
|
+
================================
|
|
1623
|
+
Audit trail (3.1) ──┐
|
|
1624
|
+
├──► Lock file (3.2)
|
|
1625
|
+
└──► CI enforcement (3.4)
|
|
1626
|
+
|
|
1627
|
+
Phase 5: Polish (week 6-7)
|
|
1628
|
+
===========================
|
|
1629
|
+
└──► Full tests + docs (4.1, 4.2)
|
|
1630
|
+
```
|
|
1631
|
+
|
|
1632
|
+
---
|
|
1633
|
+
|
|
1634
|
+
## 5. Detailed Implementation
|
|
1635
|
+
|
|
1636
|
+
### Phase 1: Core Engine (week 1-2)
|
|
1637
|
+
|
|
1638
|
+
#### 1.1 Configuration Schema (`.ai-toolkit.json`)
|
|
1639
|
+
|
|
1640
|
+
> **v1 scope:** The full schema below shows the target state. v1 implements only: `extends`, `profile`, `agents`, `rules`, `constitution`, and `enforce`. See section 6a for the v1/v2 field breakdown.
|
|
1641
|
+
|
|
1642
|
+
**Project-level config file:**
|
|
1643
|
+
|
|
1644
|
+
```json
|
|
1645
|
+
{
|
|
1646
|
+
"$schema": "https://softspark.github.io/ai-toolkit/schemas/ai-toolkit-config.json",
|
|
1647
|
+
|
|
1648
|
+
"extends": "@mycompany/ai-toolkit-config",
|
|
1649
|
+
|
|
1650
|
+
"profile": "standard",
|
|
1651
|
+
"persona": "backend-lead",
|
|
1652
|
+
"hookProfile": "strict",
|
|
1653
|
+
|
|
1654
|
+
"agents": {
|
|
1655
|
+
"enabled": ["backend-specialist", "test-engineer", "debugger"],
|
|
1656
|
+
"disabled": ["game-developer", "mobile-developer"],
|
|
1657
|
+
"custom": ["./agents/compliance-auditor.md"]
|
|
1658
|
+
},
|
|
1659
|
+
|
|
1660
|
+
"skills": {
|
|
1661
|
+
"disabled": ["/deploy", "/rollback"],
|
|
1662
|
+
"custom": ["./skills/internal-deploy/"]
|
|
1663
|
+
},
|
|
1664
|
+
|
|
1665
|
+
"rules": {
|
|
1666
|
+
"inject": ["./rules/code-review-policy.md"],
|
|
1667
|
+
"remove": []
|
|
1668
|
+
},
|
|
1669
|
+
|
|
1670
|
+
"plugins": {
|
|
1671
|
+
"required": ["security-pack", "memory-pack"],
|
|
1672
|
+
"forbidden": []
|
|
1673
|
+
},
|
|
1674
|
+
|
|
1675
|
+
"languages": ["typescript", "python"],
|
|
1676
|
+
|
|
1677
|
+
"editors": ["cursor", "windsurf", "copilot"],
|
|
1678
|
+
|
|
1679
|
+
"constitution": {
|
|
1680
|
+
"amendments": [
|
|
1681
|
+
{
|
|
1682
|
+
"article": 6,
|
|
1683
|
+
"title": "Data Sovereignty",
|
|
1684
|
+
"text": "All code generation must comply with GDPR. No personal data in prompts. No PII in generated code comments."
|
|
1685
|
+
}
|
|
1686
|
+
]
|
|
1687
|
+
},
|
|
1688
|
+
|
|
1689
|
+
"overrides": {
|
|
1690
|
+
"hooks": {
|
|
1691
|
+
"quality-check": {
|
|
1692
|
+
"override": true,
|
|
1693
|
+
"justification": "Company uses custom lint pipeline via Jenkins",
|
|
1694
|
+
"replacement": "skip"
|
|
1695
|
+
}
|
|
1696
|
+
}
|
|
1697
|
+
}
|
|
1698
|
+
}
|
|
1699
|
+
```
|
|
1700
|
+
|
|
1701
|
+
**Base config (`ai-toolkit.config.json` in npm package):**
|
|
1702
|
+
|
|
1703
|
+
```json
|
|
1704
|
+
{
|
|
1705
|
+
"$schema": "https://softspark.github.io/ai-toolkit/schemas/ai-toolkit-base-config.json",
|
|
1706
|
+
"name": "@mycompany/ai-toolkit-config",
|
|
1707
|
+
"version": "2.1.0",
|
|
1708
|
+
"description": "MyCompany standard AI coding config",
|
|
1709
|
+
|
|
1710
|
+
"extends": null,
|
|
1711
|
+
|
|
1712
|
+
"profile": "strict",
|
|
1713
|
+
"persona": "backend-lead",
|
|
1714
|
+
"hookProfile": "strict",
|
|
1715
|
+
|
|
1716
|
+
"agents": {
|
|
1717
|
+
"enabled": ["backend-specialist", "test-engineer", "code-reviewer", "security-auditor", "debugger", "documenter"],
|
|
1718
|
+
"disabled": ["game-developer"],
|
|
1719
|
+
"custom": ["./agents/compliance-auditor.md"]
|
|
1720
|
+
},
|
|
1721
|
+
|
|
1722
|
+
"rules": {
|
|
1723
|
+
"inject": [
|
|
1724
|
+
"./rules/code-review-policy.md",
|
|
1725
|
+
"./rules/deployment-checklist.md",
|
|
1726
|
+
"./rules/data-handling-policy.md"
|
|
1727
|
+
]
|
|
1728
|
+
},
|
|
1729
|
+
|
|
1730
|
+
"plugins": {
|
|
1731
|
+
"required": ["security-pack"]
|
|
1732
|
+
},
|
|
1733
|
+
|
|
1734
|
+
"languages": ["typescript"],
|
|
1735
|
+
|
|
1736
|
+
"constitution": {
|
|
1737
|
+
"amendments": [
|
|
1738
|
+
{
|
|
1739
|
+
"article": 6,
|
|
1740
|
+
"title": "Data Sovereignty",
|
|
1741
|
+
"text": "All code generation must comply with GDPR. No personal data in prompts."
|
|
1742
|
+
},
|
|
1743
|
+
{
|
|
1744
|
+
"article": 7,
|
|
1745
|
+
"title": "Audit Compliance",
|
|
1746
|
+
"text": "All AI-generated code changes must be logged to the company audit system. The governance-capture hook must remain enabled."
|
|
1747
|
+
}
|
|
1748
|
+
]
|
|
1749
|
+
},
|
|
1750
|
+
|
|
1751
|
+
"enforce": {
|
|
1752
|
+
"minHookProfile": "standard",
|
|
1753
|
+
"requiredPlugins": ["security-pack"],
|
|
1754
|
+
"forbidOverride": ["constitution", "guard-destructive", "guard-path"],
|
|
1755
|
+
"requiredAgents": ["security-auditor"]
|
|
1756
|
+
}
|
|
1757
|
+
}
|
|
1758
|
+
```
|
|
1759
|
+
|
|
1760
|
+
**`enforce` section:** Base configs can define non-overridable constraints:
|
|
1761
|
+
- `minHookProfile` — projects cannot go below this profile
|
|
1762
|
+
- `requiredPlugins` — must be installed in all projects
|
|
1763
|
+
- `forbidOverride` — these components cannot be overridden
|
|
1764
|
+
- `requiredAgents` — must be enabled in all projects
|
|
1765
|
+
|
|
1766
|
+
---
|
|
1767
|
+
|
|
1768
|
+
#### 1.2 Config Resolver
|
|
1769
|
+
|
|
1770
|
+
**Resolution sources:**
|
|
1771
|
+
|
|
1772
|
+
| Source | Syntax | Resolution |
|
|
1773
|
+
|--------|--------|------------|
|
|
1774
|
+
| npm package | `"extends": "@mycompany/ai-toolkit-config"` | `npm pack --pack-destination /tmp` + extract |
|
|
1775
|
+
| npm with version | `"extends": "@mycompany/ai-toolkit-config@^2.0.0"` | Version resolution via npm |
|
|
1776
|
+
| Git URL | `"extends": "git+https://github.com/myco/ai-config.git"` | `git clone --depth 1` to cache |
|
|
1777
|
+
| Local path | `"extends": "../shared-config"` | Resolve relative to project root |
|
|
1778
|
+
| ~~Multiple bases~~ | ~~`"extends": ["@mycompany/base", "@mycompany/typescript-extra"]`~~ | Deferred to v2 — multi-base merge ordering is a complexity trap |
|
|
1779
|
+
|
|
1780
|
+
**Cache directory:** `~/.ai-toolkit/config-cache/`
|
|
1781
|
+
```
|
|
1782
|
+
~/.ai-toolkit/config-cache/
|
|
1783
|
+
@mycompany/
|
|
1784
|
+
ai-toolkit-config/
|
|
1785
|
+
2.1.0/
|
|
1786
|
+
ai-toolkit.config.json
|
|
1787
|
+
rules/
|
|
1788
|
+
agents/
|
|
1789
|
+
```
|
|
1790
|
+
|
|
1791
|
+
**Resolution algorithm:**
|
|
1792
|
+
```python
|
|
1793
|
+
def resolve_extends(extends_value: str, project_root: str) -> list[BaseConfig]:
|
|
1794
|
+
"""Resolve extends chain into ordered list of base configs.
|
|
1795
|
+
|
|
1796
|
+
v1: single string only. Multi-base (list) deferred to v2.
|
|
1797
|
+
"""
|
|
1798
|
+
configs = []
|
|
1799
|
+
for source in [extends_value]: # v2: support list[str]
|
|
1800
|
+
if source.startswith('@') or source.startswith('npm:'):
|
|
1801
|
+
config = resolve_npm(source)
|
|
1802
|
+
elif source.startswith('git+'):
|
|
1803
|
+
config = resolve_git(source)
|
|
1804
|
+
elif source.startswith('.') or source.startswith('/'):
|
|
1805
|
+
config = resolve_local(source, project_root)
|
|
1806
|
+
else:
|
|
1807
|
+
raise ConfigError(f"Unknown extends source: {source}")
|
|
1808
|
+
|
|
1809
|
+
# Recursive: base config may also have "extends"
|
|
1810
|
+
if config.extends:
|
|
1811
|
+
parent_configs = resolve_extends(config.extends, config.root)
|
|
1812
|
+
configs.extend(parent_configs)
|
|
1813
|
+
|
|
1814
|
+
configs.append(config)
|
|
1815
|
+
|
|
1816
|
+
return configs
|
|
1817
|
+
|
|
1818
|
+
|
|
1819
|
+
def resolve_extends(extends_value: str, project_root: str,
|
|
1820
|
+
_visited: set[str] | None = None) -> list[BaseConfig]:
|
|
1821
|
+
"""Full signature with cycle detection via visited set."""
|
|
1822
|
+
if _visited is None:
|
|
1823
|
+
_visited = set()
|
|
1824
|
+
if extends_value in _visited:
|
|
1825
|
+
raise ConfigError(
|
|
1826
|
+
f"Circular extends detected: {extends_value} already in chain "
|
|
1827
|
+
f"{' → '.join(_visited)}. Check your base config's 'extends' field."
|
|
1828
|
+
)
|
|
1829
|
+
if len(_visited) >= 5:
|
|
1830
|
+
raise ConfigError(
|
|
1831
|
+
f"Extends chain too deep (max 5 levels). Chain: {' → '.join(_visited)}"
|
|
1832
|
+
)
|
|
1833
|
+
_visited.add(extends_value)
|
|
1834
|
+
# ... resolution logic as above, passing _visited to recursive calls
|
|
1835
|
+
```
|
|
1836
|
+
|
|
1837
|
+
**Max recursion depth:** 5 levels (prevent circular extends). Circular detection via visited set.
|
|
1838
|
+
|
|
1839
|
+
**Offline handling:** If the npm/git source is unavailable:
|
|
1840
|
+
1. Check cache (`~/.ai-toolkit/config-cache/`)
|
|
1841
|
+
2. If cached version found → use with warning: "Using cached config v2.1.0 (offline)"
|
|
1842
|
+
3. If not cached → error with instructions: "Run `ai-toolkit config update` when online"
|
|
1843
|
+
|
|
1844
|
+
---
|
|
1845
|
+
|
|
1846
|
+
#### 1.3 Merge Engine
|
|
1847
|
+
|
|
1848
|
+
**Layered deep merge with explicit override semantics:**
|
|
1849
|
+
|
|
1850
|
+
```python
|
|
1851
|
+
def merge_configs(base: dict, project: dict) -> dict:
|
|
1852
|
+
"""Merge project config over base config with rules."""
|
|
1853
|
+
merged = {}
|
|
1854
|
+
|
|
1855
|
+
for key in set(base.keys()) | set(project.keys()):
|
|
1856
|
+
base_val = base.get(key)
|
|
1857
|
+
proj_val = project.get(key)
|
|
1858
|
+
|
|
1859
|
+
if proj_val is None:
|
|
1860
|
+
merged[key] = base_val
|
|
1861
|
+
elif base_val is None:
|
|
1862
|
+
merged[key] = proj_val
|
|
1863
|
+
elif key == 'constitution':
|
|
1864
|
+
merged[key] = merge_constitution(base_val, proj_val)
|
|
1865
|
+
elif key == 'agents':
|
|
1866
|
+
merged[key] = merge_agents(base_val, proj_val)
|
|
1867
|
+
elif key == 'rules':
|
|
1868
|
+
merged[key] = merge_rules(base_val, proj_val)
|
|
1869
|
+
elif key == 'overrides':
|
|
1870
|
+
merged[key] = validate_overrides(base, proj_val)
|
|
1871
|
+
elif isinstance(base_val, dict) and isinstance(proj_val, dict):
|
|
1872
|
+
merged[key] = merge_configs(base_val, proj_val)
|
|
1873
|
+
elif isinstance(base_val, list) and isinstance(proj_val, list):
|
|
1874
|
+
merged[key] = list(set(base_val + proj_val)) # union
|
|
1875
|
+
else:
|
|
1876
|
+
merged[key] = proj_val # project wins for scalars
|
|
1877
|
+
|
|
1878
|
+
return merged
|
|
1879
|
+
```
|
|
1880
|
+
|
|
1881
|
+
**Agent merge rules:**
|
|
1882
|
+
```python
|
|
1883
|
+
def merge_agents(base: dict, project: dict) -> dict:
|
|
1884
|
+
"""Merge agent configs — project can enable/disable but not remove base-required."""
|
|
1885
|
+
merged_enabled = set(base.get('enabled', []))
|
|
1886
|
+
|
|
1887
|
+
# Project can add agents
|
|
1888
|
+
merged_enabled.update(project.get('enabled', []))
|
|
1889
|
+
|
|
1890
|
+
# Project can disable agents (unless base enforces them)
|
|
1891
|
+
for agent in project.get('disabled', []):
|
|
1892
|
+
if agent in base.get('enforce', {}).get('requiredAgents', []):
|
|
1893
|
+
raise ConfigError(
|
|
1894
|
+
f"Cannot disable '{agent}' — required by base config '{base['name']}'. "
|
|
1895
|
+
f"Contact your team lead to request an exemption."
|
|
1896
|
+
)
|
|
1897
|
+
merged_enabled.discard(agent)
|
|
1898
|
+
|
|
1899
|
+
return {
|
|
1900
|
+
'enabled': sorted(merged_enabled),
|
|
1901
|
+
'custom': base.get('custom', []) + project.get('custom', [])
|
|
1902
|
+
}
|
|
1903
|
+
```
|
|
1904
|
+
|
|
1905
|
+
**Override validation:**
|
|
1906
|
+
```python
|
|
1907
|
+
def validate_overrides(base: dict, overrides: dict) -> dict:
|
|
1908
|
+
"""Validate project overrides against base enforcement rules."""
|
|
1909
|
+
forbidden = set(base.get('enforce', {}).get('forbidOverride', []))
|
|
1910
|
+
|
|
1911
|
+
for key, override in overrides.items():
|
|
1912
|
+
if key in forbidden:
|
|
1913
|
+
raise ConfigError(
|
|
1914
|
+
f"Cannot override '{key}' — forbidden by base config '{base['name']}'.\n"
|
|
1915
|
+
f"Forbidden overrides: {', '.join(sorted(forbidden))}\n"
|
|
1916
|
+
f"Contact your team lead to request an exemption."
|
|
1917
|
+
)
|
|
1918
|
+
if not override.get('override'):
|
|
1919
|
+
raise ConfigError(
|
|
1920
|
+
f"Override for '{key}' requires explicit 'override: true' + 'justification' field.\n"
|
|
1921
|
+
f"This ensures intentional deviation from organizational defaults."
|
|
1922
|
+
)
|
|
1923
|
+
if not override.get('justification'):
|
|
1924
|
+
raise ConfigError(
|
|
1925
|
+
f"Override for '{key}' requires a 'justification' field explaining why.\n"
|
|
1926
|
+
f"Example: \"Company uses custom lint pipeline via Jenkins\""
|
|
1927
|
+
)
|
|
1928
|
+
|
|
1929
|
+
return overrides
|
|
1930
|
+
```
|
|
1931
|
+
|
|
1932
|
+
---
|
|
1933
|
+
|
|
1934
|
+
#### 1.4 Constitution Immutability Guard
|
|
1935
|
+
|
|
1936
|
+
**Core rule:** Base constitution articles are absolutely immutable. Projects can only ADD new articles.
|
|
1937
|
+
|
|
1938
|
+
No weakening-detection heuristic (character count, semantic analysis) — these produce false positives and are gameable. Instead, the rule is simple and absolute: if an article number exists in the base, it cannot be modified by the project.
|
|
1939
|
+
|
|
1940
|
+
```python
|
|
1941
|
+
def merge_constitution(base: dict, project: dict) -> dict:
|
|
1942
|
+
"""Merge constitution — additions only, no modifications."""
|
|
1943
|
+
base_amendments = {a['article']: a for a in base.get('amendments', [])}
|
|
1944
|
+
proj_amendments = {a['article']: a for a in project.get('amendments', [])}
|
|
1945
|
+
|
|
1946
|
+
# Toolkit articles I-V are always immutable
|
|
1947
|
+
IMMUTABLE_ARTICLES = {1, 2, 3, 4, 5}
|
|
1948
|
+
|
|
1949
|
+
merged = dict(base_amendments)
|
|
1950
|
+
|
|
1951
|
+
for article_num, amendment in proj_amendments.items():
|
|
1952
|
+
if article_num in IMMUTABLE_ARTICLES:
|
|
1953
|
+
raise ConfigError(
|
|
1954
|
+
f"Cannot modify Constitution Article {article_num} — immutable.\n"
|
|
1955
|
+
f"Articles I-V are defined by ai-toolkit and cannot be overridden.\n"
|
|
1956
|
+
f"You can ADD new articles (article 6+)."
|
|
1957
|
+
)
|
|
1958
|
+
if article_num in base_amendments:
|
|
1959
|
+
# Base articles are immutable — projects cannot modify them
|
|
1960
|
+
raise ConfigError(
|
|
1961
|
+
f"Cannot modify Constitution Article {article_num} — "
|
|
1962
|
+
f"defined by base config '{base.get('name', 'unknown')}'.\n"
|
|
1963
|
+
f"Base articles are immutable. You can ADD new articles "
|
|
1964
|
+
f"with a higher article number."
|
|
1965
|
+
)
|
|
1966
|
+
merged[article_num] = amendment
|
|
1967
|
+
|
|
1968
|
+
return {'amendments': list(merged.values())}
|
|
1969
|
+
```
|
|
1970
|
+
|
|
1971
|
+
---
|
|
1972
|
+
|
|
1973
|
+
### MVP Phase 2: Integration + Diff (week 2-3)
|
|
1974
|
+
|
|
1975
|
+
#### 2.1 Install/Update Integration
|
|
1976
|
+
|
|
1977
|
+
**Modified `install.py` flow:**
|
|
1978
|
+
|
|
1979
|
+
```python
|
|
1980
|
+
# During install --local:
|
|
1981
|
+
# 1. Check for .ai-toolkit.json in project root
|
|
1982
|
+
# 2. If found and has "extends":
|
|
1983
|
+
# a. Resolve base config(s)
|
|
1984
|
+
# b. Merge base → project
|
|
1985
|
+
# c. Validate merged config
|
|
1986
|
+
# d. Generate files from merged config
|
|
1987
|
+
# 3. If not found: proceed with current behavior (backwards compatible)
|
|
1988
|
+
```
|
|
1989
|
+
|
|
1990
|
+
**CLI flags:**
|
|
1991
|
+
```bash
|
|
1992
|
+
ai-toolkit install --local # auto-detect .ai-toolkit.json
|
|
1993
|
+
ai-toolkit install --local --config ./custom.json # explicit config file
|
|
1994
|
+
ai-toolkit update --local # re-resolve extends + update
|
|
1995
|
+
ai-toolkit update --local --refresh-base # force re-fetch base config
|
|
1996
|
+
```
|
|
1997
|
+
|
|
1998
|
+
---
|
|
1999
|
+
|
|
2000
|
+
#### 2.2 `ai-toolkit config diff`
|
|
2001
|
+
|
|
2002
|
+
**Show differences between project config and base:**
|
|
2003
|
+
|
|
2004
|
+
```bash
|
|
2005
|
+
ai-toolkit config diff
|
|
2006
|
+
|
|
2007
|
+
# Output:
|
|
2008
|
+
# Base: @mycompany/ai-toolkit-config@2.1.0
|
|
2009
|
+
#
|
|
2010
|
+
# Profile: strict (base) → standard (project) ⚠ OVERRIDE
|
|
2011
|
+
# Persona: backend-lead (base) → frontend-lead (project)
|
|
2012
|
+
# Hook Profile: strict (base) → strict (inherited)
|
|
2013
|
+
#
|
|
2014
|
+
# Agents:
|
|
2015
|
+
# + frontend-specialist (project adds)
|
|
2016
|
+
# - game-developer (base disables)
|
|
2017
|
+
# = security-auditor (base requires, cannot disable)
|
|
2018
|
+
#
|
|
2019
|
+
# Rules:
|
|
2020
|
+
# + ./rules/api-standards.md (project adds)
|
|
2021
|
+
# = code-review-policy.md (inherited from base)
|
|
2022
|
+
#
|
|
2023
|
+
# Constitution:
|
|
2024
|
+
# = Articles I-V (immutable)
|
|
2025
|
+
# = Article 6: Data Sovereignty (inherited from base)
|
|
2026
|
+
# + Article 8: API Standards (project adds)
|
|
2027
|
+
#
|
|
2028
|
+
# Overrides:
|
|
2029
|
+
# quality-check: SKIP (justification: "Custom Jenkins pipeline")
|
|
2030
|
+
```
|
|
2031
|
+
|
|
2032
|
+
---
|
|
2033
|
+
|
|
2034
|
+
#### 2.3 `ai-toolkit config validate`
|
|
2035
|
+
|
|
2036
|
+
```bash
|
|
2037
|
+
ai-toolkit config validate
|
|
2038
|
+
|
|
2039
|
+
# Checks:
|
|
2040
|
+
# ✓ .ai-toolkit.json schema valid
|
|
2041
|
+
# ✓ extends: @mycompany/ai-toolkit-config@2.1.0 resolved
|
|
2042
|
+
# ✓ No forbidden overrides
|
|
2043
|
+
# ✓ Required plugins installed: security-pack
|
|
2044
|
+
# ✓ Required agents enabled: security-auditor
|
|
2045
|
+
# ✓ Constitution articles I-V intact
|
|
2046
|
+
# ✓ Hook profile meets minimum: standard ≥ standard
|
|
2047
|
+
# ✓ All custom rule files exist
|
|
2048
|
+
# ✓ All custom agent files exist
|
|
2049
|
+
```
|
|
2050
|
+
|
|
2051
|
+
---
|
|
2052
|
+
|
|
2053
|
+
### Phase 3: CLI Polish (week 4-5)
|
|
2054
|
+
|
|
2055
|
+
#### 2.4 `ai-toolkit config init`
|
|
2056
|
+
|
|
2057
|
+
**Interactive project config setup:**
|
|
2058
|
+
|
|
2059
|
+
```bash
|
|
2060
|
+
ai-toolkit config init
|
|
2061
|
+
|
|
2062
|
+
# Flow:
|
|
2063
|
+
# 1. "Does your organization have a shared ai-toolkit config? [y/n]"
|
|
2064
|
+
# → y: "npm package name or git URL:" → resolves + validates
|
|
2065
|
+
# → n: creates minimal .ai-toolkit.json without extends
|
|
2066
|
+
# 2. "Which profile? [minimal/standard/strict]" → default from base or standard
|
|
2067
|
+
# 3. "Which persona? [none/backend-lead/frontend-lead/devops-eng/junior-dev]"
|
|
2068
|
+
# 4. Auto-detect languages from project
|
|
2069
|
+
# 5. Auto-detect editors from project files
|
|
2070
|
+
# 6. Write .ai-toolkit.json
|
|
2071
|
+
# 7. Run ai-toolkit install --local
|
|
2072
|
+
```
|
|
2073
|
+
|
|
2074
|
+
---
|
|
2075
|
+
|
|
2076
|
+
#### 2.5 `ai-toolkit config create-base`
|
|
2077
|
+
|
|
2078
|
+
**Scaffold a base config package:**
|
|
2079
|
+
|
|
2080
|
+
```bash
|
|
2081
|
+
ai-toolkit config create-base @mycompany/ai-toolkit-config
|
|
2082
|
+
|
|
2083
|
+
# Creates:
|
|
2084
|
+
# @mycompany-ai-toolkit-config/
|
|
2085
|
+
# ├── package.json (name, version, files, peerDependencies)
|
|
2086
|
+
# ├── ai-toolkit.config.json (base config with sane defaults)
|
|
2087
|
+
# ├── rules/ (empty, ready for company rules)
|
|
2088
|
+
# ├── agents/ (empty, ready for company agents)
|
|
2089
|
+
# └── README.md (setup instructions)
|
|
2090
|
+
```
|
|
2091
|
+
|
|
2092
|
+
**Generated `package.json`:**
|
|
2093
|
+
```json
|
|
2094
|
+
{
|
|
2095
|
+
"name": "@mycompany/ai-toolkit-config",
|
|
2096
|
+
"version": "1.0.0",
|
|
2097
|
+
"description": "Shared ai-toolkit configuration for MyCompany",
|
|
2098
|
+
"main": "ai-toolkit.config.json",
|
|
2099
|
+
"files": ["ai-toolkit.config.json", "rules/", "agents/"],
|
|
2100
|
+
"peerDependencies": {
|
|
2101
|
+
"@softspark/ai-toolkit": ">=1.5.0"
|
|
2102
|
+
},
|
|
2103
|
+
"keywords": ["ai-toolkit", "config", "shared"]
|
|
2104
|
+
}
|
|
2105
|
+
```
|
|
2106
|
+
|
|
2107
|
+
---
|
|
2108
|
+
|
|
2109
|
+
### Phase 4: Enterprise Features (week 5-6)
|
|
2110
|
+
|
|
2111
|
+
#### 3.1 Audit Trail
|
|
2112
|
+
|
|
2113
|
+
**`state.json` additions:**
|
|
2114
|
+
```json
|
|
2115
|
+
{
|
|
2116
|
+
"extends": {
|
|
2117
|
+
"source": "@mycompany/ai-toolkit-config",
|
|
2118
|
+
"version": "2.1.0",
|
|
2119
|
+
"resolved_at": "2026-04-10T10:30:00Z",
|
|
2120
|
+
"hash": "sha256:abc123...",
|
|
2121
|
+
"overrides_applied": [
|
|
2122
|
+
{
|
|
2123
|
+
"key": "hooks.quality-check",
|
|
2124
|
+
"action": "skip",
|
|
2125
|
+
"justification": "Custom Jenkins pipeline"
|
|
2126
|
+
}
|
|
2127
|
+
]
|
|
2128
|
+
}
|
|
2129
|
+
}
|
|
2130
|
+
```
|
|
2131
|
+
|
|
2132
|
+
---
|
|
2133
|
+
|
|
2134
|
+
#### 3.2 Lock File (`.ai-toolkit.lock.json`)
|
|
2135
|
+
|
|
2136
|
+
**Purpose:** Pin the exact resolved version of base configs for reproducible installs across team members and CI.
|
|
2137
|
+
|
|
2138
|
+
```json
|
|
2139
|
+
{
|
|
2140
|
+
"lockfileVersion": 1,
|
|
2141
|
+
"resolved": {
|
|
2142
|
+
"@mycompany/ai-toolkit-config": {
|
|
2143
|
+
"version": "2.1.0",
|
|
2144
|
+
"resolved": "https://registry.npmjs.org/@mycompany/ai-toolkit-config/-/ai-toolkit-config-2.1.0.tgz",
|
|
2145
|
+
"integrity": "sha512-abc123...",
|
|
2146
|
+
"cached": "~/.ai-toolkit/config-cache/@mycompany/ai-toolkit-config/2.1.0/"
|
|
2147
|
+
}
|
|
2148
|
+
},
|
|
2149
|
+
"generated_at": "2026-04-10T10:30:00Z",
|
|
2150
|
+
"ai_toolkit_version": "1.5.1"
|
|
2151
|
+
}
|
|
2152
|
+
```
|
|
2153
|
+
|
|
2154
|
+
**Behavior:**
|
|
2155
|
+
- `ai-toolkit install --local` → uses lock file if present (like `npm ci`)
|
|
2156
|
+
- `ai-toolkit update --local` → re-resolves and updates lock file (like `npm install`)
|
|
2157
|
+
- `ai-toolkit update --local --refresh-base` → force re-fetch ignoring cache
|
|
2158
|
+
- `.ai-toolkit.lock.json` should be committed to git (team synchronization)
|
|
2159
|
+
|
|
2160
|
+
---
|
|
2161
|
+
|
|
2162
|
+
#### 3.4 CI Enforcement
|
|
2163
|
+
|
|
2164
|
+
**`ai-toolkit config check` — for CI pipelines:**
|
|
2165
|
+
|
|
2166
|
+
```bash
|
|
2167
|
+
ai-toolkit config check
|
|
2168
|
+
|
|
2169
|
+
# Exit codes:
|
|
2170
|
+
# 0 — project complies with base config
|
|
2171
|
+
# 1 — violations found (missing required plugins, forbidden overrides, etc.)
|
|
2172
|
+
# 2 — .ai-toolkit.json not found
|
|
2173
|
+
```
|
|
2174
|
+
|
|
2175
|
+
**GitHub Actions example:**
|
|
2176
|
+
```yaml
|
|
2177
|
+
- name: AI Toolkit Governance Check
|
|
2178
|
+
run: |
|
|
2179
|
+
npx @softspark/ai-toolkit config check
|
|
2180
|
+
npx @softspark/ai-toolkit config validate --strict
|
|
2181
|
+
```
|
|
2182
|
+
|
|
2183
|
+
**What it checks:**
|
|
2184
|
+
1. Required plugins are installed
|
|
2185
|
+
2. Required agents are enabled
|
|
2186
|
+
3. No forbidden overrides applied without exemption
|
|
2187
|
+
4. Hook profile meets minimum
|
|
2188
|
+
5. Constitution articles intact
|
|
2189
|
+
6. Lock file up-to-date (warn if stale)
|
|
2190
|
+
|
|
2191
|
+
---
|
|
2192
|
+
|
|
2193
|
+
## 6. File Summary
|
|
2194
|
+
|
|
2195
|
+
| File | Action | LOC (est.) | Description |
|
|
2196
|
+
|------|--------|------------|-------------|
|
|
2197
|
+
| `scripts/config_resolver.py` | CREATE | ~400 | Resolve extends (npm, git, local path) |
|
|
2198
|
+
| `scripts/config_merger.py` | CREATE | ~350 | Layered merge engine |
|
|
2199
|
+
| `scripts/config_validator.py` | CREATE | ~200 | Schema + enforcement validation |
|
|
2200
|
+
| `scripts/config_scaffold.py` | CREATE | ~250 | create-base scaffolder |
|
|
2201
|
+
| `scripts/config_diff.py` | CREATE | ~200 | Diff viewer |
|
|
2202
|
+
| `scripts/config_check.py` | CREATE | ~150 | CI enforcement checker |
|
|
2203
|
+
| `scripts/install.py` | EDIT | +80 | Integrate extends resolution |
|
|
2204
|
+
| `bin/ai-toolkit.js` | EDIT | +40 | Register config subcommands |
|
|
2205
|
+
| `manifest.json` | EDIT | +10 | Schema references |
|
|
2206
|
+
| `kb/reference/enterprise-config-guide.md` | CREATE | ~300 | Enterprise setup guide |
|
|
2207
|
+
| `kb/reference/base-config-template/` | CREATE | ~200 | Scaffolded base config files |
|
|
2208
|
+
| `tests/test_config_resolver.bats` | CREATE | ~150 | Resolution tests |
|
|
2209
|
+
| `tests/test_config_merger.bats` | CREATE | ~200 | Merge + override tests |
|
|
2210
|
+
| `tests/test_config_immutability.bats` | CREATE | ~100 | Constitution guard tests |
|
|
2211
|
+
| `tests/test_config_cli.bats` | CREATE | ~150 | CLI command tests |
|
|
2212
|
+
| **Total** | | **~2780** | |
|
|
2213
|
+
|
|
2214
|
+
---
|
|
2215
|
+
|
|
2216
|
+
## 6a. Schema Scope (v1 vs v2)
|
|
2217
|
+
|
|
2218
|
+
v1 ships with a minimal schema. Each additional field adds merge logic, validation, diff output, and test surface. Expand based on real usage, not speculation.
|
|
2219
|
+
|
|
2220
|
+
| Field | v1 | v2 | Rationale |
|
|
2221
|
+
|-------|----|----|-----------|
|
|
2222
|
+
| `extends` | single string | array (multi-base) | Multi-base merge ordering is complex |
|
|
2223
|
+
| `profile` | yes | — | Core governance knob |
|
|
2224
|
+
| `agents` | yes | — | Most common customization |
|
|
2225
|
+
| `rules` | yes | — | Rule injection is existing feature |
|
|
2226
|
+
| `constitution` | yes | — | Key differentiator |
|
|
2227
|
+
| `enforce` | yes | — | Non-overridable constraints |
|
|
2228
|
+
| `skills` | — | yes | Less commonly customized at org level |
|
|
2229
|
+
| `plugins` | — | yes | Depends on plugin maturity |
|
|
2230
|
+
| `languages` | — | yes | Auto-detected, rarely org-level |
|
|
2231
|
+
| `editors` | — | yes | Auto-detected, rarely org-level |
|
|
2232
|
+
| `overrides` | — | yes | Complex, needs real-world feedback |
|
|
2233
|
+
| `hookProfile` / `persona` | — | yes | Low demand signal |
|
|
2234
|
+
|
|
2235
|
+
---
|
|
2236
|
+
|
|
2237
|
+
## 6b. Non-Functional Requirements
|
|
2238
|
+
|
|
2239
|
+
| Category | Requirement |
|
|
2240
|
+
|----------|-------------|
|
|
2241
|
+
| **Performance** | `install --local` with extends resolution < 5s (cached), < 15s (first fetch). Config merge < 100ms. |
|
|
2242
|
+
| **Offline** | Cached configs used when registry unavailable, with clear warning. |
|
|
2243
|
+
| **Security** | No secret exposure in config files or audit trail. npm auth via `.npmrc` (user-managed). `execFile` for npm CLI (no shell injection). |
|
|
2244
|
+
| **Error messages** | Every validation error includes: what failed, which config layer caused it, and what to do (e.g., "Contact your team lead to request an exemption"). |
|
|
2245
|
+
| **Backward compatibility** | 100% — projects without `.ai-toolkit.json` work exactly as today. Zero behavioral changes for existing users. |
|
|
2246
|
+
| **Maintainability** | Each new schema field requires: merge logic, validation, diff output, test. Budget 0.5d per new field. |
|
|
2247
|
+
| **Quality gates** | `ruff check scripts/config_*.py` (0 errors), `mypy --strict scripts/config_*.py` (0 errors). Run before every commit. |
|
|
2248
|
+
| **Type safety** | 100% public API type hints (all function signatures). >60% internal. Use `TypedDict` for config schemas, `dataclass` for resolved configs. |
|
|
2249
|
+
|
|
2250
|
+
---
|
|
2251
|
+
|
|
2252
|
+
## 7. Success Criteria (Overall)
|
|
2253
|
+
|
|
2254
|
+
| Metric | Target |
|
|
2255
|
+
|--------|--------|
|
|
2256
|
+
| Extends sources (v1) | 4 (npm, npm+version, git URL, local path) — single string only |
|
|
2257
|
+
| Merge depth | 5 levels max (recursive extends) |
|
|
2258
|
+
| Config schema | JSON Schema validated |
|
|
2259
|
+
| Constitution protection | 100% (Articles I-V immutable) |
|
|
2260
|
+
| Override justification | Required for all overrides |
|
|
2261
|
+
| Enforce constraints | 4 types (minHookProfile, requiredPlugins, forbidOverride, requiredAgents) |
|
|
2262
|
+
| Backward compatibility | 100% (projects without .ai-toolkit.json work as today) |
|
|
2263
|
+
| CI enforcement | Exit code 0/1 for governance compliance |
|
|
2264
|
+
| Lock file | Reproducible installs across team members |
|
|
2265
|
+
| Scaffold command | Ready-to-publish npm package template |
|
|
2266
|
+
| Tests | 30+ |
|
|
2267
|
+
| Offline resolution | Cached configs with warning |
|
|
2268
|
+
|
|
2269
|
+
---
|
|
2270
|
+
|
|
2271
|
+
## 8. Risks and Mitigation
|
|
2272
|
+
|
|
2273
|
+
| Risk | Probability | Impact | Mitigation |
|
|
2274
|
+
|------|-------------|--------|------------|
|
|
2275
|
+
| npm registry unavailable during install | Low | Medium | Cache + offline fallback with warning |
|
|
2276
|
+
| Circular extends chain | Low | High | Max depth 5 + visited set for cycle detection |
|
|
2277
|
+
| Base config breaks project | Medium | High | Lock file pins exact version; `ai-toolkit config diff` shows changes before update |
|
|
2278
|
+
| Override abuse (teams bypass governance) | Medium | Medium | `enforce.forbidOverride` + CI check + justification requirement |
|
|
2279
|
+
| Config schema too restrictive | Medium | Medium | Start with minimal enforcement, expand based on enterprise feedback |
|
|
2280
|
+
| Multiple base configs conflict | — | — | Deferred to v2 (single extends only in v1) |
|
|
2281
|
+
| Private npm registry authentication | Medium | Low | Use existing npm auth (`.npmrc`), document setup |
|
|
2282
|
+
| Git URL resolution slow | Low | Low | `--depth 1` clone, cache aggressively |
|
|
2283
|
+
|
|
2284
|
+
---
|
|
2285
|
+
|
|
2286
|
+
## 9. Pre-Mortem
|
|
2287
|
+
|
|
2288
|
+
1. **"Config file fatigue"** — developers already have `.eslintrc`, `tsconfig.json`, `.prettierrc`. Another `.ai-toolkit.json` may feel like bloat. Mitigation: file is optional, all features work without it. The DX gain (organizational governance without per-repo updates) justifies the file.
|
|
2289
|
+
2. **"Base config never gets updated"** — team lead creates base config, nobody maintains it. Mitigation: `ai-toolkit config check` in CI catches drift; lock file staleness warnings.
|
|
2290
|
+
3. **"Override justification is annoying"** — developers will write "needed" as justification. Mitigation: CI check can enforce minimum justification length (>20 chars); code review culture catches low-effort justifications.
|
|
2291
|
+
4. **"Merge semantics are confusing"** — "does project override or extend the base agent list?" Mitigation: explicit semantics documented in schema; `ai-toolkit config diff` shows exactly what happened.
|
|
2292
|
+
5. **"Enterprise teams want RBAC on overrides"** — who can approve overrides? Mitigation: v1 uses justification text + code review; v2 could integrate with GitHub CODEOWNERS for override approval.
|
|
2293
|
+
|
|
2294
|
+
---
|
|
2295
|
+
|
|
2296
|
+
## 10. Market Positioning
|
|
2297
|
+
|
|
2298
|
+
**Target users:**
|
|
2299
|
+
1. **Engineering managers** — enforce AI coding standards across 20+ repos without touching each one
|
|
2300
|
+
2. **Security teams** — ensure constitution + security-auditor agent is always enabled
|
|
2301
|
+
3. **Platform teams** — distribute company-specific agents, rules, and plugins via npm
|
|
2302
|
+
4. **Compliance officers** — audit trail of what AI governance rules are active in each project
|
|
2303
|
+
|
|
2304
|
+
**Competitive advantage:** No existing AI coding toolkit supports configuration inheritance. This is a unique enterprise feature that transforms ai-toolkit from a developer tool into an organizational governance platform.
|
|
2305
|
+
|
|
2306
|
+
**Revenue potential:** Enterprise teams are the primary audience for paid support/consulting around ai-toolkit. Config inheritance is the feature that makes enterprise adoption manageable.
|
|
2307
|
+
|
|
2308
|
+
---
|
|
2309
|
+
|
|
2310
|
+
## 11. Next Actions
|
|
2311
|
+
|
|
2312
|
+
**MVP (ship first, ~3.5 weeks):**
|
|
2313
|
+
1. [ ] Approve plan
|
|
2314
|
+
2. [ ] Define `.ai-toolkit.json` JSON Schema — v1 scope only (1.1)
|
|
2315
|
+
3. [ ] Implement config resolver (npm, git, local) with caching (1.2)
|
|
2316
|
+
4. [ ] Implement merge engine with override validation (1.3)
|
|
2317
|
+
5. [ ] Implement constitution immutability guard (1.4)
|
|
2318
|
+
6. [ ] Integrate into install.py flow (2.1)
|
|
2319
|
+
7. [ ] Create `config diff` viewer (2.2) — primary debugging tool
|
|
2320
|
+
8. [ ] Create `config validate` checker (2.3)
|
|
2321
|
+
9. [ ] Tests for above (4.1 partial)
|
|
2322
|
+
10. [ ] **Ship MVP → announce → measure adoption**
|
|
2323
|
+
|
|
2324
|
+
**Post-MVP (if demand validated):**
|
|
2325
|
+
11. [ ] Create `config init` interactive command (2.4)
|
|
2326
|
+
12. [ ] Create `config create-base` scaffolder (2.5)
|
|
2327
|
+
13. [ ] Add audit trail to state.json (3.1)
|
|
2328
|
+
14. [ ] Implement lock file generation + resolution (3.2)
|
|
2329
|
+
15. [ ] Create base config npm package template (3.3)
|
|
2330
|
+
16. [ ] Create CI enforcement command `config check` (3.4)
|
|
2331
|
+
17. [ ] Full tests + documentation — all 9 docs per CLAUDE.md (4.1, 4.2)
|
|
2332
|
+
|
|
2333
|
+
---
|
|
2334
|
+
|
|
2335
|
+
## 12. Future (v2)
|
|
2336
|
+
|
|
2337
|
+
| Feature | Rationale |
|
|
2338
|
+
|---------|-----------|
|
|
2339
|
+
| Multi-base extends (`"extends": [...]`) | Needs real-world feedback on merge ordering UX |
|
|
2340
|
+
| v1 deferred schema fields (skills, plugins, languages, editors, overrides, hookProfile, persona) | Expand based on actual enterprise requests |
|
|
2341
|
+
| RBAC on overrides (GitHub CODEOWNERS integration) | v1 uses justification + code review |
|
|
2342
|
+
| Semantic constitution analysis | Character-count heuristics removed in v1; revisit only if absolute immutability proves too restrictive |
|
|
2343
|
+
| `ai-toolkit config audit` (full governance report) | Depends on audit trail maturity |
|
|
2344
|
+
|
|
2345
|
+
---
|
|
2346
|
+
|
|
2347
|
+
## 13. Cross-Plan Dependencies
|
|
2348
|
+
|
|
2349
|
+
This plan shares modification targets with two other proposed plans:
|
|
2350
|
+
|
|
2351
|
+
| Shared File | This Plan | Local Dashboard Plan | Offline SLM Plan |
|
|
2352
|
+
|-------------|-----------|---------------------|-----------------|
|
|
2353
|
+
| `scripts/install.py` | +80 LOC (extends resolution) | — | +30 LOC (offline-slm profile) |
|
|
2354
|
+
| `manifest.json` | +10 LOC (schema refs) | — | +5 LOC (offline-slm profile) |
|
|
2355
|
+
| `bin/ai-toolkit.js` | +40 LOC (config subcommands) | +15 LOC (ui command) | +10 LOC (compile-slm command) |
|
|
2356
|
+
|
|
2357
|
+
**If implementing in parallel:** coordinate merge order for shared files. Recommended sequence: Offline SLM (smallest changes) → Enterprise Config → Dashboard (no install.py changes).
|
|
2358
|
+
|
|
2359
|
+
**Dashboard integration note:** If this plan ships before the Dashboard plan, the Dashboard's Config page (2.6) should display `.ai-toolkit.json` / `extends` status and the `config diff` output.
|
|
2360
|
+
|
|
2361
|
+
---
|
|
2362
|
+
|
|
2363
|
+
**Last Updated:** 2026-04-10
|
|
2364
|
+
|
|
2365
|
+
---
|
|
2366
|
+
|
|
2367
|
+
## kb/planning/local-dashboard-plan.md
|
|
2368
|
+
|
|
2369
|
+
---
|
|
2370
|
+
title: "Plan: Local Dashboard — ai-toolkit ui"
|
|
2371
|
+
category: planning
|
|
2372
|
+
service: ai-toolkit
|
|
2373
|
+
tags:
|
|
2374
|
+
- dashboard
|
|
2375
|
+
- developer-experience
|
|
2376
|
+
- web-ui
|
|
2377
|
+
- tui
|
|
2378
|
+
- configuration
|
|
2379
|
+
- visualization
|
|
2380
|
+
doc_type: plan
|
|
2381
|
+
status: proposed
|
|
2382
|
+
created: "2026-04-10"
|
|
2383
|
+
last_updated: "2026-04-10"
|
|
2384
|
+
completion: "0%"
|
|
2385
|
+
description: "Ephemeral local web dashboard for ai-toolkit. Provides visual management of agents, skills, hooks, plugins, stats, credentials, and configuration profiles. Zero external dependencies — stdlib Node.js server with embedded HTML/CSS/JS. Launched via `ai-toolkit ui`."
|
|
2386
|
+
---
|
|
2387
|
+
|
|
2388
|
+
# Plan: Local Dashboard — `ai-toolkit ui`
|
|
2389
|
+
|
|
2390
|
+
**Status:** Proposed
|
|
2391
|
+
**Completion:** 0%
|
|
2392
|
+
**Created:** 2026-04-10
|
|
2393
|
+
**Origin:** DX friction — managing 44 agents, 91 skills, 21 hooks, 11 plugin packs, and multiple profiles via CLI only creates a steep learning curve for new adopters. Visual management lowers the adoption barrier.
|
|
2394
|
+
**Estimated Effort:** 6-7 weeks (1 person)
|
|
2395
|
+
|
|
2396
|
+
---
|
|
2397
|
+
|
|
2398
|
+
## 1. Objective
|
|
2399
|
+
|
|
2400
|
+
Create an `ai-toolkit ui` command that launches an ephemeral local HTTP server serving a single-page web dashboard. The dashboard provides visual management of all toolkit components — no more memorizing CLI flags.
|
|
2401
|
+
|
|
2402
|
+
**Key design principles:**
|
|
2403
|
+
- **Zero external dependencies** — stdlib Node.js `http` module, embedded HTML/CSS/JS (same approach as `visual-server.cjs` in `/write-a-prd`). Server lives in `app/dashboard/` (not `scripts/`) — aligns with `visual-server.cjs` precedent and the "scripts/ = Python stdlib only" convention
|
|
2404
|
+
- **Ephemeral** — auto-kills after 30 minutes of inactivity (matching existing companion pattern)
|
|
2405
|
+
- **Read-write** — reads state from `~/.ai-toolkit/state.json`, `~/.claude/settings.json`, `manifest.json`; writes config changes through existing CLI commands (never mutates files directly)
|
|
2406
|
+
- **Dark theme** — premium aesthetic, responsive, glassmorphism accents
|
|
2407
|
+
- **Offline** — no CDN dependencies, no external fonts, no analytics
|
|
2408
|
+
- **Port discovery** — starts on 3141, auto-increments if busy
|
|
2409
|
+
|
|
2410
|
+
---
|
|
2411
|
+
|
|
2412
|
+
## 1a. Functional Requirements
|
|
2413
|
+
|
|
2414
|
+
| ID | Requirement | Priority | Success Metric |
|
|
2415
|
+
|----|-------------|----------|----------------|
|
|
2416
|
+
| FR1 | HTTP server with auto-kill + port discovery | Must | Starts, auto-kills after 30 min, increments port if busy |
|
|
2417
|
+
| FR2 | Read API endpoints (status, agents, skills, hooks, plugins, stats, config, mcp) | Must | All 8 endpoints return valid JSON |
|
|
2418
|
+
| FR3 | Action execution via `POST /api/action` with SSE streaming | Must | Command output streamed, exit code returned |
|
|
2419
|
+
| FR4 | Overview page with health checks + component counts | Must | Live data, matches `ai-toolkit validate` output |
|
|
2420
|
+
| FR5 | Agents page with grid view + category filter | Must | All 44 agents parsed, 10 categories filterable |
|
|
2421
|
+
| FR6 | Skills page with type/effort filter + sortable table | Must | All 91 skills, 3 type badges |
|
|
2422
|
+
| FR7 | Hooks page with lifecycle diagram + profile toggle | Should | 21 hooks, 3 profiles toggleable |
|
|
2423
|
+
| FR8 | Plugins page with install/remove action buttons | Should | Action triggers CLI via POST /api/action |
|
|
2424
|
+
| FR9 | Config page with profile/persona management | Should | Reads + writes profiles via CLI |
|
|
2425
|
+
| FR10 | Stats page with inline SVG charts | Should | Bar chart + trend line minimum (v1) |
|
|
2426
|
+
| FR11 | MCP template browser with add/remove | Could | 25 templates browseable |
|
|
2427
|
+
| FR12 | Dark theme + glassmorphism CSS design system | Must | Zero external CSS/font deps |
|
|
2428
|
+
| FR13 | Client-side SPA routing (hash-based) | Must | Navigation without page reload |
|
|
2429
|
+
| FR14 | Responsive design (desktop, tablet, mobile) | Should | 3 breakpoints, usable on mobile |
|
|
2430
|
+
| FR15 | Command allowlist enforcement on /api/action | Must | Non-allowlisted commands → 403 |
|
|
2431
|
+
|
|
2432
|
+
---
|
|
2433
|
+
|
|
2434
|
+
## 2. Architecture Overview
|
|
2435
|
+
|
|
2436
|
+
```
|
|
2437
|
+
ai-toolkit ui [--port 3141] [--no-auto-kill]
|
|
2438
|
+
|
|
2439
|
+
┌──────────────────────────────────────────────────────────┐
|
|
2440
|
+
│ Local Dashboard │
|
|
2441
|
+
│ │
|
|
2442
|
+
│ Server: stdlib Node.js http (0 deps) │
|
|
2443
|
+
│ Port: 3141 (auto-increment if busy) │
|
|
2444
|
+
│ Auto-kill: 30 min idle (configurable) │
|
|
2445
|
+
│ │
|
|
2446
|
+
│ Pages: │
|
|
2447
|
+
│ / Overview + health │
|
|
2448
|
+
│ /agents 44 agents — grid view │
|
|
2449
|
+
│ /skills 91 skills — filterable table │
|
|
2450
|
+
│ /hooks 21 hooks — lifecycle diagram │
|
|
2451
|
+
│ /plugins 11 packs — install/remove │
|
|
2452
|
+
│ /config Profile / persona / modules │
|
|
2453
|
+
│ /stats Usage analytics + charts │
|
|
2454
|
+
│ /mcp MCP templates — add/remove │
|
|
2455
|
+
│ (credentials page deferred — requires cloud-security-pack CLI) │
|
|
2456
|
+
│ │
|
|
2457
|
+
│ API (JSON, internal): │
|
|
2458
|
+
│ GET /api/status Toolkit state │
|
|
2459
|
+
│ GET /api/agents Agent catalog │
|
|
2460
|
+
│ GET /api/skills Skill catalog │
|
|
2461
|
+
│ GET /api/hooks Hook definitions │
|
|
2462
|
+
│ GET /api/plugins Plugin packs + install state │
|
|
2463
|
+
│ GET /api/stats Usage statistics │
|
|
2464
|
+
│ GET /api/config Current configuration │
|
|
2465
|
+
│ GET /api/mcp MCP templates + installed │
|
|
2466
|
+
│ (credentials endpoint deferred — see Future section) │
|
|
2467
|
+
│ POST /api/action Execute CLI command │
|
|
2468
|
+
│ │
|
|
2469
|
+
│ Static assets (inlined in server.js): │
|
|
2470
|
+
│ HTML template (single page, client-side routing) │
|
|
2471
|
+
│ CSS (dark theme, glassmorphism, responsive) │
|
|
2472
|
+
│ JS (vanilla, fetch-based, no framework) │
|
|
2473
|
+
└──────────────────────────────────────────────────────────┘
|
|
2474
|
+
```
|
|
2475
|
+
|
|
2476
|
+
### Action Execution Model
|
|
2477
|
+
|
|
2478
|
+
All write operations go through `POST /api/action`:
|
|
2479
|
+
|
|
2480
|
+
```json
|
|
2481
|
+
{
|
|
2482
|
+
"command": "plugin",
|
|
2483
|
+
"args": ["install", "memory-pack"]
|
|
2484
|
+
}
|
|
2485
|
+
```
|
|
2486
|
+
|
|
2487
|
+
The server spawns `ai-toolkit <command> <args>` as a child process, streams stdout/stderr back via SSE (Server-Sent Events), and returns exit code. This ensures:
|
|
2488
|
+
1. All existing validation logic in CLI scripts executes
|
|
2489
|
+
2. No file mutation logic duplicated in the dashboard
|
|
2490
|
+
3. Audit trail identical to CLI usage
|
|
2491
|
+
|
|
2492
|
+
---
|
|
2493
|
+
|
|
2494
|
+
## 3. Progress Tracking
|
|
2495
|
+
|
|
2496
|
+
| # | Feature | Priority | Status | Est. Time | Notes |
|
|
2497
|
+
|---|---------|----------|--------|-----------|-------|
|
|
2498
|
+
| 1.1 | HTTP server + auto-kill + port discovery | P0 | Proposed | 1d | stdlib Node.js, ephemeral lifecycle |
|
|
2499
|
+
| 1.2 | API layer — read endpoints (status, agents, skills, hooks, plugins, stats, config, mcp) | P0 | Proposed | 2d | Parse frontmatter, state.json, hooks.json |
|
|
2500
|
+
| 1.3 | API layer — action execution endpoint | P0 | Proposed | 1d | Spawn CLI + SSE streaming |
|
|
2501
|
+
| 2.1 | Overview page (health, component counts, version) | P0 | Proposed | 1.5d | Dashboard landing page |
|
|
2502
|
+
| 2.2 | Agents page (grid + detail view + category filter) | P0 | Proposed | 2d | Parse agent .md frontmatter |
|
|
2503
|
+
| 2.3 | Skills page (filterable table + type badges + effort) | P0 | Proposed | 2d | Task / hybrid / knowledge taxonomy |
|
|
2504
|
+
| 2.4 | Hooks page (lifecycle diagram + profile toggle) | P1 | Proposed | 2d | Visual event → script mapping |
|
|
2505
|
+
| 2.5 | Plugins page (install/remove/status cards) | P1 | Proposed | 2d | Action buttons trigger CLI |
|
|
2506
|
+
| 2.6 | Config page (profile/persona/modules checkboxes) | P1 | Proposed | 2d | Read/write profiles |
|
|
2507
|
+
| 2.7 | Stats page (usage charts, skill invocation heatmap) | P1 | Proposed | 3.5d | Hand-drawn SVG charts (bar, heatmap, trend), no chart library |
|
|
2508
|
+
| 2.8 | MCP page (template browser, add/remove) | P2 | Proposed | 1.5d | 25 MCP templates |
|
|
2509
|
+
| 2.9 | ~~Credentials page~~ | — | Deferred | — | Requires cloud-security-pack CLI commands (not yet implemented) |
|
|
2510
|
+
| 3.1 | CSS design system (dark theme, glassmorphism, responsive) | P0 | Proposed | 2d | Premium aesthetic, zero external deps |
|
|
2511
|
+
| 3.2 | Client-side routing + navigation | P0 | Proposed | 1d | Hash-based SPA routing |
|
|
2512
|
+
| 4.1 | CLI command registration (`ai-toolkit ui`) | P0 | Proposed | 0.5d | bin/ai-toolkit.js integration |
|
|
2513
|
+
| 4.2 | Tests (bats + node:test) | P1 | Proposed | 3d | Server lifecycle (bats) + API endpoints + SSE streaming (node:test) |
|
|
2514
|
+
| 4.3 | Documentation | P1 | Proposed | 2.5d | README, CLAUDE.md, ARCHITECTURE.md, package.json, llms.txt, llms-full.txt, AGENTS.md, skills-catalog.md, architecture-overview.md |
|
|
2515
|
+
|
|
2516
|
+
**Phasing:**
|
|
2517
|
+
- **Phase 1 (week 1-3):** Foundation — server, API, design system, overview page, agents page, skills page
|
|
2518
|
+
- **Phase 2 (week 3-5):** Interactive — hooks, plugins, config, action execution, stats
|
|
2519
|
+
- **Phase 3 (week 6-7):** Polish — MCP, responsive testing, tests, documentation (2.5d docs — all 9 docs per CLAUDE.md rules)
|
|
2520
|
+
|
|
2521
|
+
> **Demand validation gate:** Ship Phase 1 (server + overview + agents + skills) as MVP. Announce, measure adoption (track `ai-toolkit ui` invocations via `stats.json`). Only build Phase 2 interactive pages if usage > 10 sessions/week across early adopters.
|
|
2522
|
+
|
|
2523
|
+
---
|
|
2524
|
+
|
|
2525
|
+
## 4. Dependency Graph
|
|
2526
|
+
|
|
2527
|
+
```
|
|
2528
|
+
Phase 1: Foundation (week 1-3)
|
|
2529
|
+
================================
|
|
2530
|
+
HTTP server (1.1) ──────┐
|
|
2531
|
+
├──► API read layer (1.2) ──► Overview page (2.1)
|
|
2532
|
+
CSS design system (3.1) ┤ ├──► Agents page (2.2)
|
|
2533
|
+
Client routing (3.2) ───┘ └──► Skills page (2.3)
|
|
2534
|
+
|
|
2535
|
+
Phase 2: Interactive (week 3-5)
|
|
2536
|
+
=================================
|
|
2537
|
+
API action layer (1.3) ─┐
|
|
2538
|
+
├──► Hooks page (2.4)
|
|
2539
|
+
├──► Plugins page (2.5)
|
|
2540
|
+
├──► Config page (2.6)
|
|
2541
|
+
└──► Stats page (2.7)
|
|
2542
|
+
|
|
2543
|
+
Phase 3: Polish (week 6-7)
|
|
2544
|
+
===========================
|
|
2545
|
+
├──► MCP page (2.8)
|
|
2546
|
+
├──► CLI registration (4.1)
|
|
2547
|
+
└──► Tests + docs (4.2, 4.3)
|
|
2548
|
+
```
|
|
2549
|
+
|
|
2550
|
+
---
|
|
2551
|
+
|
|
2552
|
+
## 5. Detailed Implementation
|
|
2553
|
+
|
|
2554
|
+
### Phase 1: Foundation (week 1-3)
|
|
2555
|
+
|
|
2556
|
+
#### 1.1 HTTP Server + Lifecycle
|
|
2557
|
+
|
|
2558
|
+
**File:** `app/dashboard/server.js`
|
|
2559
|
+
|
|
2560
|
+
```javascript
|
|
2561
|
+
// Key design decisions:
|
|
2562
|
+
// 1. stdlib only — require('http'), require('fs'), require('path')
|
|
2563
|
+
// 2. Auto-kill timer — 30 min idle, reset on every request
|
|
2564
|
+
// 3. Port discovery — try 3141, increment until available
|
|
2565
|
+
// 4. Single-file deployment — HTML/CSS/JS embedded as template literals
|
|
2566
|
+
// 5. Same pattern as visual-brainstorming companion in /write-a-prd
|
|
2567
|
+
|
|
2568
|
+
const AUTO_KILL_MS = 30 * 60 * 1000; // 30 minutes
|
|
2569
|
+
const DEFAULT_PORT = 3141;
|
|
2570
|
+
const MAX_PORT_ATTEMPTS = 10;
|
|
2571
|
+
```
|
|
2572
|
+
|
|
2573
|
+
**CLI interface:**
|
|
2574
|
+
```bash
|
|
2575
|
+
ai-toolkit ui # open dashboard on port 3141
|
|
2576
|
+
ai-toolkit ui --port 4000 # custom port
|
|
2577
|
+
ai-toolkit ui --no-auto-kill # disable 30 min auto-kill
|
|
2578
|
+
ai-toolkit ui --open # auto-open browser (default: true)
|
|
2579
|
+
```
|
|
2580
|
+
|
|
2581
|
+
**Lifecycle:**
|
|
2582
|
+
1. Start HTTP server on available port
|
|
2583
|
+
2. Open browser via `open` (macOS) / `xdg-open` (Linux)
|
|
2584
|
+
3. Reset idle timer on every request
|
|
2585
|
+
4. After 30 min inactivity → `process.exit(0)` with console message
|
|
2586
|
+
5. `Ctrl+C` → graceful shutdown
|
|
2587
|
+
|
|
2588
|
+
**Security:**
|
|
2589
|
+
- Bind to `127.0.0.1` only (never `0.0.0.0`)
|
|
2590
|
+
- No authentication needed (localhost only, ephemeral)
|
|
2591
|
+
- POST `/api/action` validates command against allowlist (same commands as CLI)
|
|
2592
|
+
- No file uploads, no eval, no template injection
|
|
2593
|
+
|
|
2594
|
+
**Files:**
|
|
2595
|
+
|
|
2596
|
+
| File | Action | Description |
|
|
2597
|
+
|------|--------|-------------|
|
|
2598
|
+
| `app/dashboard/server.js` | CREATE | HTTP server + API + embedded UI |
|
|
2599
|
+
| `app/dashboard/api.js` | CREATE | API endpoint handlers |
|
|
2600
|
+
| `app/dashboard/assets.js` | CREATE | Embedded HTML/CSS/JS templates |
|
|
2601
|
+
|
|
2602
|
+
**Success Criteria:**
|
|
2603
|
+
- [ ] Server starts on port 3141 (or next available)
|
|
2604
|
+
- [ ] Auto-kills after 30 min idle
|
|
2605
|
+
- [ ] Browser auto-opens on launch
|
|
2606
|
+
- [ ] Binds to 127.0.0.1 only
|
|
2607
|
+
- [ ] `Ctrl+C` graceful shutdown
|
|
2608
|
+
|
|
2609
|
+
---
|
|
2610
|
+
|
|
2611
|
+
#### 1.2 API Layer — Read Endpoints
|
|
2612
|
+
|
|
2613
|
+
All endpoints return JSON. Data sources:
|
|
2614
|
+
|
|
2615
|
+
| Endpoint | Source | Description |
|
|
2616
|
+
|----------|--------|-------------|
|
|
2617
|
+
| `GET /api/status` | `~/.ai-toolkit/state.json` + `manifest.json` | Version, profile, installed modules |
|
|
2618
|
+
| `GET /api/agents` | `app/agents/*.md` frontmatter | Name, description, tools, triggers, category |
|
|
2619
|
+
| `GET /api/skills` | `app/skills/*/SKILL.md` frontmatter | Name, type, effort, agent, description |
|
|
2620
|
+
| `GET /api/hooks` | `app/hooks.json` | Event, script, description, profile |
|
|
2621
|
+
| `GET /api/plugins` | `app/plugins/*/plugin.json` + state | Name, domain, status, installed?, components |
|
|
2622
|
+
| `GET /api/stats` | `~/.ai-toolkit/stats.json` | Skill invocation counts, dates |
|
|
2623
|
+
| `GET /api/config` | `~/.ai-toolkit/state.json` + settings | Profile, persona, modules, hook profile |
|
|
2624
|
+
| `GET /api/mcp` | `app/mcp-templates/*.json` + `.mcp.json` | Available templates, installed servers |
|
|
2625
|
+
| ~~`GET /api/credentials`~~ | — | Deferred — requires cloud-security-pack CLI |
|
|
2626
|
+
|
|
2627
|
+
**Frontmatter parser:** Reuse the YAML-subset parser pattern from existing `scripts/frontmatter.py` — port to JS (simple `---` delimited key-value extraction, covers all toolkit frontmatter which is flat YAML).
|
|
2628
|
+
|
|
2629
|
+
**Response format (example):**
|
|
2630
|
+
```json
|
|
2631
|
+
{
|
|
2632
|
+
"agents": [
|
|
2633
|
+
{
|
|
2634
|
+
"name": "backend-specialist",
|
|
2635
|
+
"description": "Expert backend architect for Node.js, Python, PHP...",
|
|
2636
|
+
"tools": ["Read", "Write", "Edit", "Bash", "Grep", "Glob"],
|
|
2637
|
+
"triggers": ["backend", "server", "api", "endpoint"],
|
|
2638
|
+
"category": "development",
|
|
2639
|
+
"file": "app/agents/backend-specialist.md"
|
|
2640
|
+
}
|
|
2641
|
+
],
|
|
2642
|
+
"meta": { "count": 44, "categories": 10 }
|
|
2643
|
+
}
|
|
2644
|
+
```
|
|
2645
|
+
|
|
2646
|
+
**Success Criteria:**
|
|
2647
|
+
- [ ] All 8 read endpoints return valid JSON
|
|
2648
|
+
- [ ] Agent frontmatter parsed for all 44 agents
|
|
2649
|
+
- [ ] Skill frontmatter parsed for all 91 skills
|
|
2650
|
+
- [ ] Stats endpoint handles missing stats.json gracefully
|
|
2651
|
+
|
|
2652
|
+
---
|
|
2653
|
+
|
|
2654
|
+
#### 1.3 API Layer — Action Execution
|
|
2655
|
+
|
|
2656
|
+
**Endpoint:** `POST /api/action`
|
|
2657
|
+
|
|
2658
|
+
```json
|
|
2659
|
+
// Request
|
|
2660
|
+
{
|
|
2661
|
+
"command": "plugin",
|
|
2662
|
+
"args": ["install", "memory-pack"]
|
|
2663
|
+
}
|
|
2664
|
+
|
|
2665
|
+
// Response (SSE stream)
|
|
2666
|
+
event: stdout
|
|
2667
|
+
data: Installing memory-pack...
|
|
2668
|
+
|
|
2669
|
+
event: stdout
|
|
2670
|
+
data: ✓ Hooks installed
|
|
2671
|
+
|
|
2672
|
+
event: done
|
|
2673
|
+
data: {"exitCode": 0, "duration": 1234}
|
|
2674
|
+
```
|
|
2675
|
+
|
|
2676
|
+
**Command allowlist:**
|
|
2677
|
+
```javascript
|
|
2678
|
+
const ALLOWED_COMMANDS = [
|
|
2679
|
+
'plugin install', 'plugin remove', 'plugin update', 'plugin clean',
|
|
2680
|
+
'mcp add', 'mcp remove',
|
|
2681
|
+
'install --profile', 'install --persona',
|
|
2682
|
+
'update', 'validate', 'doctor', 'doctor --fix',
|
|
2683
|
+
'stats', 'stats --reset',
|
|
2684
|
+
// 'credentials add/remove/test' — deferred until cloud-security-pack CLI ships
|
|
2685
|
+
];
|
|
2686
|
+
```
|
|
2687
|
+
|
|
2688
|
+
**Security:** Commands not in allowlist → 403. No shell injection — args passed as array to `execFile`, never interpolated into a string.
|
|
2689
|
+
|
|
2690
|
+
**Success Criteria:**
|
|
2691
|
+
- [ ] SSE streaming of command output
|
|
2692
|
+
- [ ] Exit code returned in final event
|
|
2693
|
+
- [ ] Command allowlist enforced
|
|
2694
|
+
- [ ] No shell injection possible
|
|
2695
|
+
- [ ] Concurrent commands rejected (one at a time)
|
|
2696
|
+
|
|
2697
|
+
---
|
|
2698
|
+
|
|
2699
|
+
### Phase 2: UI Pages (week 3-5)
|
|
2700
|
+
|
|
2701
|
+
#### 2.1 Overview Page
|
|
2702
|
+
|
|
2703
|
+
The landing page — first thing the user sees.
|
|
2704
|
+
|
|
2705
|
+
**Layout:**
|
|
2706
|
+
```
|
|
2707
|
+
┌─────────────────────────────────────────────────┐
|
|
2708
|
+
│ ai-toolkit v1.5.1 [● running] │
|
|
2709
|
+
├─────────────────────────────────────────────────┤
|
|
2710
|
+
│ │
|
|
2711
|
+
│ ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐ │
|
|
2712
|
+
│ │ 44 │ │ 91 │ │ 21 │ │ 11 │ │
|
|
2713
|
+
│ │agents│ │skills│ │hooks │ │packs │ │
|
|
2714
|
+
│ └──────┘ └──────┘ └──────┘ └──────┘ │
|
|
2715
|
+
│ │
|
|
2716
|
+
│ Profile: standard Persona: (none) │
|
|
2717
|
+
│ Hook Profile: standard │
|
|
2718
|
+
│ Node: v22.x Python: 3.12 │
|
|
2719
|
+
│ │
|
|
2720
|
+
│ ┌─ Health ──────────────────────────────────┐ │
|
|
2721
|
+
│ │ ✓ Constitution symlinked │ │
|
|
2722
|
+
│ │ ✓ Hooks installed (21/21) │ │
|
|
2723
|
+
│ │ ✓ Agents symlinked (44/44) │ │
|
|
2724
|
+
│ │ ⚠ 2 plugins not installed │ │
|
|
2725
|
+
│ │ ✓ MCP servers: 3 configured │ │
|
|
2726
|
+
│ └───────────────────────────────────────────┘ │
|
|
2727
|
+
│ │
|
|
2728
|
+
│ ┌─ Recent Activity ─────────────────────────┐ │
|
|
2729
|
+
│ │ /review — 12 invocations (last: 2h ago) │ │
|
|
2730
|
+
│ │ /commit — 8 invocations (last: 5h ago) │ │
|
|
2731
|
+
│ │ /test — 6 invocations (last: 1d ago) │ │
|
|
2732
|
+
│ └───────────────────────────────────────────┘ │
|
|
2733
|
+
└─────────────────────────────────────────────────┘
|
|
2734
|
+
```
|
|
2735
|
+
|
|
2736
|
+
**Data sources:** `/api/status` + `/api/stats` + `/api/plugins`
|
|
2737
|
+
|
|
2738
|
+
**Success Criteria:**
|
|
2739
|
+
- [ ] Component counts rendered from live data
|
|
2740
|
+
- [ ] Health checks match `ai-toolkit validate` output
|
|
2741
|
+
- [ ] Recent activity from stats.json
|
|
2742
|
+
- [ ] Responsive on mobile-width screens
|
|
2743
|
+
|
|
2744
|
+
---
|
|
2745
|
+
|
|
2746
|
+
#### 2.2 Agents Page
|
|
2747
|
+
|
|
2748
|
+
**Layout:** Grid of agent cards, filterable by category (10 categories).
|
|
2749
|
+
|
|
2750
|
+
**Card design:**
|
|
2751
|
+
```
|
|
2752
|
+
┌─────────────────────────────────┐
|
|
2753
|
+
│ 🔧 backend-specialist │
|
|
2754
|
+
│ │
|
|
2755
|
+
│ Expert backend architect for │
|
|
2756
|
+
│ Node.js, Python, PHP... │
|
|
2757
|
+
│ │
|
|
2758
|
+
│ Tools: Read, Write, Edit, Bash │
|
|
2759
|
+
│ Triggers: backend, server, api │
|
|
2760
|
+
│ │
|
|
2761
|
+
│ [View Definition] │
|
|
2762
|
+
└─────────────────────────────────┘
|
|
2763
|
+
```
|
|
2764
|
+
|
|
2765
|
+
**Features:**
|
|
2766
|
+
- Category filter pills (development, security, data, infrastructure, etc.)
|
|
2767
|
+
- Search by name/description/trigger
|
|
2768
|
+
- Click card → modal with full agent definition (rendered markdown)
|
|
2769
|
+
- Agent count per category in filter pills
|
|
2770
|
+
|
|
2771
|
+
---
|
|
2772
|
+
|
|
2773
|
+
#### 2.3 Skills Page
|
|
2774
|
+
|
|
2775
|
+
**Layout:** Filterable table with type badges.
|
|
2776
|
+
|
|
2777
|
+
| Skill | Type | Effort | Agent | Description |
|
|
2778
|
+
|-------|------|--------|-------|-------------|
|
|
2779
|
+
| /commit | task | medium | — | Structured commit with linting |
|
|
2780
|
+
| /review | hybrid | high | code-reviewer | Code review: quality, security |
|
|
2781
|
+
| clean-code | knowledge | — | — | Auto-loaded code quality patterns |
|
|
2782
|
+
|
|
2783
|
+
**Features:**
|
|
2784
|
+
- Filter by type (task / hybrid / knowledge)
|
|
2785
|
+
- Filter by effort (low / medium / high / max)
|
|
2786
|
+
- Search by name/description
|
|
2787
|
+
- Sort by any column
|
|
2788
|
+
- Badge colors: task=blue, hybrid=purple, knowledge=green
|
|
2789
|
+
|
|
2790
|
+
---
|
|
2791
|
+
|
|
2792
|
+
#### 2.4 Hooks Page
|
|
2793
|
+
|
|
2794
|
+
**Layout:** Lifecycle diagram with profile toggle (minimal/standard/strict).
|
|
2795
|
+
|
|
2796
|
+
**Visualization:**
|
|
2797
|
+
```
|
|
2798
|
+
Event Timeline (horizontal flow):
|
|
2799
|
+
PreToolUse ──► ToolUse ──► PostToolUse ──► Notification
|
|
2800
|
+
│ │ │ │
|
|
2801
|
+
guard-path.sh (tool exec) quality-check.sh track-usage.sh
|
|
2802
|
+
guard-destructive.sh session-context.sh
|
|
2803
|
+
```
|
|
2804
|
+
|
|
2805
|
+
**Features:**
|
|
2806
|
+
- Lifecycle event diagram — visual mapping of event → hook script(s)
|
|
2807
|
+
- Profile toggle — show/hide hooks per profile (minimal hides most, strict shows all)
|
|
2808
|
+
- Hook detail panel — click hook name to see: script path, description, when it fires, which profile includes it
|
|
2809
|
+
- Profile diff — highlight which hooks are added/removed between profiles
|
|
2810
|
+
- Status indicators — green (installed), red (missing script), yellow (overridden by plugin)
|
|
2811
|
+
|
|
2812
|
+
**Data source:** `app/hooks.json` + `~/.ai-toolkit/hooks/` filesystem check
|
|
2813
|
+
|
|
2814
|
+
**Success Criteria:**
|
|
2815
|
+
- [ ] All 21 hooks rendered with correct lifecycle event
|
|
2816
|
+
- [ ] Profile toggle filters hooks correctly
|
|
2817
|
+
- [ ] Missing hook scripts shown with warning indicator
|
|
2818
|
+
- [ ] Hook detail panel shows script content preview
|
|
2819
|
+
|
|
2820
|
+
---
|
|
2821
|
+
|
|
2822
|
+
#### 2.5 Plugins Page
|
|
2823
|
+
|
|
2824
|
+
**Layout:** Cards with install/remove buttons.
|
|
2825
|
+
|
|
2826
|
+
```
|
|
2827
|
+
┌─────────────────────────────────┐
|
|
2828
|
+
│ 🧩 memory-pack ● installed │
|
|
2829
|
+
│ │
|
|
2830
|
+
│ SQLite-based persistent memory │
|
|
2831
|
+
│ with FTS5 search across sessions│
|
|
2832
|
+
│ │
|
|
2833
|
+
│ Skills: 1 Hooks: 2 Agents: 0 │
|
|
2834
|
+
│ DB size: 2.4 MB Obs: 1,234 │
|
|
2835
|
+
│ │
|
|
2836
|
+
│ [Update] [Clean] [Remove] │
|
|
2837
|
+
└─────────────────────────────────┘
|
|
2838
|
+
```
|
|
2839
|
+
|
|
2840
|
+
**Actions:** Install, update, remove, clean — all via `POST /api/action` → `ai-toolkit plugin <action> <name>`. SSE output shown in a slide-out console panel.
|
|
2841
|
+
|
|
2842
|
+
---
|
|
2843
|
+
|
|
2844
|
+
#### 2.6 Config Page
|
|
2845
|
+
|
|
2846
|
+
**Layout:** Form with current configuration, editable.
|
|
2847
|
+
|
|
2848
|
+
**Sections:**
|
|
2849
|
+
1. **Profile selector:** minimal / standard / strict (radio buttons)
|
|
2850
|
+
2. **Persona selector:** none / backend-lead / frontend-lead / devops-eng / junior-dev
|
|
2851
|
+
3. **Hook profile:** minimal / standard / strict
|
|
2852
|
+
4. **Installed modules:** checkboxes for each module from manifest.json
|
|
2853
|
+
5. **Language rules:** detected languages + override checkboxes
|
|
2854
|
+
6. **Editor configs:** which editors are configured (read-only status)
|
|
2855
|
+
|
|
2856
|
+
**Save:** Generates and executes the equivalent `ai-toolkit install --profile X --persona Y --modules A,B,C` command.
|
|
2857
|
+
|
|
2858
|
+
---
|
|
2859
|
+
|
|
2860
|
+
#### 2.7 Stats Page
|
|
2861
|
+
|
|
2862
|
+
**Layout:** Usage analytics with inline SVG charts.
|
|
2863
|
+
|
|
2864
|
+
**Charts (SVG, hand-drawn — no chart library):**
|
|
2865
|
+
1. **Skill invocation bar chart** — top 15 most-used skills
|
|
2866
|
+
2. **Invocation heatmap** — 7x24 grid (day of week × hour) showing when skills are used
|
|
2867
|
+
3. **Effort distribution** — pie chart of low/medium/high/max invocations
|
|
2868
|
+
4. **Trend line** — daily invocations over last 30 days
|
|
2869
|
+
|
|
2870
|
+
**Data source:** `~/.ai-toolkit/stats.json` (written by `track-usage.sh` hook)
|
|
2871
|
+
|
|
2872
|
+
**SVG implementation notes:**
|
|
2873
|
+
- All charts rendered as inline SVG strings in `assets.js` — no external chart library
|
|
2874
|
+
- Bar chart: `<rect>` elements with calculated heights, axis labels as `<text>`
|
|
2875
|
+
- Heatmap: 7x24 grid of `<rect>` with color intensity mapped to invocation count (0=transparent, max=accent-primary)
|
|
2876
|
+
- Trend line: `<polyline>` with data points as `<circle>`, area fill via `<polygon>`
|
|
2877
|
+
- Pie chart: `<path>` arcs calculated from percentages (use `Math.cos`/`Math.sin` for arc endpoints)
|
|
2878
|
+
- All charts use CSS custom properties for colors (respects design system tokens)
|
|
2879
|
+
|
|
2880
|
+
**Fallback:** If hand-drawn SVG proves too complex for 4 chart types in 3.5d budget, reduce to 2 charts (bar + trend) for v1 and defer heatmap + pie to v2.
|
|
2881
|
+
|
|
2882
|
+
**Success Criteria:**
|
|
2883
|
+
- [ ] All 4 chart types render from live stats.json data
|
|
2884
|
+
- [ ] Empty state when stats.json missing: "No usage data yet — skills will appear here after first use"
|
|
2885
|
+
- [ ] Charts responsive (SVG viewBox scales to container width)
|
|
2886
|
+
- [ ] Tooltip on hover showing exact counts
|
|
2887
|
+
|
|
2888
|
+
---
|
|
2889
|
+
|
|
2890
|
+
### Phase 3: Polish (week 6-7)
|
|
2891
|
+
|
|
2892
|
+
#### 3.1 CSS Design System
|
|
2893
|
+
|
|
2894
|
+
**Design tokens:**
|
|
2895
|
+
```css
|
|
2896
|
+
:root {
|
|
2897
|
+
/* Dark theme palette */
|
|
2898
|
+
--bg-primary: #0f0f14;
|
|
2899
|
+
--bg-secondary: #1a1a24;
|
|
2900
|
+
--bg-card: rgba(255, 255, 255, 0.04);
|
|
2901
|
+
--bg-glass: rgba(255, 255, 255, 0.06);
|
|
2902
|
+
--border-glass: rgba(255, 255, 255, 0.08);
|
|
2903
|
+
|
|
2904
|
+
/* Accent colors (derived from SoftSpark brand) */
|
|
2905
|
+
--accent-primary: #6366f1; /* indigo */
|
|
2906
|
+
--accent-secondary: #8b5cf6; /* violet */
|
|
2907
|
+
--accent-success: #10b981; /* emerald */
|
|
2908
|
+
--accent-warning: #f59e0b; /* amber */
|
|
2909
|
+
--accent-danger: #ef4444; /* red */
|
|
2910
|
+
|
|
2911
|
+
/* Typography */
|
|
2912
|
+
--font-sans: system-ui, -apple-system, sans-serif;
|
|
2913
|
+
--font-mono: 'SF Mono', 'Cascadia Code', monospace;
|
|
2914
|
+
|
|
2915
|
+
/* Spacing scale */
|
|
2916
|
+
--space-xs: 4px;
|
|
2917
|
+
--space-sm: 8px;
|
|
2918
|
+
--space-md: 16px;
|
|
2919
|
+
--space-lg: 24px;
|
|
2920
|
+
--space-xl: 32px;
|
|
2921
|
+
|
|
2922
|
+
/* Glassmorphism */
|
|
2923
|
+
--glass-blur: 12px;
|
|
2924
|
+
--glass-bg: rgba(255, 255, 255, 0.05);
|
|
2925
|
+
--glass-border: rgba(255, 255, 255, 0.1);
|
|
2926
|
+
}
|
|
2927
|
+
```
|
|
2928
|
+
|
|
2929
|
+
**Components:**
|
|
2930
|
+
- Cards with glassmorphism backdrop-filter
|
|
2931
|
+
- Badge pills (type, effort, status)
|
|
2932
|
+
- Sidebar navigation with active state
|
|
2933
|
+
- Console output panel (slide-out, monospace)
|
|
2934
|
+
- Modal overlays for detail views
|
|
2935
|
+
- Toast notifications for action results
|
|
2936
|
+
- Skeleton loading states
|
|
2937
|
+
|
|
2938
|
+
**Responsive breakpoints:**
|
|
2939
|
+
- Desktop: 1200px+ (sidebar + content)
|
|
2940
|
+
- Tablet: 768px-1199px (collapsed sidebar)
|
|
2941
|
+
- Mobile: <768px (hamburger menu, stacked cards)
|
|
2942
|
+
|
|
2943
|
+
---
|
|
2944
|
+
|
|
2945
|
+
#### 4.1 CLI Command Registration
|
|
2946
|
+
|
|
2947
|
+
**File:** `bin/ai-toolkit.js` — add to `COMMANDS` and `SPECIAL_HANDLERS`:
|
|
2948
|
+
|
|
2949
|
+
```javascript
|
|
2950
|
+
// COMMANDS
|
|
2951
|
+
'ui': 'Launch local web dashboard for visual toolkit management',
|
|
2952
|
+
|
|
2953
|
+
// SPECIAL_HANDLERS
|
|
2954
|
+
'ui': (args) => {
|
|
2955
|
+
const serverPath = path.join(TOOLKIT_DIR, 'app', 'dashboard', 'server.js');
|
|
2956
|
+
const child = require('child_process').spawn('node', [serverPath, ...args], {
|
|
2957
|
+
stdio: 'inherit',
|
|
2958
|
+
env: { ...process.env, TOOLKIT_DIR }
|
|
2959
|
+
});
|
|
2960
|
+
child.on('exit', (code) => process.exit(code || 0));
|
|
2961
|
+
},
|
|
2962
|
+
```
|
|
2963
|
+
|
|
2964
|
+
**Files to modify:**
|
|
2965
|
+
|
|
2966
|
+
| File | Action | Description |
|
|
2967
|
+
|------|--------|-------------|
|
|
2968
|
+
| `bin/ai-toolkit.js` | EDIT | Register `ui` command |
|
|
2969
|
+
| `app/dashboard/server.js` | CREATE | HTTP server (main) |
|
|
2970
|
+
| `app/dashboard/api.js` | CREATE | API handlers |
|
|
2971
|
+
| `app/dashboard/assets.js` | CREATE | Embedded HTML/CSS/JS |
|
|
2972
|
+
| `app/dashboard/frontmatter.js` | CREATE | JS frontmatter parser |
|
|
2973
|
+
| `tests/test_dashboard.bats` | CREATE | Server lifecycle tests |
|
|
2974
|
+
| `tests/test_dashboard_api.bats` | CREATE | API endpoint tests |
|
|
2975
|
+
|
|
2976
|
+
---
|
|
2977
|
+
|
|
2978
|
+
## 6. File Summary
|
|
2979
|
+
|
|
2980
|
+
| File | Action | LOC (est.) | Description |
|
|
2981
|
+
|------|--------|------------|-------------|
|
|
2982
|
+
| `app/dashboard/server.js` | CREATE | ~300 | HTTP server, lifecycle, routing |
|
|
2983
|
+
| `app/dashboard/api.js` | CREATE | ~400 | All API endpoint handlers |
|
|
2984
|
+
| `app/dashboard/assets.js` | CREATE | ~2800 | Embedded HTML + CSS + JS. Split internally: `getOverviewHTML()`, `getAgentsHTML()`, etc. — one exported function per page. Single file, multiple functions (not multiple files — preserves single-`require` deployment). If file exceeds 3000 LOC, extract to `assets/` directory with `index.js` barrel |
|
|
2985
|
+
| `app/dashboard/frontmatter.js` | CREATE | ~80 | Frontmatter parser (JS port) |
|
|
2986
|
+
| `bin/ai-toolkit.js` | EDIT | +15 | Register `ui` command |
|
|
2987
|
+
| `tests/test_dashboard.bats` | CREATE | ~100 | Server lifecycle tests |
|
|
2988
|
+
| `tests/test_dashboard_api.bats` | CREATE | ~150 | API endpoint tests |
|
|
2989
|
+
| **Total** | | **~3845** | |
|
|
2990
|
+
|
|
2991
|
+
---
|
|
2992
|
+
|
|
2993
|
+
## 7. Success Criteria (Overall)
|
|
2994
|
+
|
|
2995
|
+
| Metric | Target |
|
|
2996
|
+
|--------|--------|
|
|
2997
|
+
| Pages | 8 (overview, agents, skills, hooks, plugins, config, stats, mcp) |
|
|
2998
|
+
| API endpoints | 9 (8 read + 1 action) |
|
|
2999
|
+
| External dependencies | 0 (stdlib Node.js only) |
|
|
3000
|
+
| Server startup time | < 500ms |
|
|
3001
|
+
| Auto-kill | 30 min idle (configurable) |
|
|
3002
|
+
| Responsive breakpoints | 3 (desktop, tablet, mobile) |
|
|
3003
|
+
| Tests | 25+ |
|
|
3004
|
+
| Browser support | Chrome, Firefox, Safari (modern, no IE) |
|
|
3005
|
+
|
|
3006
|
+
---
|
|
3007
|
+
|
|
3008
|
+
## 7a. Non-Functional Requirements
|
|
3009
|
+
|
|
3010
|
+
| Category | Requirement |
|
|
3011
|
+
|----------|-------------|
|
|
3012
|
+
| **Performance** | Server startup < 500ms. Page render < 200ms. API responses < 100ms. |
|
|
3013
|
+
| **Memory** | Max RSS < 100MB (including embedded assets). No unbounded caching. |
|
|
3014
|
+
| **Concurrency** | Max 1 concurrent action execution. Queue or reject additional requests with 429. |
|
|
3015
|
+
| **Security** | Bind `127.0.0.1` only. CSP header: `default-src 'self' 'unsafe-inline'`. No CORS headers (same-origin only). Command allowlist enforced server-side. `execFile` with array args (no shell). |
|
|
3016
|
+
| **Accessibility** | Keyboard navigation for all interactive elements. Focus management on page transitions. Minimum 4.5:1 contrast ratio (WCAG AA). ARIA labels on icon-only buttons. |
|
|
3017
|
+
| **Error handling** | Missing/corrupt data files → graceful empty state with message. Server crash → exit code 1 with stderr diagnostic. API errors → JSON `{ "error": "..." }` with HTTP status. |
|
|
3018
|
+
| **Graceful degradation** | If `stats.json` missing → stats page shows "No data yet". If `state.json` missing → overview shows defaults. If agent/skill .md unreadable → skip with warning in console. |
|
|
3019
|
+
|
|
3020
|
+
---
|
|
3021
|
+
|
|
3022
|
+
## 7b. Rollback & Feature Flag
|
|
3023
|
+
|
|
3024
|
+
The dashboard is purely additive — removing it is trivial:
|
|
3025
|
+
1. Delete `app/dashboard/` directory
|
|
3026
|
+
2. Remove `ui` from `COMMANDS` and `SPECIAL_HANDLERS` in `bin/ai-toolkit.js`
|
|
3027
|
+
3. No config files, no state files, no hooks to clean up
|
|
3028
|
+
|
|
3029
|
+
**Disable without removal:** `ai-toolkit ui --disabled` could print "Dashboard disabled" and exit. Alternatively, skip registering the `ui` command in `bin/ai-toolkit.js` behind a `manifest.json` module flag so users can opt out via `--skip dashboard`.
|
|
3030
|
+
|
|
3031
|
+
---
|
|
3032
|
+
|
|
3033
|
+
## 7c. Testing Strategy
|
|
3034
|
+
|
|
3035
|
+
| Layer | Framework | Coverage |
|
|
3036
|
+
|-------|-----------|----------|
|
|
3037
|
+
| Server lifecycle (start, port discovery, auto-kill, shutdown) | bats | 5+ tests |
|
|
3038
|
+
| API endpoints (all 8 read + action) | `node:test` (stdlib) | 15+ tests |
|
|
3039
|
+
| SSE streaming (stdout/stderr/done events) | `node:test` | 3+ tests |
|
|
3040
|
+
| Command allowlist enforcement | `node:test` | 5+ tests |
|
|
3041
|
+
| Frontmatter parser edge cases | `node:test` | 5+ tests |
|
|
3042
|
+
| Integration: full pipeline (start → API → action → stop) | `node:test` | 3+ tests |
|
|
3043
|
+
| **Total** | | **36+** |
|
|
3044
|
+
|
|
3045
|
+
`node:test` is stdlib (Node 18+), zero dependencies. Bats tests are for CLI-level integration (process start/stop). JS tests cover API correctness.
|
|
3046
|
+
|
|
3047
|
+
---
|
|
3048
|
+
|
|
3049
|
+
## 7d. Discoverability
|
|
3050
|
+
|
|
3051
|
+
| Touchpoint | Action |
|
|
3052
|
+
|------------|--------|
|
|
3053
|
+
| `ai-toolkit install` output | Print banner: `Run 'ai-toolkit ui' to explore agents, skills, and plugins visually.` |
|
|
3054
|
+
| `ai-toolkit help` | Include `ui` in command list |
|
|
3055
|
+
| README.md | Screenshot/GIF of dashboard overview page |
|
|
3056
|
+
| First-run detection | If `stats.json` is empty, show a "Try the dashboard" suggestion after `ai-toolkit install` |
|
|
3057
|
+
|
|
3058
|
+
---
|
|
3059
|
+
|
|
3060
|
+
## 8. Risks and Mitigation
|
|
3061
|
+
|
|
3062
|
+
| Risk | Probability | Impact | Mitigation |
|
|
3063
|
+
|------|-------------|--------|------------|
|
|
3064
|
+
| Assets.js file too large (embedded HTML/CSS/JS) | Medium | Low | Split into multiple template modules, lazy-load pages |
|
|
3065
|
+
| Port conflict on 3141 | Low | Low | Auto-increment port, show clear message |
|
|
3066
|
+
| SSE not supported in old browsers | Low | Low | Fallback to polling for action results |
|
|
3067
|
+
| Frontmatter parser edge cases | Low | Medium | Match exact patterns used in existing toolkit metadata |
|
|
3068
|
+
| Config writes break installation | Low | High | All writes go through existing CLI commands — never direct file mutation |
|
|
3069
|
+
| Stats.json missing or empty | Medium | Low | Graceful empty state with "No usage data yet" message |
|
|
3070
|
+
|
|
3071
|
+
---
|
|
3072
|
+
|
|
3073
|
+
## 9. Pre-Mortem
|
|
3074
|
+
|
|
3075
|
+
1. **"Too many features per page"** — Dashboard tries to show everything. Users may feel overwhelmed. Mitigation: progressive disclosure — overview page shows summary only, detail pages are opt-in.
|
|
3076
|
+
2. **"Asset file becomes unmaintainable"** — ~2800 LOC of embedded HTML/CSS/JS is hard to iterate on. Mitigation: split into `assets/overview.js`, `assets/agents.js`, etc. with a build-free concatenation in server.js.
|
|
3077
|
+
3. **"Nobody uses it"** — CLI users may prefer CLI. Mitigation: dashboard is opt-in, never required. Add `ai-toolkit ui` suggestion to `ai-toolkit install` output for new users.
|
|
3078
|
+
4. **"Action execution feels disconnected"** — SSE console output may confuse users unfamiliar with CLI. Mitigation: rich UI feedback (progress bars, success/error toasts) layered on top of raw output.
|
|
3079
|
+
5. **"Charts look bad without a library"** — Hand-drawn SVG charts may look amateur. Mitigation: keep charts simple (bar + heatmap), use consistent design tokens, test extensively.
|
|
3080
|
+
|
|
3081
|
+
---
|
|
3082
|
+
|
|
3083
|
+
## 10. Next Actions
|
|
3084
|
+
|
|
3085
|
+
1. [ ] Approve plan
|
|
3086
|
+
2. [ ] Create `app/dashboard/server.js` with HTTP server + lifecycle (1.1)
|
|
3087
|
+
3. [ ] Create API layer — read endpoints (1.2)
|
|
3088
|
+
4. [ ] Create CSS design system + client routing (3.1, 3.2)
|
|
3089
|
+
5. [ ] Build Overview page (2.1)
|
|
3090
|
+
6. [ ] Build Agents page (2.2)
|
|
3091
|
+
7. [ ] Build Skills page (2.3)
|
|
3092
|
+
8. [ ] Add action execution API (1.3)
|
|
3093
|
+
9. [ ] Build Hooks, Plugins, Config pages (2.4, 2.5, 2.6)
|
|
3094
|
+
10. [ ] Build Stats page with SVG charts (2.7)
|
|
3095
|
+
11. [ ] Build MCP page (2.8)
|
|
3096
|
+
12. [ ] Register CLI command (4.1)
|
|
3097
|
+
13. [ ] Tests — bats + node:test (4.2)
|
|
3098
|
+
14. [ ] Documentation — all 9 docs per CLAUDE.md rules (4.3)
|
|
3099
|
+
|
|
3100
|
+
---
|
|
3101
|
+
|
|
3102
|
+
## 11. Future (Deferred)
|
|
3103
|
+
|
|
3104
|
+
| Feature | Reason for deferral | Prerequisite |
|
|
3105
|
+
|---------|-------------------|--------------|
|
|
3106
|
+
| Credentials page (2.9) | CLI has no `credentials` commands yet | cloud-security-pack CLI implementation |
|
|
3107
|
+
| `ai-toolkit ui --static` | Generate a single standalone HTML file (no server) — covers 80% of catalog value at 20% complexity | Post-v1 evaluation of actual usage patterns |
|
|
3108
|
+
| Guided onboarding flow | Interactive wizard for new users | Post-v1, based on user feedback |
|
|
3109
|
+
| Light theme toggle | Dark-only for v1 | CSS variable architecture makes this easy later |
|
|
3110
|
+
|
|
3111
|
+
---
|
|
3112
|
+
|
|
3113
|
+
## 10. Market Positioning
|
|
3114
|
+
|
|
3115
|
+
**Target users:**
|
|
3116
|
+
1. **New adopters** — developers evaluating ai-toolkit who want to understand what's included before committing to a profile
|
|
3117
|
+
2. **Team leads** — visual overview of which agents, skills, and plugins are active across team setups
|
|
3118
|
+
3. **Plugin explorers** — developers browsing available plugin packs without memorizing CLI commands
|
|
3119
|
+
4. **Onboarding** — new team members getting oriented with the toolkit's capabilities
|
|
3120
|
+
|
|
3121
|
+
**Competitive advantage:** No existing AI coding toolkit provides a zero-dependency ephemeral web dashboard for visual management. Backstage is 1000x heavier (requires Kubernetes, PostgreSQL). TUI tools (mise, lazygit) lack visual richness. The ephemeral auto-kill design means zero operational burden.
|
|
3122
|
+
|
|
3123
|
+
**Discovery opportunity:** The dashboard serves as a self-documenting catalog — users discover agents and skills they didn't know existed, increasing toolkit utilization.
|
|
3124
|
+
|
|
3125
|
+
---
|
|
3126
|
+
|
|
3127
|
+
## 12. Cross-Plan Dependencies
|
|
3128
|
+
|
|
3129
|
+
This plan shares modification targets with two other proposed plans:
|
|
3130
|
+
|
|
3131
|
+
| Shared File | This Plan | Enterprise Config Plan | Offline SLM Plan |
|
|
3132
|
+
|-------------|-----------|----------------------|-----------------|
|
|
3133
|
+
| `bin/ai-toolkit.js` | +15 LOC (ui command) | +40 LOC (config subcommands) | +10 LOC (compile-slm command) |
|
|
3134
|
+
|
|
3135
|
+
**If Enterprise Config ships first:** Dashboard Config page (2.6) should display `.ai-toolkit.json` / `extends` status and `config diff` output. Add an API endpoint `GET /api/config/extends` reading resolved extends state from `state.json`.
|
|
3136
|
+
|
|
3137
|
+
**If Offline SLM ships first:** Dashboard Overview page (2.1) should display `offline-slm` profile status and link to compiled output. Stats page may show limited data (SLM providers don't emit hook-based stats).
|
|
3138
|
+
|
|
3139
|
+
---
|
|
3140
|
+
|
|
3141
|
+
**Last Updated:** 2026-04-10
|
|
3142
|
+
|
|
3143
|
+
---
|
|
3144
|
+
|
|
3145
|
+
## kb/planning/offline-slm-profile-plan.md
|
|
3146
|
+
|
|
3147
|
+
---
|
|
3148
|
+
title: "Plan: Offline-First SLM Profile — Lightweight Mode for Local Models"
|
|
3149
|
+
category: planning
|
|
3150
|
+
service: ai-toolkit
|
|
3151
|
+
tags:
|
|
3152
|
+
- offline
|
|
3153
|
+
- slm
|
|
3154
|
+
- small-language-models
|
|
3155
|
+
- ollama
|
|
3156
|
+
- lm-studio
|
|
3157
|
+
- profile
|
|
3158
|
+
- context-optimization
|
|
3159
|
+
- privacy
|
|
3160
|
+
doc_type: plan
|
|
3161
|
+
status: proposed
|
|
3162
|
+
created: "2026-04-10"
|
|
3163
|
+
last_updated: "2026-04-10"
|
|
3164
|
+
completion: "0%"
|
|
3165
|
+
description: "Lightweight profile for ai-toolkit optimized for Small Language Models (SLMs) running locally via Ollama, LM Studio, or similar. Compiles a minimal instruction set that fits within 4K-8K system prompt budgets while preserving critical safety guardrails. Targets air-gapped, privacy-first, and cost-sensitive development workflows."
|
|
3166
|
+
---
|
|
3167
|
+
|
|
3168
|
+
# Plan: Offline-First SLM Profile — Lightweight Mode for Local Models
|
|
3169
|
+
|
|
3170
|
+
**Status:** Proposed
|
|
3171
|
+
**Completion:** 0%
|
|
3172
|
+
**Created:** 2026-04-10
|
|
3173
|
+
**Origin:** Enterprise IP security requirements (air-gapped environments), cost-sensitive solo developers, and the growing adoption of local models (Ollama, LM Studio, llamafile). Current toolkit emits 20K+ token system prompts that exceed SLM context windows and degrade small model performance.
|
|
3174
|
+
**Estimated Effort:** 4-5 weeks (1 person)
|
|
3175
|
+
|
|
3176
|
+
---
|
|
3177
|
+
|
|
3178
|
+
## 1. Objective
|
|
3179
|
+
|
|
3180
|
+
Create a `--profile offline-slm` install profile and a `scripts/compile_slm.py` compiler that produces a minimal, high-signal instruction set optimized for Small Language Models (8B-32B parameters). The compiled output preserves critical safety guardrails while stripping agent orchestration, multi-agent coordination, and complex skill routing that SLMs cannot handle.
|
|
3181
|
+
|
|
3182
|
+
**Key design principles:**
|
|
3183
|
+
- **Token budget** — compiled output fits within 4K tokens (system prompt), with optional 8K mode for larger SLMs
|
|
3184
|
+
- **Safety-preserved** — Constitution Articles I-V always included (non-negotiable)
|
|
3185
|
+
- **Single-agent focus** — no multi-agent orchestration, no /swarm, no /teams
|
|
3186
|
+
- **Deterministic compilation** — same input → same output, no LLM involved in compilation
|
|
3187
|
+
- **Model-aware** — detects model size from Ollama API or manual flag and adjusts verbosity
|
|
3188
|
+
- **Platform-agnostic** — outputs plain markdown consumable by any local inference engine
|
|
3189
|
+
- **Hooks stripped** — SLM providers don't support lifecycle hooks; rules compile into system prompt
|
|
3190
|
+
|
|
3191
|
+
---
|
|
3192
|
+
|
|
3193
|
+
## 1a. Functional Requirements
|
|
3194
|
+
|
|
3195
|
+
| ID | Requirement | Priority | Success Metric |
|
|
3196
|
+
|----|-------------|----------|----------------|
|
|
3197
|
+
| FR1 | Token counter (stdlib-only, ±10% accuracy target) | Must | Conservative estimate, no external deps |
|
|
3198
|
+
| FR2 | Component parser + scorer with safety-priority ranking | Must | Constitution=1.0, all components scored |
|
|
3199
|
+
| FR3 | Compression engine with 4 levels (ultra-light, light, standard, extended) | Must | Each level strips progressively less |
|
|
3200
|
+
| FR4 | Budget packer (greedy knapsack by score/size ratio) | Must | Output ≤ budget × 0.95 in all cases |
|
|
3201
|
+
| FR5 | Markdown emitter with safety-first structure | Must | Constitution always first in output |
|
|
3202
|
+
| FR6 | `--profile offline-slm` install integration | Must | `install.py` + `manifest.json` updated |
|
|
3203
|
+
| FR7 | `compile-slm` CLI command with flags | Must | `--budget`, `--model-size`, `--persona`, `--lang`, `--output`, `--format`, `--dry-run` |
|
|
3204
|
+
| FR8 | Constitution always included (non-negotiable) | Must | Compilation fails if constitution exceeds budget alone |
|
|
3205
|
+
| FR9 | Model size detection from Ollama API | Should | Auto-detect with graceful fallback to `14b` |
|
|
3206
|
+
| FR10 | Persona-aware compilation (boost relevant skills) | Should | Persona skills ranked higher |
|
|
3207
|
+
| FR11 | Language-aware compilation (include matching rules only) | Should | Non-matching language rules excluded |
|
|
3208
|
+
| FR12 | Integration guides for 4 platforms (Ollama, LM Studio, Aider, Continue.dev) | Should | Step-by-step setup per platform |
|
|
3209
|
+
| FR13 | Compile quality validator (post-compilation checks) | Should | FAIL on missing constitution, budget exceeded |
|
|
3210
|
+
| FR14 | 4 output formats (raw markdown, Ollama Modelfile, JSON string, Aider-compatible) | Should | Each format usable by target tool |
|
|
3211
|
+
| FR15 | `--dry-run` output showing included components + token counts | Should | Table: component, score, tokens, included? |
|
|
3212
|
+
|
|
3213
|
+
---
|
|
3214
|
+
|
|
3215
|
+
## 2. Architecture Overview
|
|
3216
|
+
|
|
3217
|
+
```
|
|
3218
|
+
ai-toolkit install --profile offline-slm [--model-size 8b|14b|32b|70b]
|
|
3219
|
+
ai-toolkit compile-slm [--budget 4096] [--persona backend-lead] [--lang typescript]
|
|
3220
|
+
|
|
3221
|
+
┌──────────────────────────────────────────────────────────┐
|
|
3222
|
+
│ offline-slm Profile │
|
|
3223
|
+
│ │
|
|
3224
|
+
│ Compiler: scripts/compile_slm.py │
|
|
3225
|
+
│ Input: full toolkit (agents, skills, rules, constitution)│
|
|
3226
|
+
│ Output: single compiled .md file for system prompt │
|
|
3227
|
+
│ │
|
|
3228
|
+
│ Token Budget Tiers: │
|
|
3229
|
+
│ ultra-light (2K) — safety + persona only │
|
|
3230
|
+
│ light (4K) — safety + persona + top skills + rules │
|
|
3231
|
+
│ standard (8K) — safety + persona + full skills + rules │
|
|
3232
|
+
│ extended (16K) — near-full toolkit (for 32B+ models) │
|
|
3233
|
+
│ │
|
|
3234
|
+
│ Output Files: │
|
|
3235
|
+
│ ~/.ai-toolkit/compiled/slm-system-prompt.md │
|
|
3236
|
+
│ ~/.ai-toolkit/compiled/slm-skills-reference.md │
|
|
3237
|
+
│ CLAUDE.md (or equivalent) — auto-generated │
|
|
3238
|
+
│ │
|
|
3239
|
+
│ Integration Targets: │
|
|
3240
|
+
│ Ollama (modelfile SYSTEM directive) │
|
|
3241
|
+
│ LM Studio (system prompt field) │
|
|
3242
|
+
│ llamafile (--system-prompt flag) │
|
|
3243
|
+
│ Open WebUI (system prompt setting) │
|
|
3244
|
+
│ Aider (--system-prompt-file flag) │
|
|
3245
|
+
│ Continue.dev (system prompt in config) │
|
|
3246
|
+
└──────────────────────────────────────────────────────────┘
|
|
3247
|
+
```
|
|
3248
|
+
|
|
3249
|
+
### Compilation Pipeline
|
|
3250
|
+
|
|
3251
|
+
```
|
|
3252
|
+
Full Toolkit (20K+ tokens)
|
|
3253
|
+
│
|
|
3254
|
+
▼
|
|
3255
|
+
┌─────────────────┐
|
|
3256
|
+
│ 1. Parse Phase │ Read all agents, skills, rules, constitution
|
|
3257
|
+
└────────┬────────┘
|
|
3258
|
+
▼
|
|
3259
|
+
┌─────────────────┐
|
|
3260
|
+
│ 2. Rank Phase │ Score components by: safety criticality × usage frequency × persona relevance
|
|
3261
|
+
└────────┬────────┘
|
|
3262
|
+
▼
|
|
3263
|
+
┌─────────────────┐
|
|
3264
|
+
│ 3. Compress Phase│ Strip: examples, rationalization tables, related skills, verbose headers
|
|
3265
|
+
└────────┬────────┘
|
|
3266
|
+
▼
|
|
3267
|
+
┌─────────────────┐
|
|
3268
|
+
│ 4. Budget Phase │ Pack highest-scoring components until token budget reached
|
|
3269
|
+
└────────┬────────┘
|
|
3270
|
+
▼
|
|
3271
|
+
┌─────────────────┐
|
|
3272
|
+
│ 5. Emit Phase │ Write compiled .md + integration instructions
|
|
3273
|
+
└─────────────────┘
|
|
3274
|
+
```
|
|
3275
|
+
|
|
3276
|
+
---
|
|
3277
|
+
|
|
3278
|
+
## 3. Progress Tracking
|
|
3279
|
+
|
|
3280
|
+
| # | Feature | Priority | Status | Est. Time | Notes |
|
|
3281
|
+
|---|---------|----------|--------|-----------|-------|
|
|
3282
|
+
| 1.1 | Token counter (tiktoken-free, word-based estimator) | P0 | Proposed | 0.5d | ~0.75 tokens/word heuristic (stdlib only) |
|
|
3283
|
+
| 1.2 | Component parser + scorer | P0 | Proposed | 2d | Parse frontmatter, score by criticality/frequency/persona |
|
|
3284
|
+
| 1.3 | Compression engine | P0 | Proposed | 2d | Strip examples, rationalization tables, headers |
|
|
3285
|
+
| 1.4 | Budget packer | P0 | Proposed | 1d | Greedy knapsack by score/size ratio |
|
|
3286
|
+
| 1.5 | Emitter (markdown output) | P0 | Proposed | 1d | Clean compiled .md file |
|
|
3287
|
+
| 2.1 | Profile integration (`--profile offline-slm`) | P0 | Proposed | 1.5d | Install.py + manifest.json + state.json |
|
|
3288
|
+
| 2.2 | CLI command (`ai-toolkit compile-slm`) | P0 | Proposed | 1d | Standalone compilation with flags |
|
|
3289
|
+
| 2.3 | Model size detection (Ollama API) | P1 | Proposed | 1d | Auto-detect model params from `ollama list` |
|
|
3290
|
+
| 2.4 | Persona-aware compilation | P1 | Proposed | 1.5d | Boost persona-relevant skills in ranking |
|
|
3291
|
+
| 2.5 | Language-aware compilation | P1 | Proposed | 1d | Include only matching language rules |
|
|
3292
|
+
| 3.1 | Integration guides (Ollama, LM Studio, Aider, Continue) | P1 | Proposed | 1.5d | Step-by-step per platform |
|
|
3293
|
+
| 3.2 | Compile quality validator | P1 | Proposed | 1d | Verify output covers constitution, fits budget |
|
|
3294
|
+
| 3.3 | Tests | P1 | Proposed | 3d | Unit: compilation determinism, budget compliance, 4 compression levels × 4 output formats, constitution guard. Integration: `compile-slm --model-size 8b`, verify output fits 2048 tokens + constitution present end-to-end. Target: 40+ tests |
|
|
3295
|
+
| 3.4 | Documentation | P1 | Proposed | 2.5d | All 9 docs per CLAUDE.md: README, CLAUDE.md, ARCHITECTURE.md, package.json, llms.txt, llms-full.txt, AGENTS.md, skills-catalog.md, architecture-overview.md + integration guide |
|
|
3296
|
+
|
|
3297
|
+
**Phasing:**
|
|
3298
|
+
- **Phase 1 (week 1-2):** Compiler — parser, scorer, compressor, packer, emitter
|
|
3299
|
+
- **Phase 2 (week 2-3):** Integration — profile, CLI, model detection, persona/language awareness
|
|
3300
|
+
- **Phase 3 (week 3-4):** Polish — integration guides, validator, tests, documentation
|
|
3301
|
+
|
|
3302
|
+
> **Demand validation gate:** Ship Phase 1 + basic Phase 2 (compiler + profile + CLI with `--budget` and `--model-size` flags) as MVP. Test with 3 real models (8B, 14B, 32B). Only build persona/language-aware compilation and platform-specific integration guides if MVP validation confirms output quality.
|
|
3303
|
+
|
|
3304
|
+
---
|
|
3305
|
+
|
|
3306
|
+
## 4. Dependency Graph
|
|
3307
|
+
|
|
3308
|
+
```
|
|
3309
|
+
Phase 1: Compiler (week 1-2)
|
|
3310
|
+
============================
|
|
3311
|
+
Token counter (1.1) ────┐
|
|
3312
|
+
├──► Compression engine (1.3) ──► Budget packer (1.4) ──► Emitter (1.5)
|
|
3313
|
+
Component parser (1.2) ──┘
|
|
3314
|
+
|
|
3315
|
+
Phase 2: Integration (week 2-3)
|
|
3316
|
+
================================
|
|
3317
|
+
Profile integration (2.1) ──┐
|
|
3318
|
+
├──► CLI command (2.2)
|
|
3319
|
+
Model detection (2.3) ──────┤
|
|
3320
|
+
Persona-aware (2.4) ────────┤
|
|
3321
|
+
Language-aware (2.5) ────────┘
|
|
3322
|
+
|
|
3323
|
+
Phase 3: Polish (week 3-4)
|
|
3324
|
+
===========================
|
|
3325
|
+
├──► Integration guides (3.1)
|
|
3326
|
+
├──► Compile validator (3.2)
|
|
3327
|
+
└──► Tests + docs (3.3, 3.4)
|
|
3328
|
+
```
|
|
3329
|
+
|
|
3330
|
+
---
|
|
3331
|
+
|
|
3332
|
+
## 5. Detailed Implementation
|
|
3333
|
+
|
|
3334
|
+
### Phase 1: Compiler Engine (week 1-2)
|
|
3335
|
+
|
|
3336
|
+
#### 1.1 Token Counter
|
|
3337
|
+
|
|
3338
|
+
**Stdlib-only token estimation** — no tiktoken, no external dependencies.
|
|
3339
|
+
|
|
3340
|
+
```python
|
|
3341
|
+
def estimate_tokens(text: str) -> int:
|
|
3342
|
+
"""Estimate token count from text without external dependencies.
|
|
3343
|
+
|
|
3344
|
+
Uses two heuristics and returns the higher (conservative) estimate:
|
|
3345
|
+
1. Word-based: ~0.75 tokens/word for English prose
|
|
3346
|
+
2. Char-based: ~1 token per 4 chars (more accurate for code-heavy content)
|
|
3347
|
+
|
|
3348
|
+
Accuracy target: ±10% vs tiktoken cl100k_base. To be validated on 50 toolkit files before shipping.
|
|
3349
|
+
"""
|
|
3350
|
+
word_est = int(len(text.split()) * 0.75)
|
|
3351
|
+
char_est = len(text) // 4
|
|
3352
|
+
# Code blocks have higher token density — adjust
|
|
3353
|
+
code_blocks = text.count('```')
|
|
3354
|
+
code_penalty = code_blocks * 15
|
|
3355
|
+
return max(word_est, char_est) + code_penalty
|
|
3356
|
+
```
|
|
3357
|
+
|
|
3358
|
+
**Why not tiktoken:** tiktoken requires a C extension and network download of the BPE file. This violates the stdlib-only constraint and fails in air-gapped environments (which is literally the target audience for this feature).
|
|
3359
|
+
|
|
3360
|
+
**Accuracy target:** ±10% vs tiktoken cl100k_base. Using `max(word, char)` gives a conservative estimate. We pack to budget × 0.95 (5% safety margin) to absorb estimation error.
|
|
3361
|
+
|
|
3362
|
+
---
|
|
3363
|
+
|
|
3364
|
+
#### 1.2 Component Parser + Scorer
|
|
3365
|
+
|
|
3366
|
+
**Parse all toolkit components into a unified scoring table:**
|
|
3367
|
+
|
|
3368
|
+
```python
|
|
3369
|
+
@dataclass
|
|
3370
|
+
class Component:
|
|
3371
|
+
name: str
|
|
3372
|
+
type: str # 'constitution', 'agent', 'skill', 'rule', 'hook-equivalent'
|
|
3373
|
+
source_file: str
|
|
3374
|
+
full_text: str
|
|
3375
|
+
compressed_text: str # after stripping (populated by compressor)
|
|
3376
|
+
tokens_full: int
|
|
3377
|
+
tokens_compressed: int
|
|
3378
|
+
score: float # 0.0 - 1.0
|
|
3379
|
+
|
|
3380
|
+
# Scoring factors
|
|
3381
|
+
safety_criticality: float # 0.0-1.0 (constitution=1.0, guard hooks=0.9)
|
|
3382
|
+
usage_frequency: float # 0.0-1.0 (from stats.json, normalized)
|
|
3383
|
+
persona_relevance: float # 0.0-1.0 (match against active persona)
|
|
3384
|
+
language_relevance: float # 0.0-1.0 (match against project language)
|
|
3385
|
+
```
|
|
3386
|
+
|
|
3387
|
+
**Scoring formula:**
|
|
3388
|
+
```python
|
|
3389
|
+
score = (
|
|
3390
|
+
safety_criticality * 0.40 + # Safety always dominates — non-negotiable content gets priority
|
|
3391
|
+
usage_frequency * 0.25 + # Frequently used = valuable — from stats.json invocation counts
|
|
3392
|
+
persona_relevance * 0.20 + # Persona-matched = valuable — e.g. backend-lead boosts API skills
|
|
3393
|
+
language_relevance * 0.15 # Language-matched = contextual — include only relevant rules
|
|
3394
|
+
)
|
|
3395
|
+
# Weight rationale: safety must dominate (0.40) to guarantee constitution + guard rules always fit.
|
|
3396
|
+
# Usage + persona (0.45 combined) ensure the most practical content fills remaining budget.
|
|
3397
|
+
# Language (0.15) is a tiebreaker — most projects use 1-2 languages.
|
|
3398
|
+
# Weights are compile-time constants in v1. If empirical testing (5 standard tasks across
|
|
3399
|
+
# 3 model sizes) shows suboptimal results, expose as --score-weights flag in v2.
|
|
3400
|
+
```
|
|
3401
|
+
|
|
3402
|
+
**Fixed-score components (always included):**
|
|
3403
|
+
|
|
3404
|
+
| Component | Score | Reason |
|
|
3405
|
+
|-----------|-------|--------|
|
|
3406
|
+
| Constitution (Articles I-V) | 1.0 | Non-negotiable safety |
|
|
3407
|
+
| Guard hooks (destructive, path) | 0.95 | Core safety rules (compiled as text, not hooks) |
|
|
3408
|
+
| Active persona definition | 0.90 | User-selected identity |
|
|
3409
|
+
| Active language rules | 0.85 | Project-specific quality gates |
|
|
3410
|
+
|
|
3411
|
+
**Dynamic-score components:**
|
|
3412
|
+
|
|
3413
|
+
| Component | Base Score | Adjusted By |
|
|
3414
|
+
|-----------|-----------|-------------|
|
|
3415
|
+
| Individual skills | 0.3-0.7 | Usage frequency + persona fit |
|
|
3416
|
+
| Agent definitions | 0.2-0.6 | Persona relevance (only 1 agent in SLM mode) |
|
|
3417
|
+
| Knowledge skills | 0.2-0.5 | Language match + persona match |
|
|
3418
|
+
| Iron Law rules | 0.7 | Always high (quality enforcement) |
|
|
3419
|
+
|
|
3420
|
+
---
|
|
3421
|
+
|
|
3422
|
+
#### 1.3 Compression Engine
|
|
3423
|
+
|
|
3424
|
+
**Strip low-signal content while preserving semantics:**
|
|
3425
|
+
|
|
3426
|
+
| Strip Target | Savings (est.) | Example |
|
|
3427
|
+
|-------------|---------------|---------|
|
|
3428
|
+
| `## Common Rationalizations` tables | 200-400 tokens/skill | 15 skills have these tables |
|
|
3429
|
+
| `## Related Skills` sections | 50-100 tokens/skill | Routing not useful for SLMs |
|
|
3430
|
+
| `## Verification Checklist` (keep 1-liner summary) | 100-200 tokens/agent | Compress to "Verify: tests pass, no placeholders" |
|
|
3431
|
+
| Markdown headers (collapse hierarchy) | 20-50 tokens/file | `### 2.1.3 Sub-feature` → plain paragraph |
|
|
3432
|
+
| Example code blocks (keep first, strip rest) | 100-500 tokens/skill | Keep 1 example max |
|
|
3433
|
+
| Frontmatter (YAML) | 50-100 tokens/file | Strip entirely from compiled output |
|
|
3434
|
+
| Agent `## Allowed CLI Commands` lists | 200-400 tokens/agent | Not needed when agent won't execute them |
|
|
3435
|
+
| Multi-agent coordination instructions | 300-500 tokens | SLM = single agent, no /orchestrate |
|
|
3436
|
+
| Effort-based budgeting rules | 100 tokens | SLM doesn't manage budgets |
|
|
3437
|
+
|
|
3438
|
+
**Compression levels:**
|
|
3439
|
+
|
|
3440
|
+
```python
|
|
3441
|
+
COMPRESSION_LEVELS = {
|
|
3442
|
+
'ultra-light': {
|
|
3443
|
+
'strip_examples': True,
|
|
3444
|
+
'strip_rationalizations': True,
|
|
3445
|
+
'strip_related_skills': True,
|
|
3446
|
+
'strip_verification': True,
|
|
3447
|
+
'strip_agent_commands': True,
|
|
3448
|
+
'strip_multi_agent': True,
|
|
3449
|
+
'max_skills': 5, # Only top 5 skills by score
|
|
3450
|
+
'max_agents': 0, # No agent definitions (persona only)
|
|
3451
|
+
'include_rules': False,
|
|
3452
|
+
},
|
|
3453
|
+
'light': {
|
|
3454
|
+
'strip_examples': True,
|
|
3455
|
+
'strip_rationalizations': True,
|
|
3456
|
+
'strip_related_skills': True,
|
|
3457
|
+
'strip_verification': 'summary', # 1-liner
|
|
3458
|
+
'strip_agent_commands': True,
|
|
3459
|
+
'strip_multi_agent': True,
|
|
3460
|
+
'max_skills': 10,
|
|
3461
|
+
'max_agents': 1, # Persona agent only
|
|
3462
|
+
'include_rules': True,
|
|
3463
|
+
},
|
|
3464
|
+
'standard': {
|
|
3465
|
+
'strip_examples': 'first-only', # Keep 1 example
|
|
3466
|
+
'strip_rationalizations': True,
|
|
3467
|
+
'strip_related_skills': True,
|
|
3468
|
+
'strip_verification': 'summary',
|
|
3469
|
+
'strip_agent_commands': True,
|
|
3470
|
+
'strip_multi_agent': True,
|
|
3471
|
+
'max_skills': 20,
|
|
3472
|
+
'max_agents': 3,
|
|
3473
|
+
'include_rules': True,
|
|
3474
|
+
},
|
|
3475
|
+
'extended': {
|
|
3476
|
+
'strip_examples': 'first-only',
|
|
3477
|
+
'strip_rationalizations': 'first-only',
|
|
3478
|
+
'strip_related_skills': False,
|
|
3479
|
+
'strip_verification': False,
|
|
3480
|
+
'strip_agent_commands': False,
|
|
3481
|
+
'strip_multi_agent': True, # Still stripped for SLMs
|
|
3482
|
+
'max_skills': 40,
|
|
3483
|
+
'max_agents': 5,
|
|
3484
|
+
'include_rules': True,
|
|
3485
|
+
},
|
|
3486
|
+
}
|
|
3487
|
+
```
|
|
3488
|
+
|
|
3489
|
+
---
|
|
3490
|
+
|
|
3491
|
+
#### 1.4 Budget Packer
|
|
3492
|
+
|
|
3493
|
+
**Greedy knapsack algorithm:** Sort components by `score / compressed_tokens` ratio (value density), pack until budget exhausted.
|
|
3494
|
+
|
|
3495
|
+
```python
|
|
3496
|
+
def pack_components(components: list[Component], budget: int) -> list[Component]:
|
|
3497
|
+
"""Pack highest-value components into token budget."""
|
|
3498
|
+
# Fixed components always included (constitution, persona, language rules)
|
|
3499
|
+
fixed = [c for c in components if c.score >= 0.85]
|
|
3500
|
+
remaining_budget = budget - sum(c.tokens_compressed for c in fixed)
|
|
3501
|
+
|
|
3502
|
+
# Sort remaining by value density
|
|
3503
|
+
dynamic = sorted(
|
|
3504
|
+
[c for c in components if c.score < 0.85],
|
|
3505
|
+
key=lambda c: c.score / max(c.tokens_compressed, 1),
|
|
3506
|
+
reverse=True
|
|
3507
|
+
)
|
|
3508
|
+
|
|
3509
|
+
packed = list(fixed)
|
|
3510
|
+
for comp in dynamic:
|
|
3511
|
+
if comp.tokens_compressed <= remaining_budget:
|
|
3512
|
+
packed.append(comp)
|
|
3513
|
+
remaining_budget -= comp.tokens_compressed
|
|
3514
|
+
|
|
3515
|
+
return packed
|
|
3516
|
+
```
|
|
3517
|
+
|
|
3518
|
+
**Budget validation:** After packing, verify total tokens ≤ budget × 0.95 (5% safety margin for tokenizer estimation error).
|
|
3519
|
+
|
|
3520
|
+
**Constitution budget guard:** Before packing dynamic components, verify that fixed components (constitution + persona + language rules) fit within the budget. If `sum(fixed.tokens_compressed) > budget`, fail with: `"Constitution + safety rules alone exceed {budget} token budget. Minimum safe budget: {required}. Use --budget {required} or higher."` This prevents silent omission of safety-critical content.
|
|
3521
|
+
|
|
3522
|
+
---
|
|
3523
|
+
|
|
3524
|
+
#### 1.5 Emitter
|
|
3525
|
+
|
|
3526
|
+
**Output:** Single markdown file structured for maximum SLM comprehension.
|
|
3527
|
+
|
|
3528
|
+
```markdown
|
|
3529
|
+
# AI Coding Assistant — System Instructions
|
|
3530
|
+
|
|
3531
|
+
## Safety Rules (MANDATORY)
|
|
3532
|
+
[Compiled constitution — always first, highest attention position]
|
|
3533
|
+
|
|
3534
|
+
## Your Identity
|
|
3535
|
+
[Compiled persona — who you are, what you focus on]
|
|
3536
|
+
|
|
3537
|
+
## Coding Standards
|
|
3538
|
+
[Compiled language rules — active language only]
|
|
3539
|
+
|
|
3540
|
+
## Key Skills
|
|
3541
|
+
[Top N skill summaries — compressed, actionable]
|
|
3542
|
+
|
|
3543
|
+
## Quality Checklist
|
|
3544
|
+
[Compiled from Iron Laws + verification — bullet points only]
|
|
3545
|
+
```
|
|
3546
|
+
|
|
3547
|
+
**Why this structure:**
|
|
3548
|
+
- Safety first = maximum attention weight in transformer architecture
|
|
3549
|
+
- Identity second = establishes persona before task instructions
|
|
3550
|
+
- Standards = project-specific rules that shape code output
|
|
3551
|
+
- Skills at end = reference material, lower attention needed
|
|
3552
|
+
|
|
3553
|
+
---
|
|
3554
|
+
|
|
3555
|
+
### Phase 2: Integration (week 2-3)
|
|
3556
|
+
|
|
3557
|
+
#### 2.1 Profile Integration
|
|
3558
|
+
|
|
3559
|
+
**manifest.json addition:**
|
|
3560
|
+
```json
|
|
3561
|
+
{
|
|
3562
|
+
"profiles": {
|
|
3563
|
+
"offline-slm": ["core"],
|
|
3564
|
+
"offline-slm-extended": ["core", "agents"]
|
|
3565
|
+
}
|
|
3566
|
+
}
|
|
3567
|
+
```
|
|
3568
|
+
|
|
3569
|
+
**Install behavior:**
|
|
3570
|
+
```bash
|
|
3571
|
+
ai-toolkit install --profile offline-slm
|
|
3572
|
+
|
|
3573
|
+
# What happens:
|
|
3574
|
+
# 1. Standard install of core components
|
|
3575
|
+
# 2. Runs compile_slm.py with auto-detected settings
|
|
3576
|
+
# 3. Writes compiled output to ~/.ai-toolkit/compiled/
|
|
3577
|
+
# 4. Generates integration instructions for detected local model tools
|
|
3578
|
+
# 5. state.json records profile as "offline-slm"
|
|
3579
|
+
```
|
|
3580
|
+
|
|
3581
|
+
**No hooks installed:** SLM providers (Ollama, LM Studio) don't support lifecycle hooks. The critical hook behavior (destructive command guard, path guard) is compiled into the system prompt text as rules.
|
|
3582
|
+
|
|
3583
|
+
---
|
|
3584
|
+
|
|
3585
|
+
#### 2.2 CLI Command
|
|
3586
|
+
|
|
3587
|
+
```bash
|
|
3588
|
+
ai-toolkit compile-slm # auto-detect model, default budget
|
|
3589
|
+
ai-toolkit compile-slm --budget 4096 # explicit token budget
|
|
3590
|
+
ai-toolkit compile-slm --budget 8192 --persona backend-lead # persona + budget
|
|
3591
|
+
ai-toolkit compile-slm --model-size 8b # auto-select budget for 8B model
|
|
3592
|
+
ai-toolkit compile-slm --model-size 32b # auto-select budget for 32B model
|
|
3593
|
+
ai-toolkit compile-slm --lang typescript,python # include specific language rules
|
|
3594
|
+
ai-toolkit compile-slm --output ./my-system-prompt.md # custom output path
|
|
3595
|
+
ai-toolkit compile-slm --dry-run # show what would be included + token counts (table format below)
|
|
3596
|
+
|
|
3597
|
+
# --dry-run output format:
|
|
3598
|
+
# Budget: 4096 tokens | Level: light | Persona: backend-lead
|
|
3599
|
+
# ┌────────────────────────────┬──────────┬────────┬──────────┐
|
|
3600
|
+
# │ Component │ Score │ Tokens │ Included │
|
|
3601
|
+
# ├────────────────────────────┼──────────┼────────┼──────────┤
|
|
3602
|
+
# │ Constitution (Articles I-V)│ 1.00 │ 420 │ YES │
|
|
3603
|
+
# │ Persona: backend-lead │ 0.90 │ 180 │ YES │
|
|
3604
|
+
# │ Rule: coding-style │ 0.85 │ 310 │ YES │
|
|
3605
|
+
# │ Skill: /review │ 0.68 │ 290 │ YES │
|
|
3606
|
+
# │ ... │ ... │ ... │ ... │
|
|
3607
|
+
# │ Skill: /deploy │ 0.22 │ 350 │ NO (budget)│
|
|
3608
|
+
# └────────────────────────────┴──────────┴────────┴──────────┘
|
|
3609
|
+
# Total: 3,840 / 4,096 tokens (93.7% utilization)
|
|
3610
|
+
ai-toolkit compile-slm --format ollama # output as Ollama Modelfile SYSTEM block
|
|
3611
|
+
ai-toolkit compile-slm --format json-string # JSON-escaped string (for config files)
|
|
3612
|
+
ai-toolkit compile-slm --format raw # plain markdown (default)
|
|
3613
|
+
```
|
|
3614
|
+
|
|
3615
|
+
**Model size → budget mapping:**
|
|
3616
|
+
|
|
3617
|
+
Note: budget is about *effective instruction following capacity*, not context window. A 128K-context 8B model can *hold* 16K system prompt tokens, but cannot *follow* them reliably. Empirically, SLMs degrade when system prompt exceeds ~10-15% of their effective capacity.
|
|
3618
|
+
|
|
3619
|
+
```python
|
|
3620
|
+
MODEL_BUDGETS = {
|
|
3621
|
+
'7b': {'budget': 2048, 'level': 'ultra-light'}, # Llama 3.1 8B, Mistral 7B
|
|
3622
|
+
'8b': {'budget': 2048, 'level': 'ultra-light'},
|
|
3623
|
+
'14b': {'budget': 4096, 'level': 'light'}, # Qwen 2.5 14B, Phi-3 14B
|
|
3624
|
+
'32b': {'budget': 8192, 'level': 'standard'}, # Qwen 2.5 32B, Mixtral 8x7B
|
|
3625
|
+
'70b': {'budget': 16384, 'level': 'extended'}, # Llama 3.1 70B
|
|
3626
|
+
}
|
|
3627
|
+
```
|
|
3628
|
+
|
|
3629
|
+
---
|
|
3630
|
+
|
|
3631
|
+
#### 2.3 Model Size Detection
|
|
3632
|
+
|
|
3633
|
+
**Auto-detect from Ollama:**
|
|
3634
|
+
```python
|
|
3635
|
+
def detect_model_size() -> str | None:
|
|
3636
|
+
"""Detect running model size from Ollama API."""
|
|
3637
|
+
try:
|
|
3638
|
+
# curl http://localhost:11434/api/tags
|
|
3639
|
+
resp = urllib.request.urlopen('http://localhost:11434/api/tags', timeout=2)
|
|
3640
|
+
data = json.loads(resp.read())
|
|
3641
|
+
models = data.get('models', [])
|
|
3642
|
+
if models:
|
|
3643
|
+
# Extract parameter count from model name: "llama3.1:8b" → "8b"
|
|
3644
|
+
latest = models[0]['name']
|
|
3645
|
+
match = re.search(r'(\d+)[bB]', latest)
|
|
3646
|
+
if match:
|
|
3647
|
+
return match.group(0).lower()
|
|
3648
|
+
except (urllib.error.URLError, TimeoutError, json.JSONDecodeError):
|
|
3649
|
+
pass
|
|
3650
|
+
return None
|
|
3651
|
+
```
|
|
3652
|
+
|
|
3653
|
+
**Fallback:** If no model detected, use `14b` defaults (4K budget, light compression). User can override with `--model-size`.
|
|
3654
|
+
|
|
3655
|
+
---
|
|
3656
|
+
|
|
3657
|
+
### Phase 3: Polish (week 3-4)
|
|
3658
|
+
|
|
3659
|
+
#### 3.1 Integration Guides
|
|
3660
|
+
|
|
3661
|
+
Per-platform setup instructions generated by the compiler.
|
|
3662
|
+
|
|
3663
|
+
**Ollama:**
|
|
3664
|
+
```bash
|
|
3665
|
+
# 1. Compile system prompt
|
|
3666
|
+
ai-toolkit compile-slm --format ollama --model-size 8b > Modelfile.ai-toolkit
|
|
3667
|
+
|
|
3668
|
+
# 2. Create custom model
|
|
3669
|
+
ollama create my-coder -f Modelfile.ai-toolkit
|
|
3670
|
+
|
|
3671
|
+
# 3. Use
|
|
3672
|
+
ollama run my-coder "implement the payment API"
|
|
3673
|
+
```
|
|
3674
|
+
|
|
3675
|
+
**LM Studio:**
|
|
3676
|
+
```
|
|
3677
|
+
1. ai-toolkit compile-slm --model-size 14b
|
|
3678
|
+
2. Open LM Studio → Chat → System Prompt
|
|
3679
|
+
3. Paste contents of ~/.ai-toolkit/compiled/slm-system-prompt.md
|
|
3680
|
+
```
|
|
3681
|
+
|
|
3682
|
+
**Aider:**
|
|
3683
|
+
```bash
|
|
3684
|
+
ai-toolkit compile-slm --output .aider.system-prompt.md --model-size 32b
|
|
3685
|
+
aider --model ollama/qwen2.5-coder:32b --system-prompt-file .aider.system-prompt.md
|
|
3686
|
+
```
|
|
3687
|
+
|
|
3688
|
+
**Continue.dev:**
|
|
3689
|
+
```bash
|
|
3690
|
+
# 1. Compile to a local file
|
|
3691
|
+
ai-toolkit compile-slm --model-size 14b
|
|
3692
|
+
|
|
3693
|
+
# 2. In .continue/config.json, paste the compiled content into systemMessage
|
|
3694
|
+
# (Continue.dev does not support file references — content must be inline)
|
|
3695
|
+
# Use: ai-toolkit compile-slm --format json-string to get escaped output
|
|
3696
|
+
```
|
|
3697
|
+
|
|
3698
|
+
---
|
|
3699
|
+
|
|
3700
|
+
#### 3.2 Compile Quality Validator
|
|
3701
|
+
|
|
3702
|
+
Post-compilation checks:
|
|
3703
|
+
|
|
3704
|
+
| Check | Severity | Description |
|
|
3705
|
+
|-------|----------|-------------|
|
|
3706
|
+
| Constitution present | FAIL | Articles I-V must be in output |
|
|
3707
|
+
| Budget exceeded | FAIL | Token count > budget |
|
|
3708
|
+
| Persona missing (when specified) | WARN | Persona definition not included |
|
|
3709
|
+
| No language rules included | WARN | Project language not detected |
|
|
3710
|
+
| Less than 3 skills included | WARN | Very minimal — may be too sparse |
|
|
3711
|
+
| Output empty | FAIL | Compilation produced no content |
|
|
3712
|
+
|
|
3713
|
+
---
|
|
3714
|
+
|
|
3715
|
+
## 6. File Summary
|
|
3716
|
+
|
|
3717
|
+
| File | Action | LOC (est.) | Description |
|
|
3718
|
+
|------|--------|------------|-------------|
|
|
3719
|
+
| `scripts/compile_slm.py` | CREATE | ~500 | Main compiler — orchestrates pipeline: parse → score → compress → pack → emit. Contains `Component` dataclass, scorer, and budget packer |
|
|
3720
|
+
| `scripts/slm_token_counter.py` | CREATE | ~50 | Token estimation (stdlib only) — `estimate_tokens()` function used by compiler and validator |
|
|
3721
|
+
| `scripts/slm_compression.py` | CREATE | ~300 | Compression engine — strip/summarize functions per content type, compression level configs (`COMPRESSION_LEVELS` dict) |
|
|
3722
|
+
| `scripts/slm_integration.py` | CREATE | ~150 | Platform-specific output formatters — Ollama Modelfile, JSON-escaped string, raw markdown, Aider-compatible |
|
|
3723
|
+
| `bin/ai-toolkit.js` | EDIT | +10 | Register `compile-slm` command |
|
|
3724
|
+
| `scripts/install.py` | EDIT | +30 | Handle `--profile offline-slm` |
|
|
3725
|
+
| `manifest.json` | EDIT | +5 | Add offline-slm profile |
|
|
3726
|
+
| `kb/reference/offline-slm-guide.md` | CREATE | ~200 | Integration guides for all platforms |
|
|
3727
|
+
| `tests/test_compile_slm.bats` | CREATE | ~150 | Compilation tests |
|
|
3728
|
+
| `tests/test_slm_budgets.bats` | CREATE | ~80 | Budget compliance tests |
|
|
3729
|
+
| **Total** | | **~1575** | |
|
|
3730
|
+
|
|
3731
|
+
---
|
|
3732
|
+
|
|
3733
|
+
## 6a. Non-Functional Requirements
|
|
3734
|
+
|
|
3735
|
+
| Category | Requirement |
|
|
3736
|
+
|----------|-------------|
|
|
3737
|
+
| **Performance** | Compilation < 2 seconds. No network calls during compilation (all data local). |
|
|
3738
|
+
| **Accuracy** | Token estimation ±10% vs tiktoken cl100k_base. Budget compliance: output ≤ budget × 0.95. |
|
|
3739
|
+
| **Determinism** | Same input (agents, skills, rules, persona, language, budget) → identical output. No randomness. |
|
|
3740
|
+
| **Security** | Constitution Articles I-V always present in output — compilation fails if they exceed budget alone. |
|
|
3741
|
+
| **Offline** | Zero network dependencies. Ollama auto-detection gracefully fails to manual fallback. |
|
|
3742
|
+
| **Portability** | Output is plain markdown — consumable by any tool accepting a system prompt string/file. |
|
|
3743
|
+
| **Quality gates** | `ruff check scripts/compile_slm.py scripts/slm_*.py` (0 errors), `mypy --strict scripts/compile_slm.py scripts/slm_*.py` (0 errors). Run before every commit. |
|
|
3744
|
+
| **Type safety** | 100% public API type hints (all function signatures). `Component` dataclass fully typed. Scoring functions use typed parameters, not bare `dict`. |
|
|
3745
|
+
|
|
3746
|
+
---
|
|
3747
|
+
|
|
3748
|
+
## 6b. Cache Invalidation & Recompile Triggers
|
|
3749
|
+
|
|
3750
|
+
Compiled output (`~/.ai-toolkit/compiled/slm-system-prompt.md`) is a **derived artifact** — it must be recompiled when inputs change:
|
|
3751
|
+
|
|
3752
|
+
| Trigger | Action |
|
|
3753
|
+
|---------|--------|
|
|
3754
|
+
| `ai-toolkit update` | Auto-recompile if profile is `offline-slm` |
|
|
3755
|
+
| `ai-toolkit install --profile offline-slm` | Always compile |
|
|
3756
|
+
| Agent/skill/rule files changed (detected via mtime) | Warn: "Compiled SLM prompt may be stale. Run `ai-toolkit compile-slm`" |
|
|
3757
|
+
| Manual `ai-toolkit compile-slm` | Always recompile |
|
|
3758
|
+
|
|
3759
|
+
Compiled output includes a header comment: `<!-- Compiled: 2026-04-10T10:30:00Z | Budget: 4096 | Level: light | Persona: backend-lead -->` for staleness detection.
|
|
3760
|
+
|
|
3761
|
+
---
|
|
3762
|
+
|
|
3763
|
+
## 6b-bis. Rollback & Removal
|
|
3764
|
+
|
|
3765
|
+
The offline-slm feature is purely additive — removing it is trivial:
|
|
3766
|
+
1. Delete `scripts/compile_slm.py`, `scripts/slm_token_counter.py`, `scripts/slm_compression.py`, `scripts/slm_integration.py`
|
|
3767
|
+
2. Remove `"offline-slm"` and `"offline-slm-extended"` from `manifest.json` profiles
|
|
3768
|
+
3. Remove `compile-slm` from `SCRIPT_COMMANDS` in `bin/ai-toolkit.js`
|
|
3769
|
+
4. Delete `~/.ai-toolkit/compiled/` directory (user-side)
|
|
3770
|
+
5. No hooks, no state files, no config entries to clean up
|
|
3771
|
+
|
|
3772
|
+
---
|
|
3773
|
+
|
|
3774
|
+
## 6c. Quality Gate Degradation Notice
|
|
3775
|
+
|
|
3776
|
+
**Important:** The `offline-slm` profile strips lifecycle hooks because SLM providers don't support them. This means:
|
|
3777
|
+
|
|
3778
|
+
- No pre-commit quality check (ruff/tsc/mypy)
|
|
3779
|
+
- No destructive command interception (guard hooks)
|
|
3780
|
+
- No session context preservation
|
|
3781
|
+
|
|
3782
|
+
Guard hook behavior is **compiled into the system prompt as text rules** — the SLM is *instructed* not to run destructive commands, but unlike hook-based enforcement, this is advisory, not blocking.
|
|
3783
|
+
|
|
3784
|
+
Documentation must clearly state: **"SLM mode trades enforcement for guidance. Safety rules are present but not machine-enforced."**
|
|
3785
|
+
|
|
3786
|
+
For teams needing enforcement, recommend `--profile offline-slm` combined with a Git pre-commit hook (`.git/hooks/pre-commit`) that runs lint/type-check independently of the AI tool.
|
|
3787
|
+
|
|
3788
|
+
---
|
|
3789
|
+
|
|
3790
|
+
## 7. Success Criteria (Overall)
|
|
3791
|
+
|
|
3792
|
+
| Metric | Target |
|
|
3793
|
+
|--------|--------|
|
|
3794
|
+
| Budget tiers | 4 (ultra-light 2K, light 4K, standard 8K, extended 16K) |
|
|
3795
|
+
| Token budget compliance | 100% (output ≤ budget in all cases) |
|
|
3796
|
+
| Constitution inclusion | 100% (always present, all 5 articles) |
|
|
3797
|
+
| Compilation time | < 2 seconds |
|
|
3798
|
+
| Model size auto-detection | Ollama API (with graceful fallback) |
|
|
3799
|
+
| Output formats | 4 (raw markdown, Ollama Modelfile, JSON-escaped string, Aider-compatible) |
|
|
3800
|
+
| Integration guides | 4 platforms (Ollama, LM Studio, Aider, Continue.dev) |
|
|
3801
|
+
| Persona support | 4 personas (backend-lead, frontend-lead, devops-eng, junior-dev) |
|
|
3802
|
+
| Language rule support | 13 languages (all existing rules) |
|
|
3803
|
+
| External dependencies | 0 (stdlib Python only) |
|
|
3804
|
+
| Tests | 40+ |
|
|
3805
|
+
|
|
3806
|
+
---
|
|
3807
|
+
|
|
3808
|
+
## 8. Risks and Mitigation
|
|
3809
|
+
|
|
3810
|
+
| Risk | Probability | Impact | Mitigation |
|
|
3811
|
+
|------|-------------|--------|------------|
|
|
3812
|
+
| Token estimation too inaccurate | Medium | Medium | Validate against tiktoken on 50 files, target ±10% |
|
|
3813
|
+
| Compiled prompt too compressed — loses meaning | Medium | High | Validate with 8B model on 5 standard tasks; if quality drops, increase minimum budget |
|
|
3814
|
+
| Ollama API changes | Low | Low | Graceful fallback to manual `--model-size` |
|
|
3815
|
+
| User expects full toolkit features with SLM | Medium | Medium | Clear documentation: "SLM mode = safety + coding standards, not multi-agent orchestration" |
|
|
3816
|
+
| Air-gapped environment can't run `ai-toolkit compile-slm` | Low | Medium | Pre-compile during `install` when network is available; compiled output is self-contained |
|
|
3817
|
+
| Constitution text changes break compiled cache | Low | Low | Recompile on every `ai-toolkit update` |
|
|
3818
|
+
|
|
3819
|
+
---
|
|
3820
|
+
|
|
3821
|
+
## 9. Pre-Mortem
|
|
3822
|
+
|
|
3823
|
+
1. **"Compiled prompt is too generic"** — without the full agent definitions, the SLM may produce generic code that doesn't match project conventions. Mitigation: Language rules and persona have highest priority after constitution — they provide project-specific context.
|
|
3824
|
+
2. **"Users expect /review to work with 8B model"** — Skill invocations won't be available via SLM providers that lack hook support. Mitigation: Compiled output includes skill *knowledge* as rules, not invocable commands. Clear docs: "Skills are compiled as coding standards, not slash commands."
|
|
3825
|
+
3. **"Model detects wrong size"** — Ollama model naming is inconsistent (`llama3.1:8b` vs `codellama:7b-instruct`). Mitigation: regex extracts any `\d+[bB]` pattern; fallback to `14b` if ambiguous.
|
|
3826
|
+
4. **"Other local inference tools emerge"** — Jan.ai, GPT4All, Tabby, etc. Mitigation: Raw markdown output works with any tool that accepts a system prompt file.
|
|
3827
|
+
5. **"Nobody uses the feature"** — SLM adoption among professional developers may be niche. Mitigation: Low effort (3-4 weeks), high signaling value ("we support air-gapped environments"), enterprise sales enabler.
|
|
3828
|
+
|
|
3829
|
+
---
|
|
3830
|
+
|
|
3831
|
+
## 10. Market Positioning
|
|
3832
|
+
|
|
3833
|
+
**Target users:**
|
|
3834
|
+
1. **Enterprise (air-gapped)** — financial services, defense, healthcare firms that cannot send code to cloud LLM APIs
|
|
3835
|
+
2. **Privacy-conscious solo devs** — developers who don't want code leaving their machine
|
|
3836
|
+
3. **Cost-sensitive teams** — startups that can't afford Anthropic/OpenAI API costs at scale
|
|
3837
|
+
4. **Offline-first** — developers working on trains, planes, or in regions with poor connectivity
|
|
3838
|
+
|
|
3839
|
+
**Competitive advantage:** No existing AI coding toolkit provides a compilation pipeline that adapts its instruction set to model capacity. This is a first-mover feature.
|
|
3840
|
+
|
|
3841
|
+
---
|
|
3842
|
+
|
|
3843
|
+
## 11. Next Actions
|
|
3844
|
+
|
|
3845
|
+
1. [ ] Approve plan
|
|
3846
|
+
2. [ ] Implement token counter (1.1)
|
|
3847
|
+
3. [ ] Implement component parser + scorer (1.2)
|
|
3848
|
+
4. [ ] Implement compression engine (1.3)
|
|
3849
|
+
5. [ ] Implement budget packer + emitter (1.4, 1.5)
|
|
3850
|
+
6. [ ] Integrate profile into install.py + manifest.json (2.1)
|
|
3851
|
+
7. [ ] Create CLI command `compile-slm` (2.2)
|
|
3852
|
+
8. [ ] Add Ollama model detection (2.3)
|
|
3853
|
+
9. [ ] Add persona + language aware compilation (2.4, 2.5)
|
|
3854
|
+
10. [ ] Write integration guides for 4 platforms (3.1)
|
|
3855
|
+
11. [ ] Compile quality validator (3.2)
|
|
3856
|
+
12. [ ] Tests + documentation (3.3, 3.4)
|
|
3857
|
+
|
|
3858
|
+
---
|
|
3859
|
+
|
|
3860
|
+
## 12. Future
|
|
3861
|
+
|
|
3862
|
+
| Feature | Rationale |
|
|
3863
|
+
|---------|-----------|
|
|
3864
|
+
| Automatic recompile on file changes (file watcher) | v1 uses manual recompile + staleness warning |
|
|
3865
|
+
| Quality benchmarks (run 5 standard tasks, measure output quality per budget tier) | Validate compilation quality empirically before shipping |
|
|
3866
|
+
| Plugin-aware compilation (include memory-pack prompts if installed) | Depends on plugin system maturity |
|
|
3867
|
+
| Model-specific prompt templates (different SLM families prefer different instruction styles) | Needs empirical testing across model families |
|
|
3868
|
+
| `--profile offline-slm --enforce-git-hooks` (install git pre-commit hook for quality gates) | Compensates for stripped lifecycle hooks |
|
|
3869
|
+
|
|
3870
|
+
---
|
|
3871
|
+
|
|
3872
|
+
## 13. Cross-Plan Dependencies
|
|
3873
|
+
|
|
3874
|
+
This plan shares modification targets with two other proposed plans:
|
|
3875
|
+
|
|
3876
|
+
| Shared File | This Plan | Enterprise Config Plan | Local Dashboard Plan |
|
|
3877
|
+
|-------------|-----------|----------------------|---------------------|
|
|
3878
|
+
| `scripts/install.py` | +30 LOC (offline-slm profile) | +80 LOC (extends resolution) | — |
|
|
3879
|
+
| `manifest.json` | +5 LOC (offline-slm profile) | +10 LOC (schema refs) | — |
|
|
3880
|
+
| `bin/ai-toolkit.js` | +10 LOC (compile-slm command) | +40 LOC (config subcommands) | +15 LOC (ui command) |
|
|
3881
|
+
|
|
3882
|
+
**If implementing in parallel:** this plan has the smallest changes to shared files — merge first to minimize conflicts.
|
|
3883
|
+
|
|
3884
|
+
**Enterprise Config interaction:** If Enterprise Config ships, `compile-slm` should respect the `extends` chain — compile the merged config, not just local. Add a `--ignore-extends` flag for air-gapped environments without access to the base config.
|
|
3885
|
+
|
|
3886
|
+
---
|
|
3887
|
+
|
|
3888
|
+
**Last Updated:** 2026-04-10
|
|
3889
|
+
|
|
3890
|
+
---
|
|
3891
|
+
|
|
269
3892
|
## kb/procedures/maintenance-sop.md
|
|
270
3893
|
|
|
271
3894
|
---
|
|
@@ -275,7 +3898,7 @@ service: ai-toolkit
|
|
|
275
3898
|
tags: [sop, maintenance, agents, skills, install]
|
|
276
3899
|
version: "1.4.2"
|
|
277
3900
|
created: "2026-03-23"
|
|
278
|
-
last_updated: "2026-04-
|
|
3901
|
+
last_updated: "2026-04-10"
|
|
279
3902
|
description: "Standard operating procedures for installing, maintaining, and evolving the ai-toolkit."
|
|
280
3903
|
---
|
|
281
3904
|
|
|
@@ -306,12 +3929,15 @@ ai-toolkit install --local --editors cursor,aider # specific editors onl
|
|
|
306
3929
|
|
|
307
3930
|
Supported editors: `cursor`, `windsurf`, `cline`, `roo`, `aider`, `augment`, `copilot`, `antigravity`.
|
|
308
3931
|
|
|
309
|
-
To restrict which language rules are injected
|
|
3932
|
+
To restrict which language rules are injected, use `--lang`:
|
|
310
3933
|
|
|
311
3934
|
```bash
|
|
312
3935
|
ai-toolkit install --local --lang python,typescript
|
|
3936
|
+
ai-toolkit install --local --lang python --editors all # language rules propagated to all editors
|
|
313
3937
|
```
|
|
314
3938
|
|
|
3939
|
+
When `--editors` is combined with `--lang` (or auto-detected languages), language rules are propagated to all configured editors as `ai-toolkit-lang-<lang>` files — not just Claude's `CLAUDE.md`. Similarly, registered custom rules (`~/.ai-toolkit/rules/`) are propagated to directory-based editor configs as `ai-toolkit-custom-<name>` files.
|
|
3940
|
+
|
|
315
3941
|
**Note:** Hooks are global-only — merged into `~/.claude/settings.json` by `ai-toolkit install`. Project-local `--local` does not install hooks; any legacy `.claude/hooks.json` is removed automatically.
|
|
316
3942
|
|
|
317
3943
|
**Input validation (v1.4.2):** `--only`, `--skip`, `--editors`, and `--lang` are validated on input; an invalid value exits with a clear error before any changes are made.
|
|
@@ -391,7 +4017,10 @@ ai-toolkit add-rule ./my-project-rules.md
|
|
|
391
4017
|
# → copies to ~/.ai-toolkit/rules/my-project-rules.md
|
|
392
4018
|
|
|
393
4019
|
ai-toolkit update
|
|
394
|
-
# → injects the rule into ~/.claude/CLAUDE.md
|
|
4020
|
+
# → injects the rule into ~/.claude/CLAUDE.md and all global editor configs
|
|
4021
|
+
|
|
4022
|
+
ai-toolkit update --local
|
|
4023
|
+
# → also propagates as ai-toolkit-custom-<name> to directory-based editors (Cursor, Windsurf, Cline, Roo, Augment, Antigravity)
|
|
395
4024
|
```
|
|
396
4025
|
|
|
397
4026
|
To unregister (removes from registry **and** strips the block from CLAUDE.md):
|
|
@@ -479,7 +4108,7 @@ State tracked in `~/.ai-toolkit/plugins.json`.
|
|
|
479
4108
|
|
|
480
4109
|
Follow the `documentation-standards` knowledge skill (`app/skills/documentation-standards/SKILL.md`) for full spec. Quick checklist:
|
|
481
4110
|
|
|
482
|
-
1. **Choose category directory:** `kb/reference/`, `kb/howto/`, `kb/procedures/`, `kb/troubleshooting/`,
|
|
4111
|
+
1. **Choose category directory:** `kb/reference/`, `kb/howto/`, `kb/procedures/`, `kb/troubleshooting/`, `kb/best-practices/`, or `kb/planning/`
|
|
483
4112
|
2. **Create file:** kebab-case name, no dates in filename
|
|
484
4113
|
3. **Add frontmatter** with all 7 required fields: `title`, `category`, `service`, `tags`, `created`, `last_updated`, `description`
|
|
485
4114
|
4. **Write in English**
|
|
@@ -601,6 +4230,288 @@ What `uninstall` does:
|
|
|
601
4230
|
|
|
602
4231
|
---
|
|
603
4232
|
|
|
4233
|
+
## kb/procedures/release-preparation-sop.md
|
|
4234
|
+
|
|
4235
|
+
---
|
|
4236
|
+
title: "SOP: Release Preparation"
|
|
4237
|
+
category: procedures
|
|
4238
|
+
service: ai-toolkit
|
|
4239
|
+
tags: [sop, release, version, publish, changelog, semver]
|
|
4240
|
+
version: "1.5.0"
|
|
4241
|
+
created: "2026-04-10"
|
|
4242
|
+
last_updated: "2026-04-10"
|
|
4243
|
+
description: "Step-by-step checklist for preparing a new ai-toolkit release — version sync, changelog, artifact regeneration, validation, and tagging. Run BEFORE every git tag."
|
|
4244
|
+
---
|
|
4245
|
+
|
|
4246
|
+
# SOP: Release Preparation
|
|
4247
|
+
|
|
4248
|
+
Complete checklist for preparing a new `@softspark/ai-toolkit` release.
|
|
4249
|
+
Run this **before** tagging. After tagging and publishing, run the
|
|
4250
|
+
[Release Verification SOP](release-verification-sop.md) to smoke-test.
|
|
4251
|
+
|
|
4252
|
+
**Pipeline:**
|
|
4253
|
+
```
|
|
4254
|
+
Release Preparation (this SOP) → git tag → CI publish → Release Verification SOP
|
|
4255
|
+
```
|
|
4256
|
+
|
|
4257
|
+
**Time:** 5-10 minutes
|
|
4258
|
+
|
|
4259
|
+
---
|
|
4260
|
+
|
|
4261
|
+
## Quick Checklist (TL;DR)
|
|
4262
|
+
|
|
4263
|
+
```bash
|
|
4264
|
+
# 1. Decide version bump
|
|
4265
|
+
# patch (1.4.2 → 1.4.3): bugfix, typo, doc fix
|
|
4266
|
+
# minor (1.4.2 → 1.5.0): new feature, new skill, new flag
|
|
4267
|
+
# major (1.4.2 → 2.0.0): breaking change
|
|
4268
|
+
|
|
4269
|
+
# 2. Sync version across all files
|
|
4270
|
+
python3 scripts/sync_version.py X.Y.Z # if script exists, else manual
|
|
4271
|
+
|
|
4272
|
+
# 3. Write CHANGELOG.md entry
|
|
4273
|
+
# 4. Regenerate artifacts
|
|
4274
|
+
python3 scripts/generate_agents_md.py > AGENTS.md
|
|
4275
|
+
python3 scripts/generate_llms_txt.py > llms.txt
|
|
4276
|
+
python3 scripts/generate_llms_txt.py --full > llms-full.txt
|
|
4277
|
+
|
|
4278
|
+
# 5. Validate + audit + test
|
|
4279
|
+
python3 scripts/validate.py --strict && python3 scripts/audit_skills.py --ci && npm test
|
|
4280
|
+
|
|
4281
|
+
# 6. Commit + tag + push
|
|
4282
|
+
git add -A && git commit -m "chore: release vX.Y.Z"
|
|
4283
|
+
git tag vX.Y.Z
|
|
4284
|
+
git push origin main --tags
|
|
4285
|
+
```
|
|
4286
|
+
|
|
4287
|
+
---
|
|
4288
|
+
|
|
4289
|
+
## Phase 1: Determine Version Bump
|
|
4290
|
+
|
|
4291
|
+
Follow [Semantic Versioning](https://semver.org/):
|
|
4292
|
+
|
|
4293
|
+
| Change Type | Bump | Examples |
|
|
4294
|
+
|-------------|------|---------|
|
|
4295
|
+
| Bugfix, typo, doc-only | **patch** | Fix install flag, correct description |
|
|
4296
|
+
| New feature, skill, agent, flag | **minor** | Add `/hipaa-validate`, add `--output json` |
|
|
4297
|
+
| Breaking CLI change, removed skill, config format change | **major** | Rename `install` to `setup`, remove skill |
|
|
4298
|
+
|
|
4299
|
+
**Rule:** When in doubt, bump minor.
|
|
4300
|
+
|
|
4301
|
+
---
|
|
4302
|
+
|
|
4303
|
+
## Phase 2: Sync Version in All Files
|
|
4304
|
+
|
|
4305
|
+
The canonical version lives in `package.json`. These files **must** match:
|
|
4306
|
+
|
|
4307
|
+
### Mandatory sync (every release)
|
|
4308
|
+
|
|
4309
|
+
| File | Field | How to update |
|
|
4310
|
+
|------|-------|---------------|
|
|
4311
|
+
| `package.json` | `"version": "X.Y.Z"` | Edit directly |
|
|
4312
|
+
| `manifest.json` | `"version": "X.Y.Z"` | Edit directly |
|
|
4313
|
+
| `app/.claude-plugin/plugin.json` | `"version": "X.Y.Z"` | Edit directly |
|
|
4314
|
+
|
|
4315
|
+
### Auto-synced (no manual action)
|
|
4316
|
+
|
|
4317
|
+
| File | Mechanism |
|
|
4318
|
+
|------|-----------|
|
|
4319
|
+
| `package-lock.json` | Regenerated by `npm install --package-lock-only` |
|
|
4320
|
+
|
|
4321
|
+
### Conditional sync (only if the doc was modified in this release)
|
|
4322
|
+
|
|
4323
|
+
| File | Field | When to update |
|
|
4324
|
+
|------|-------|---------------|
|
|
4325
|
+
| `kb/procedures/maintenance-sop.md` | frontmatter `version:` | If SOP content changed |
|
|
4326
|
+
| `kb/reference/skills-catalog.md` | frontmatter `version:` | If skills added/removed |
|
|
4327
|
+
| `kb/reference/agents-catalog.md` | frontmatter `version:` | If agents added/removed |
|
|
4328
|
+
| `kb/reference/hooks-catalog.md` | frontmatter `version:` | If hooks changed |
|
|
4329
|
+
| `kb/reference/architecture-overview.md` | frontmatter `version:` | If architecture changed |
|
|
4330
|
+
| `kb/reference/distribution-model.md` | frontmatter `version:` | If install model changed |
|
|
4331
|
+
| `kb/reference/global-install-model.md` | frontmatter `version:` | If install model changed |
|
|
4332
|
+
|
|
4333
|
+
> **Note:** KB `version:` fields track the **document version**, not the toolkit version.
|
|
4334
|
+
> Only bump them when the document content actually changes in this release.
|
|
4335
|
+
|
|
4336
|
+
### Count sync (if skills/agents/hooks changed)
|
|
4337
|
+
|
|
4338
|
+
| File | What to check |
|
|
4339
|
+
|------|---------------|
|
|
4340
|
+
| `package.json` | `"description"` — skill/agent count |
|
|
4341
|
+
| `README.md` | Badge counts, "What You Get" table |
|
|
4342
|
+
| `app/ARCHITECTURE.md` | Section headings with counts |
|
|
4343
|
+
|
|
4344
|
+
> **Tip:** `validate.py --strict` and `npm test` (metadata contract tests) catch
|
|
4345
|
+
> count drift automatically. If tests pass, counts are correct.
|
|
4346
|
+
|
|
4347
|
+
### Verification command
|
|
4348
|
+
|
|
4349
|
+
After syncing, verify all mandatory files match:
|
|
4350
|
+
|
|
4351
|
+
```bash
|
|
4352
|
+
VERSION=$(python3 -c "import json; print(json.load(open('package.json'))['version'])")
|
|
4353
|
+
echo "Target: $VERSION"
|
|
4354
|
+
echo "manifest.json: $(python3 -c "import json; print(json.load(open('manifest.json'))['version'])")"
|
|
4355
|
+
echo "plugin.json: $(python3 -c "import json; print(json.load(open('app/.claude-plugin/plugin.json'))['version'])")"
|
|
4356
|
+
echo "package-lock.json: $(python3 -c "import json; print(json.load(open('package-lock.json'))['version'])")"
|
|
4357
|
+
```
|
|
4358
|
+
|
|
4359
|
+
All four must print the same version. If not, fix before proceeding.
|
|
4360
|
+
|
|
4361
|
+
---
|
|
4362
|
+
|
|
4363
|
+
## Phase 3: Write CHANGELOG Entry
|
|
4364
|
+
|
|
4365
|
+
Add entry at the top of `CHANGELOG.md` (after the header, before previous release):
|
|
4366
|
+
|
|
4367
|
+
```markdown
|
|
4368
|
+
## vX.Y.Z — Short Title (YYYY-MM-DD)
|
|
4369
|
+
|
|
4370
|
+
### Added
|
|
4371
|
+
- **Feature name** — description
|
|
4372
|
+
|
|
4373
|
+
### Changed
|
|
4374
|
+
- **What changed** — old behavior → new behavior
|
|
4375
|
+
|
|
4376
|
+
### Fixed
|
|
4377
|
+
- **Bug description** — what was broken and how it's fixed
|
|
4378
|
+
|
|
4379
|
+
### Removed
|
|
4380
|
+
- **What was removed** — migration path if any
|
|
4381
|
+
```
|
|
4382
|
+
|
|
4383
|
+
**Rules:**
|
|
4384
|
+
- Use **bold** for feature names
|
|
4385
|
+
- Start descriptions with a verb (Added, Changed, Fixed, Removed)
|
|
4386
|
+
- Reference skill names with backticks and slash: `/hipaa-validate`
|
|
4387
|
+
- Include script names: `scripts/hipaa_scan.py`
|
|
4388
|
+
- Include count changes: `Skill count: 91 → 92`
|
|
4389
|
+
- Date format: `YYYY-MM-DD`
|
|
4390
|
+
- Title: short, descriptive, no version number repetition
|
|
4391
|
+
|
|
4392
|
+
---
|
|
4393
|
+
|
|
4394
|
+
## Phase 4: Regenerate Artifacts
|
|
4395
|
+
|
|
4396
|
+
```bash
|
|
4397
|
+
python3 scripts/generate_agents_md.py > AGENTS.md
|
|
4398
|
+
python3 scripts/generate_llms_txt.py > llms.txt
|
|
4399
|
+
python3 scripts/generate_llms_txt.py --full > llms-full.txt
|
|
4400
|
+
```
|
|
4401
|
+
|
|
4402
|
+
Check if anything actually changed:
|
|
4403
|
+
|
|
4404
|
+
```bash
|
|
4405
|
+
git diff --stat AGENTS.md llms.txt llms-full.txt
|
|
4406
|
+
```
|
|
4407
|
+
|
|
4408
|
+
If no diff, the artifacts are already current. If there is a diff, stage them.
|
|
4409
|
+
|
|
4410
|
+
---
|
|
4411
|
+
|
|
4412
|
+
## Phase 5: Validate, Audit, Test
|
|
4413
|
+
|
|
4414
|
+
Run the full quality gate:
|
|
4415
|
+
|
|
4416
|
+
```bash
|
|
4417
|
+
python3 scripts/validate.py --strict
|
|
4418
|
+
python3 scripts/audit_skills.py --ci
|
|
4419
|
+
npm test
|
|
4420
|
+
```
|
|
4421
|
+
|
|
4422
|
+
**Expected results:**
|
|
4423
|
+
- `validate.py`: `Errors: 0 | Warnings: 0 | VALIDATION PASSED`
|
|
4424
|
+
- `audit_skills.py`: `HIGH: 0 | WARN: 0` (INFO is acceptable)
|
|
4425
|
+
- `npm test`: `1..N` with zero `not ok`
|
|
4426
|
+
|
|
4427
|
+
**One-liner:**
|
|
4428
|
+
```bash
|
|
4429
|
+
python3 scripts/validate.py --strict && python3 scripts/audit_skills.py --ci && npm test
|
|
4430
|
+
```
|
|
4431
|
+
|
|
4432
|
+
**If tests fail:** Fix the issue, do NOT skip. Common failures:
|
|
4433
|
+
- Stale counts → re-run `generate:all` or fix README/ARCHITECTURE
|
|
4434
|
+
- Missing frontmatter → add to new KB docs
|
|
4435
|
+
- Broken symlink → `ai-toolkit doctor --fix`
|
|
4436
|
+
|
|
4437
|
+
---
|
|
4438
|
+
|
|
4439
|
+
## Phase 6: Commit
|
|
4440
|
+
|
|
4441
|
+
Stage all release files:
|
|
4442
|
+
|
|
4443
|
+
```bash
|
|
4444
|
+
git add package.json manifest.json app/.claude-plugin/plugin.json
|
|
4445
|
+
git add package-lock.json
|
|
4446
|
+
git add CHANGELOG.md
|
|
4447
|
+
git add AGENTS.md llms.txt llms-full.txt
|
|
4448
|
+
git add -p # review and stage any other changes
|
|
4449
|
+
```
|
|
4450
|
+
|
|
4451
|
+
Commit:
|
|
4452
|
+
|
|
4453
|
+
```bash
|
|
4454
|
+
git commit -m "chore: release vX.Y.Z"
|
|
4455
|
+
```
|
|
4456
|
+
|
|
4457
|
+
---
|
|
4458
|
+
|
|
4459
|
+
## Phase 7: Tag and Push
|
|
4460
|
+
|
|
4461
|
+
```bash
|
|
4462
|
+
git tag vX.Y.Z
|
|
4463
|
+
git push origin main --tags
|
|
4464
|
+
```
|
|
4465
|
+
|
|
4466
|
+
This triggers `.github/workflows/publish.yml` which:
|
|
4467
|
+
1. Runs `validate.py --strict`
|
|
4468
|
+
2. Runs `npm test`
|
|
4469
|
+
3. Publishes to npm as `@softspark/ai-toolkit@X.Y.Z`
|
|
4470
|
+
|
|
4471
|
+
**After CI completes:** Run the [Release Verification SOP](release-verification-sop.md)
|
|
4472
|
+
to smoke-test the published package.
|
|
4473
|
+
|
|
4474
|
+
---
|
|
4475
|
+
|
|
4476
|
+
## Rollback
|
|
4477
|
+
|
|
4478
|
+
If a bad release was published:
|
|
4479
|
+
|
|
4480
|
+
```bash
|
|
4481
|
+
# Unpublish from npm (within 72h)
|
|
4482
|
+
npm unpublish @softspark/ai-toolkit@X.Y.Z
|
|
4483
|
+
|
|
4484
|
+
# Or deprecate (preferred — doesn't break existing installs)
|
|
4485
|
+
npm deprecate @softspark/ai-toolkit@X.Y.Z "Known issue: <description>. Use vA.B.C instead."
|
|
4486
|
+
|
|
4487
|
+
# Delete tag
|
|
4488
|
+
git tag -d vX.Y.Z
|
|
4489
|
+
git push origin --delete vX.Y.Z
|
|
4490
|
+
```
|
|
4491
|
+
|
|
4492
|
+
---
|
|
4493
|
+
|
|
4494
|
+
## Checklist Summary
|
|
4495
|
+
|
|
4496
|
+
| # | Step | Command / Action | Pass Criteria |
|
|
4497
|
+
|---|------|-----------------|---------------|
|
|
4498
|
+
| 1 | Version bump type | Decide patch/minor/major | — |
|
|
4499
|
+
| 2 | `package.json` version | Edit `"version"` | Matches target |
|
|
4500
|
+
| 3 | `manifest.json` version | Edit `"version"` | Matches target |
|
|
4501
|
+
| 4 | `plugin.json` version | Edit `"version"` | Matches target |
|
|
4502
|
+
| 5 | `package-lock.json` | `npm install --package-lock-only` | Matches target |
|
|
4503
|
+
| 6 | Count sync | Check `package.json` description, README | `validate.py` passes |
|
|
4504
|
+
| 7 | CHANGELOG.md | Add release entry | Entry exists for vX.Y.Z |
|
|
4505
|
+
| 8 | Regenerate artifacts | `generate_agents_md.py`, `generate_llms_txt.py` | No unexpected diff |
|
|
4506
|
+
| 9 | Validate | `validate.py --strict` | 0 errors, 0 warnings |
|
|
4507
|
+
| 10 | Security audit | `audit_skills.py --ci` | 0 HIGH |
|
|
4508
|
+
| 11 | Tests | `npm test` | All pass |
|
|
4509
|
+
| 12 | Commit | `git commit` | Clean working tree |
|
|
4510
|
+
| 13 | Tag | `git tag vX.Y.Z` | Tag exists |
|
|
4511
|
+
| 14 | Push | `git push origin main --tags` | CI triggered |
|
|
4512
|
+
|
|
4513
|
+
---
|
|
4514
|
+
|
|
604
4515
|
## kb/procedures/release-verification-sop.md
|
|
605
4516
|
|
|
606
4517
|
---
|