@boshu2/vibe-check 2.2.0 → 2.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (44) hide show
  1. package/.agents/plans/2025-12-28-ai-safety-integration-plan.md +326 -0
  2. package/.agents/research/2025-12-28-ai-platform-security-integration.md +295 -0
  3. package/.beads/README.md +81 -0
  4. package/.beads/config.yaml +62 -0
  5. package/.beads/interactions.jsonl +0 -0
  6. package/.beads/issues.jsonl +9 -0
  7. package/.beads/metadata.json +4 -0
  8. package/.gitattributes +3 -0
  9. package/AGENTS.md +40 -0
  10. package/CHANGELOG.md +43 -1
  11. package/CLAUDE.md +75 -0
  12. package/README.md +46 -1
  13. package/dist/ai-safety/contract-drift.d.ts +14 -0
  14. package/dist/ai-safety/contract-drift.d.ts.map +1 -0
  15. package/dist/ai-safety/contract-drift.js +230 -0
  16. package/dist/ai-safety/contract-drift.js.map +1 -0
  17. package/dist/ai-safety/index.d.ts +43 -0
  18. package/dist/ai-safety/index.d.ts.map +1 -0
  19. package/dist/ai-safety/index.js +177 -0
  20. package/dist/ai-safety/index.js.map +1 -0
  21. package/dist/ai-safety/scope-violation.d.ts +18 -0
  22. package/dist/ai-safety/scope-violation.d.ts.map +1 -0
  23. package/dist/ai-safety/scope-violation.js +150 -0
  24. package/dist/ai-safety/scope-violation.js.map +1 -0
  25. package/dist/ai-safety/secret-leakage.d.ts +18 -0
  26. package/dist/ai-safety/secret-leakage.d.ts.map +1 -0
  27. package/dist/ai-safety/secret-leakage.js +188 -0
  28. package/dist/ai-safety/secret-leakage.js.map +1 -0
  29. package/dist/ai-safety/token-spiral.d.ts +17 -0
  30. package/dist/ai-safety/token-spiral.d.ts.map +1 -0
  31. package/dist/ai-safety/token-spiral.js +183 -0
  32. package/dist/ai-safety/token-spiral.js.map +1 -0
  33. package/dist/ai-safety/types.d.ts +122 -0
  34. package/dist/ai-safety/types.d.ts.map +1 -0
  35. package/dist/ai-safety/types.js +32 -0
  36. package/dist/ai-safety/types.js.map +1 -0
  37. package/dist/commands/session.d.ts +9 -0
  38. package/dist/commands/session.d.ts.map +1 -1
  39. package/dist/commands/session.js +42 -0
  40. package/dist/commands/session.js.map +1 -1
  41. package/dist/commands/watch.d.ts.map +1 -1
  42. package/dist/commands/watch.js +59 -0
  43. package/dist/commands/watch.js.map +1 -1
  44. package/package.json +4 -1
@@ -0,0 +1,326 @@
1
+ ---
2
+ date: 2025-12-28
3
+ type: Plan
4
+ topic: "AI Safety Detection Module for vibe-check"
5
+ research: ".agents/research/2025-12-28-ai-platform-security-integration.md"
6
+ tags: [plan, ai-safety, llm-validation, security, vibe-check]
7
+ status: READY_FOR_APPROVAL
8
+ ---
9
+
10
+ # Plan: AI Safety Detection Module
11
+
12
+ **Created:** 2025-12-28
13
+ **Research:** `.agents/research/2025-12-28-ai-platform-security-integration.md`
14
+ **Vibe Level:** L3 (Multiple files, following existing patterns)
15
+
16
+ ---
17
+
18
+ ## Overview
19
+
20
+ Add a new `src/ai-safety/` module to vibe-check that detects LLM-specific antipatterns in commit history: **secret leakage**, **scope violations**, **contract drift**, and **token spirals**. This extends the existing inner-loop failure detection with patterns ported from ai-platform's production security stack.
21
+
22
+ ---
23
+
24
+ ## Approach
25
+
26
+ Port security validation patterns from `ai-platform/tests/agents/test_agent_security.py` and adapt them for git-based analysis. Follow the existing `src/inner-loop/` module structure exactly.
27
+
28
+ **Why this approach:**
29
+ 1. **Proven patterns:** ai-platform patterns are battle-tested for classified environments
30
+ 2. **Non-breaking:** Extends existing architecture without replacing anything
31
+ 3. **Familiar structure:** Same module organization as inner-loop
32
+ 4. **Zero new dependencies:** Pure TypeScript, uses existing git analysis
33
+
34
+ ---
35
+
36
+ ## Features
37
+
38
+ ### Feature 1: AI Safety Types and Orchestrator
39
+
40
+ **Priority:** P0 (Foundation - must come first)
41
+ **Type:** feature
42
+ **Depends On:** None
43
+
44
+ **Acceptance Criteria:**
45
+ - [ ] `src/ai-safety/types.ts` defines all interfaces
46
+ - [ ] `src/ai-safety/index.ts` exports orchestrator function
47
+ - [ ] `AISafetyAnalysis` interface matches inner-loop pattern
48
+ - [ ] Config supports enable/disable per detector
49
+ - [ ] Compiles without errors
50
+
51
+ **Files Affected:**
52
+ - `src/ai-safety/types.ts` - New file: Type definitions
53
+ - `src/ai-safety/index.ts` - New file: Orchestrator + exports
54
+
55
+ **Test Strategy:** `npm run build` passes, types are importable
56
+
57
+ ---
58
+
59
+ ### Feature 2: Secret Leakage Detector
60
+
61
+ **Priority:** P1 (Highest value, immediate utility)
62
+ **Type:** feature
63
+ **Depends On:** Feature 1
64
+
65
+ **Acceptance Criteria:**
66
+ - [ ] Detects OpenAI keys (`sk-[a-zA-Z0-9]{48}`)
67
+ - [ ] Detects GitHub PATs (`ghp_[a-zA-Z0-9]{36}`)
68
+ - [ ] Detects GitLab PATs (`glpat-[a-zA-Z0-9]{20}`)
69
+ - [ ] Detects AWS keys (`AKIA[0-9A-Z]{16}`)
70
+ - [ ] Detects Slack tokens (`xox[baprs]-*`)
71
+ - [ ] Detects generic patterns (password=, secret=)
72
+ - [ ] Reports file, line context, and pattern matched
73
+ - [ ] Critical severity by default
74
+
75
+ **Files Affected:**
76
+ - `src/ai-safety/secret-leakage.ts` - New file: Secret detection logic
77
+ - `src/ai-safety/index.ts` - Wire into orchestrator
78
+
79
+ **Test Strategy:**
80
+ 1. Create test with fake secrets in commit diffs
81
+ 2. Verify detection of each pattern
82
+ 3. Verify no false positives on similar-looking strings
83
+
84
+ ---
85
+
86
+ ### Feature 3: Scope Violation Detector
87
+
88
+ **Priority:** P2 (High value, needs config)
89
+ **Type:** feature
90
+ **Depends On:** Feature 1
91
+
92
+ **Acceptance Criteria:**
93
+ - [ ] Detects commits touching files outside declared scope
94
+ - [ ] Scope defined via config file (`.vibe-check/scope.yaml`) or CLI
95
+ - [ ] Supports glob patterns for scope definition
96
+ - [ ] Reports which files violated scope
97
+ - [ ] Warning severity by default
98
+
99
+ **Files Affected:**
100
+ - `src/ai-safety/scope-violation.ts` - New file: Scope check logic
101
+ - `src/ai-safety/index.ts` - Wire into orchestrator
102
+ - `src/types.ts` - Add AISafetyConfig to main types
103
+
104
+ **Test Strategy:**
105
+ 1. Define scope as `src/feature/**`
106
+ 2. Create commit touching `src/other/file.ts`
107
+ 3. Verify scope violation detected
108
+
109
+ ---
110
+
111
+ ### Feature 4: Contract Drift Detector
112
+
113
+ **Priority:** P2 (Medium value, pattern analysis)
114
+ **Type:** feature
115
+ **Depends On:** Feature 1
116
+
117
+ **Acceptance Criteria:**
118
+ - [ ] Tracks commit message format compliance over time
119
+ - [ ] Detects degradation from conventional commits format
120
+ - [ ] Tracks entropy (message quality declining)
121
+ - [ ] Reports drift percentage and trend direction
122
+ - [ ] Warning severity when >30% drift from baseline
123
+
124
+ **Files Affected:**
125
+ - `src/ai-safety/contract-drift.ts` - New file: Format drift logic
126
+ - `src/ai-safety/index.ts` - Wire into orchestrator
127
+
128
+ **Test Strategy:**
129
+ 1. Provide commits with good format then degrading format
130
+ 2. Verify drift percentage calculated
131
+ 3. Verify warning triggered at threshold
132
+
133
+ ---
134
+
135
+ ### Feature 5: Token Spiral Estimator
136
+
137
+ **Priority:** P3 (Lower value, estimation-based)
138
+ **Type:** feature
139
+ **Depends On:** Feature 1
140
+
141
+ **Acceptance Criteria:**
142
+ - [ ] Estimates token usage from commit size
143
+ - [ ] Uses heuristic: ~4 chars per token
144
+ - [ ] Tracks cumulative session tokens
145
+ - [ ] Detects explosion patterns (>2x baseline)
146
+ - [ ] Info severity (advisory, not critical)
147
+
148
+ **Files Affected:**
149
+ - `src/ai-safety/token-spiral.ts` - New file: Token estimation logic
150
+ - `src/ai-safety/index.ts` - Wire into orchestrator
151
+
152
+ **Test Strategy:**
153
+ 1. Provide small commits then huge commit
154
+ 2. Verify explosion detected
155
+ 3. Verify reasonable token estimates
156
+
157
+ ---
158
+
159
+ ### Feature 6: Integration with Session End
160
+
161
+ **Priority:** P1 (Makes feature visible to users)
162
+ **Type:** feature
163
+ **Depends On:** Features 1-2 (minimum)
164
+
165
+ **Acceptance Criteria:**
166
+ - [ ] `session end` JSON output includes `ai_safety` section
167
+ - [ ] Terminal output shows AI safety warnings
168
+ - [ ] Critical findings show in red
169
+ - [ ] Recommendations generated for each finding
170
+
171
+ **Files Affected:**
172
+ - `src/commands/session.ts` - Add AI safety analysis call
173
+ - `src/output/terminal.ts` - Format AI safety for terminal
174
+ - `src/output/json.ts` - Format AI safety for JSON
175
+
176
+ **Test Strategy:** `vibe-check session end --format json` includes ai_safety section
177
+
178
+ ---
179
+
180
+ ### Feature 7: Integration with Watch Command
181
+
182
+ **Priority:** P2 (Real-time value)
183
+ **Type:** feature
184
+ **Depends On:** Feature 2 (secret detection)
185
+
186
+ **Acceptance Criteria:**
187
+ - [ ] Watch mode alerts on secret detection immediately
188
+ - [ ] Uses red/bold for critical findings
189
+ - [ ] Shows file and pattern matched
190
+ - [ ] Rate-limited (don't spam alerts)
191
+
192
+ **Files Affected:**
193
+ - `src/commands/watch.ts` - Add secret detection to watch loop
194
+
195
+ **Test Strategy:** Run watch, commit file with fake secret, see immediate alert
196
+
197
+ ---
198
+
199
+ ### Feature 8: Tests and Documentation
200
+
201
+ **Priority:** P2 (Quality assurance)
202
+ **Type:** task
203
+ **Depends On:** Features 1-5
204
+
205
+ **Acceptance Criteria:**
206
+ - [ ] Unit tests for each detector
207
+ - [ ] Integration test for orchestrator
208
+ - [ ] CLAUDE.md updated with ai-safety commands
209
+ - [ ] README.md mentions AI safety features
210
+
211
+ **Files Affected:**
212
+ - `tests/ai-safety.test.ts` - New file: Test suite
213
+ - `CLAUDE.md` - Update documentation
214
+ - `README.md` - Feature description
215
+
216
+ **Test Strategy:** `npm test` passes with >80% coverage on new module
217
+
218
+ ---
219
+
220
+ ## Implementation Order
221
+
222
+ | Step | Feature | Depends On | Validation |
223
+ |------|---------|------------|------------|
224
+ | 1 | Types and Orchestrator | - | `npm run build` passes |
225
+ | 2 | Secret Leakage | Step 1 | Test with fake secrets |
226
+ | 3 | Session Integration | Steps 1-2 | `session end` shows findings |
227
+ | 4 | Scope Violation | Step 1 | Test with scope config |
228
+ | 5 | Contract Drift | Step 1 | Test with degrading commits |
229
+ | 6 | Token Spiral | Step 1 | Test with varying sizes |
230
+ | 7 | Watch Integration | Step 2 | Real-time alert test |
231
+ | 8 | Tests & Docs | Steps 1-6 | `npm test` passes |
232
+
233
+ ---
234
+
235
+ ## Beads Issues to Create
236
+
237
+ After approval, these issues will be created:
238
+
239
+ | ID | Title | Type | Priority | Depends On |
240
+ |----|-------|------|----------|------------|
241
+ | TBD | Epic: AI Safety Detection Module | epic | P1 | - |
242
+ | TBD | AI Safety types and orchestrator | feature | P0 | Epic |
243
+ | TBD | Secret leakage detector | feature | P1 | Types |
244
+ | TBD | Session end AI safety integration | feature | P1 | Secret detector |
245
+ | TBD | Scope violation detector | feature | P2 | Types |
246
+ | TBD | Contract drift detector | feature | P2 | Types |
247
+ | TBD | Token spiral estimator | feature | P3 | Types |
248
+ | TBD | Watch command AI safety integration | feature | P2 | Secret detector |
249
+ | TBD | Tests and documentation | task | P2 | All features |
250
+
251
+ ---
252
+
253
+ ## File Structure After Implementation
254
+
255
+ ```
256
+ src/ai-safety/
257
+ ├── index.ts # Orchestrator: analyzeAISafety()
258
+ ├── types.ts # Interfaces: AISafetyAnalysis, configs
259
+ ├── secret-leakage.ts # Regex-based secret detection
260
+ ├── scope-violation.ts # Glob-based scope checking
261
+ ├── contract-drift.ts # Commit format degradation
262
+ └── token-spiral.ts # Token usage estimation
263
+
264
+ tests/ai-safety.test.ts # Full test suite
265
+ ```
266
+
267
+ ---
268
+
269
+ ## Key Interfaces (Preview)
270
+
271
+ ```typescript
272
+ // src/ai-safety/types.ts
273
+ export interface AISafetyAnalysis {
274
+ secretLeakage: SecretLeakageResult;
275
+ scopeViolations: ScopeViolationResult;
276
+ contractDrift: ContractDriftResult;
277
+ tokenSpiral: TokenSpiralResult;
278
+ summary: {
279
+ totalIssues: number;
280
+ criticalIssues: number;
281
+ warningIssues: number;
282
+ overallHealth: 'healthy' | 'warning' | 'critical';
283
+ };
284
+ recommendations: string[];
285
+ }
286
+
287
+ export interface SecretLeakageResult {
288
+ detected: boolean;
289
+ findings: SecretFinding[];
290
+ message: string;
291
+ }
292
+
293
+ export interface SecretFinding {
294
+ commitHash: string;
295
+ file: string;
296
+ pattern: string;
297
+ lineContext: string;
298
+ severity: 'critical' | 'warning';
299
+ }
300
+ ```
301
+
302
+ ---
303
+
304
+ ## Rollback Procedure
305
+
306
+ If something goes wrong:
307
+
308
+ 1. **Module not loading:** Revert `src/ai-safety/` directory
309
+ 2. **Session command broken:** Revert `src/commands/session.ts` changes
310
+ 3. **Watch command broken:** Revert `src/commands/watch.ts` changes
311
+ 4. **Build fails:** Check `src/ai-safety/index.ts` exports
312
+
313
+ All changes are additive - existing functionality is not modified, only extended.
314
+
315
+ ---
316
+
317
+ ## Next Steps
318
+
319
+ 1. **Review and approve this plan**
320
+ 2. **Create beads issues** (commands below)
321
+ 3. **`bd ready`** to see unblocked issues
322
+ 4. **`/implement`** to execute first issue
323
+
324
+ ---
325
+
326
+ **Output:** .agents/plans/2025-12-28-ai-safety-integration-plan.md
@@ -0,0 +1,295 @@
1
+ ---
2
+ date: 2025-12-28
3
+ type: Research
4
+ topic: "Integrating ai-platform security and LLM validation into vibe-check"
5
+ tags: [research, security, llm-validation, integration, ai-platform, vibe-check]
6
+ status: COMPLETE
7
+ ---
8
+
9
+ # Research: Integrating AI-Platform Security & LLM Validation into Vibe-Check
10
+
11
+ **Created:** 2025-12-28
12
+ **Goal:** Understand how to integrate the security and LLM validation components from ai-platform into vibe-check for detecting AI agent misbehavior patterns.
13
+
14
+ ---
15
+
16
+ ## Executive Summary
17
+
18
+ The **ai-platform** repository (`/Users/fullerbt/workspaces/work/ai-platform`) contains a comprehensive security and LLM validation stack built for production use in classified environments. Key components include: **RBAC access control**, **audit logging** (SIEM-ready), **agent execution tracking** (Prometheus metrics), **rate limiting with token budgets**, and **contract-driven agent testing**.
19
+
20
+ **Recommendation:** Integrate these concepts into vibe-check as a new `ai-safety/` or `agent-validation/` module, extending the existing inner-loop failure pattern detection to include LLM-specific antipatterns like **prompt injection attempts**, **hallucination markers**, **scope violations**, and **contract drift**.
21
+
22
+ ---
23
+
24
+ ## Current State
25
+
26
+ ### What Exists in ai-platform
27
+
28
+ | Component | Location | Purpose |
29
+ |-----------|----------|---------|
30
+ | **Access Control** | `services/gateway/access_control.py` | RBAC for agents/tools via OIDC groups |
31
+ | **Audit Logging** | `services/gateway/audit.py` | JSON request/response logging for SIEM |
32
+ | **Agent Audit** | `services/gateway/agent_audit.py` | Agent execution tracking + Prometheus metrics |
33
+ | **Rate Limiting** | `services/gateway/rate_limit.py` | Per-user rate limits + token budgets |
34
+ | **Config Validator** | `services/etl/app/config_validator.py` | Startup config health checks |
35
+ | **Security Tests** | `tests/agents/test_agent_security.py` | Boundary violation detection |
36
+ | **Contract Tests** | `tests/agents/test_agent_contract_validation.py` | Output contract compliance |
37
+ | **Platform Validator** | `services/etl/scripts/validate-platform.py` | End-to-end LLM platform checks |
38
+
39
+ ### Key Patterns Worth Porting to vibe-check
40
+
41
+ 1. **Agent Scope Boundary Detection** (`test_agent_security.py`)
42
+ - Detects when agents attempt actions outside their declared scope
43
+ - Pattern: Check if commit changes files/APIs not in the agent's domain
44
+
45
+ 2. **Hallucination Markers** (`test_agent_security.py:50-70`)
46
+ - Detects file path hallucination (made-up paths)
47
+ - Pattern: Commits referencing non-existent files or phantom dependencies
48
+
49
+ 3. **Secret Leakage Detection** (`test_agent_security.py:120-150`)
50
+ - Regex patterns for API keys, tokens, credentials
51
+ - Pattern: Commits accidentally exposing secrets
52
+
53
+ 4. **Contract Drift Detection** (`test_agent_contract_validation.py`)
54
+ - Validates agent responses match expected structure
55
+ - Pattern: Detect when AI outputs stop conforming to expected formats
56
+
57
+ 5. **Token Budget Tracking** (`rate_limit.py`)
58
+ - Tracks token consumption per user/session
59
+ - Pattern: Detect runaway token usage (context explosion)
60
+
61
+ ### Key Files
62
+
63
+ | File | Purpose | Relevance to vibe-check |
64
+ |------|---------|-------------------------|
65
+ | `services/gateway/access_control.py` | RBAC via OIDC groups | Could map to commit author scope detection |
66
+ | `services/gateway/audit.py` | JSON audit logging | Structure for storing AI interaction events |
67
+ | `services/gateway/agent_audit.py` | Prometheus metrics + Langfuse traces | Pattern for tracking AI agent behavior over time |
68
+ | `tests/agents/test_agent_security.py` | Security test patterns | Regex patterns and violation detection logic |
69
+ | `services/gateway/rate_limit.py` | Token budget tracking | Pattern for detecting context window abuse |
70
+
71
+ ### Existing Patterns in vibe-check
72
+
73
+ vibe-check already has:
74
+ - **Inner-loop failure detection** (`src/inner-loop/`) with 4 detectors
75
+ - **Spiral detection** with pattern regexes in `watch.ts`
76
+ - **Session tracking** with baseline comparison
77
+ - **NDJSON storage** for historical pattern analysis
78
+ - **Prometheus-like metrics** (not exposed, but structured similarly)
79
+
80
+ ---
81
+
82
+ ## Findings
83
+
84
+ ### Finding 1: Security Validation Patterns Are Git-Analyzable
85
+
86
+ **Evidence:** The ai-platform security tests detect violations by analyzing:
87
+ - Commit messages for intent signals
88
+ - File changes for scope violations
89
+ - Code content for secret patterns
90
+
91
+ **Implication:** These patterns can be adapted for vibe-check's commit-based analysis:
92
+ ```
93
+ ai-platform pattern → vibe-check integration
94
+ -------------------------------------------
95
+ Scope violation test → Detect commits touching files outside declared domain
96
+ Secret leakage test → Detect commits adding API keys/tokens
97
+ Hallucination test → Detect commits referencing non-existent paths
98
+ Contract drift test → Detect commits breaking expected output formats
99
+ ```
100
+
101
+ ### Finding 2: Agent Audit Structure Is Session-Compatible
102
+
103
+ **Evidence:** `agent_audit.py` tracks events in a structure similar to vibe-check sessions:
104
+ ```python
105
+ audit_record = {
106
+ "correlation_id": str, # → session_id
107
+ "agent_name": str, # → (new field)
108
+ "user_identity": str, # → (already tracked)
109
+ "tokens_used": { # → (new metric)
110
+ "input": int,
111
+ "output": int,
112
+ "total": int
113
+ },
114
+ "tool_invocations": [...], # → (map to commit file changes)
115
+ "duration_ms": int # → (session duration)
116
+ }
117
+ ```
118
+
119
+ **Implication:** Can extend `active-session.json` with agent/LLM metadata.
120
+
121
+ ### Finding 3: Rate Limiting = Token Spiral Detection
122
+
123
+ **Evidence:** `rate_limit.py` implements token budget tracking:
124
+ ```python
125
+ @dataclass
126
+ class UserRateLimit:
127
+ max_tokens_per_minute: int = 100_000
128
+ max_tokens_per_hour: int = 1_000_000
129
+ max_tokens_per_day: int = 10_000_000
130
+ ```
131
+
132
+ **Implication:** vibe-check could detect "token spirals" - sessions where token consumption explodes, indicating:
133
+ - Context window abuse
134
+ - Prompt stuffing
135
+ - Repeated failed attempts (AI trying same thing repeatedly)
136
+
137
+ ### Finding 4: Secret Detection Regex Is Ready to Use
138
+
139
+ **Evidence:** `test_agent_security.py:120-150`:
140
+ ```python
141
+ SECRET_PATTERNS = [
142
+ r'sk-[a-zA-Z0-9]{48}', # OpenAI
143
+ r'ghp_[a-zA-Z0-9]{36}', # GitHub PAT
144
+ r'glpat-[a-zA-Z0-9]{20}', # GitLab PAT
145
+ r'AKIA[0-9A-Z]{16}', # AWS Access Key
146
+ r'xox[baprs]-[a-zA-Z0-9-]+', # Slack tokens
147
+ ]
148
+ ```
149
+
150
+ **Implication:** Can add a `secret-leakage.ts` detector to inner-loop.
151
+
152
+ ### Finding 5: Contract Validation Is Output-Pattern Detection
153
+
154
+ **Evidence:** `test_agent_contract_validation.py` validates agent outputs against YAML-defined contracts:
155
+ ```yaml
156
+ agents:
157
+ - name: code-review
158
+ output_contract:
159
+ required_sections: ["summary", "issues", "suggestions"]
160
+ max_length: 5000
161
+ ```
162
+
163
+ **Implication:** vibe-check could detect "contract drift" - when AI commits start deviating from expected patterns:
164
+ - Commit messages becoming less structured
165
+ - PR descriptions missing required sections
166
+ - Code comments becoming vague or absent
167
+
168
+ ---
169
+
170
+ ## Constraints
171
+
172
+ | Constraint | Impact | Mitigation |
173
+ |------------|--------|------------|
174
+ | ai-platform is Python, vibe-check is TypeScript | Can't share code directly | Port patterns/logic, not code |
175
+ | ai-platform runs in Kubernetes, vibe-check is CLI | Different runtime context | Adapt to git-based detection |
176
+ | ai-platform has Prometheus/Langfuse dependencies | Heavy deps for CLI tool | Use file-based storage, optionally export |
177
+ | Token tracking requires LLM API integration | vibe-check only sees git | Estimate tokens from commit size/complexity |
178
+
179
+ ---
180
+
181
+ ## Risks
182
+
183
+ | Risk | Likelihood | Impact | Mitigation |
184
+ |------|------------|--------|------------|
185
+ | Over-engineering simple CLI tool | Medium | High | Start with 2-3 highest-value patterns |
186
+ | False positives on secret detection | Medium | Medium | Require confidence threshold, allow suppression |
187
+ | Token estimation inaccuracy | High | Low | Use as relative signal, not absolute |
188
+ | Breaking existing inner-loop interface | Low | High | Extend, don't replace |
189
+
190
+ ---
191
+
192
+ ## Recommendation
193
+
194
+ **Approach:** Add a new `src/ai-safety/` module to vibe-check with 4 new detectors:
195
+
196
+ ### Phase 1: High-Value Ports (Recommended First)
197
+
198
+ 1. **Secret Leakage Detector** (`src/ai-safety/secret-leakage.ts`)
199
+ - Port regex patterns from `test_agent_security.py`
200
+ - Scan commit diffs for exposed secrets
201
+ - Integrate into `session end` output
202
+
203
+ 2. **Scope Violation Detector** (`src/ai-safety/scope-violation.ts`)
204
+ - Detect commits touching files outside declared domain
205
+ - Requires domain configuration (similar to access_control.py mappings)
206
+ - Warning when agent strays from its lane
207
+
208
+ ### Phase 2: Medium-Value Ports
209
+
210
+ 3. **Contract Drift Detector** (`src/ai-safety/contract-drift.ts`)
211
+ - Detect when commit message patterns degrade
212
+ - Track deviation from conventional commit format
213
+ - Alert when AI stops following established patterns
214
+
215
+ 4. **Token Spiral Estimator** (`src/ai-safety/token-spiral.ts`)
216
+ - Estimate token usage from commit size/complexity
217
+ - Detect sessions with exploding context
218
+ - Use as relative metric (not absolute)
219
+
220
+ ### Integration Points
221
+
222
+ ```typescript
223
+ // src/ai-safety/index.ts
224
+ export interface AISafetyAnalysis {
225
+ secretsDetected: SecretLeakageResult;
226
+ scopeViolations: ScopeViolationResult;
227
+ contractDrift: ContractDriftResult;
228
+ tokenSpiral: TokenSpiralResult;
229
+ summary: {
230
+ totalIssues: number;
231
+ criticalIssues: number;
232
+ warningIssues: number;
233
+ overallHealth: 'healthy' | 'warning' | 'critical';
234
+ };
235
+ recommendations: string[];
236
+ }
237
+
238
+ export function analyzeAISafety(
239
+ commits: Commit[],
240
+ filesPerCommit: Map<string, string[]>,
241
+ config?: AISafetyConfig
242
+ ): AISafetyAnalysis;
243
+ ```
244
+
245
+ **Integration into existing commands:**
246
+ - `session end` → Add `ai_safety` section to JSON output
247
+ - `watch` → Add real-time alerts for secret detection
248
+ - `insights` → Add AI safety pattern history
249
+
250
+ ### Why This Approach
251
+
252
+ 1. **Non-breaking:** Extends existing architecture, doesn't replace
253
+ 2. **High-value first:** Secret leakage is immediately useful
254
+ 3. **Familiar patterns:** Same structure as existing inner-loop detectors
255
+ 4. **Low deps:** No new dependencies required (pure TypeScript)
256
+ 5. **Git-native:** Works with existing commit-based analysis
257
+
258
+ ---
259
+
260
+ ## Alternatives Considered
261
+
262
+ 1. **Import ai-platform as dependency** - Rejected: Different language, heavy deps
263
+ 2. **Create shared npm/pip packages** - Rejected: Over-engineering for 4 patterns
264
+ 3. **Keep validation in ai-platform only** - Rejected: vibe-check users need these patterns
265
+ 4. **Port entire audit system** - Rejected: Too complex, different runtime
266
+
267
+ ---
268
+
269
+ ## Next Steps
270
+
271
+ 1. Run `/plan` to create implementation plan from this research
272
+ 2. Plan will create beads issues for each detector
273
+ 3. Implement in priority order: secrets → scope → contract → token
274
+
275
+ ---
276
+
277
+ ## Appendix: Key Source File References
278
+
279
+ ### ai-platform Security Components
280
+ - `/Users/fullerbt/workspaces/work/ai-platform/services/gateway/access_control.py` - RBAC implementation
281
+ - `/Users/fullerbt/workspaces/work/ai-platform/services/gateway/audit.py` - Audit logging
282
+ - `/Users/fullerbt/workspaces/work/ai-platform/services/gateway/agent_audit.py` - Agent execution tracking
283
+ - `/Users/fullerbt/workspaces/work/ai-platform/services/gateway/rate_limit.py` - Rate limiting
284
+ - `/Users/fullerbt/workspaces/work/ai-platform/tests/agents/test_agent_security.py` - Security tests
285
+ - `/Users/fullerbt/workspaces/work/ai-platform/tests/agents/test_agent_contract_validation.py` - Contract tests
286
+
287
+ ### vibe-check Integration Points
288
+ - `/Users/fullerbt/workspaces/personal/vibe-check/src/inner-loop/index.ts` - Existing detector orchestrator
289
+ - `/Users/fullerbt/workspaces/personal/vibe-check/src/commands/session.ts` - Session end output
290
+ - `/Users/fullerbt/workspaces/personal/vibe-check/src/commands/watch.ts` - Real-time monitoring
291
+ - `/Users/fullerbt/workspaces/personal/vibe-check/src/types.ts` - Type definitions
292
+
293
+ ---
294
+
295
+ **Output:** .agents/research/2025-12-28-ai-platform-security-integration.md
@@ -0,0 +1,81 @@
1
+ # Beads - AI-Native Issue Tracking
2
+
3
+ Welcome to Beads! This repository uses **Beads** for issue tracking - a modern, AI-native tool designed to live directly in your codebase alongside your code.
4
+
5
+ ## What is Beads?
6
+
7
+ Beads is issue tracking that lives in your repo, making it perfect for AI coding agents and developers who want their issues close to their code. No web UI required - everything works through the CLI and integrates seamlessly with git.
8
+
9
+ **Learn more:** [github.com/steveyegge/beads](https://github.com/steveyegge/beads)
10
+
11
+ ## Quick Start
12
+
13
+ ### Essential Commands
14
+
15
+ ```bash
16
+ # Create new issues
17
+ bd create "Add user authentication"
18
+
19
+ # View all issues
20
+ bd list
21
+
22
+ # View issue details
23
+ bd show <issue-id>
24
+
25
+ # Update issue status
26
+ bd update <issue-id> --status in_progress
27
+ bd update <issue-id> --status done
28
+
29
+ # Sync with git remote
30
+ bd sync
31
+ ```
32
+
33
+ ### Working with Issues
34
+
35
+ Issues in Beads are:
36
+ - **Git-native**: Stored in `.beads/issues.jsonl` and synced like code
37
+ - **AI-friendly**: CLI-first design works perfectly with AI coding agents
38
+ - **Branch-aware**: Issues can follow your branch workflow
39
+ - **Always in sync**: Auto-syncs with your commits
40
+
41
+ ## Why Beads?
42
+
43
+ ✨ **AI-Native Design**
44
+ - Built specifically for AI-assisted development workflows
45
+ - CLI-first interface works seamlessly with AI coding agents
46
+ - No context switching to web UIs
47
+
48
+ 🚀 **Developer Focused**
49
+ - Issues live in your repo, right next to your code
50
+ - Works offline, syncs when you push
51
+ - Fast, lightweight, and stays out of your way
52
+
53
+ 🔧 **Git Integration**
54
+ - Automatic sync with git commits
55
+ - Branch-aware issue tracking
56
+ - Intelligent JSONL merge resolution
57
+
58
+ ## Get Started with Beads
59
+
60
+ Try Beads in your own projects:
61
+
62
+ ```bash
63
+ # Install Beads
64
+ curl -sSL https://raw.githubusercontent.com/steveyegge/beads/main/scripts/install.sh | bash
65
+
66
+ # Initialize in your repo
67
+ bd init
68
+
69
+ # Create your first issue
70
+ bd create "Try out Beads"
71
+ ```
72
+
73
+ ## Learn More
74
+
75
+ - **Documentation**: [github.com/steveyegge/beads/docs](https://github.com/steveyegge/beads/tree/main/docs)
76
+ - **Quick Start Guide**: Run `bd quickstart`
77
+ - **Examples**: [github.com/steveyegge/beads/examples](https://github.com/steveyegge/beads/tree/main/examples)
78
+
79
+ ---
80
+
81
+ *Beads: Issue tracking that moves at the speed of thought* ⚡