@boshu2/vibe-check 2.2.1 → 2.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/plans/2025-12-28-ai-safety-integration-plan.md +326 -0
- package/.agents/research/2025-12-28-ai-platform-security-integration.md +295 -0
- package/.beads/README.md +81 -0
- package/.beads/config.yaml +62 -0
- package/.beads/interactions.jsonl +0 -0
- package/.beads/issues.jsonl +9 -0
- package/.beads/metadata.json +4 -0
- package/.gitattributes +3 -0
- package/AGENTS.md +40 -0
- package/CHANGELOG.md +42 -0
- package/CLAUDE.md +75 -0
- package/dist/ai-safety/contract-drift.d.ts +14 -0
- package/dist/ai-safety/contract-drift.d.ts.map +1 -0
- package/dist/ai-safety/contract-drift.js +230 -0
- package/dist/ai-safety/contract-drift.js.map +1 -0
- package/dist/ai-safety/index.d.ts +43 -0
- package/dist/ai-safety/index.d.ts.map +1 -0
- package/dist/ai-safety/index.js +177 -0
- package/dist/ai-safety/index.js.map +1 -0
- package/dist/ai-safety/scope-violation.d.ts +18 -0
- package/dist/ai-safety/scope-violation.d.ts.map +1 -0
- package/dist/ai-safety/scope-violation.js +150 -0
- package/dist/ai-safety/scope-violation.js.map +1 -0
- package/dist/ai-safety/secret-leakage.d.ts +18 -0
- package/dist/ai-safety/secret-leakage.d.ts.map +1 -0
- package/dist/ai-safety/secret-leakage.js +188 -0
- package/dist/ai-safety/secret-leakage.js.map +1 -0
- package/dist/ai-safety/token-spiral.d.ts +17 -0
- package/dist/ai-safety/token-spiral.d.ts.map +1 -0
- package/dist/ai-safety/token-spiral.js +183 -0
- package/dist/ai-safety/token-spiral.js.map +1 -0
- package/dist/ai-safety/types.d.ts +122 -0
- package/dist/ai-safety/types.d.ts.map +1 -0
- package/dist/ai-safety/types.js +32 -0
- package/dist/ai-safety/types.js.map +1 -0
- package/dist/commands/session.d.ts +9 -0
- package/dist/commands/session.d.ts.map +1 -1
- package/dist/commands/session.js +42 -0
- package/dist/commands/session.js.map +1 -1
- package/dist/commands/watch.d.ts.map +1 -1
- package/dist/commands/watch.js +59 -0
- package/dist/commands/watch.js.map +1 -1
- package/package.json +4 -1
|
@@ -0,0 +1,326 @@
|
|
|
1
|
+
---
|
|
2
|
+
date: 2025-12-28
|
|
3
|
+
type: Plan
|
|
4
|
+
topic: "AI Safety Detection Module for vibe-check"
|
|
5
|
+
research: ".agents/research/2025-12-28-ai-platform-security-integration.md"
|
|
6
|
+
tags: [plan, ai-safety, llm-validation, security, vibe-check]
|
|
7
|
+
status: READY_FOR_APPROVAL
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
# Plan: AI Safety Detection Module
|
|
11
|
+
|
|
12
|
+
**Created:** 2025-12-28
|
|
13
|
+
**Research:** `.agents/research/2025-12-28-ai-platform-security-integration.md`
|
|
14
|
+
**Vibe Level:** L3 (Multiple files, following existing patterns)
|
|
15
|
+
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
## Overview
|
|
19
|
+
|
|
20
|
+
Add a new `src/ai-safety/` module to vibe-check that detects LLM-specific antipatterns in commit history: **secret leakage**, **scope violations**, **contract drift**, and **token spirals**. This extends the existing inner-loop failure detection with patterns ported from ai-platform's production security stack.
|
|
21
|
+
|
|
22
|
+
---
|
|
23
|
+
|
|
24
|
+
## Approach
|
|
25
|
+
|
|
26
|
+
Port security validation patterns from `ai-platform/tests/agents/test_agent_security.py` and adapt them for git-based analysis. Follow the existing `src/inner-loop/` module structure exactly.
|
|
27
|
+
|
|
28
|
+
**Why this approach:**
|
|
29
|
+
1. **Proven patterns:** ai-platform patterns are battle-tested for classified environments
|
|
30
|
+
2. **Non-breaking:** Extends existing architecture without replacing anything
|
|
31
|
+
3. **Familiar structure:** Same module organization as inner-loop
|
|
32
|
+
4. **Zero new dependencies:** Pure TypeScript, uses existing git analysis
|
|
33
|
+
|
|
34
|
+
---
|
|
35
|
+
|
|
36
|
+
## Features
|
|
37
|
+
|
|
38
|
+
### Feature 1: AI Safety Types and Orchestrator
|
|
39
|
+
|
|
40
|
+
**Priority:** P0 (Foundation - must come first)
|
|
41
|
+
**Type:** feature
|
|
42
|
+
**Depends On:** None
|
|
43
|
+
|
|
44
|
+
**Acceptance Criteria:**
|
|
45
|
+
- [ ] `src/ai-safety/types.ts` defines all interfaces
|
|
46
|
+
- [ ] `src/ai-safety/index.ts` exports orchestrator function
|
|
47
|
+
- [ ] `AISafetyAnalysis` interface matches inner-loop pattern
|
|
48
|
+
- [ ] Config supports enable/disable per detector
|
|
49
|
+
- [ ] Compiles without errors
|
|
50
|
+
|
|
51
|
+
**Files Affected:**
|
|
52
|
+
- `src/ai-safety/types.ts` - New file: Type definitions
|
|
53
|
+
- `src/ai-safety/index.ts` - New file: Orchestrator + exports
|
|
54
|
+
|
|
55
|
+
**Test Strategy:** `npm run build` passes, types are importable
|
|
56
|
+
|
|
57
|
+
---
|
|
58
|
+
|
|
59
|
+
### Feature 2: Secret Leakage Detector
|
|
60
|
+
|
|
61
|
+
**Priority:** P1 (Highest value, immediate utility)
|
|
62
|
+
**Type:** feature
|
|
63
|
+
**Depends On:** Feature 1
|
|
64
|
+
|
|
65
|
+
**Acceptance Criteria:**
|
|
66
|
+
- [ ] Detects OpenAI keys (`sk-[a-zA-Z0-9]{48}`)
|
|
67
|
+
- [ ] Detects GitHub PATs (`ghp_[a-zA-Z0-9]{36}`)
|
|
68
|
+
- [ ] Detects GitLab PATs (`glpat-[a-zA-Z0-9]{20}`)
|
|
69
|
+
- [ ] Detects AWS keys (`AKIA[0-9A-Z]{16}`)
|
|
70
|
+
- [ ] Detects Slack tokens (`xox[baprs]-*`)
|
|
71
|
+
- [ ] Detects generic patterns (password=, secret=)
|
|
72
|
+
- [ ] Reports file, line context, and pattern matched
|
|
73
|
+
- [ ] Critical severity by default
|
|
74
|
+
|
|
75
|
+
**Files Affected:**
|
|
76
|
+
- `src/ai-safety/secret-leakage.ts` - New file: Secret detection logic
|
|
77
|
+
- `src/ai-safety/index.ts` - Wire into orchestrator
|
|
78
|
+
|
|
79
|
+
**Test Strategy:**
|
|
80
|
+
1. Create test with fake secrets in commit diffs
|
|
81
|
+
2. Verify detection of each pattern
|
|
82
|
+
3. Verify no false positives on similar-looking strings
|
|
83
|
+
|
|
84
|
+
---
|
|
85
|
+
|
|
86
|
+
### Feature 3: Scope Violation Detector
|
|
87
|
+
|
|
88
|
+
**Priority:** P2 (High value, needs config)
|
|
89
|
+
**Type:** feature
|
|
90
|
+
**Depends On:** Feature 1
|
|
91
|
+
|
|
92
|
+
**Acceptance Criteria:**
|
|
93
|
+
- [ ] Detects commits touching files outside declared scope
|
|
94
|
+
- [ ] Scope defined via config file (`.vibe-check/scope.yaml`) or CLI
|
|
95
|
+
- [ ] Supports glob patterns for scope definition
|
|
96
|
+
- [ ] Reports which files violated scope
|
|
97
|
+
- [ ] Warning severity by default
|
|
98
|
+
|
|
99
|
+
**Files Affected:**
|
|
100
|
+
- `src/ai-safety/scope-violation.ts` - New file: Scope check logic
|
|
101
|
+
- `src/ai-safety/index.ts` - Wire into orchestrator
|
|
102
|
+
- `src/types.ts` - Add AISafetyConfig to main types
|
|
103
|
+
|
|
104
|
+
**Test Strategy:**
|
|
105
|
+
1. Define scope as `src/feature/**`
|
|
106
|
+
2. Create commit touching `src/other/file.ts`
|
|
107
|
+
3. Verify scope violation detected
|
|
108
|
+
|
|
109
|
+
---
|
|
110
|
+
|
|
111
|
+
### Feature 4: Contract Drift Detector
|
|
112
|
+
|
|
113
|
+
**Priority:** P2 (Medium value, pattern analysis)
|
|
114
|
+
**Type:** feature
|
|
115
|
+
**Depends On:** Feature 1
|
|
116
|
+
|
|
117
|
+
**Acceptance Criteria:**
|
|
118
|
+
- [ ] Tracks commit message format compliance over time
|
|
119
|
+
- [ ] Detects degradation from conventional commits format
|
|
120
|
+
- [ ] Tracks entropy (message quality declining)
|
|
121
|
+
- [ ] Reports drift percentage and trend direction
|
|
122
|
+
- [ ] Warning severity when >30% drift from baseline
|
|
123
|
+
|
|
124
|
+
**Files Affected:**
|
|
125
|
+
- `src/ai-safety/contract-drift.ts` - New file: Format drift logic
|
|
126
|
+
- `src/ai-safety/index.ts` - Wire into orchestrator
|
|
127
|
+
|
|
128
|
+
**Test Strategy:**
|
|
129
|
+
1. Provide commits with good format then degrading format
|
|
130
|
+
2. Verify drift percentage calculated
|
|
131
|
+
3. Verify warning triggered at threshold
|
|
132
|
+
|
|
133
|
+
---
|
|
134
|
+
|
|
135
|
+
### Feature 5: Token Spiral Estimator
|
|
136
|
+
|
|
137
|
+
**Priority:** P3 (Lower value, estimation-based)
|
|
138
|
+
**Type:** feature
|
|
139
|
+
**Depends On:** Feature 1
|
|
140
|
+
|
|
141
|
+
**Acceptance Criteria:**
|
|
142
|
+
- [ ] Estimates token usage from commit size
|
|
143
|
+
- [ ] Uses heuristic: ~4 chars per token
|
|
144
|
+
- [ ] Tracks cumulative session tokens
|
|
145
|
+
- [ ] Detects explosion patterns (>2x baseline)
|
|
146
|
+
- [ ] Info severity (advisory, not critical)
|
|
147
|
+
|
|
148
|
+
**Files Affected:**
|
|
149
|
+
- `src/ai-safety/token-spiral.ts` - New file: Token estimation logic
|
|
150
|
+
- `src/ai-safety/index.ts` - Wire into orchestrator
|
|
151
|
+
|
|
152
|
+
**Test Strategy:**
|
|
153
|
+
1. Provide small commits then huge commit
|
|
154
|
+
2. Verify explosion detected
|
|
155
|
+
3. Verify reasonable token estimates
|
|
156
|
+
|
|
157
|
+
---
|
|
158
|
+
|
|
159
|
+
### Feature 6: Integration with Session End
|
|
160
|
+
|
|
161
|
+
**Priority:** P1 (Makes feature visible to users)
|
|
162
|
+
**Type:** feature
|
|
163
|
+
**Depends On:** Features 1-2 (minimum)
|
|
164
|
+
|
|
165
|
+
**Acceptance Criteria:**
|
|
166
|
+
- [ ] `session end` JSON output includes `ai_safety` section
|
|
167
|
+
- [ ] Terminal output shows AI safety warnings
|
|
168
|
+
- [ ] Critical findings show in red
|
|
169
|
+
- [ ] Recommendations generated for each finding
|
|
170
|
+
|
|
171
|
+
**Files Affected:**
|
|
172
|
+
- `src/commands/session.ts` - Add AI safety analysis call
|
|
173
|
+
- `src/output/terminal.ts` - Format AI safety for terminal
|
|
174
|
+
- `src/output/json.ts` - Format AI safety for JSON
|
|
175
|
+
|
|
176
|
+
**Test Strategy:** `vibe-check session end --format json` includes ai_safety section
|
|
177
|
+
|
|
178
|
+
---
|
|
179
|
+
|
|
180
|
+
### Feature 7: Integration with Watch Command
|
|
181
|
+
|
|
182
|
+
**Priority:** P2 (Real-time value)
|
|
183
|
+
**Type:** feature
|
|
184
|
+
**Depends On:** Feature 2 (secret detection)
|
|
185
|
+
|
|
186
|
+
**Acceptance Criteria:**
|
|
187
|
+
- [ ] Watch mode alerts on secret detection immediately
|
|
188
|
+
- [ ] Uses red/bold for critical findings
|
|
189
|
+
- [ ] Shows file and pattern matched
|
|
190
|
+
- [ ] Rate-limited (don't spam alerts)
|
|
191
|
+
|
|
192
|
+
**Files Affected:**
|
|
193
|
+
- `src/commands/watch.ts` - Add secret detection to watch loop
|
|
194
|
+
|
|
195
|
+
**Test Strategy:** Run watch, commit file with fake secret, see immediate alert
|
|
196
|
+
|
|
197
|
+
---
|
|
198
|
+
|
|
199
|
+
### Feature 8: Tests and Documentation
|
|
200
|
+
|
|
201
|
+
**Priority:** P2 (Quality assurance)
|
|
202
|
+
**Type:** task
|
|
203
|
+
**Depends On:** Features 1-5
|
|
204
|
+
|
|
205
|
+
**Acceptance Criteria:**
|
|
206
|
+
- [ ] Unit tests for each detector
|
|
207
|
+
- [ ] Integration test for orchestrator
|
|
208
|
+
- [ ] CLAUDE.md updated with ai-safety commands
|
|
209
|
+
- [ ] README.md mentions AI safety features
|
|
210
|
+
|
|
211
|
+
**Files Affected:**
|
|
212
|
+
- `tests/ai-safety.test.ts` - New file: Test suite
|
|
213
|
+
- `CLAUDE.md` - Update documentation
|
|
214
|
+
- `README.md` - Feature description
|
|
215
|
+
|
|
216
|
+
**Test Strategy:** `npm test` passes with >80% coverage on new module
|
|
217
|
+
|
|
218
|
+
---
|
|
219
|
+
|
|
220
|
+
## Implementation Order
|
|
221
|
+
|
|
222
|
+
| Step | Feature | Depends On | Validation |
|
|
223
|
+
|------|---------|------------|------------|
|
|
224
|
+
| 1 | Types and Orchestrator | - | `npm run build` passes |
|
|
225
|
+
| 2 | Secret Leakage | Step 1 | Test with fake secrets |
|
|
226
|
+
| 3 | Session Integration | Steps 1-2 | `session end` shows findings |
|
|
227
|
+
| 4 | Scope Violation | Step 1 | Test with scope config |
|
|
228
|
+
| 5 | Contract Drift | Step 1 | Test with degrading commits |
|
|
229
|
+
| 6 | Token Spiral | Step 1 | Test with varying sizes |
|
|
230
|
+
| 7 | Watch Integration | Step 2 | Real-time alert test |
|
|
231
|
+
| 8 | Tests & Docs | Steps 1-6 | `npm test` passes |
|
|
232
|
+
|
|
233
|
+
---
|
|
234
|
+
|
|
235
|
+
## Beads Issues to Create
|
|
236
|
+
|
|
237
|
+
After approval, these issues will be created:
|
|
238
|
+
|
|
239
|
+
| ID | Title | Type | Priority | Depends On |
|
|
240
|
+
|----|-------|------|----------|------------|
|
|
241
|
+
| TBD | Epic: AI Safety Detection Module | epic | P1 | - |
|
|
242
|
+
| TBD | AI Safety types and orchestrator | feature | P0 | Epic |
|
|
243
|
+
| TBD | Secret leakage detector | feature | P1 | Types |
|
|
244
|
+
| TBD | Session end AI safety integration | feature | P1 | Secret detector |
|
|
245
|
+
| TBD | Scope violation detector | feature | P2 | Types |
|
|
246
|
+
| TBD | Contract drift detector | feature | P2 | Types |
|
|
247
|
+
| TBD | Token spiral estimator | feature | P3 | Types |
|
|
248
|
+
| TBD | Watch command AI safety integration | feature | P2 | Secret detector |
|
|
249
|
+
| TBD | Tests and documentation | task | P2 | All features |
|
|
250
|
+
|
|
251
|
+
---
|
|
252
|
+
|
|
253
|
+
## File Structure After Implementation
|
|
254
|
+
|
|
255
|
+
```
|
|
256
|
+
src/ai-safety/
|
|
257
|
+
├── index.ts # Orchestrator: analyzeAISafety()
|
|
258
|
+
├── types.ts # Interfaces: AISafetyAnalysis, configs
|
|
259
|
+
├── secret-leakage.ts # Regex-based secret detection
|
|
260
|
+
├── scope-violation.ts # Glob-based scope checking
|
|
261
|
+
├── contract-drift.ts # Commit format degradation
|
|
262
|
+
└── token-spiral.ts # Token usage estimation
|
|
263
|
+
|
|
264
|
+
tests/ai-safety.test.ts # Full test suite
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
---
|
|
268
|
+
|
|
269
|
+
## Key Interfaces (Preview)
|
|
270
|
+
|
|
271
|
+
```typescript
|
|
272
|
+
// src/ai-safety/types.ts
|
|
273
|
+
export interface AISafetyAnalysis {
|
|
274
|
+
secretLeakage: SecretLeakageResult;
|
|
275
|
+
scopeViolations: ScopeViolationResult;
|
|
276
|
+
contractDrift: ContractDriftResult;
|
|
277
|
+
tokenSpiral: TokenSpiralResult;
|
|
278
|
+
summary: {
|
|
279
|
+
totalIssues: number;
|
|
280
|
+
criticalIssues: number;
|
|
281
|
+
warningIssues: number;
|
|
282
|
+
overallHealth: 'healthy' | 'warning' | 'critical';
|
|
283
|
+
};
|
|
284
|
+
recommendations: string[];
|
|
285
|
+
}
|
|
286
|
+
|
|
287
|
+
export interface SecretLeakageResult {
|
|
288
|
+
detected: boolean;
|
|
289
|
+
findings: SecretFinding[];
|
|
290
|
+
message: string;
|
|
291
|
+
}
|
|
292
|
+
|
|
293
|
+
export interface SecretFinding {
|
|
294
|
+
commitHash: string;
|
|
295
|
+
file: string;
|
|
296
|
+
pattern: string;
|
|
297
|
+
lineContext: string;
|
|
298
|
+
severity: 'critical' | 'warning';
|
|
299
|
+
}
|
|
300
|
+
```
|
|
301
|
+
|
|
302
|
+
---
|
|
303
|
+
|
|
304
|
+
## Rollback Procedure
|
|
305
|
+
|
|
306
|
+
If something goes wrong:
|
|
307
|
+
|
|
308
|
+
1. **Module not loading:** Revert `src/ai-safety/` directory
|
|
309
|
+
2. **Session command broken:** Revert `src/commands/session.ts` changes
|
|
310
|
+
3. **Watch command broken:** Revert `src/commands/watch.ts` changes
|
|
311
|
+
4. **Build fails:** Check `src/ai-safety/index.ts` exports
|
|
312
|
+
|
|
313
|
+
All changes are additive - existing functionality is not modified, only extended.
|
|
314
|
+
|
|
315
|
+
---
|
|
316
|
+
|
|
317
|
+
## Next Steps
|
|
318
|
+
|
|
319
|
+
1. **Review and approve this plan**
|
|
320
|
+
2. **Create beads issues** (commands below)
|
|
321
|
+
3. **`bd ready`** to see unblocked issues
|
|
322
|
+
4. **`/implement`** to execute first issue
|
|
323
|
+
|
|
324
|
+
---
|
|
325
|
+
|
|
326
|
+
**Output:** .agents/plans/2025-12-28-ai-safety-integration-plan.md
|
|
@@ -0,0 +1,295 @@
|
|
|
1
|
+
---
|
|
2
|
+
date: 2025-12-28
|
|
3
|
+
type: Research
|
|
4
|
+
topic: "Integrating ai-platform security and LLM validation into vibe-check"
|
|
5
|
+
tags: [research, security, llm-validation, integration, ai-platform, vibe-check]
|
|
6
|
+
status: COMPLETE
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# Research: Integrating AI-Platform Security & LLM Validation into Vibe-Check
|
|
10
|
+
|
|
11
|
+
**Created:** 2025-12-28
|
|
12
|
+
**Goal:** Understand how to integrate the security and LLM validation components from ai-platform into vibe-check for detecting AI agent misbehavior patterns.
|
|
13
|
+
|
|
14
|
+
---
|
|
15
|
+
|
|
16
|
+
## Executive Summary
|
|
17
|
+
|
|
18
|
+
The **ai-platform** repository (`/Users/fullerbt/workspaces/work/ai-platform`) contains a comprehensive security and LLM validation stack built for production use in classified environments. Key components include: **RBAC access control**, **audit logging** (SIEM-ready), **agent execution tracking** (Prometheus metrics), **rate limiting with token budgets**, and **contract-driven agent testing**.
|
|
19
|
+
|
|
20
|
+
**Recommendation:** Integrate these concepts into vibe-check as a new `ai-safety/` or `agent-validation/` module, extending the existing inner-loop failure pattern detection to include LLM-specific antipatterns like **prompt injection attempts**, **hallucination markers**, **scope violations**, and **contract drift**.
|
|
21
|
+
|
|
22
|
+
---
|
|
23
|
+
|
|
24
|
+
## Current State
|
|
25
|
+
|
|
26
|
+
### What Exists in ai-platform
|
|
27
|
+
|
|
28
|
+
| Component | Location | Purpose |
|
|
29
|
+
|-----------|----------|---------|
|
|
30
|
+
| **Access Control** | `services/gateway/access_control.py` | RBAC for agents/tools via OIDC groups |
|
|
31
|
+
| **Audit Logging** | `services/gateway/audit.py` | JSON request/response logging for SIEM |
|
|
32
|
+
| **Agent Audit** | `services/gateway/agent_audit.py` | Agent execution tracking + Prometheus metrics |
|
|
33
|
+
| **Rate Limiting** | `services/gateway/rate_limit.py` | Per-user rate limits + token budgets |
|
|
34
|
+
| **Config Validator** | `services/etl/app/config_validator.py` | Startup config health checks |
|
|
35
|
+
| **Security Tests** | `tests/agents/test_agent_security.py` | Boundary violation detection |
|
|
36
|
+
| **Contract Tests** | `tests/agents/test_agent_contract_validation.py` | Output contract compliance |
|
|
37
|
+
| **Platform Validator** | `services/etl/scripts/validate-platform.py` | End-to-end LLM platform checks |
|
|
38
|
+
|
|
39
|
+
### Key Patterns Worth Porting to vibe-check
|
|
40
|
+
|
|
41
|
+
1. **Agent Scope Boundary Detection** (`test_agent_security.py`)
|
|
42
|
+
- Detects when agents attempt actions outside their declared scope
|
|
43
|
+
- Pattern: Check if commit changes files/APIs not in the agent's domain
|
|
44
|
+
|
|
45
|
+
2. **Hallucination Markers** (`test_agent_security.py:50-70`)
|
|
46
|
+
- Detects file path hallucination (made-up paths)
|
|
47
|
+
- Pattern: Commits referencing non-existent files or phantom dependencies
|
|
48
|
+
|
|
49
|
+
3. **Secret Leakage Detection** (`test_agent_security.py:120-150`)
|
|
50
|
+
- Regex patterns for API keys, tokens, credentials
|
|
51
|
+
- Pattern: Commits accidentally exposing secrets
|
|
52
|
+
|
|
53
|
+
4. **Contract Drift Detection** (`test_agent_contract_validation.py`)
|
|
54
|
+
- Validates agent responses match expected structure
|
|
55
|
+
- Pattern: Detect when AI outputs stop conforming to expected formats
|
|
56
|
+
|
|
57
|
+
5. **Token Budget Tracking** (`rate_limit.py`)
|
|
58
|
+
- Tracks token consumption per user/session
|
|
59
|
+
- Pattern: Detect runaway token usage (context explosion)
|
|
60
|
+
|
|
61
|
+
### Key Files
|
|
62
|
+
|
|
63
|
+
| File | Purpose | Relevance to vibe-check |
|
|
64
|
+
|------|---------|-------------------------|
|
|
65
|
+
| `services/gateway/access_control.py` | RBAC via OIDC groups | Could map to commit author scope detection |
|
|
66
|
+
| `services/gateway/audit.py` | JSON audit logging | Structure for storing AI interaction events |
|
|
67
|
+
| `services/gateway/agent_audit.py` | Prometheus metrics + Langfuse traces | Pattern for tracking AI agent behavior over time |
|
|
68
|
+
| `tests/agents/test_agent_security.py` | Security test patterns | Regex patterns and violation detection logic |
|
|
69
|
+
| `services/gateway/rate_limit.py` | Token budget tracking | Pattern for detecting context window abuse |
|
|
70
|
+
|
|
71
|
+
### Existing Patterns in vibe-check
|
|
72
|
+
|
|
73
|
+
vibe-check already has:
|
|
74
|
+
- **Inner-loop failure detection** (`src/inner-loop/`) with 4 detectors
|
|
75
|
+
- **Spiral detection** with pattern regexes in `watch.ts`
|
|
76
|
+
- **Session tracking** with baseline comparison
|
|
77
|
+
- **NDJSON storage** for historical pattern analysis
|
|
78
|
+
- **Prometheus-like metrics** (not exposed, but structured similarly)
|
|
79
|
+
|
|
80
|
+
---
|
|
81
|
+
|
|
82
|
+
## Findings
|
|
83
|
+
|
|
84
|
+
### Finding 1: Security Validation Patterns Are Git-Analyzable
|
|
85
|
+
|
|
86
|
+
**Evidence:** The ai-platform security tests detect violations by analyzing:
|
|
87
|
+
- Commit messages for intent signals
|
|
88
|
+
- File changes for scope violations
|
|
89
|
+
- Code content for secret patterns
|
|
90
|
+
|
|
91
|
+
**Implication:** These patterns can be adapted for vibe-check's commit-based analysis:
|
|
92
|
+
```
|
|
93
|
+
ai-platform pattern → vibe-check integration
|
|
94
|
+
-------------------------------------------
|
|
95
|
+
Scope violation test → Detect commits touching files outside declared domain
|
|
96
|
+
Secret leakage test → Detect commits adding API keys/tokens
|
|
97
|
+
Hallucination test → Detect commits referencing non-existent paths
|
|
98
|
+
Contract drift test → Detect commits breaking expected output formats
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
### Finding 2: Agent Audit Structure Is Session-Compatible
|
|
102
|
+
|
|
103
|
+
**Evidence:** `agent_audit.py` tracks events in a structure similar to vibe-check sessions:
|
|
104
|
+
```python
|
|
105
|
+
audit_record = {
|
|
106
|
+
"correlation_id": str, # → session_id
|
|
107
|
+
"agent_name": str, # → (new field)
|
|
108
|
+
"user_identity": str, # → (already tracked)
|
|
109
|
+
"tokens_used": { # → (new metric)
|
|
110
|
+
"input": int,
|
|
111
|
+
"output": int,
|
|
112
|
+
"total": int
|
|
113
|
+
},
|
|
114
|
+
"tool_invocations": [...], # → (map to commit file changes)
|
|
115
|
+
"duration_ms": int # → (session duration)
|
|
116
|
+
}
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
**Implication:** Can extend `active-session.json` with agent/LLM metadata.
|
|
120
|
+
|
|
121
|
+
### Finding 3: Rate Limiting = Token Spiral Detection
|
|
122
|
+
|
|
123
|
+
**Evidence:** `rate_limit.py` implements token budget tracking:
|
|
124
|
+
```python
|
|
125
|
+
@dataclass
|
|
126
|
+
class UserRateLimit:
|
|
127
|
+
max_tokens_per_minute: int = 100_000
|
|
128
|
+
max_tokens_per_hour: int = 1_000_000
|
|
129
|
+
max_tokens_per_day: int = 10_000_000
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
**Implication:** vibe-check could detect "token spirals" - sessions where token consumption explodes, indicating:
|
|
133
|
+
- Context window abuse
|
|
134
|
+
- Prompt stuffing
|
|
135
|
+
- Repeated failed attempts (AI trying same thing repeatedly)
|
|
136
|
+
|
|
137
|
+
### Finding 4: Secret Detection Regex Is Ready to Use
|
|
138
|
+
|
|
139
|
+
**Evidence:** `test_agent_security.py:120-150`:
|
|
140
|
+
```python
|
|
141
|
+
SECRET_PATTERNS = [
|
|
142
|
+
r'sk-[a-zA-Z0-9]{48}', # OpenAI
|
|
143
|
+
r'ghp_[a-zA-Z0-9]{36}', # GitHub PAT
|
|
144
|
+
r'glpat-[a-zA-Z0-9]{20}', # GitLab PAT
|
|
145
|
+
r'AKIA[0-9A-Z]{16}', # AWS Access Key
|
|
146
|
+
r'xox[baprs]-[a-zA-Z0-9-]+', # Slack tokens
|
|
147
|
+
]
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
**Implication:** Can add a `secret-leakage.ts` detector to inner-loop.
|
|
151
|
+
|
|
152
|
+
### Finding 5: Contract Validation Is Output-Pattern Detection
|
|
153
|
+
|
|
154
|
+
**Evidence:** `test_agent_contract_validation.py` validates agent outputs against YAML-defined contracts:
|
|
155
|
+
```yaml
|
|
156
|
+
agents:
|
|
157
|
+
- name: code-review
|
|
158
|
+
output_contract:
|
|
159
|
+
required_sections: ["summary", "issues", "suggestions"]
|
|
160
|
+
max_length: 5000
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
**Implication:** vibe-check could detect "contract drift" - when AI commits start deviating from expected patterns:
|
|
164
|
+
- Commit messages becoming less structured
|
|
165
|
+
- PR descriptions missing required sections
|
|
166
|
+
- Code comments becoming vague or absent
|
|
167
|
+
|
|
168
|
+
---
|
|
169
|
+
|
|
170
|
+
## Constraints
|
|
171
|
+
|
|
172
|
+
| Constraint | Impact | Mitigation |
|
|
173
|
+
|------------|--------|------------|
|
|
174
|
+
| ai-platform is Python, vibe-check is TypeScript | Can't share code directly | Port patterns/logic, not code |
|
|
175
|
+
| ai-platform runs in Kubernetes, vibe-check is CLI | Different runtime context | Adapt to git-based detection |
|
|
176
|
+
| ai-platform has Prometheus/Langfuse dependencies | Heavy deps for CLI tool | Use file-based storage, optionally export |
|
|
177
|
+
| Token tracking requires LLM API integration | vibe-check only sees git | Estimate tokens from commit size/complexity |
|
|
178
|
+
|
|
179
|
+
---
|
|
180
|
+
|
|
181
|
+
## Risks
|
|
182
|
+
|
|
183
|
+
| Risk | Likelihood | Impact | Mitigation |
|
|
184
|
+
|------|------------|--------|------------|
|
|
185
|
+
| Over-engineering simple CLI tool | Medium | High | Start with 2-3 highest-value patterns |
|
|
186
|
+
| False positives on secret detection | Medium | Medium | Require confidence threshold, allow suppression |
|
|
187
|
+
| Token estimation inaccuracy | High | Low | Use as relative signal, not absolute |
|
|
188
|
+
| Breaking existing inner-loop interface | Low | High | Extend, don't replace |
|
|
189
|
+
|
|
190
|
+
---
|
|
191
|
+
|
|
192
|
+
## Recommendation
|
|
193
|
+
|
|
194
|
+
**Approach:** Add a new `src/ai-safety/` module to vibe-check with 4 new detectors:
|
|
195
|
+
|
|
196
|
+
### Phase 1: High-Value Ports (Recommended First)
|
|
197
|
+
|
|
198
|
+
1. **Secret Leakage Detector** (`src/ai-safety/secret-leakage.ts`)
|
|
199
|
+
- Port regex patterns from `test_agent_security.py`
|
|
200
|
+
- Scan commit diffs for exposed secrets
|
|
201
|
+
- Integrate into `session end` output
|
|
202
|
+
|
|
203
|
+
2. **Scope Violation Detector** (`src/ai-safety/scope-violation.ts`)
|
|
204
|
+
- Detect commits touching files outside declared domain
|
|
205
|
+
- Requires domain configuration (similar to access_control.py mappings)
|
|
206
|
+
- Warning when agent strays from its lane
|
|
207
|
+
|
|
208
|
+
### Phase 2: Medium-Value Ports
|
|
209
|
+
|
|
210
|
+
3. **Contract Drift Detector** (`src/ai-safety/contract-drift.ts`)
|
|
211
|
+
- Detect when commit message patterns degrade
|
|
212
|
+
- Track deviation from conventional commit format
|
|
213
|
+
- Alert when AI stops following established patterns
|
|
214
|
+
|
|
215
|
+
4. **Token Spiral Estimator** (`src/ai-safety/token-spiral.ts`)
|
|
216
|
+
- Estimate token usage from commit size/complexity
|
|
217
|
+
- Detect sessions with exploding context
|
|
218
|
+
- Use as relative metric (not absolute)
|
|
219
|
+
|
|
220
|
+
### Integration Points
|
|
221
|
+
|
|
222
|
+
```typescript
|
|
223
|
+
// src/ai-safety/index.ts
|
|
224
|
+
export interface AISafetyAnalysis {
|
|
225
|
+
secretsDetected: SecretLeakageResult;
|
|
226
|
+
scopeViolations: ScopeViolationResult;
|
|
227
|
+
contractDrift: ContractDriftResult;
|
|
228
|
+
tokenSpiral: TokenSpiralResult;
|
|
229
|
+
summary: {
|
|
230
|
+
totalIssues: number;
|
|
231
|
+
criticalIssues: number;
|
|
232
|
+
warningIssues: number;
|
|
233
|
+
overallHealth: 'healthy' | 'warning' | 'critical';
|
|
234
|
+
};
|
|
235
|
+
recommendations: string[];
|
|
236
|
+
}
|
|
237
|
+
|
|
238
|
+
export function analyzeAISafety(
|
|
239
|
+
commits: Commit[],
|
|
240
|
+
filesPerCommit: Map<string, string[]>,
|
|
241
|
+
config?: AISafetyConfig
|
|
242
|
+
): AISafetyAnalysis;
|
|
243
|
+
```
|
|
244
|
+
|
|
245
|
+
**Integration into existing commands:**
|
|
246
|
+
- `session end` → Add `ai_safety` section to JSON output
|
|
247
|
+
- `watch` → Add real-time alerts for secret detection
|
|
248
|
+
- `insights` → Add AI safety pattern history
|
|
249
|
+
|
|
250
|
+
### Why This Approach
|
|
251
|
+
|
|
252
|
+
1. **Non-breaking:** Extends existing architecture, doesn't replace
|
|
253
|
+
2. **High-value first:** Secret leakage is immediately useful
|
|
254
|
+
3. **Familiar patterns:** Same structure as existing inner-loop detectors
|
|
255
|
+
4. **Low deps:** No new dependencies required (pure TypeScript)
|
|
256
|
+
5. **Git-native:** Works with existing commit-based analysis
|
|
257
|
+
|
|
258
|
+
---
|
|
259
|
+
|
|
260
|
+
## Alternatives Considered
|
|
261
|
+
|
|
262
|
+
1. **Import ai-platform as dependency** - Rejected: Different language, heavy deps
|
|
263
|
+
2. **Create shared npm/pip packages** - Rejected: Over-engineering for 4 patterns
|
|
264
|
+
3. **Keep validation in ai-platform only** - Rejected: vibe-check users need these patterns
|
|
265
|
+
4. **Port entire audit system** - Rejected: Too complex, different runtime
|
|
266
|
+
|
|
267
|
+
---
|
|
268
|
+
|
|
269
|
+
## Next Steps
|
|
270
|
+
|
|
271
|
+
1. Run `/plan` to create implementation plan from this research
|
|
272
|
+
2. Plan will create beads issues for each detector
|
|
273
|
+
3. Implement in priority order: secrets → scope → contract → token
|
|
274
|
+
|
|
275
|
+
---
|
|
276
|
+
|
|
277
|
+
## Appendix: Key Source File References
|
|
278
|
+
|
|
279
|
+
### ai-platform Security Components
|
|
280
|
+
- `/Users/fullerbt/workspaces/work/ai-platform/services/gateway/access_control.py` - RBAC implementation
|
|
281
|
+
- `/Users/fullerbt/workspaces/work/ai-platform/services/gateway/audit.py` - Audit logging
|
|
282
|
+
- `/Users/fullerbt/workspaces/work/ai-platform/services/gateway/agent_audit.py` - Agent execution tracking
|
|
283
|
+
- `/Users/fullerbt/workspaces/work/ai-platform/services/gateway/rate_limit.py` - Rate limiting
|
|
284
|
+
- `/Users/fullerbt/workspaces/work/ai-platform/tests/agents/test_agent_security.py` - Security tests
|
|
285
|
+
- `/Users/fullerbt/workspaces/work/ai-platform/tests/agents/test_agent_contract_validation.py` - Contract tests
|
|
286
|
+
|
|
287
|
+
### vibe-check Integration Points
|
|
288
|
+
- `/Users/fullerbt/workspaces/personal/vibe-check/src/inner-loop/index.ts` - Existing detector orchestrator
|
|
289
|
+
- `/Users/fullerbt/workspaces/personal/vibe-check/src/commands/session.ts` - Session end output
|
|
290
|
+
- `/Users/fullerbt/workspaces/personal/vibe-check/src/commands/watch.ts` - Real-time monitoring
|
|
291
|
+
- `/Users/fullerbt/workspaces/personal/vibe-check/src/types.ts` - Type definitions
|
|
292
|
+
|
|
293
|
+
---
|
|
294
|
+
|
|
295
|
+
**Output:** .agents/research/2025-12-28-ai-platform-security-integration.md
|
package/.beads/README.md
ADDED
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
# Beads - AI-Native Issue Tracking
|
|
2
|
+
|
|
3
|
+
Welcome to Beads! This repository uses **Beads** for issue tracking - a modern, AI-native tool designed to live directly in your codebase alongside your code.
|
|
4
|
+
|
|
5
|
+
## What is Beads?
|
|
6
|
+
|
|
7
|
+
Beads is issue tracking that lives in your repo, making it perfect for AI coding agents and developers who want their issues close to their code. No web UI required - everything works through the CLI and integrates seamlessly with git.
|
|
8
|
+
|
|
9
|
+
**Learn more:** [github.com/steveyegge/beads](https://github.com/steveyegge/beads)
|
|
10
|
+
|
|
11
|
+
## Quick Start
|
|
12
|
+
|
|
13
|
+
### Essential Commands
|
|
14
|
+
|
|
15
|
+
```bash
|
|
16
|
+
# Create new issues
|
|
17
|
+
bd create "Add user authentication"
|
|
18
|
+
|
|
19
|
+
# View all issues
|
|
20
|
+
bd list
|
|
21
|
+
|
|
22
|
+
# View issue details
|
|
23
|
+
bd show <issue-id>
|
|
24
|
+
|
|
25
|
+
# Update issue status
|
|
26
|
+
bd update <issue-id> --status in_progress
|
|
27
|
+
bd update <issue-id> --status done
|
|
28
|
+
|
|
29
|
+
# Sync with git remote
|
|
30
|
+
bd sync
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
### Working with Issues
|
|
34
|
+
|
|
35
|
+
Issues in Beads are:
|
|
36
|
+
- **Git-native**: Stored in `.beads/issues.jsonl` and synced like code
|
|
37
|
+
- **AI-friendly**: CLI-first design works perfectly with AI coding agents
|
|
38
|
+
- **Branch-aware**: Issues can follow your branch workflow
|
|
39
|
+
- **Always in sync**: Auto-syncs with your commits
|
|
40
|
+
|
|
41
|
+
## Why Beads?
|
|
42
|
+
|
|
43
|
+
✨ **AI-Native Design**
|
|
44
|
+
- Built specifically for AI-assisted development workflows
|
|
45
|
+
- CLI-first interface works seamlessly with AI coding agents
|
|
46
|
+
- No context switching to web UIs
|
|
47
|
+
|
|
48
|
+
🚀 **Developer Focused**
|
|
49
|
+
- Issues live in your repo, right next to your code
|
|
50
|
+
- Works offline, syncs when you push
|
|
51
|
+
- Fast, lightweight, and stays out of your way
|
|
52
|
+
|
|
53
|
+
🔧 **Git Integration**
|
|
54
|
+
- Automatic sync with git commits
|
|
55
|
+
- Branch-aware issue tracking
|
|
56
|
+
- Intelligent JSONL merge resolution
|
|
57
|
+
|
|
58
|
+
## Get Started with Beads
|
|
59
|
+
|
|
60
|
+
Try Beads in your own projects:
|
|
61
|
+
|
|
62
|
+
```bash
|
|
63
|
+
# Install Beads
|
|
64
|
+
curl -sSL https://raw.githubusercontent.com/steveyegge/beads/main/scripts/install.sh | bash
|
|
65
|
+
|
|
66
|
+
# Initialize in your repo
|
|
67
|
+
bd init
|
|
68
|
+
|
|
69
|
+
# Create your first issue
|
|
70
|
+
bd create "Try out Beads"
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
## Learn More
|
|
74
|
+
|
|
75
|
+
- **Documentation**: [github.com/steveyegge/beads/docs](https://github.com/steveyegge/beads/tree/main/docs)
|
|
76
|
+
- **Quick Start Guide**: Run `bd quickstart`
|
|
77
|
+
- **Examples**: [github.com/steveyegge/beads/examples](https://github.com/steveyegge/beads/tree/main/examples)
|
|
78
|
+
|
|
79
|
+
---
|
|
80
|
+
|
|
81
|
+
*Beads: Issue tracking that moves at the speed of thought* ⚡
|