runward 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/NOTICE.md +13 -0
- package/README.md +128 -0
- package/dist/cli.js +76 -0
- package/dist/commands/check.js +50 -0
- package/dist/commands/doctor.js +75 -0
- package/dist/commands/init.js +90 -0
- package/dist/commands/status.js +49 -0
- package/dist/commands/update.js +48 -0
- package/dist/lib/constants.js +2 -0
- package/dist/lib/mission.js +96 -0
- package/dist/lib/paths.js +28 -0
- package/dist/lib/styles.js +60 -0
- package/dist/lib/tools.js +59 -0
- package/dist/lib/write.js +30 -0
- package/package.json +29 -0
- package/templates/mission/adr/ADR-0000-template.md +38 -0
- package/templates/mission/architecture.md +60 -0
- package/templates/mission/decision-matrix.md +37 -0
- package/templates/mission/evaluation-rubric.md +60 -0
- package/templates/mission/floor.md +47 -0
- package/templates/mission/framing.md +58 -0
- package/templates/mission/mission-contract.md +58 -0
- package/templates/mission/observability-schema.md +53 -0
- package/templates/mission/port-contract.md +63 -0
- package/templates/mission/reference-stack.md +70 -0
- package/templates/mission/runbook.md +70 -0
- package/templates/mission/shared-bricks.md +74 -0
- package/templates/mission/threat-model.md +53 -0
- package/templates/rules/async-job-guardrails.md +95 -0
- package/templates/rules/async-post-turn-pipeline.md +96 -0
- package/templates/rules/async-scheduled-maintenance.md +95 -0
- package/templates/rules/cache-three-tier-architecture.md +44 -0
- package/templates/rules/checklist-day-zero-project.md +97 -0
- package/templates/rules/checklist-pre-production-observability.md +99 -0
- package/templates/rules/checklist-pre-production-performance.md +109 -0
- package/templates/rules/checklist-pre-production-resilience.md +99 -0
- package/templates/rules/checklist-pre-production-security.md +86 -0
- package/templates/rules/config-secrets-boundary.md +57 -0
- package/templates/rules/config-typing-zod.md +78 -0
- package/templates/rules/contracts-governance.md +64 -0
- package/templates/rules/data-memory-consolidation.md +81 -0
- package/templates/rules/data-memory-invalidation.md +90 -0
- package/templates/rules/data-memory-scoring.md +154 -0
- package/templates/rules/data-migrations-forward-only.md +55 -0
- package/templates/rules/data-orphan-cleanup.md +142 -0
- package/templates/rules/data-ttl-types.md +100 -0
- package/templates/rules/eval-loop.md +52 -0
- package/templates/rules/frontier-deterministic-boundary.md +91 -0
- package/templates/rules/hexa-adapter-pattern.md +85 -0
- package/templates/rules/hexa-architecture.md +84 -0
- package/templates/rules/hexa-move-deterministic-out.md +93 -0
- package/templates/rules/hexa-recommended-stack.md +33 -0
- package/templates/rules/hexa-typescript-native.md +80 -0
- package/templates/rules/observability-alert-configuration.md +160 -0
- package/templates/rules/observability-llm-metrics.md +150 -0
- package/templates/rules/observability-startup-provider-log.md +128 -0
- package/templates/rules/observability-structured-json-logs.md +138 -0
- package/templates/rules/patterns-memory-router-tiered.md +139 -0
- package/templates/rules/patterns-prompt-compiler.md +126 -0
- package/templates/rules/patterns-request-id-propagation.md +137 -0
- package/templates/rules/process-adr-and-journal.md +50 -0
- package/templates/rules/provider-llm-auto-detection.md +54 -0
- package/templates/rules/provider-no-crash-missing-env.md +67 -0
- package/templates/rules/quality-codebase-metrics.md +115 -0
- package/templates/rules/quality-zod-input-validation.md +131 -0
- package/templates/rules/resilience-fail-open.md +65 -0
- package/templates/rules/resilience-multi-provider-fallback.md +100 -0
- package/templates/rules/resilience-retry-backoff.md +115 -0
- package/templates/rules/resilience-retryable-errors.md +107 -0
- package/templates/rules/routing-confidence-upgrade.md +89 -0
- package/templates/rules/routing-model-cost-ratios.md +68 -0
- package/templates/rules/routing-smart-complexity.md +131 -0
- package/templates/rules/scaling-db-connection-pooling.md +121 -0
- package/templates/rules/scaling-distributed-rate-limiting.md +141 -0
- package/templates/rules/scaling-state-externalization.md +84 -0
- package/templates/rules/security-prompt-injection.md +63 -0
- package/templates/rules/state-event-sourcing.md +66 -0
- package/templates/rules/tools-registry-pattern.md +153 -0
- package/templates/rules/tools-scope-atomicity.md +103 -0
- package/templates/targets/AGENTS.md +32 -0
- package/templates/workflows/architect.md +50 -0
- package/templates/workflows/brownfield.md +52 -0
- package/templates/workflows/decision-loop.md +52 -0
- package/templates/workflows/floor.md +60 -0
- package/templates/workflows/frame.md +69 -0
- package/templates/workflows/govern.md +55 -0
- package/templates/workflows/handover.md +55 -0
- package/templates/workflows/iterate.md +52 -0
- package/templates/workflows/method.md +53 -0
- package/templates/workflows/review.md +59 -0
|
@@ -0,0 +1,142 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Orphan Data Cleanup
|
|
3
|
+
impact: MEDIUM
|
|
4
|
+
impactDescription: Prevents data bloat from orphaned records
|
|
5
|
+
tags: [data, cleanup, maintenance, integrity]
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
## Orphan Data Cleanup
|
|
9
|
+
|
|
10
|
+
Orphaned records waste storage and can cause bugs. Regular cleanup is essential.
|
|
11
|
+
|
|
12
|
+
**Common Orphan Types:**
|
|
13
|
+
|
|
14
|
+
| Orphan Type | Detection | Action |
|
|
15
|
+
|-------------|-----------|--------|
|
|
16
|
+
| Project without owner | Cron weekly | Notify admin, archive after 30d |
|
|
17
|
+
| Messages without session | Cron daily | Anonymize + archive |
|
|
18
|
+
| Files (S3) without DB ref | Cron monthly | Quarantine then delete |
|
|
19
|
+
| Broken DB relationships | Cron weekly | Rebuild or delete |
|
|
20
|
+
| Unused embeddings | Cron monthly | Delete |
|
|
21
|
+
|
|
22
|
+
**Implementation:**
|
|
23
|
+
|
|
24
|
+
```typescript
|
|
25
|
+
// Orphan detection and cleanup service
|
|
26
|
+
class OrphanCleanupService {
|
|
27
|
+
async findOrphanProjects(): Promise<Project[]> {
|
|
28
|
+
return db.projects.findMany({
|
|
29
|
+
where: {
|
|
30
|
+
OR: [
|
|
31
|
+
{ ownerId: null },
|
|
32
|
+
{ owner: { deletedAt: { not: null } } },
|
|
33
|
+
],
|
|
34
|
+
orphanedAt: null, // Not already marked
|
|
35
|
+
},
|
|
36
|
+
});
|
|
37
|
+
}
|
|
38
|
+
|
|
39
|
+
async handleOrphanProjects() {
|
|
40
|
+
const orphans = await this.findOrphanProjects();
|
|
41
|
+
|
|
42
|
+
for (const project of orphans) {
|
|
43
|
+
// Mark as orphaned, set deadline
|
|
44
|
+
await db.projects.update({
|
|
45
|
+
where: { id: project.id },
|
|
46
|
+
data: {
|
|
47
|
+
orphanedAt: new Date(),
|
|
48
|
+
archiveDeadline: addDays(new Date(), 30),
|
|
49
|
+
},
|
|
50
|
+
});
|
|
51
|
+
|
|
52
|
+
// Notify admins
|
|
53
|
+
await notifyAdmins('orphan_project', {
|
|
54
|
+
projectId: project.id,
|
|
55
|
+
projectName: project.name,
|
|
56
|
+
archiveDeadline: addDays(new Date(), 30),
|
|
57
|
+
});
|
|
58
|
+
}
|
|
59
|
+
|
|
60
|
+
logger.info('Orphan projects processed', { count: orphans.length });
|
|
61
|
+
}
|
|
62
|
+
|
|
63
|
+
async archiveExpiredOrphans() {
|
|
64
|
+
const expired = await db.projects.findMany({
|
|
65
|
+
where: {
|
|
66
|
+
archiveDeadline: { lt: new Date() },
|
|
67
|
+
archivedAt: null,
|
|
68
|
+
},
|
|
69
|
+
});
|
|
70
|
+
|
|
71
|
+
for (const project of expired) {
|
|
72
|
+
await this.archiveProject(project);
|
|
73
|
+
}
|
|
74
|
+
}
|
|
75
|
+
|
|
76
|
+
async findOrphanFiles(): Promise<string[]> {
|
|
77
|
+
// List all S3 files
|
|
78
|
+
const s3Files = await s3.listObjects({ Bucket: 'uploads' });
|
|
79
|
+
|
|
80
|
+
// Get all referenced files from DB
|
|
81
|
+
const referencedFiles = await db.files.findMany({
|
|
82
|
+
select: { s3Key: true },
|
|
83
|
+
});
|
|
84
|
+
const referencedSet = new Set(referencedFiles.map(f => f.s3Key));
|
|
85
|
+
|
|
86
|
+
// Find orphans
|
|
87
|
+
return s3Files
|
|
88
|
+
.filter(f => !referencedSet.has(f.Key))
|
|
89
|
+
.map(f => f.Key);
|
|
90
|
+
}
|
|
91
|
+
|
|
92
|
+
async cleanupOrphanFiles() {
|
|
93
|
+
const orphans = await this.findOrphanFiles();
|
|
94
|
+
|
|
95
|
+
// Move to quarantine first (safety)
|
|
96
|
+
for (const key of orphans) {
|
|
97
|
+
await s3.copyObject({
|
|
98
|
+
CopySource: `uploads/${key}`,
|
|
99
|
+
Bucket: 'uploads-quarantine',
|
|
100
|
+
Key: key,
|
|
101
|
+
});
|
|
102
|
+
}
|
|
103
|
+
|
|
104
|
+
logger.info('Orphan files quarantined', { count: orphans.length });
|
|
105
|
+
|
|
106
|
+
// Delete from main bucket after 30 days in quarantine
|
|
107
|
+
// (separate cron job)
|
|
108
|
+
}
|
|
109
|
+
}
|
|
110
|
+
|
|
111
|
+
// Schedule cleanup jobs
|
|
112
|
+
cron.schedule('0 2 * * 0', () => orphanService.handleOrphanProjects()); // Weekly
|
|
113
|
+
cron.schedule('0 3 * * *', () => orphanService.archiveExpiredOrphans()); // Daily
|
|
114
|
+
cron.schedule('0 4 1 * *', () => orphanService.cleanupOrphanFiles()); // Monthly
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
**Database Integrity Check:**
|
|
118
|
+
|
|
119
|
+
```typescript
|
|
120
|
+
async function checkDatabaseIntegrity() {
|
|
121
|
+
const issues: IntegrityIssue[] = [];
|
|
122
|
+
|
|
123
|
+
// Check foreign key references
|
|
124
|
+
const brokenRefs = await db.$queryRaw`
|
|
125
|
+
SELECT m.id, m.session_id
|
|
126
|
+
FROM messages m
|
|
127
|
+
LEFT JOIN sessions s ON m.session_id = s.id
|
|
128
|
+
WHERE s.id IS NULL
|
|
129
|
+
`;
|
|
130
|
+
|
|
131
|
+
if (brokenRefs.length > 0) {
|
|
132
|
+
issues.push({
|
|
133
|
+
type: 'broken_reference',
|
|
134
|
+
table: 'messages',
|
|
135
|
+
count: brokenRefs.length,
|
|
136
|
+
ids: brokenRefs.map(r => r.id),
|
|
137
|
+
});
|
|
138
|
+
}
|
|
139
|
+
|
|
140
|
+
return issues;
|
|
141
|
+
}
|
|
142
|
+
```
|
|
@@ -0,0 +1,100 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: TTL by Data Type
|
|
3
|
+
impact: MEDIUM
|
|
4
|
+
impactDescription: Ensures proper data lifecycle management and compliance
|
|
5
|
+
tags: [data, lifecycle, compliance, ttl]
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
## TTL by Data Type
|
|
9
|
+
|
|
10
|
+
Different data types require different retention policies. Define TTLs at the schema level.
|
|
11
|
+
|
|
12
|
+
**Example TTLs:**
|
|
13
|
+
|
|
14
|
+
> The durations below are **examples — set actual retention periods with legal counsel; they are jurisdiction-specific** (statutory retention duties, storage-limitation principles, sector rules).
|
|
15
|
+
|
|
16
|
+
| Data Type | TTL | Action at Expiration |
|
|
17
|
+
|-----------|-----|---------------------|
|
|
18
|
+
| Active chat sessions | Session duration | Archive + summarize |
|
|
19
|
+
| Archived sessions | 2 years | Pseudonymize then delete |
|
|
20
|
+
| Active projects | Project duration | Archive |
|
|
21
|
+
| Completed projects | 5 years | Pseudonymize personal data |
|
|
22
|
+
| Agent memories | Progressive decay | Prune if score < threshold |
|
|
23
|
+
| Audit logs | 5 years (legal) | Cold archive |
|
|
24
|
+
| Technical logs | 90 days | Delete |
|
|
25
|
+
| API tokens | 90 days | Revoke |
|
|
26
|
+
| Password reset | 1 hour | Delete |
|
|
27
|
+
|
|
28
|
+
**Implementation:**
|
|
29
|
+
|
|
30
|
+
```typescript
|
|
31
|
+
// Schema with TTL metadata
|
|
32
|
+
interface DataSchema {
|
|
33
|
+
type: string;
|
|
34
|
+
ttlDays: number | null; // null = never expires
|
|
35
|
+
expirationAction: 'delete' | 'archive' | 'pseudonymize';
|
|
36
|
+
}
|
|
37
|
+
|
|
38
|
+
const dataSchemas: Record<string, DataSchema> = {
|
|
39
|
+
chatSession: { type: 'chatSession', ttlDays: null, expirationAction: 'archive' },
|
|
40
|
+
archivedSession: { type: 'archivedSession', ttlDays: 730, expirationAction: 'pseudonymize' },
|
|
41
|
+
auditLog: { type: 'auditLog', ttlDays: 1825, expirationAction: 'archive' },
|
|
42
|
+
technicalLog: { type: 'technicalLog', ttlDays: 90, expirationAction: 'delete' },
|
|
43
|
+
memory: { type: 'memory', ttlDays: null, expirationAction: 'delete' }, // Score-based
|
|
44
|
+
};
|
|
45
|
+
|
|
46
|
+
// Cleanup cron job
|
|
47
|
+
async function runDataCleanup() {
|
|
48
|
+
for (const [type, schema] of Object.entries(dataSchemas)) {
|
|
49
|
+
if (!schema.ttlDays) continue;
|
|
50
|
+
|
|
51
|
+
const cutoff = new Date(Date.now() - schema.ttlDays * 24 * 60 * 60 * 1000);
|
|
52
|
+
|
|
53
|
+
switch (schema.expirationAction) {
|
|
54
|
+
case 'delete':
|
|
55
|
+
await db.deleteWhere(type, { createdAt: { lt: cutoff } });
|
|
56
|
+
break;
|
|
57
|
+
case 'archive':
|
|
58
|
+
await db.archiveWhere(type, { createdAt: { lt: cutoff } });
|
|
59
|
+
break;
|
|
60
|
+
case 'pseudonymize':
|
|
61
|
+
await db.pseudonymizeWhere(type, { createdAt: { lt: cutoff } });
|
|
62
|
+
break;
|
|
63
|
+
}
|
|
64
|
+
|
|
65
|
+
logger.info(`Data cleanup completed for ${type}`, { cutoff, action: schema.expirationAction });
|
|
66
|
+
}
|
|
67
|
+
}
|
|
68
|
+
|
|
69
|
+
// Run daily
|
|
70
|
+
cron.schedule('0 3 * * *', runDataCleanup);
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
**Pseudonymization Pattern (this is NOT anonymization):**
|
|
74
|
+
|
|
75
|
+
A hashed email is still personal data: the hash is a stable pseudonym that can be linked back to the person (dictionary attack on known addresses, or simply joining on the same hash elsewhere). Data-protection regimes such as the GDPR treat pseudonymized data as personal data, with all obligations attached. True **anonymization** is irreversible — outright deletion of identifying fields, or aggregation into counts that no longer describe an individual.
|
|
76
|
+
|
|
77
|
+
```typescript
|
|
78
|
+
// Pseudonymization: reversible in principle, still personal data.
|
|
79
|
+
async function pseudonymizeRecord(record: UserData): Promise<PseudonymizedData> {
|
|
80
|
+
return {
|
|
81
|
+
...record,
|
|
82
|
+
email: hashEmail(record.email), // stable pseudonym, NOT anonymous
|
|
83
|
+
name: null,
|
|
84
|
+
phone: null,
|
|
85
|
+
address: null,
|
|
86
|
+
// Keep non-personal fields for analytics
|
|
87
|
+
createdAt: record.createdAt,
|
|
88
|
+
projectCount: record.projectCount,
|
|
89
|
+
};
|
|
90
|
+
}
|
|
91
|
+
|
|
92
|
+
// Anonymization: irreversible. Either delete the record outright, or
|
|
93
|
+
// keep only aggregates that no longer describe an individual.
|
|
94
|
+
async function anonymizeIntoAggregates(records: UserData[]): Promise<void> {
|
|
95
|
+
await db.analytics.increment('projects_completed', sum(records.map(r => r.projectCount)));
|
|
96
|
+
await db.deleteMany(records.map(r => r.id)); // individual rows are gone
|
|
97
|
+
}
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
Decide per field, with legal counsel, whether pseudonymization is enough (data stays in scope of the regulation) or true anonymization is required.
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Evaluation Loop (Test + Evaluate, Hold-out, Anchored Judge)
|
|
3
|
+
impact: CRITICAL
|
|
4
|
+
impactDescription: Gives a non-deterministic system a trustworthy way to detect behavioural regressions without letting it optimise its own grader
|
|
5
|
+
tags: [evaluation, testing, llm, quality, governance]
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
## Evaluation Loop
|
|
9
|
+
|
|
10
|
+
> **A deterministic system is tested. An agentic system is tested *and* evaluated.**
|
|
11
|
+
|
|
12
|
+
The code around the agent is verified like any software (unit, integration, contract). The agent's behaviour is non-deterministic, so it needs a second instrument: an evaluation rubric replayed over scenarios, scored over time.
|
|
13
|
+
|
|
14
|
+
**Two distinct instruments, never confused:**
|
|
15
|
+
|
|
16
|
+
| | Test | Evaluation |
|
|
17
|
+
|---|---|---|
|
|
18
|
+
| Subject | Deterministic code | Agent behaviour |
|
|
19
|
+
| Verdict | Pass / fail | Score on a rubric |
|
|
20
|
+
| Guarantee | Hard | Behavioural signal |
|
|
21
|
+
| Owns | Safety, authorisation, schema, idempotency | Faithfulness, clarity, abstention quality |
|
|
22
|
+
|
|
23
|
+
**Abstention is scored.** For a grounded system, refusing to invent a figure counts as much as recalling one. A hallucinated number where the agent should have abstained is the worst outcome, scored zero, even if the rest is perfect.
|
|
24
|
+
|
|
25
|
+
**The hold-out is non-manipulable.** A naive closed loop that optimises the judge's score drifts toward what flatters the judge without improving real quality. What makes self-tuning safe is a guard the optimiser never sees: a hold-out with verifiable ground truth that triggers a rollback when the hallucination rate climbs. The hold-out is only valid where ground truth is verifiable; for purely subjective quality, the net is a human sample and the self-tuning envelope stays narrow.
|
|
26
|
+
|
|
27
|
+
**The judge is anchored.** The model that scores behavioural quality is frozen in version; on any judge change you replay an anchoring set to recalibrate, otherwise the quality series becomes incomparable in silence. The judge covers only irreducible behavioural quality. The hard floor (is the figure exactly the one in the store, is the source the right one) stays deterministic, compared directly to the store, never handed to the judge.
|
|
28
|
+
|
|
29
|
+
**No chorus of verifiers replaces a deterministic guard.** Stacking fallible LLM checkers that re-read each other does not manufacture a hard property: their errors correlate (shared data, shared blind spots), an adversarial input that fools one fools the chorus, and each check is one more slow, costly inference where a deterministic control is free and certain. Deterministic for what admits a guarantee; judge only for behavioural quality; always under the hold-out.
|
|
30
|
+
|
|
31
|
+
**The loop is open at the action boundary.** It produces signals and recommendations. Acting on them goes through a bounded, audited tuning envelope or a human decision, never an autonomous rewrite of the agent.
|
|
32
|
+
|
|
33
|
+
**Eval runbook (per scenario):**
|
|
34
|
+
|
|
35
|
+
```
|
|
36
|
+
Add describe the multi-turn conversation, the final question,
|
|
37
|
+
the expected and forbidden terms, the capability targeted.
|
|
38
|
+
Run replay the turns in order against the current system,
|
|
39
|
+
ask the final question, compare to expected/forbidden terms.
|
|
40
|
+
Score apply the rubric, track the score per capability over time
|
|
41
|
+
to catch behavioural regressions.
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
**Continuous faithfulness metric (monitoring, not gating):** alongside the deterministic guard, run a faithfulness metric of the RAGAS / FActScore family over a traffic sample: decompose each synthesis into atomic claims, verify each against the served facts and their source, aggregate into a per-request score and a time series. The guard blocks; the metric alerts. They are complementary.
|
|
45
|
+
|
|
46
|
+
**Checklist:**
|
|
47
|
+
|
|
48
|
+
- [ ] Tests and evaluation are separate; safety lives in tests, not in the rubric.
|
|
49
|
+
- [ ] Abstention is a scored capability.
|
|
50
|
+
- [ ] A non-manipulable hold-out with verifiable ground truth gates regressions.
|
|
51
|
+
- [ ] The judge is version-anchored; the deterministic floor never goes to the judge.
|
|
52
|
+
- [ ] Acting on eval signals is bounded-and-audited or human, never autonomous.
|
|
@@ -0,0 +1,91 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Deterministic Boundary of the Model
|
|
3
|
+
impact: CRITICAL
|
|
4
|
+
impactDescription: Keeps every fact, figure and decision that can be checked out of the model, so the system is verifiable and cannot hallucinate load-bearing values
|
|
5
|
+
tags: [architecture, llm, frontier, grounding, safety, determinism]
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
## Deterministic Boundary of the Model
|
|
9
|
+
|
|
10
|
+
> **The model writes prose. The program owns the facts.**
|
|
11
|
+
> Anything that can be retrieved, computed, validated or decided deterministically must live in code, never in the model's free generation. The model is rented and non-deterministic; the boundary around it is owned and tested.
|
|
12
|
+
|
|
13
|
+
This is the strongest form of the LLM Boundary Principle: not only move deterministic *logic* out (routing, classification), but move deterministic *truth* out. The model is allowed to phrase, summarise and connect. It is never the source of a number, a citation, a price, a date, or an authorisation.
|
|
14
|
+
|
|
15
|
+
**Why structured outputs are not enough:** constrained decoding and JSON schema guarantee the *shape* of the output, never its *meaning*. A model can emit a perfectly valid JSON number that is invented. Shape validation and truth validation are two different jobs.
|
|
16
|
+
|
|
17
|
+
**Incorrect:**
|
|
18
|
+
|
|
19
|
+
```typescript
|
|
20
|
+
// The model is asked to "report the figures". It will, including ones it invents.
|
|
21
|
+
const answer = await model.complete(`
|
|
22
|
+
Summarise the company's AI budget from these notes: ${notes}
|
|
23
|
+
Give exact percentages and the source.
|
|
24
|
+
`);
|
|
25
|
+
// BAD: Any % in `answer` is the model's word, unverifiable, potentially hallucinated.
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
**Correct (data-to-text / slot-filling):**
|
|
29
|
+
|
|
30
|
+
```typescript
|
|
31
|
+
// 1. Facts come from a structured store, each with value, unit, period, source.
|
|
32
|
+
const facts = await portFacts.query({ topic, edition }); // deterministic
|
|
33
|
+
|
|
34
|
+
// 2. The model writes the narrative and emits SLOTS, never raw numbers.
|
|
35
|
+
// Prompt: "Refer to a served fact as {f0}, {f1}... Never write a figure yourself."
|
|
36
|
+
const draft = await model.complete(buildPrompt(question, facts));
|
|
37
|
+
|
|
38
|
+
// 3. The program substitutes value + unit + period into each slot.
|
|
39
|
+
const answer = fillSlots(draft, facts); // deterministic
|
|
40
|
+
|
|
41
|
+
// 4. A guard rejects the response if any number in it is not grounded in served facts.
|
|
42
|
+
assertGrounded(answer, facts); // fail-closed
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
**The grounding guard (executable specification of the boundary):**
|
|
46
|
+
|
|
47
|
+
```typescript
|
|
48
|
+
// Extract numbers that are not part of a word/identifier, normalise FR decimals.
|
|
49
|
+
const NUMBER = /(?<![\p{L}\d-])\d+(?:[.,]\d+)?/gu;
|
|
50
|
+
|
|
51
|
+
// Grounded set = fact values ∪ numbers appearing in served fact labels ∪ plausible years.
|
|
52
|
+
function groundedSet(facts: Pick<Fact, "value" | "indicator">[]): Set<string> {
|
|
53
|
+
const set = new Set<string>();
|
|
54
|
+
for (const f of facts) {
|
|
55
|
+
set.add(normalise(f.value));
|
|
56
|
+
for (const n of f.indicator.match(NUMBER) ?? []) set.add(normalise(n));
|
|
57
|
+
}
|
|
58
|
+
return set;
|
|
59
|
+
}
|
|
60
|
+
|
|
61
|
+
// Reject any figure in the synthesis that is not in the grounded set (years excepted).
|
|
62
|
+
export function assertGrounded(text: string, facts: Pick<Fact, "value" | "indicator">[]): void {
|
|
63
|
+
const grounded = groundedSet(facts);
|
|
64
|
+
for (const n of text.match(NUMBER) ?? []) {
|
|
65
|
+
const v = normalise(n);
|
|
66
|
+
if (!grounded.has(v) && !isPlausibleYear(v)) {
|
|
67
|
+
throw new GroundingError(`Ungrounded figure in synthesis: ${n}`);
|
|
68
|
+
}
|
|
69
|
+
}
|
|
70
|
+
}
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
**Layered defence, by maturity of the model wiring:**
|
|
74
|
+
|
|
75
|
+
| Stage | Mechanism | Guarantee |
|
|
76
|
+
|-------|-----------|-----------|
|
|
77
|
+
| Floor | Guard checks every figure against the served facts (value ∪ label numbers ∪ years) | No invented number reaches the user |
|
|
78
|
+
| Wired with structured outputs | Model emits `{fN}` slots, program substitutes value+unit+period | Attribution is exact (AIS: Attributable to Identified Sources) |
|
|
79
|
+
| Monitoring | Faithfulness metric (RAGAS / FActScore family) over a traffic sample | Drift in narrative faithfulness is measured and alerted, not blocked |
|
|
80
|
+
|
|
81
|
+
The guard **guarantees and blocks** ("can we serve this?"). The metric **measures and alerts** ("how faithful is the prose over time?"). They are complementary, never redundant.
|
|
82
|
+
|
|
83
|
+
**Boundary checklist for any feature that surfaces facts:**
|
|
84
|
+
|
|
85
|
+
- [ ] Every load-bearing value comes from a typed store, not from the model.
|
|
86
|
+
- [ ] The model's output passes a grounding guard before it reaches a user.
|
|
87
|
+
- [ ] The guard **fails closed**: on doubt it rejects, it does not pass through.
|
|
88
|
+
- [ ] Abstention is a first-class answer: "not in the corpus" beats a plausible guess.
|
|
89
|
+
- [ ] The guard has unit tests that feed realistic facts *with their labels* (the spec is executable).
|
|
90
|
+
|
|
91
|
+
**Where this sits in the Runward method:** the deterministic boundary is the practical form of the guiding principle — the deterministic stays out of the model. It is also a security control: a hallucinated figure is an integrity failure, not just a quality one.
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: New Feature = New Adapter
|
|
3
|
+
impact: HIGH
|
|
4
|
+
impactDescription: Keeps core business logic clean and external integrations isolated
|
|
5
|
+
tags: [architecture, hexagonal, adapters, integration]
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
## New Feature = New Adapter
|
|
9
|
+
|
|
10
|
+
When adding a new external integration, always create a new adapter. Never mix external service logic into core.
|
|
11
|
+
|
|
12
|
+
**The Rule:**
|
|
13
|
+
|
|
14
|
+
> **New external service → New adapter implementing existing or new port**
|
|
15
|
+
|
|
16
|
+
**Incorrect:**
|
|
17
|
+
|
|
18
|
+
```typescript
|
|
19
|
+
// Business logic mixed with external service
|
|
20
|
+
// core/application/analyze-project.ts
|
|
21
|
+
import { WebSearchClient } from 'web-search'; // BAD: External import in core
|
|
22
|
+
|
|
23
|
+
export class AnalyzeProjectUseCase {
|
|
24
|
+
async execute(projectId: string) {
|
|
25
|
+
// BAD: Direct external service usage in core
|
|
26
|
+
const web-search = new WebSearchClient(process.env.SEARCH_API_KEY);
|
|
27
|
+
const searchResults = await web-search.search(query);
|
|
28
|
+
// ...
|
|
29
|
+
}
|
|
30
|
+
}
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
**Correct:**
|
|
34
|
+
|
|
35
|
+
```typescript
|
|
36
|
+
// 1. Define port in core
|
|
37
|
+
// core/ports/out/search-gateway.port.ts
|
|
38
|
+
export interface SearchGateway {
|
|
39
|
+
search(query: string, options?: SearchOptions): Promise<SearchResult[]>;
|
|
40
|
+
}
|
|
41
|
+
|
|
42
|
+
// 2. Use case depends on port (interface), not implementation
|
|
43
|
+
// core/application/analyze-project.ts
|
|
44
|
+
export class AnalyzeProjectUseCase {
|
|
45
|
+
constructor(private searchGateway: SearchGateway) {}
|
|
46
|
+
|
|
47
|
+
async execute(projectId: string) {
|
|
48
|
+
const searchResults = await this.searchGateway.search(query);
|
|
49
|
+
// ...
|
|
50
|
+
}
|
|
51
|
+
}
|
|
52
|
+
|
|
53
|
+
// 3. Adapter implements port
|
|
54
|
+
// adapters/secondary/search/web-search.adapter.ts
|
|
55
|
+
import { WebSearchClient } from 'web-search';
|
|
56
|
+
import { SearchGateway, SearchResult } from '@/core/ports/out/search-gateway.port';
|
|
57
|
+
|
|
58
|
+
export class WebSearchAdapter implements SearchGateway {
|
|
59
|
+
private client: WebSearchClient;
|
|
60
|
+
|
|
61
|
+
constructor(apiKey: string) {
|
|
62
|
+
this.client = new WebSearchClient(apiKey);
|
|
63
|
+
}
|
|
64
|
+
|
|
65
|
+
async search(query: string): Promise<SearchResult[]> {
|
|
66
|
+
const results = await this.client.search(query);
|
|
67
|
+
return results.map(this.mapToSearchResult);
|
|
68
|
+
}
|
|
69
|
+
}
|
|
70
|
+
|
|
71
|
+
// 4. Alternative adapter for testing or different provider
|
|
72
|
+
// adapters/secondary/search/mock.adapter.ts
|
|
73
|
+
export class MockSearchAdapter implements SearchGateway {
|
|
74
|
+
async search(query: string): Promise<SearchResult[]> {
|
|
75
|
+
return [{ title: 'Mock result', url: 'https://example.com' }];
|
|
76
|
+
}
|
|
77
|
+
}
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
**Benefits:**
|
|
81
|
+
|
|
82
|
+
- Core is testable without external services
|
|
83
|
+
- Easy to swap providers (the search provider → another provider)
|
|
84
|
+
- Clear contracts between layers
|
|
85
|
+
- External service changes don't affect core logic
|
|
@@ -0,0 +1,84 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Hexagonal Architecture Structure
|
|
3
|
+
impact: HIGH
|
|
4
|
+
impactDescription: Enables testability, maintainability, and clean dependency management
|
|
5
|
+
tags: [architecture, hexagonal, structure, testing]
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
## Hexagonal Architecture Structure
|
|
9
|
+
|
|
10
|
+
Organize code into layers with clear boundaries. Dependencies point inward. External services are adapters.
|
|
11
|
+
|
|
12
|
+
**Incorrect:**
|
|
13
|
+
|
|
14
|
+
```typescript
|
|
15
|
+
// Everything mixed together
|
|
16
|
+
src/
|
|
17
|
+
├── api/
|
|
18
|
+
│ └── projects.ts // Contains business logic + DB + LLM calls
|
|
19
|
+
├── utils/
|
|
20
|
+
│ └── helpers.ts // Random functions
|
|
21
|
+
└── types.ts // All types in one file
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
**Correct:**
|
|
25
|
+
|
|
26
|
+
```
|
|
27
|
+
src/
|
|
28
|
+
├── core/ # UNTOUCHABLE - Pure business logic
|
|
29
|
+
│ ├── domain/ # Entities, value objects, domain events
|
|
30
|
+
│ │ ├── project.ts
|
|
31
|
+
│ │ └── user.ts
|
|
32
|
+
│ ├── application/ # Use cases (orchestration)
|
|
33
|
+
│ │ ├── create-project.ts
|
|
34
|
+
│ │ └── analyze-project.ts
|
|
35
|
+
│ └── ports/ # Interfaces (contracts)
|
|
36
|
+
│ ├── in/ # Primary ports (how app is used)
|
|
37
|
+
│ │ └── project-service.port.ts
|
|
38
|
+
│ └── out/ # Secondary ports (what app needs)
|
|
39
|
+
│ ├── project-repository.port.ts
|
|
40
|
+
│ └── llm-gateway.port.ts
|
|
41
|
+
│
|
|
42
|
+
├── adapters/
|
|
43
|
+
│ ├── primary/ # How the outside world calls us
|
|
44
|
+
│ │ ├── rest/ # REST API
|
|
45
|
+
│ │ ├── graphql/ # GraphQL API
|
|
46
|
+
│ │ └── cli/ # CLI commands
|
|
47
|
+
│ └── secondary/ # How we call the outside world
|
|
48
|
+
│ ├── database/ # Postgres, graph store adapters
|
|
49
|
+
│ ├── llm/ # model provider adapters
|
|
50
|
+
│ └── external/ # Third-party APIs
|
|
51
|
+
│
|
|
52
|
+
└── infrastructure/
|
|
53
|
+
├── config/ # Zod-typed configuration
|
|
54
|
+
├── middleware/ # Request middleware
|
|
55
|
+
└── observability/ # Logging, metrics, tracing
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
**Port Example:**
|
|
59
|
+
|
|
60
|
+
```typescript
|
|
61
|
+
// core/ports/out/llm-gateway.port.ts
|
|
62
|
+
export interface LLMGateway {
|
|
63
|
+
complete(prompt: string, options: CompletionOptions): Promise<string>;
|
|
64
|
+
embed(text: string): Promise<number[]>;
|
|
65
|
+
}
|
|
66
|
+
|
|
67
|
+
// adapters/secondary/model/provider.adapter.ts
|
|
68
|
+
export class ModelAdapter implements LLMGateway {
|
|
69
|
+
async complete(prompt: string, options: CompletionOptions): Promise<string> {
|
|
70
|
+
const response = await this.client.messages.create({...});
|
|
71
|
+
return response.content[0].text;
|
|
72
|
+
}
|
|
73
|
+
}
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
**Dependency Rule:**
|
|
77
|
+
|
|
78
|
+
```
|
|
79
|
+
External World → Adapters → Ports → Core Domain
|
|
80
|
+
↑ ↓
|
|
81
|
+
(implements) (depends on)
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
Core never imports from adapters. Adapters implement ports defined in core.
|
|
@@ -0,0 +1,93 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Move the Deterministic out of the Model
|
|
3
|
+
impact: CRITICAL
|
|
4
|
+
impactDescription: Reduces LLM costs and latency by moving deterministic logic out of LLM calls
|
|
5
|
+
tags: [architecture, llm, cost-optimization, performance]
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
## LLM Boundary Principle
|
|
9
|
+
|
|
10
|
+
> **Everything that can be deterministic, testable, and cacheable must be moved out of the LLM.**
|
|
11
|
+
|
|
12
|
+
LLMs are expensive, slow, and non-deterministic. Move everything possible to code.
|
|
13
|
+
|
|
14
|
+
**Incorrect:**
|
|
15
|
+
|
|
16
|
+
```typescript
|
|
17
|
+
// LLM does classification AND response
|
|
18
|
+
const response = await llm.complete({
|
|
19
|
+
prompt: `
|
|
20
|
+
Analyze this message and determine:
|
|
21
|
+
1. Is it casual, task-oriented, or complex?
|
|
22
|
+
2. What data sources are needed?
|
|
23
|
+
3. Generate the response.
|
|
24
|
+
|
|
25
|
+
Message: ${userMessage}
|
|
26
|
+
`
|
|
27
|
+
});
|
|
28
|
+
// BAD: Pays full LLM cost for classification
|
|
29
|
+
// BAD: Non-deterministic classification
|
|
30
|
+
// BAD: Can't test classification logic
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
**Correct:**
|
|
34
|
+
|
|
35
|
+
```typescript
|
|
36
|
+
// Step 1: Deterministic classification (regex)
|
|
37
|
+
const regexResult = classifyWithRegex(userMessage);
|
|
38
|
+
if (regexResult.confidence >= 0.6) {
|
|
39
|
+
complexity = regexResult; // $0, ~1ms
|
|
40
|
+
} else {
|
|
41
|
+
// Step 2: Cheap LLM classification (Fast)
|
|
42
|
+
complexity = await classifyWithFastModel(userMessage); // ~$0.0003, ~100ms
|
|
43
|
+
}
|
|
44
|
+
|
|
45
|
+
// Step 3: Deterministic routing
|
|
46
|
+
const model = selectModel(complexity);
|
|
47
|
+
const dataSources = selectDataSources(complexity);
|
|
48
|
+
|
|
49
|
+
// Step 4: Only use expensive LLM for actual generation
|
|
50
|
+
const response = await llm.complete({
|
|
51
|
+
model,
|
|
52
|
+
prompt: buildPrompt(userMessage, dataSources),
|
|
53
|
+
});
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
**What to Move Out:**
|
|
57
|
+
|
|
58
|
+
| Logic | Method | Savings |
|
|
59
|
+
|-------|--------|---------|
|
|
60
|
+
| Request classification | Regex + fallback Fast | -40% classification cost |
|
|
61
|
+
| Model selection | Deterministic rules | -55% average cost |
|
|
62
|
+
| Memory source selection | Router by complexity | -76% latency |
|
|
63
|
+
| Format validation | Zod schemas | $0 (vs LLM validation) |
|
|
64
|
+
| Error detection | Pattern matching | Instant feedback |
|
|
65
|
+
|
|
66
|
+
**Implementation Pattern:**
|
|
67
|
+
|
|
68
|
+
```typescript
|
|
69
|
+
// The LLM should only do what requires intelligence
|
|
70
|
+
interface LLMBoundary {
|
|
71
|
+
// OUT: Deterministic pre-processing
|
|
72
|
+
classify(message: string): ComplexityLevel;
|
|
73
|
+
selectModel(complexity: ComplexityLevel): Model;
|
|
74
|
+
selectSources(complexity: ComplexityLevel): DataSource[];
|
|
75
|
+
buildPrompt(message: string, context: Context): string;
|
|
76
|
+
|
|
77
|
+
// IN: Actual LLM call
|
|
78
|
+
generate(prompt: string, model: Model): Promise<string>;
|
|
79
|
+
|
|
80
|
+
// OUT: Deterministic post-processing
|
|
81
|
+
validateOutput(output: string, schema: ZodSchema): Result;
|
|
82
|
+
formatResponse(result: Result): Response;
|
|
83
|
+
}
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
**Testing Benefits:**
|
|
87
|
+
|
|
88
|
+
- Classification logic: 100% unit testable
|
|
89
|
+
- Routing logic: 100% unit testable
|
|
90
|
+
- Only generation needs integration tests
|
|
91
|
+
- Faster test suite, more reliable CI
|
|
92
|
+
|
|
93
|
+
**The stronger form: move deterministic *truth* out, not only deterministic *logic*.** This rule moves classification, routing and validation out of the model. Its critical extension is to also keep load-bearing facts and figures out of the model's free generation. See `frontier-deterministic-boundary`.
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: Stack Posture (Sober Default plus Trigger), Not a Fixed Stack
|
|
3
|
+
impact: MEDIUM
|
|
4
|
+
impactDescription: Replaces a frozen product stack with a portable rule for choosing technology by evidence, so the rule does not age with any vendor
|
|
5
|
+
tags: [architecture, stack, decision, portability]
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
## Stack Posture, Not a Fixed Stack
|
|
9
|
+
|
|
10
|
+
A method describes functions, never products, so it ages slowly. This rule follows the same discipline: it does not prescribe a vendor stack. It prescribes how to choose one.
|
|
11
|
+
|
|
12
|
+
> **Start with the smallest stack that proves value on real traffic. Cross a frontier (new service, new database, new provider) only on an objective trigger.**
|
|
13
|
+
|
|
14
|
+
**Default posture at the floor:**
|
|
15
|
+
|
|
16
|
+
| Concern | Sober default | Cross the frontier when (trigger) |
|
|
17
|
+
|---|---|---|
|
|
18
|
+
| Language | One typed language for the core, chosen by the team's fluency and the ecosystem | A second runtime earns a measured, isolated need (e.g. a Python sidecar for OCR/NLP) |
|
|
19
|
+
| Topology | Modular monolith, single process | A real, measured reason to split a service (independent scaling, isolation, team boundary) |
|
|
20
|
+
| Model | One provider behind a `ModelPort`, ideally via a gateway | Availability fallback, or a validated promotion in shadow deployment |
|
|
21
|
+
| Persistence | One relational store (Postgres) for facts; vectors in the same store (pgvector) | Volume that overflows the single store; a real graph traversal becomes central |
|
|
22
|
+
| State / scale | In-process, single instance | Horizontal scaling needed -> externalise state first |
|
|
23
|
+
| Async | Inline | A workload genuinely needs scheduling or backpressure |
|
|
24
|
+
|
|
25
|
+
**How to record a stack choice:** each line above is a decision with a default and a trigger. When a trigger fires, the choice is made in an ADR with the options and a preference order, not improvised. Until the trigger fires, you do not pre-build and you do not re-debate.
|
|
26
|
+
|
|
27
|
+
**What does not belong in the core, whatever the stack:**
|
|
28
|
+
|
|
29
|
+
- No vendor SDK imported from the domain (model, database, search are adapters).
|
|
30
|
+
- No framework owning your control flow that you cannot see through; prefer thin abstractions you own.
|
|
31
|
+
- No optional service that crashes the app when absent.
|
|
32
|
+
|
|
33
|
+
When you pin a concrete stack for a mission, record it as an illustration of the method, not as a stack to copy.
|