@prism-d1/cli 1.0.79 → 1.0.80
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/assets/github-workflows/prism-ai-metrics.yml +83 -122
- package/dist/assets/github-workflows/prism-eval-gate-kiro.yml +66 -0
- package/dist/assets/github-workflows/prism-eval-gate.yml +1 -1
- package/dist/assets/gitlab-workflows/.gitlab-ci.yml +0 -3
- package/dist/assets/gitlab-workflows/prism-ai-metrics.yml +53 -56
- package/dist/assets/infra/bin/app.ts +1 -0
- package/dist/assets/infra/lib/constructs/otel-collector-construct.ts +4 -0
- package/dist/assets/infra/lib/dashboard-stack.ts +378 -1235
- package/dist/assets/infra/lib/lambda/event-schema.ts +144 -0
- package/dist/assets/infra/lib/lambda/metrics-processor.ts +124 -36
- package/dist/assets/infra/lib/lambda/otel-receiver.ts +249 -35
- package/dist/assets/infra/lib/lambda/security-remediation-tracker.ts +17 -2
- package/dist/assets/infra/lib/lambda/velocity-widget.ts +1483 -0
- package/dist/assets/infra/lib/metrics-pipeline-stack.ts +110 -119
- package/dist/assets/infra/package.json +2 -1
- package/dist/assets/infra/tsconfig.json +9 -2
- package/dist/src/commands/bootstrapper/install-eval-harness.d.ts +8 -2
- package/dist/src/commands/bootstrapper/install-eval-harness.d.ts.map +1 -1
- package/dist/src/commands/bootstrapper/install-eval-harness.js +32 -32
- package/dist/src/commands/bootstrapper/install-eval-harness.js.map +1 -1
- package/dist/src/commands/bootstrapper/install-github-workflows.d.ts.map +1 -1
- package/dist/src/commands/bootstrapper/install-github-workflows.js +7 -2
- package/dist/src/commands/bootstrapper/install-github-workflows.js.map +1 -1
- package/dist/src/commands/bootstrapper/install-gitlab-workflows.d.ts.map +1 -1
- package/dist/src/commands/bootstrapper/install-gitlab-workflows.js +7 -1
- package/dist/src/commands/bootstrapper/install-gitlab-workflows.js.map +1 -1
- package/dist/src/utils/region.d.ts +50 -0
- package/dist/src/utils/region.d.ts.map +1 -0
- package/dist/src/utils/region.js +62 -0
- package/dist/src/utils/region.js.map +1 -0
- package/package.json +2 -2
- package/dist/assets/eval-harness/README.md +0 -144
- package/dist/assets/github-workflows/README.md +0 -110
- package/dist/assets/github-workflows/prism-dora-weekly.yml +0 -334
- package/dist/assets/gitlab-workflows/prism-dora-weekly.yml +0 -181
- package/dist/assets/infra/lib/lambda/defect-correlator.ts +0 -142
- package/dist/assets/infra/lib/lambda/spec-to-code-calculator.ts +0 -123
- package/dist/assets/metric-hooks/README.md +0 -107
|
@@ -1,181 +0,0 @@
|
|
|
1
|
-
# prism-dora-weekly.yml — Weekly DORA + AI-DORA metrics assessment.
|
|
2
|
-
#
|
|
3
|
-
# Include this file in your .gitlab-ci.yml:
|
|
4
|
-
# include:
|
|
5
|
-
# - local: '.prism/gitlab-workflows/prism-dora-weekly.yml'
|
|
6
|
-
#
|
|
7
|
-
# Required setup:
|
|
8
|
-
# - CI/CD variable PRISM_METRICS_ROLE_ARN (IAM role with events:PutEvents + cloudwatch:PutMetricData)
|
|
9
|
-
# - CI/CD variable AWS_REGION (default: us-west-2)
|
|
10
|
-
# - .prism/config.json committed to the repo
|
|
11
|
-
# - Pipeline schedule configured: weekly on Monday at 09:00 UTC
|
|
12
|
-
|
|
13
|
-
prism-dora-weekly:
|
|
14
|
-
stage: test
|
|
15
|
-
image:
|
|
16
|
-
name: amazon/aws-cli:latest
|
|
17
|
-
entrypoint: [""]
|
|
18
|
-
id_tokens:
|
|
19
|
-
GITLAB_OIDC_TOKEN:
|
|
20
|
-
aud: https://gitlab.com
|
|
21
|
-
rules:
|
|
22
|
-
- if: '$CI_PIPELINE_SOURCE == "schedule"'
|
|
23
|
-
- if: '$CI_PIPELINE_SOURCE == "web"'
|
|
24
|
-
when: manual
|
|
25
|
-
variables:
|
|
26
|
-
AWS_DEFAULT_REGION: us-west-2
|
|
27
|
-
before_script:
|
|
28
|
-
- yum install -y -q jq bc git
|
|
29
|
-
- git fetch --unshallow || true
|
|
30
|
-
- |
|
|
31
|
-
# Assume AWS role via GitLab OIDC
|
|
32
|
-
if [[ -n "${PRISM_METRICS_ROLE_ARN}" && -n "${GITLAB_OIDC_TOKEN}" ]]; then
|
|
33
|
-
echo "Assuming role via OIDC: ${PRISM_METRICS_ROLE_ARN}"
|
|
34
|
-
if CREDS=$(aws sts assume-role-with-web-identity \
|
|
35
|
-
--role-arn "${PRISM_METRICS_ROLE_ARN}" \
|
|
36
|
-
--role-session-name "gitlab-ci-${CI_JOB_ID}" \
|
|
37
|
-
--web-identity-token "${GITLAB_OIDC_TOKEN}" \
|
|
38
|
-
--query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' \
|
|
39
|
-
--output text 2>&1); then
|
|
40
|
-
export AWS_ACCESS_KEY_ID=$(echo "${CREDS}" | awk '{print $1}')
|
|
41
|
-
export AWS_SECRET_ACCESS_KEY=$(echo "${CREDS}" | awk '{print $2}')
|
|
42
|
-
export AWS_SESSION_TOKEN=$(echo "${CREDS}" | awk '{print $3}')
|
|
43
|
-
echo "✓ OIDC credentials acquired."
|
|
44
|
-
else
|
|
45
|
-
echo "⚠️ OIDC assume-role failed: ${CREDS}"
|
|
46
|
-
fi
|
|
47
|
-
elif [[ -n "${PRISM_METRICS_ROLE_ARN}" ]]; then
|
|
48
|
-
echo "⚠️ PRISM_METRICS_ROLE_ARN set but GITLAB_OIDC_TOKEN is empty."
|
|
49
|
-
fi
|
|
50
|
-
script:
|
|
51
|
-
- |
|
|
52
|
-
# Skip if no AWS credentials available
|
|
53
|
-
if ! aws sts get-caller-identity &>/dev/null; then
|
|
54
|
-
echo "⚠️ No AWS credentials. Set PRISM_METRICS_ROLE_ARN or configure AWS access."
|
|
55
|
-
exit 0
|
|
56
|
-
fi
|
|
57
|
-
|
|
58
|
-
TEAM_ID=$(jq -r '.team_id // "no_team"' .prism/config.json 2>/dev/null || echo "no_team")
|
|
59
|
-
SINCE_DATE=$(date -u -d '7 days ago' +"%Y-%m-%d")
|
|
60
|
-
REPO="${CI_PROJECT_PATH}"
|
|
61
|
-
NAMESPACE="PRISM/D1/Velocity"
|
|
62
|
-
|
|
63
|
-
# ---------------------------------------------------------------
|
|
64
|
-
# 1. Deployment Frequency — merged MRs to default branch (past 7 days)
|
|
65
|
-
# ---------------------------------------------------------------
|
|
66
|
-
SINCE_ISO=$(date -u -d '7 days ago' +"%Y-%m-%dT%H:%M:%SZ")
|
|
67
|
-
MERGED_MRS=$(curl -s --header "PRIVATE-TOKEN: ${GITLAB_TOKEN:-${CI_JOB_TOKEN}}" \
|
|
68
|
-
"${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/merge_requests?state=merged&updated_after=${SINCE_ISO}&per_page=100" 2>/dev/null || echo "[]")
|
|
69
|
-
DEPLOY_COUNT=$(echo "${MERGED_MRS}" | jq 'length')
|
|
70
|
-
DEPLOY_FREQUENCY=$(echo "scale=2; ${DEPLOY_COUNT} / 7" | bc -l | sed 's/^\./0./')
|
|
71
|
-
|
|
72
|
-
# ---------------------------------------------------------------
|
|
73
|
-
# 2. Lead Time — avg time from MR open to merge
|
|
74
|
-
# ---------------------------------------------------------------
|
|
75
|
-
AVG_LEAD_TIME_HOURS=$(echo "${MERGED_MRS}" | jq '
|
|
76
|
-
if length == 0 then 0
|
|
77
|
-
else [.[] | select(.merged_at != null and .created_at != null) |
|
|
78
|
-
((.merged_at | sub("\\.[0-9]+Z$"; "Z") | strptime("%Y-%m-%dT%H:%M:%SZ") | mktime) -
|
|
79
|
-
(.created_at | sub("\\.[0-9]+Z$"; "Z") | strptime("%Y-%m-%dT%H:%M:%SZ") | mktime)) / 3600
|
|
80
|
-
] | if length > 0 then add / length | . * 100 | round / 100 else 0 end
|
|
81
|
-
end')
|
|
82
|
-
|
|
83
|
-
# ---------------------------------------------------------------
|
|
84
|
-
# 3. Change Failure Rate — MRs with "revert|hotfix|rollback" in title
|
|
85
|
-
# ---------------------------------------------------------------
|
|
86
|
-
FAILURE_MRS=$(echo "${MERGED_MRS}" | jq '[.[] | select(.title // "" | test("revert|hotfix|rollback"; "i"))] | length')
|
|
87
|
-
[[ "${DEPLOY_COUNT}" -gt 0 ]] && CHANGE_FAILURE_RATE=$(echo "scale=4; ${FAILURE_MRS} / ${DEPLOY_COUNT}" | bc -l | sed 's/^\./0./') || CHANGE_FAILURE_RATE="0"
|
|
88
|
-
|
|
89
|
-
# ---------------------------------------------------------------
|
|
90
|
-
# 4. MTTR — avg time for hotfix/revert MRs
|
|
91
|
-
# ---------------------------------------------------------------
|
|
92
|
-
MTTR_HOURS=$(echo "${MERGED_MRS}" | jq '
|
|
93
|
-
[.[] | select(.title // "" | test("revert|hotfix|rollback"; "i")) |
|
|
94
|
-
select(.merged_at != null and .created_at != null) |
|
|
95
|
-
((.merged_at | sub("\\.[0-9]+Z$"; "Z") | strptime("%Y-%m-%dT%H:%M:%SZ") | mktime) -
|
|
96
|
-
(.created_at | sub("\\.[0-9]+Z$"; "Z") | strptime("%Y-%m-%dT%H:%M:%SZ") | mktime)) / 3600
|
|
97
|
-
] | if length > 0 then add / length | . * 100 | round / 100 else 0 end')
|
|
98
|
-
|
|
99
|
-
# ---------------------------------------------------------------
|
|
100
|
-
# 5. AI-DORA metrics from commit trailers
|
|
101
|
-
# ---------------------------------------------------------------
|
|
102
|
-
TOTAL_COMMITS=$(git log --since="${SINCE_DATE}" --oneline "${CI_DEFAULT_BRANCH}" 2>/dev/null | wc -l | tr -d ' ')
|
|
103
|
-
AI_GENERATED=0; AI_ASSISTED=0; HUMAN_ONLY=0; SPEC_REFS=0
|
|
104
|
-
CLAUDE_CODE_COUNT=0; KIRO_COUNT=0; Q_DEV_COUNT=0
|
|
105
|
-
|
|
106
|
-
for SHA in $(git log --since="${SINCE_DATE}" --format='%H' "${CI_DEFAULT_BRANCH}" 2>/dev/null); do
|
|
107
|
-
MSG=$(git log -1 --format='%B' "${SHA}")
|
|
108
|
-
echo "${MSG}" | grep -q '^AI-Origin: ai-generated' && AI_GENERATED=$((AI_GENERATED + 1))
|
|
109
|
-
echo "${MSG}" | grep -q '^AI-Origin: ai-assisted' && AI_ASSISTED=$((AI_ASSISTED + 1))
|
|
110
|
-
echo "${MSG}" | grep -q '^AI-Origin: human' && HUMAN_ONLY=$((HUMAN_ONLY + 1))
|
|
111
|
-
echo "${MSG}" | grep -q '^Spec-Ref:' && SPEC_REFS=$((SPEC_REFS + 1))
|
|
112
|
-
TOOL=$(echo "${MSG}" | grep -oP '(?<=^AI-Tool: ).+' || true)
|
|
113
|
-
case "${TOOL}" in
|
|
114
|
-
*claude-code*) CLAUDE_CODE_COUNT=$((CLAUDE_CODE_COUNT + 1)) ;;
|
|
115
|
-
*kiro*) KIRO_COUNT=$((KIRO_COUNT + 1)) ;;
|
|
116
|
-
*q-developer*) Q_DEV_COUNT=$((Q_DEV_COUNT + 1)) ;;
|
|
117
|
-
esac
|
|
118
|
-
done
|
|
119
|
-
|
|
120
|
-
UNTAGGED=$((TOTAL_COMMITS - AI_GENERATED - AI_ASSISTED - HUMAN_ONLY))
|
|
121
|
-
[[ "${UNTAGGED}" -lt 0 ]] && UNTAGGED=0
|
|
122
|
-
AI_TOTAL=$((AI_GENERATED + AI_ASSISTED))
|
|
123
|
-
[[ "${TOTAL_COMMITS}" -gt 0 ]] && AI_ADOPTION_RATE=$(echo "scale=4; ${AI_TOTAL} / ${TOTAL_COMMITS}" | bc -l | sed 's/^\./0./') || AI_ADOPTION_RATE="0"
|
|
124
|
-
[[ "${TOTAL_COMMITS}" -gt 0 ]] && SPEC_COVERAGE=$(echo "scale=4; ${SPEC_REFS} / ${TOTAL_COMMITS}" | bc -l | sed 's/^\./0./') || SPEC_COVERAGE="0"
|
|
125
|
-
|
|
126
|
-
# ---------------------------------------------------------------
|
|
127
|
-
# Emit assessment event
|
|
128
|
-
# ---------------------------------------------------------------
|
|
129
|
-
TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
|
|
130
|
-
|
|
131
|
-
ASSESSMENT_EVENT=$(jq -n \
|
|
132
|
-
--arg team_id "${TEAM_ID}" --arg repo "${REPO}" --arg timestamp "${TIMESTAMP}" \
|
|
133
|
-
--argjson deploy_count "${DEPLOY_COUNT}" --argjson deploy_frequency "${DEPLOY_FREQUENCY}" \
|
|
134
|
-
--argjson avg_lead_time_hours "${AVG_LEAD_TIME_HOURS}" \
|
|
135
|
-
--argjson change_failure_rate "${CHANGE_FAILURE_RATE}" --argjson mttr_hours "${MTTR_HOURS}" \
|
|
136
|
-
--argjson total_commits "${TOTAL_COMMITS}" --argjson ai_generated "${AI_GENERATED}" \
|
|
137
|
-
--argjson ai_assisted "${AI_ASSISTED}" --argjson human_only "${HUMAN_ONLY}" \
|
|
138
|
-
--argjson untagged "${UNTAGGED}" --argjson ai_adoption_rate "${AI_ADOPTION_RATE}" \
|
|
139
|
-
--argjson spec_coverage "${SPEC_COVERAGE}" \
|
|
140
|
-
--argjson claude_code "${CLAUDE_CODE_COUNT}" --argjson kiro "${KIRO_COUNT}" --argjson q_dev "${Q_DEV_COUNT}" \
|
|
141
|
-
'{team_id:$team_id, repo:$repo, timestamp:$timestamp, prism_level:2,
|
|
142
|
-
metric:{name:"weekly_assessment",value:1,unit:"assessment"},
|
|
143
|
-
dora:{deployment_frequency:$deploy_frequency, deployments:$deploy_count,
|
|
144
|
-
lead_time_hours:$avg_lead_time_hours, change_failure_rate:$change_failure_rate,
|
|
145
|
-
mttr_hours:$mttr_hours},
|
|
146
|
-
ai_dora:{ai_adoption_rate:$ai_adoption_rate, ai_to_merge_ratio:$ai_adoption_rate,
|
|
147
|
-
total_commits:$total_commits, ai_generated_commits:$ai_generated,
|
|
148
|
-
ai_assisted_commits:$ai_assisted, human_commits:$human_only,
|
|
149
|
-
untagged_commits:$untagged, spec_coverage:$spec_coverage,
|
|
150
|
-
tool_breakdown:{claude_code:$claude_code, kiro:$kiro, q_developer:$q_dev}}}')
|
|
151
|
-
|
|
152
|
-
aws events put-events --region "${AWS_DEFAULT_REGION}" \
|
|
153
|
-
--entries "[{\"Source\":\"prism.d1.velocity\",\"DetailType\":\"prism.d1.assessment\",
|
|
154
|
-
\"EventBusName\":\"prism-d1-metrics\",
|
|
155
|
-
\"Detail\":$(echo "${ASSESSMENT_EVENT}" | jq -c '.' | jq -Rs '.')}]"
|
|
156
|
-
|
|
157
|
-
# Emit CloudWatch metrics
|
|
158
|
-
for METRIC_DATA in \
|
|
159
|
-
"AIAdoptionRate:${AI_ADOPTION_RATE}:Percent" \
|
|
160
|
-
"SpecCoverage:${SPEC_COVERAGE}:Percent" \
|
|
161
|
-
"ClaudeCodeCommits:${CLAUDE_CODE_COUNT}:Count" \
|
|
162
|
-
"KiroCommits:${KIRO_COUNT}:Count" \
|
|
163
|
-
"QDeveloperCommits:${Q_DEV_COUNT}:Count" \
|
|
164
|
-
"UntaggedCommits:${UNTAGGED}:Count"; do
|
|
165
|
-
METRIC_NAME="${METRIC_DATA%%:*}"; REST="${METRIC_DATA#*:}"
|
|
166
|
-
METRIC_VALUE="${REST%%:*}"; METRIC_UNIT="${REST#*:}"
|
|
167
|
-
aws cloudwatch put-metric-data --region "${AWS_DEFAULT_REGION}" --namespace "${NAMESPACE}" \
|
|
168
|
-
--metric-data "[{\"MetricName\":\"${METRIC_NAME}\",\"Value\":${METRIC_VALUE},
|
|
169
|
-
\"Unit\":\"${METRIC_UNIT}\",\"Timestamp\":\"${TIMESTAMP}\",
|
|
170
|
-
\"Dimensions\":[{\"Name\":\"TeamId\",\"Value\":\"${TEAM_ID}\"},{\"Name\":\"Repository\",\"Value\":\"${REPO}\"}]}]"
|
|
171
|
-
done
|
|
172
|
-
|
|
173
|
-
echo "=== DORA Metrics ==="
|
|
174
|
-
echo "Deployment Frequency: ${DEPLOY_FREQUENCY}/day"
|
|
175
|
-
echo "Lead Time: ${AVG_LEAD_TIME_HOURS} hours"
|
|
176
|
-
echo "Change Failure Rate: ${CHANGE_FAILURE_RATE}"
|
|
177
|
-
echo "MTTR: ${MTTR_HOURS} hours"
|
|
178
|
-
echo "=== AI-DORA Metrics ==="
|
|
179
|
-
echo "AI Adoption Rate: ${AI_ADOPTION_RATE}"
|
|
180
|
-
echo "Spec Coverage: ${SPEC_COVERAGE}"
|
|
181
|
-
echo "Total Commits: ${TOTAL_COMMITS}"
|
|
@@ -1,142 +0,0 @@
|
|
|
1
|
-
import {
|
|
2
|
-
DynamoDBClient,
|
|
3
|
-
QueryCommand,
|
|
4
|
-
} from '@aws-sdk/client-dynamodb';
|
|
5
|
-
import {
|
|
6
|
-
EventBridgeClient,
|
|
7
|
-
PutEventsCommand,
|
|
8
|
-
} from '@aws-sdk/client-eventbridge';
|
|
9
|
-
|
|
10
|
-
const dynamoClient = new DynamoDBClient({});
|
|
11
|
-
const eventBridgeClient = new EventBridgeClient({});
|
|
12
|
-
|
|
13
|
-
const EVENTS_TABLE = process.env.EVENTS_TABLE!;
|
|
14
|
-
const EVENT_BUS_NAME = process.env.EVENT_BUS_NAME!;
|
|
15
|
-
const LOOKBACK_HOURS = parseInt(process.env.LOOKBACK_HOURS ?? '24', 10);
|
|
16
|
-
|
|
17
|
-
interface DeployEvent {
|
|
18
|
-
source: string;
|
|
19
|
-
'detail-type': string;
|
|
20
|
-
detail: {
|
|
21
|
-
team_id: string;
|
|
22
|
-
repo: string;
|
|
23
|
-
timestamp: string;
|
|
24
|
-
dora?: {
|
|
25
|
-
change_failure_rate?: number;
|
|
26
|
-
};
|
|
27
|
-
};
|
|
28
|
-
}
|
|
29
|
-
|
|
30
|
-
/**
|
|
31
|
-
* Triggered by failed deployment events.
|
|
32
|
-
* Queries recent commits by AI origin, calculates AI vs human defect rates,
|
|
33
|
-
* and emits a prism.d1.quality event.
|
|
34
|
-
*/
|
|
35
|
-
export async function handler(event: DeployEvent): Promise<void> {
|
|
36
|
-
const detail = event.detail;
|
|
37
|
-
|
|
38
|
-
// Only process deployment failures
|
|
39
|
-
const cfr = detail.dora?.change_failure_rate ?? 0;
|
|
40
|
-
if (cfr === 0) {
|
|
41
|
-
console.log('Deployment succeeded, no defect correlation needed');
|
|
42
|
-
return;
|
|
43
|
-
}
|
|
44
|
-
|
|
45
|
-
console.log(`Processing failed deployment for ${detail.team_id}/${detail.repo}`);
|
|
46
|
-
|
|
47
|
-
const deployTime = new Date(detail.timestamp);
|
|
48
|
-
const lookbackStart = new Date(deployTime.getTime() - LOOKBACK_HOURS * 60 * 60 * 1000);
|
|
49
|
-
|
|
50
|
-
// Query recent commit events for this repo
|
|
51
|
-
try {
|
|
52
|
-
const result = await dynamoClient.send(
|
|
53
|
-
new QueryCommand({
|
|
54
|
-
TableName: EVENTS_TABLE,
|
|
55
|
-
KeyConditionExpression: 'pk = :pk AND sk BETWEEN :start AND :end',
|
|
56
|
-
ExpressionAttributeValues: {
|
|
57
|
-
':pk': { S: `${detail.team_id}#${detail.repo}` },
|
|
58
|
-
':start': { S: lookbackStart.toISOString() },
|
|
59
|
-
':end': { S: deployTime.toISOString() },
|
|
60
|
-
},
|
|
61
|
-
}),
|
|
62
|
-
);
|
|
63
|
-
|
|
64
|
-
if (!result.Items || result.Items.length === 0) {
|
|
65
|
-
console.log('No recent commits found in lookback window');
|
|
66
|
-
return;
|
|
67
|
-
}
|
|
68
|
-
|
|
69
|
-
// Count commits by AI origin
|
|
70
|
-
let aiCommitCount = 0;
|
|
71
|
-
let humanCommitCount = 0;
|
|
72
|
-
|
|
73
|
-
for (const item of result.Items) {
|
|
74
|
-
const detailType = item.detail_type?.S ?? '';
|
|
75
|
-
if (detailType !== 'prism.d1.commit') continue;
|
|
76
|
-
|
|
77
|
-
const data = JSON.parse(item.data?.S ?? '{}');
|
|
78
|
-
const origin = data.ai_context?.origin ?? 'human';
|
|
79
|
-
|
|
80
|
-
if (origin === 'ai-generated' || origin === 'ai-assisted') {
|
|
81
|
-
aiCommitCount++;
|
|
82
|
-
} else {
|
|
83
|
-
humanCommitCount++;
|
|
84
|
-
}
|
|
85
|
-
}
|
|
86
|
-
|
|
87
|
-
const totalCommits = aiCommitCount + humanCommitCount;
|
|
88
|
-
if (totalCommits === 0) {
|
|
89
|
-
console.log('No commit events in lookback window');
|
|
90
|
-
return;
|
|
91
|
-
}
|
|
92
|
-
|
|
93
|
-
// Calculate defect rates using proportional distribution.
|
|
94
|
-
// Since we can't attribute a specific failure to a specific commit,
|
|
95
|
-
// we distribute the change failure rate proportionally by code origin.
|
|
96
|
-
// This gives a directional signal: if 80% of commits are AI and the
|
|
97
|
-
// deploy failed, AI code carries 80% of the attributed failure weight.
|
|
98
|
-
const aiDefectRate = aiCommitCount > 0 ? (cfr * aiCommitCount) / totalCommits : 0;
|
|
99
|
-
const humanDefectRate = humanCommitCount > 0 ? (cfr * humanCommitCount) / totalCommits : 0;
|
|
100
|
-
|
|
101
|
-
const qualityEvent = {
|
|
102
|
-
team_id: detail.team_id,
|
|
103
|
-
repo: detail.repo,
|
|
104
|
-
timestamp: detail.timestamp,
|
|
105
|
-
prism_level: 2,
|
|
106
|
-
metric: {
|
|
107
|
-
name: 'post_merge_defect_rate',
|
|
108
|
-
value: cfr,
|
|
109
|
-
unit: 'percent',
|
|
110
|
-
},
|
|
111
|
-
ai_dora: {
|
|
112
|
-
post_merge_defect_rate: cfr,
|
|
113
|
-
},
|
|
114
|
-
quality: {
|
|
115
|
-
deployment_id: `deploy-${Date.now()}`,
|
|
116
|
-
ai_defect_rate: Math.round(aiDefectRate * 10000) / 10000,
|
|
117
|
-
human_defect_rate: Math.round(humanDefectRate * 10000) / 10000,
|
|
118
|
-
total_ai_commits: aiCommitCount,
|
|
119
|
-
total_human_commits: humanCommitCount,
|
|
120
|
-
},
|
|
121
|
-
};
|
|
122
|
-
|
|
123
|
-
await eventBridgeClient.send(
|
|
124
|
-
new PutEventsCommand({
|
|
125
|
-
Entries: [
|
|
126
|
-
{
|
|
127
|
-
Source: 'prism.d1.velocity',
|
|
128
|
-
DetailType: 'prism.d1.quality',
|
|
129
|
-
EventBusName: EVENT_BUS_NAME,
|
|
130
|
-
Detail: JSON.stringify(qualityEvent),
|
|
131
|
-
},
|
|
132
|
-
],
|
|
133
|
-
}),
|
|
134
|
-
);
|
|
135
|
-
|
|
136
|
-
console.log(
|
|
137
|
-
`Emitted quality event: AI defect rate ${aiDefectRate.toFixed(4)}, Human defect rate ${humanDefectRate.toFixed(4)}, ${aiCommitCount} AI commits, ${humanCommitCount} human commits`,
|
|
138
|
-
);
|
|
139
|
-
} catch (err) {
|
|
140
|
-
console.error('Failed to correlate defects:', err);
|
|
141
|
-
}
|
|
142
|
-
}
|
|
@@ -1,123 +0,0 @@
|
|
|
1
|
-
import {
|
|
2
|
-
DynamoDBClient,
|
|
3
|
-
QueryCommand,
|
|
4
|
-
} from '@aws-sdk/client-dynamodb';
|
|
5
|
-
import {
|
|
6
|
-
EventBridgeClient,
|
|
7
|
-
PutEventsCommand,
|
|
8
|
-
} from '@aws-sdk/client-eventbridge';
|
|
9
|
-
|
|
10
|
-
const dynamoClient = new DynamoDBClient({});
|
|
11
|
-
const eventBridgeClient = new EventBridgeClient({});
|
|
12
|
-
|
|
13
|
-
const EVENTS_TABLE = process.env.EVENTS_TABLE!;
|
|
14
|
-
const EVENT_BUS_NAME = process.env.EVENT_BUS_NAME!;
|
|
15
|
-
|
|
16
|
-
interface PrMergedEvent {
|
|
17
|
-
source: string;
|
|
18
|
-
'detail-type': string;
|
|
19
|
-
detail: {
|
|
20
|
-
team_id: string;
|
|
21
|
-
repo: string;
|
|
22
|
-
timestamp: string;
|
|
23
|
-
ai_context?: {
|
|
24
|
-
tool: string;
|
|
25
|
-
model: string;
|
|
26
|
-
origin: string;
|
|
27
|
-
};
|
|
28
|
-
pr?: {
|
|
29
|
-
spec_ref?: string;
|
|
30
|
-
merged_at?: string;
|
|
31
|
-
};
|
|
32
|
-
};
|
|
33
|
-
}
|
|
34
|
-
|
|
35
|
-
/**
|
|
36
|
-
* Triggered by merged PR events that have a Spec-Ref.
|
|
37
|
-
* Calculates spec-to-code hours by finding the earliest commit
|
|
38
|
-
* referencing the same spec and computing the time delta to PR merge.
|
|
39
|
-
*/
|
|
40
|
-
export async function handler(event: PrMergedEvent): Promise<void> {
|
|
41
|
-
const detail = event.detail;
|
|
42
|
-
const specRef = detail.pr?.spec_ref;
|
|
43
|
-
|
|
44
|
-
if (!specRef) {
|
|
45
|
-
console.log('No spec reference in PR event, skipping');
|
|
46
|
-
return;
|
|
47
|
-
}
|
|
48
|
-
|
|
49
|
-
console.log(`Calculating spec-to-code for spec: ${specRef}`);
|
|
50
|
-
|
|
51
|
-
// Query for the earliest commit event in this repo that references this spec.
|
|
52
|
-
// spec_ref is stored as a top-level DynamoDB attribute by the metrics-processor.
|
|
53
|
-
try {
|
|
54
|
-
const result = await dynamoClient.send(
|
|
55
|
-
new QueryCommand({
|
|
56
|
-
TableName: EVENTS_TABLE,
|
|
57
|
-
KeyConditionExpression: 'pk = :pk',
|
|
58
|
-
FilterExpression: 'spec_ref = :spec_ref AND detail_type = :dt',
|
|
59
|
-
ExpressionAttributeValues: {
|
|
60
|
-
':pk': { S: `${detail.team_id}#${detail.repo}` },
|
|
61
|
-
':spec_ref': { S: specRef },
|
|
62
|
-
':dt': { S: 'prism.d1.commit' },
|
|
63
|
-
},
|
|
64
|
-
ScanIndexForward: true, // Oldest first
|
|
65
|
-
Limit: 1,
|
|
66
|
-
}),
|
|
67
|
-
);
|
|
68
|
-
|
|
69
|
-
if (!result.Items || result.Items.length === 0) {
|
|
70
|
-
console.log(`No commit events found referencing spec: ${specRef}`);
|
|
71
|
-
return;
|
|
72
|
-
}
|
|
73
|
-
|
|
74
|
-
const earliestSpecCommitTime = result.Items[0].sk?.S;
|
|
75
|
-
if (!earliestSpecCommitTime) {
|
|
76
|
-
console.log('Could not determine spec commit timestamp');
|
|
77
|
-
return;
|
|
78
|
-
}
|
|
79
|
-
|
|
80
|
-
const mergedAt = detail.pr?.merged_at ?? detail.timestamp;
|
|
81
|
-
const specTime = new Date(earliestSpecCommitTime);
|
|
82
|
-
const mergeTime = new Date(mergedAt);
|
|
83
|
-
const specToCodeHours = (mergeTime.getTime() - specTime.getTime()) / (1000 * 60 * 60);
|
|
84
|
-
|
|
85
|
-
if (specToCodeHours < 0) {
|
|
86
|
-
console.log('Negative spec-to-code time, likely data issue');
|
|
87
|
-
return;
|
|
88
|
-
}
|
|
89
|
-
|
|
90
|
-
const specEvent = {
|
|
91
|
-
team_id: detail.team_id,
|
|
92
|
-
repo: detail.repo,
|
|
93
|
-
timestamp: detail.timestamp,
|
|
94
|
-
prism_level: 3,
|
|
95
|
-
metric: {
|
|
96
|
-
name: 'spec_to_code_hours',
|
|
97
|
-
value: Math.round(specToCodeHours * 100) / 100,
|
|
98
|
-
unit: 'hours',
|
|
99
|
-
},
|
|
100
|
-
ai_context: detail.ai_context,
|
|
101
|
-
ai_dora: {
|
|
102
|
-
spec_to_code_hours: Math.round(specToCodeHours * 100) / 100,
|
|
103
|
-
},
|
|
104
|
-
};
|
|
105
|
-
|
|
106
|
-
await eventBridgeClient.send(
|
|
107
|
-
new PutEventsCommand({
|
|
108
|
-
Entries: [
|
|
109
|
-
{
|
|
110
|
-
Source: 'prism.d1.velocity',
|
|
111
|
-
DetailType: 'prism.d1.pr',
|
|
112
|
-
EventBusName: EVENT_BUS_NAME,
|
|
113
|
-
Detail: JSON.stringify(specEvent),
|
|
114
|
-
},
|
|
115
|
-
],
|
|
116
|
-
}),
|
|
117
|
-
);
|
|
118
|
-
|
|
119
|
-
console.log(`Spec-to-code: ${specToCodeHours.toFixed(2)} hours for spec ${specRef}`);
|
|
120
|
-
} catch (err) {
|
|
121
|
-
console.error('Failed to calculate spec-to-code hours:', err);
|
|
122
|
-
}
|
|
123
|
-
}
|
|
@@ -1,107 +0,0 @@
|
|
|
1
|
-
# Metric Hooks
|
|
2
|
-
|
|
3
|
-
A `prepare-commit-msg` git hook that automatically tags commits with AI origin metadata and token usage.
|
|
4
|
-
|
|
5
|
-
## What It Does
|
|
6
|
-
|
|
7
|
-
Every commit gets trailers appended to the message:
|
|
8
|
-
|
|
9
|
-
```
|
|
10
|
-
feat: add order creation endpoint
|
|
11
|
-
|
|
12
|
-
AI-Origin: ai-generated
|
|
13
|
-
AI-Tool: claude-code
|
|
14
|
-
AI-Model: us.anthropic.claude-sonnet-4-5-20250929-v1:0
|
|
15
|
-
AI-Input-Tokens: 12450
|
|
16
|
-
AI-Output-Tokens: 3200
|
|
17
|
-
AI-Cost: $0.08
|
|
18
|
-
Spec-Ref: specs/create-order-endpoint.md
|
|
19
|
-
```
|
|
20
|
-
|
|
21
|
-
These trailers are read by the `prism-ai-metrics.yml` GitHub workflow on PR merge to emit metrics to EventBridge.
|
|
22
|
-
|
|
23
|
-
## Installation
|
|
24
|
-
|
|
25
|
-
```bash
|
|
26
|
-
bash prism-cli bootstrapper install-git-hooks --team-id my-team
|
|
27
|
-
```
|
|
28
|
-
|
|
29
|
-
Or interactively (prompts for team ID):
|
|
30
|
-
|
|
31
|
-
```bash
|
|
32
|
-
bash prism-cli bootstrapper install-git-hooks
|
|
33
|
-
```
|
|
34
|
-
|
|
35
|
-
To remove:
|
|
36
|
-
|
|
37
|
-
```bash
|
|
38
|
-
bash prism-cli bootstrapper install-git-hooks --uninstall
|
|
39
|
-
```
|
|
40
|
-
|
|
41
|
-
The installer also registers a Claude Code `SessionStart` hook in
|
|
42
|
-
`~/.claude/settings.json` (served by `prism-cli git claude-session-context`).
|
|
43
|
-
It captures the Claude session id into the environment so commits made during a
|
|
44
|
-
Claude Code session are attributed correctly.
|
|
45
|
-
|
|
46
|
-
## Prerequisites
|
|
47
|
-
|
|
48
|
-
- **prism-cli** — Runs the hook logic (`prism-cli git commit-trailers`). Install: `npm install -g @prism-d1/cli`
|
|
49
|
-
- **codeburn** *(optional)* — Token usage tracking for non-Kiro tools. Install: `npm install -g codeburn` (or `brew install codeburn` on macOS). Kiro sessions are parsed directly by prism-cli, no codeburn needed.
|
|
50
|
-
|
|
51
|
-
> The hook is a thin bash delegator to prism-cli. It requires only `git` and
|
|
52
|
-
> `bash` (with `prism-cli` on PATH) — **no `jq`, `bc`, or `sed`** — so it works
|
|
53
|
-
> on Linux, macOS, and Windows Git Bash. If `prism-cli` is not installed, the
|
|
54
|
-
> commit proceeds normally without trailers.
|
|
55
|
-
|
|
56
|
-
## How AI Detection Works
|
|
57
|
-
|
|
58
|
-
The hook (via `prism-cli git commit-trailers`) checks for AI tool involvement, in order:
|
|
59
|
-
|
|
60
|
-
1. **Claude Code**: `CLAUDE_CODE` or `CLAUDE_CODE_SESSION_ID` environment variable
|
|
61
|
-
2. **Kiro**: `KIRO_SESSION_ID` / `KIRO_SESSION` env var, `TERM_PROGRAM=kiro` (IDE terminal), or a `kiro` path in `VSCODE_GIT_ASKPASS_NODE` / `GIT_ASKPASS` (Source Control panel commits)
|
|
62
|
-
3. **Q Developer**: `Q_DEVELOPER_SESSION` environment variable
|
|
63
|
-
4. **Default**: No indicators → `AI-Origin: human`
|
|
64
|
-
|
|
65
|
-
## Token Tracking
|
|
66
|
-
|
|
67
|
-
When an AI tool is detected, the hook computes a per-commit delta:
|
|
68
|
-
|
|
69
|
-
1. Collects lifetime token totals — Kiro sessions are parsed directly by prism-cli; other tools use `codeburn report -p all --format json`
|
|
70
|
-
2. Compares against a snapshot from the previous commit (`~/.prism/tokentracker/<project-basename>.json`)
|
|
71
|
-
3. Writes the delta as `AI-Input-Tokens` and `AI-Output-Tokens` trailers
|
|
72
|
-
4. Saves the new snapshot for next time
|
|
73
|
-
|
|
74
|
-
If no usage data is available (e.g. codeburn absent for a non-Kiro tool) or no AI tool is detected, token trailers are omitted.
|
|
75
|
-
|
|
76
|
-
## Configuration
|
|
77
|
-
|
|
78
|
-
The installer creates `.prism/config.json`:
|
|
79
|
-
|
|
80
|
-
```json
|
|
81
|
-
{
|
|
82
|
-
"team_id": "your-team",
|
|
83
|
-
"max_tokens": 1000000,
|
|
84
|
-
"max_cost": 100
|
|
85
|
-
}
|
|
86
|
-
```
|
|
87
|
-
|
|
88
|
-
| Field | Description | Default |
|
|
89
|
-
|-------|-------------|---------|
|
|
90
|
-
| `team_id` | Team identifier for metric attribution | *(required)* |
|
|
91
|
-
| `max_tokens` | Max input/output tokens per commit (capped at this value) | `1000000` |
|
|
92
|
-
| `max_cost` | Max cost in USD per commit (capped at this value) | `100` |
|
|
93
|
-
|
|
94
|
-
Set custom bounds at install time:
|
|
95
|
-
|
|
96
|
-
```bash
|
|
97
|
-
prism-cli bootstrapper install-git-hooks --team-id my-team --max-tokens 500000 --max-cost 50
|
|
98
|
-
```
|
|
99
|
-
|
|
100
|
-
Values exceeding bounds are clamped to the configured maximum. The workflow (`prism-ai-metrics.yml`) applies a second layer of enforcement, discarding values above 1M tokens / $100 to zero.
|
|
101
|
-
|
|
102
|
-
## Safety
|
|
103
|
-
|
|
104
|
-
- Never blocks a commit — exits 0 even if prism-cli or codeburn is missing or errors
|
|
105
|
-
- Only appends trailers — never modifies code
|
|
106
|
-
- Skips merge and squash commits
|
|
107
|
-
- Won't duplicate trailers if already present
|