@thinkingai/ae-cli 6.0.54 → 6.0.56
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +24 -3
- package/README.zh.md +24 -3
- package/dist/{auth-MBWRZTB6.js → auth-U7MNUUEF.js} +37 -13
- package/dist/{automation-NWWVLWG2.js → automation-K32V2NKP.js} +94 -38
- package/dist/{capability-7SSN6PNZ.js → capability-VV7RHB3B.js} +8 -7
- package/dist/{chunk-CT7QDJS6.js → chunk-2IYIYSUO.js} +2 -2
- package/dist/{chunk-PLE722AJ.js → chunk-2UWS5CUP.js} +62 -35
- package/dist/{chunk-ZIQWCZTS.js → chunk-3TQA5RXE.js} +3 -3
- package/dist/{chunk-KFNMQTZI.js → chunk-ELCR235M.js} +26 -19
- package/dist/{chunk-FMNNGNYO.js → chunk-F6NIPIT5.js} +7 -6
- package/dist/{chunk-D6ACFILC.js → chunk-GI5BMEZO.js} +1 -1
- package/dist/chunk-JJJIP2XQ.js +105 -0
- package/dist/{chunk-5XJYLEVC.js → chunk-KHS64JZQ.js} +4 -4
- package/dist/{chunk-UQ77JXYH.js → chunk-LMCYSVVA.js} +12 -13
- package/dist/{chunk-EOYSKMXN.js → chunk-MXV3PYC3.js} +14 -6
- package/dist/{chunk-7QXISROD.js → chunk-P6VFCH3P.js} +4 -2
- package/dist/{chunk-GIK6EDUG.js → chunk-P7KL7IOH.js} +41 -13
- package/dist/{chunk-AP7YP2NX.js → chunk-QSAR3GHY.js} +2 -2
- package/dist/{chunk-S6M42KND.js → chunk-T44M3PLZ.js} +1 -1
- package/dist/{chunk-HVHR4SCP.js → chunk-UUD5RB36.js} +3 -3
- package/dist/{chunk-HBLK3ZHW.js → chunk-W2PTY5NN.js} +3 -3
- package/dist/chunk-X7DRLE62.js +233 -0
- package/dist/{chunk-WNGFSIOB.js → chunk-ZW4VMY5H.js} +103 -6
- package/dist/{community-report-client-PEERQCX2.js → community-report-client-S2K3A3Q6.js} +1 -1
- package/dist/{config-FF6YVJKI.js → config-ASSAAIOI.js} +10 -9
- package/dist/{context-YAE3QXP3.js → context-6RWLVOM5.js} +4 -4
- package/dist/{data-integration-HESZET7E.js → data-integration-EXSUPUDZ.js} +6 -6
- package/dist/evaluation-5YT4QLCL.js +3714 -0
- package/dist/index.js +93 -49
- package/dist/{local-data-upload-client-TWMHFLUA.js → local-data-upload-client-7HCPGIS3.js} +1 -1
- package/dist/{memory-62BJOULV.js → memory-VCYLVPMI.js} +6 -6
- package/dist/{metadata-7CUMWIB4.js → metadata-TIWIOFTU.js} +8 -8
- package/dist/{model-SBV2JFJI.js → model-DPJHSD73.js} +4 -4
- package/dist/{personal-semantic-preference-6SRCYPGE.js → personal-semantic-preference-SBKU2LOK.js} +6 -6
- package/dist/{project-semantic-YYEM4OSU.js → project-semantic-SKNWLH3I.js} +6 -6
- package/dist/{sync-SU33KYX7.js → sync-7PP6KQ75.js} +7 -7
- package/dist/{te-agent-F5QBZSQA.js → te-agent-L2BYNPHM.js} +753 -94
- package/dist/{te-analysis-U4I2W2WC.js → te-analysis-RJB6VJC4.js} +411 -79
- package/dist/{te-community-PFLDLPP4.js → te-community-QN3WMPNT.js} +6 -4
- package/dist/{te-dataops-FOQSRIAZ.js → te-dataops-UKD2XN3E.js} +8 -6
- package/dist/{te-engage-ZX76CNAB.js → te-engage-KPTN4N2U.js} +6 -6
- package/dist/{te-kb-MHDOST62.js → te-kb-SFZI3W5T.js} +7 -7
- package/dist/{te-system-J2FQF5ER.js → te-system-IE7NCKC3.js} +10 -9
- package/dist/{te-team-5UOZPTU3.js → te-team-4KHCFM2I.js} +63 -41
- package/dist/{things-VPPXXLHB.js → things-7JJUVAWL.js} +7 -7
- package/dist/{update-EEUTEI6R.js → update-AX66IFMJ.js} +4 -4
- package/dist/{workspace-O3EEIYQM.js → workspace-Q5NMEKEN.js} +6 -6
- package/package.json +11 -7
- package/skills/ae-agent/SKILL.md +22 -3
- package/skills/ae-agent/references/command_index.md +8 -0
- package/skills/ae-agent/references/resource-package.md +53 -0
- package/skills/ae-analysis/SKILL.md +12 -3
- package/skills/ae-analysis/references/adhoc_export.md +1 -1
- package/skills/ae-analysis/references/agent_review_preflight.md +4 -0
- package/skills/ae-analysis/references/agent_review_submit_to_page.md +4 -0
- package/skills/ae-analysis/references/ai_models/event.md +58 -2
- package/skills/ae-analysis/references/ai_models/rank_list.md +1 -0
- package/skills/ae-analysis/references/ai_models/retention.md +2 -0
- package/skills/ae-analysis/references/ai_models/revenue.md +2 -0
- package/skills/ae-analysis/references/analysis_data_export.md +11 -1
- package/skills/ae-analysis/references/asset_batch_info_export.md +2 -2
- package/skills/ae-analysis/references/asset_batch_sql_export.md +2 -2
- package/skills/ae-analysis/references/asset_export.md +1 -1
- package/skills/ae-analysis/references/bi_panel_page_data_export.md +1 -1
- package/skills/ae-analysis/references/check_export.md +1 -1
- package/skills/ae-analysis/references/command_index.md +34 -25
- package/skills/ae-analysis/references/dashboard_report_data_export.md +2 -0
- package/skills/ae-analysis/references/dashboard_report_get.md +6 -2
- package/skills/ae-analysis/references/dashboard_report_update.md +24 -5
- package/skills/ae-analysis/references/feedback_submit.md +1 -1
- package/skills/ae-analysis/references/governance_recommendation_export.md +10 -2
- package/skills/ae-analysis/references/live_data_export.md +1 -1
- package/skills/ae-analysis/references/metric_create.md +29 -0
- package/skills/ae-analysis/references/metric_get.md +2 -0
- package/skills/ae-analysis/references/metric_update.md +2 -0
- package/skills/ae-analysis/references/operation_record_export.md +1 -1
- package/skills/ae-analysis/references/plan_change_log_export.md +1 -1
- package/skills/ae-analysis/references/plan_export.md +1 -1
- package/skills/ae-analysis/references/project_info_get.md +4 -0
- package/skills/ae-analysis/references/project_member_group_add_members.md +35 -0
- package/skills/ae-analysis/references/project_member_group_create.md +25 -0
- package/skills/ae-analysis/references/project_member_group_delete.md +34 -0
- package/skills/ae-analysis/references/project_member_group_list.md +29 -0
- package/skills/ae-analysis/references/project_member_group_members.md +28 -0
- package/skills/ae-analysis/references/project_member_group_remove_members.md +35 -0
- package/skills/ae-analysis/references/project_member_group_rename.md +26 -0
- package/skills/ae-analysis/references/project_semantic_knowledge_wiki_plan_schema.md +6 -1
- package/skills/ae-analysis/references/report_data_export.md +1 -1
- package/skills/ae-analysis/references/scenario_append.md +25 -0
- package/skills/ae-analysis/references/scenario_collect.md +35 -0
- package/skills/ae-analysis/references/scenario_evaluation.md +30 -0
- package/skills/ae-analysis/scripts/project-semantic-knowledge-wiki/build-project-semantic-wiki.mjs +73 -4
- package/skills/ae-analysis/scripts/project-semantic-knowledge-wiki/default-compile-schema.md +5 -1
- package/skills/ae-analysis/scripts/project-semantic-knowledge-wiki/execution-contract.mjs +173 -0
- package/skills/ae-analysis/scripts/project-semantic-knowledge-wiki/generate-build-ir.mjs +109 -64
- package/skills/ae-automation/SKILL.md +2 -2
- package/skills/ae-automation/references/cli.md +18 -1
- package/skills/ae-capability/SKILL.md +7 -3
- package/skills/ae-evaluation/SKILL.md +83 -0
- package/skills/ae-evaluation/references/access-and-members.md +55 -0
- package/skills/ae-evaluation/references/connections.md +78 -0
- package/skills/ae-evaluation/references/datasets.md +57 -0
- package/skills/ae-evaluation/references/evaluators.md +81 -0
- package/skills/ae-evaluation/references/files.md +33 -0
- package/skills/ae-evaluation/references/integrations.md +39 -0
- package/skills/ae-evaluation/references/runs.md +160 -0
- package/skills/ae-evaluation/references/schedules.md +31 -0
- package/skills/ae-metadata/SKILL.md +2 -1
- package/skills/ae-metadata/references/metadata_data_table_download.md +7 -1
- package/skills/ae-metadata/references/metadata_data_table_get.md +2 -0
- package/skills/ae-team/SKILL.md +2 -2
- package/skills/ae-team/references/run-start.md +1 -0
- package/skills/ae-team/references/run-watch.md +7 -5
- package/skills/ae-things/SKILL.md +53 -51
- package/skills/ae-things/references/cli.md +56 -54
- package/skills/ae-things-progress/SKILL.md +20 -20
- package/skills/ae-things-review/SKILL.md +19 -19
- package/dist/chunk-4655PUNX.js +0 -43
|
@@ -3,6 +3,9 @@
|
|
|
3
3
|
import { createHash, randomBytes } from 'node:crypto';
|
|
4
4
|
import fs from 'node:fs';
|
|
5
5
|
import path from 'node:path';
|
|
6
|
+
import { conditionTree, renderConditionTree, reportContract, reportSignature,
|
|
7
|
+
semanticValue, enrichMetricEvidence, renderExecutionContract, validateSqlExplanations,
|
|
8
|
+
renderSqlExplanations } from './execution-contract.mjs';
|
|
6
9
|
|
|
7
10
|
const args = parseArgs(process.argv.slice(2));
|
|
8
11
|
const packageRoot = path.resolve(requiredArg(args, 'asset-package'));
|
|
@@ -99,13 +102,19 @@ const sqlReportSemanticsById = new Map(array(domainPlan.sql_report_semantics)
|
|
|
99
102
|
.map((value) => object(value))
|
|
100
103
|
.map((value) => [text(value.report_id), value])
|
|
101
104
|
.filter(([id]) => id));
|
|
105
|
+
for (const report of reports) {
|
|
106
|
+
report.calculation_context = calculationContextForKb(report);
|
|
107
|
+
report.metric_occurrences = enrichMetricEvidence(array(report.metric_occurrences), report.calculation_context);
|
|
108
|
+
}
|
|
102
109
|
const semanticConflicts = detectSemanticConflicts();
|
|
103
110
|
const reportConflicts = semanticConflicts.reportConflicts;
|
|
111
|
+
const reportSettingVariants = semanticConflicts.executionVariants;
|
|
104
112
|
const metadataSemanticConflicts = semanticConflicts.metadataConflicts;
|
|
105
113
|
const metadataMissingTitles = metadata.filter((item) => !text(item.title));
|
|
106
114
|
const reportsInBusinessDomains = reports.filter((report) => domainsForReport(report).length);
|
|
107
115
|
const reportsInTechnicalAppendices = reports.filter((report) => appendicesForReport(report).length);
|
|
108
116
|
const warnings = [
|
|
117
|
+
...[...reportSettingVariants.keys()].map(id => `Report ${id} has matching calculation evidence but different saved or uninterpreted settings; verify explicit overrides before reuse.`),
|
|
109
118
|
...(text(descriptor.asset_scope ?? manifest.asset_scope) !== 'governed'
|
|
110
119
|
? [`The package scope is ${text(descriptor.asset_scope ?? manifest.asset_scope) || 'unknown'}; unauthenticated assets may be present.`]
|
|
111
120
|
: []),
|
|
@@ -533,6 +542,9 @@ function validateSqlSemanticPlan(plan) {
|
|
|
533
542
|
const summary = object(summaryValue);
|
|
534
543
|
const reportId = text(summary.report_id);
|
|
535
544
|
if (!sqlReportsById.has(reportId)) continue;
|
|
545
|
+
try {
|
|
546
|
+
validateSqlExplanations(summary, sqlTextFromCalculationContext(calculationContextForKb(sqlReportsById.get(reportId))));
|
|
547
|
+
} catch (error) { fail(`SQL report ${reportId}: ${error.message}`); }
|
|
536
548
|
if (text(summary.definition_state) !== 'valid') continue;
|
|
537
549
|
const issues = sqlSemanticQualityIssues(summary);
|
|
538
550
|
if (issues.length) {
|
|
@@ -1036,13 +1048,29 @@ function renderDashboardAgentUsageSummary(dashboard, reportRows, notes, domain,
|
|
|
1036
1048
|
}
|
|
1037
1049
|
|
|
1038
1050
|
function renderReportAgentUsageSummary(report, metadataRefs, calculationContext, sqlSummary) {
|
|
1051
|
+
const explanation = renderSqlExplanations(sqlSummary);
|
|
1052
|
+
const coverage = text(report.definition_kind) === 'SQL' && !array(sqlSummary?.calculation_rules).length
|
|
1053
|
+
? '\n\n- Calculation coverage: unknown for this legacy summary. Verify the current SQL for operand filters, denominator, units, rounding, time baselines and excluded rows before execution.' : '';
|
|
1054
|
+
return renderReportAuthoredSummary(report, metadataRefs, calculationContext, sqlSummary)
|
|
1055
|
+
+ (explanation ? `\n\n### Calculation evidence\n\n${explanation}` : '')
|
|
1056
|
+
+ coverage
|
|
1057
|
+
+ renderExecutionContract(reportContract(report, calculationContext))
|
|
1058
|
+
+ (reportSettingVariants.has(text(report.resource_key)) ? '\n- Saved-setting variant: matching calculation evidence does not establish identical defaults. Verify or explicitly override the required settings before reuse.\n' : '');
|
|
1059
|
+
}
|
|
1060
|
+
|
|
1061
|
+
function renderReportAuthoredSummary(report, metadataRefs, calculationContext, sqlSummary) {
|
|
1039
1062
|
const isSql = text(report.definition_kind) === 'SQL';
|
|
1040
1063
|
if (isSql) return renderSqlReportAgentUsageSummary(report, sqlSummary);
|
|
1064
|
+
const modelConfig = reportContract(report, calculationContext).model_config;
|
|
1065
|
+
if (array(modelConfig.userCrowds).length || modelConfig.windowsGapValue != null) {
|
|
1066
|
+
const model = array(modelConfig.userCrowds).length ? 'Saved population analysis' : 'Saved interval analysis';
|
|
1067
|
+
return `- Model: ${model}.\n- Purpose: ${text(report.description || report.summary) || 'Verify the business question against the saved model settings below.'}\n- Result semantics: use the complete population conditions or ordered event roles, window and selected statistics in the execution contract.\n- Missing configuration, entity scope and historical state remain unknown; do not infer ordinary event metrics from model endpoints.`;
|
|
1068
|
+
}
|
|
1041
1069
|
const purpose = usefulSemanticText(sqlSummary?.business_purpose) || reportPurpose(report);
|
|
1042
1070
|
const questions = usefulSemanticList(sqlSummary?.applicable_questions)
|
|
1043
1071
|
? array(sqlSummary.applicable_questions).map(renderPlainSemanticItem)
|
|
1044
1072
|
: reportApplicableQuestions(report);
|
|
1045
|
-
const metrics =
|
|
1073
|
+
const metrics = array(report.metric_occurrences).map(occurrenceSentence);
|
|
1046
1074
|
const parameters = reportParameterSummaryItems(calculationContext);
|
|
1047
1075
|
const dimensions = reportDimensionSummaryItems(report, sqlSummary, calculationContext, metadataRefs);
|
|
1048
1076
|
const filters = reportFilterSummaryItems(report, calculationContext);
|
|
@@ -1083,18 +1111,29 @@ function renderSqlReportAgentUsageSummary(report, summary) {
|
|
|
1083
1111
|
}
|
|
1084
1112
|
|
|
1085
1113
|
function renderSqlSemanticItem(value, summary) {
|
|
1114
|
+
const item = object(value);
|
|
1115
|
+
if (Object.keys(item).length && item.required == null && item.default_value == null && !Array.isArray(item.options)) {
|
|
1116
|
+
return [text(item.name) ? `\`${text(item.name)}\`` : '', sanitizeVisibleSqlSemanticText(item.meaning)].filter(Boolean).join(':')
|
|
1117
|
+
+ (item.evidence ? ` [source: ${jsonInline(item.evidence)}]` : '')
|
|
1118
|
+
+ (item.identity_state === 'unknown' ? ' [output identity unverified; match the current SQL projection before use]' : '');
|
|
1119
|
+
}
|
|
1086
1120
|
return replaceSqlParameterMentions(renderSemanticItem(value), summary);
|
|
1087
1121
|
}
|
|
1088
1122
|
|
|
1089
1123
|
function renderSqlInputParameterItem(value) {
|
|
1090
|
-
return sanitizeVisibleSqlSemanticText(
|
|
1124
|
+
if (!value || typeof value !== 'object') return sanitizeVisibleSqlSemanticText(value);
|
|
1125
|
+
return renderSemanticItem({ ...value, meaning: sanitizeVisibleSqlSemanticText(value.meaning) });
|
|
1091
1126
|
}
|
|
1092
1127
|
|
|
1093
1128
|
function replaceSqlParameterMentions(value, summary) {
|
|
1094
|
-
let result =
|
|
1129
|
+
let result = text(value);
|
|
1095
1130
|
for (const [name, meaning] of sqlParameterMeaningEntries(summary)) {
|
|
1096
|
-
const pattern = new RegExp(
|
|
1131
|
+
const pattern = new RegExp(`\\$\\{(?:[A-Za-z]+:)?${escapeRegExp(name)}\\}`, 'gu');
|
|
1097
1132
|
result = result.replace(pattern, meaning);
|
|
1133
|
+
if (/^(?:selector|variable|text|number|date)\d+$/i.test(name)) {
|
|
1134
|
+
const generatedReference = new RegExp(`(?<![A-Za-z0-9_#$\x60])${escapeRegExp(name)}(?![A-Za-z0-9_#$\x60])`, 'gu');
|
|
1135
|
+
result = result.replace(generatedReference, meaning);
|
|
1136
|
+
}
|
|
1098
1137
|
}
|
|
1099
1138
|
return sanitizeVisibleSqlSemanticText(result)
|
|
1100
1139
|
.replace(/\bLIMIT\s+([^,;。]+?)限制/g, '按$1限制')
|
|
@@ -1135,11 +1174,6 @@ function genericSqlParameterMeaning(name) {
|
|
|
1135
1174
|
return normalized;
|
|
1136
1175
|
}
|
|
1137
1176
|
|
|
1138
|
-
function isGeneratedSqlParameterName(name) {
|
|
1139
|
-
return /^(?:selector|variable|text|number|partdate)\d*$/i.test(text(name))
|
|
1140
|
-
|| /^date\d+$/i.test(text(name));
|
|
1141
|
-
}
|
|
1142
|
-
|
|
1143
1177
|
function escapeRegExp(value) {
|
|
1144
1178
|
return text(value).replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
|
|
1145
1179
|
}
|
|
@@ -1161,7 +1195,7 @@ function renderPlainSemanticItem(value) {
|
|
|
1161
1195
|
if (item.default_value != null && text(item.default_value)) extras.push(`default=${text(item.default_value)}`);
|
|
1162
1196
|
const expressionKind = text(item.expression_kind);
|
|
1163
1197
|
if (expressionKind) extras.push(expressionKind);
|
|
1164
|
-
if (isInputParameterSemanticItem(item) && meaning && name
|
|
1198
|
+
if (isInputParameterSemanticItem(item) && meaning && name) {
|
|
1165
1199
|
return [`${meaning}(\`${name}\`)`, extras.join(',')].filter(Boolean).join(':');
|
|
1166
1200
|
}
|
|
1167
1201
|
if (isInputParameterSemanticItem(item) && meaning) {
|
|
@@ -1190,6 +1224,12 @@ function reportDimensionSummaryItems(report, summary = null, calculationContext
|
|
|
1190
1224
|
const context = calculationContext || calculationContextForKb(report);
|
|
1191
1225
|
const facts = object(context.sql_semantic_facts);
|
|
1192
1226
|
const refs = metadataRefs || metadataRefsForReport(report);
|
|
1227
|
+
if (text(report.definition_kind) !== 'SQL') {
|
|
1228
|
+
return reportContract(report, context).groups.map(group => {
|
|
1229
|
+
const row = object(group);
|
|
1230
|
+
return namedCodeFact(row.columnDesc || row.columnName || text(group), row.columnName || text(group));
|
|
1231
|
+
});
|
|
1232
|
+
}
|
|
1193
1233
|
const candidates = [
|
|
1194
1234
|
...array(summary?.output_fields)
|
|
1195
1235
|
.filter((item) => text(object(item).semantic_role) === 'dimension')
|
|
@@ -1207,14 +1247,18 @@ function reportDimensionSummaryItems(report, summary = null, calculationContext
|
|
|
1207
1247
|
function reportParameterSummaryItems(calculationContext) {
|
|
1208
1248
|
const context = object(calculationContext);
|
|
1209
1249
|
return [
|
|
1250
|
+
...(Object.keys(object(context.time_scope)).length ? [`Saved time settings: ${JSON.stringify(context.time_scope)}`] : []),
|
|
1210
1251
|
...array(context.sql_params).map((item) => renderPlainSemanticItem({ name: object(item).name || object(item).paramName, meaning: object(item).displayName || object(item).placeholder })),
|
|
1211
1252
|
].filter(Boolean).slice(0, 12);
|
|
1212
1253
|
}
|
|
1213
1254
|
|
|
1214
1255
|
function reportFilterSummaryItems(report, calculationContext) {
|
|
1215
1256
|
const items = [
|
|
1216
|
-
...array(report.metric_occurrences).flatMap((item) =>
|
|
1217
|
-
|
|
1257
|
+
...array(report.metric_occurrences).flatMap((item) => [
|
|
1258
|
+
...filterSummaryLines(item.filters, item.relation),
|
|
1259
|
+
...array(item.custom_filters).flatMap((filter) => filterSummaryLines(filter).map(line => `Operand ${filter.index}: ${line}`)),
|
|
1260
|
+
]),
|
|
1261
|
+
...filterSummaryLines(object(calculationContext).filters, object(calculationContext).relation),
|
|
1218
1262
|
...array(object(calculationContext).event_view?.filts).flatMap((item) => filterSummaryLines(item)),
|
|
1219
1263
|
...array(object(calculationContext).visual_view?.filters).flatMap((item) => filterSummaryLines(item)),
|
|
1220
1264
|
...array(object(calculationContext).key_predicate_fragments).map(compactSqlFragment),
|
|
@@ -1224,7 +1268,17 @@ function reportFilterSummaryItems(report, calculationContext) {
|
|
|
1224
1268
|
|
|
1225
1269
|
function completeSqlAgentSummary(report, planSummary, facts) {
|
|
1226
1270
|
const base = object(planSummary);
|
|
1227
|
-
if (Object.keys(base).length)
|
|
1271
|
+
if (Object.keys(base).length) {
|
|
1272
|
+
const names = new Set(array(object(facts.final_select).fields).map(field => {
|
|
1273
|
+
if (field.name) return field.name;
|
|
1274
|
+
const expression = text(field.expression);
|
|
1275
|
+
return /^(?:[A-Za-z_][\w$#@.]*|"[^"]+")$/.test(expression) ? unquoteIdentifier(expression) : null;
|
|
1276
|
+
}).filter(Boolean));
|
|
1277
|
+
return { summary_source: 'semantic_plan', ...base,
|
|
1278
|
+
output_fields: array(base.output_fields).map(field => typeof field === 'object' && !names.has(field.name)
|
|
1279
|
+
? { ...field, identity_state: 'unknown' } : field),
|
|
1280
|
+
};
|
|
1281
|
+
}
|
|
1228
1282
|
return null;
|
|
1229
1283
|
}
|
|
1230
1284
|
|
|
@@ -1380,7 +1434,7 @@ function renderSemanticItem(value) {
|
|
|
1380
1434
|
.filter(([key]) => !['name', 'field', 'key', 'column', 'title', 'meaning', 'description', 'remark', 'purpose'].includes(key))
|
|
1381
1435
|
.map(([key, itemValue]) => `${key}=${inlineSemanticValue(itemValue)}`)
|
|
1382
1436
|
.join(', ');
|
|
1383
|
-
if (isInputParameterSemanticItem(item) && meaning && name
|
|
1437
|
+
if (isInputParameterSemanticItem(item) && meaning && name) {
|
|
1384
1438
|
return [`${meaning}(\`${name}\`)`, details].filter(Boolean).join(':') || jsonInline(item);
|
|
1385
1439
|
}
|
|
1386
1440
|
if (isInputParameterSemanticItem(item) && meaning) {
|
|
@@ -1391,12 +1445,9 @@ function renderSemanticItem(value) {
|
|
|
1391
1445
|
|
|
1392
1446
|
function isInputParameterSemanticItem(item) {
|
|
1393
1447
|
const row = object(item);
|
|
1394
|
-
const name = text(row.name ?? row.field ?? row.key ?? row.column ?? row.title);
|
|
1395
|
-
const meaning = text(row.meaning ?? row.description ?? row.remark ?? row.purpose);
|
|
1396
1448
|
return row.required != null
|
|
1397
1449
|
|| row.default_value != null
|
|
1398
|
-
|| Array.isArray(row.options)
|
|
1399
|
-
|| (Boolean(meaning) && /^(?:selector|variable|date|partdate|text|number)\d*$/i.test(name));
|
|
1450
|
+
|| Array.isArray(row.options);
|
|
1400
1451
|
}
|
|
1401
1452
|
|
|
1402
1453
|
function inlineSemanticValue(value) {
|
|
@@ -1434,6 +1485,11 @@ function reportDetailCalculationContext(report) {
|
|
|
1434
1485
|
if (Object.keys(eventView).length) result.event_view = eventView;
|
|
1435
1486
|
const events = jsonObject(rawDefinition.events);
|
|
1436
1487
|
if (array(events.sqlVoParams).length) result.sql_params = events.sqlVoParams;
|
|
1488
|
+
if (rawDefinition.events != null) {
|
|
1489
|
+
try { result.source_events = typeof rawDefinition.events === 'string' ? JSON.parse(rawDefinition.events) : rawDefinition.events; }
|
|
1490
|
+
catch { result.source_events = null; }
|
|
1491
|
+
}
|
|
1492
|
+
if (eventView.sqlViewParams != null) result.saved_parameters = eventView.sqlViewParams;
|
|
1437
1493
|
return result;
|
|
1438
1494
|
}
|
|
1439
1495
|
|
|
@@ -1857,6 +1913,9 @@ function metadataDefinitionLines(item) {
|
|
|
1857
1913
|
if (text(definition.entity_mapping_scope)) lines.push(`实体映射范围: ${text(definition.entity_mapping_scope)}`);
|
|
1858
1914
|
const sourceVersion = text(object(item.raw).sourceVersion ?? object(item.raw).source_version);
|
|
1859
1915
|
if (sourceVersion) lines.push(`来源版本: ${sourceVersion}`);
|
|
1916
|
+
for (const key of ['rule', 'rules', 'expression', 'dependent_events', 'dependent_properties']) {
|
|
1917
|
+
if (definition[key] != null) lines.push(`Rule evidence (${key}): ${jsonInline(definition[key])}`);
|
|
1918
|
+
}
|
|
1860
1919
|
return lines;
|
|
1861
1920
|
}
|
|
1862
1921
|
|
|
@@ -2020,11 +2079,17 @@ function eventBasisLabel(value) {
|
|
|
2020
2079
|
|
|
2021
2080
|
function detectSemanticConflicts() {
|
|
2022
2081
|
const reportConflicts = new Map();
|
|
2082
|
+
const executionVariants = new Map();
|
|
2023
2083
|
const metadataConflicts = new Map();
|
|
2024
2084
|
for (const [key, values] of groupBySemanticKey(reports, reportSemanticKey)) {
|
|
2025
2085
|
if (values.length < 2) continue;
|
|
2026
|
-
const signatures = new Set(values.map(
|
|
2027
|
-
if (signatures.size <= 1)
|
|
2086
|
+
const signatures = new Set(values.map(report => reportSignature(report, report.calculation_context, false)));
|
|
2087
|
+
if (signatures.size <= 1) {
|
|
2088
|
+
if (new Set(values.map(reportSemanticSignature)).size > 1) {
|
|
2089
|
+
for (const report of values) executionVariants.set(text(report.resource_key), true);
|
|
2090
|
+
}
|
|
2091
|
+
continue;
|
|
2092
|
+
}
|
|
2028
2093
|
const summary = semanticConflictSummary('报表', key, values, (row) => reportSemanticSignatureLabel(row));
|
|
2029
2094
|
for (const report of values) appendConflict(reportConflicts, text(report.resource_key), summary);
|
|
2030
2095
|
}
|
|
@@ -2036,7 +2101,7 @@ function detectSemanticConflicts() {
|
|
|
2036
2101
|
const summary = semanticConflictSummary('元数据', key, values, (row) => metadataSemanticSignatureLabel(row));
|
|
2037
2102
|
for (const item of values) appendConflict(metadataConflicts, text(item.evidence_id), summary);
|
|
2038
2103
|
}
|
|
2039
|
-
return { reportConflicts, metadataConflicts };
|
|
2104
|
+
return { reportConflicts, metadataConflicts, executionVariants };
|
|
2040
2105
|
}
|
|
2041
2106
|
|
|
2042
2107
|
function groupBySemanticKey(values, keyFn) {
|
|
@@ -2105,25 +2170,7 @@ function normalizeSemanticIntent(value) {
|
|
|
2105
2170
|
}
|
|
2106
2171
|
|
|
2107
2172
|
function reportSemanticSignature(report) {
|
|
2108
|
-
|
|
2109
|
-
return JSON.stringify({ kind: 'SQL', sql: normalizeSql(text(object(report.calculation_context).sql)) });
|
|
2110
|
-
}
|
|
2111
|
-
return JSON.stringify({
|
|
2112
|
-
kind: text(report.definition_kind) || text(report.report_model),
|
|
2113
|
-
metrics: array(report.metric_occurrences).map(metricSemanticSignature).sort(),
|
|
2114
|
-
filters: filterSummaryLines(object(report.calculation_context).filters).sort(),
|
|
2115
|
-
groups: array(object(report.calculation_context).group_by).map(groupLabel).sort(),
|
|
2116
|
-
});
|
|
2117
|
-
}
|
|
2118
|
-
|
|
2119
|
-
function metricSemanticSignature(item) {
|
|
2120
|
-
return JSON.stringify({
|
|
2121
|
-
event_name: text(item.event_name),
|
|
2122
|
-
aggregation_code: text(item.aggregation_code),
|
|
2123
|
-
measure_key: text(item.measure_key),
|
|
2124
|
-
formula_expression: text(item.formula_expression),
|
|
2125
|
-
filters: filterSummaryLines(item.filters ?? item.custom_filters).sort(),
|
|
2126
|
-
});
|
|
2173
|
+
return reportSignature(report);
|
|
2127
2174
|
}
|
|
2128
2175
|
|
|
2129
2176
|
function reportSemanticSignatureLabel(report) {
|
|
@@ -2146,6 +2193,7 @@ function metadataSemanticSignature(item) {
|
|
|
2146
2193
|
select_type: text(definition.select_type),
|
|
2147
2194
|
source_type: text(definition.source_type),
|
|
2148
2195
|
tag_type: text(definition.tag_type || definition.tag_sub_type),
|
|
2196
|
+
rules: semanticValue(Object.fromEntries(['rule', 'rules', 'expression', 'dependent_events', 'dependent_properties'].filter(key => definition[key] != null).map(key => [key, definition[key]]))),
|
|
2149
2197
|
});
|
|
2150
2198
|
}
|
|
2151
2199
|
|
|
@@ -2173,7 +2221,10 @@ function collectFilterValues(value) {
|
|
|
2173
2221
|
function renderOccurrence(item, index) {
|
|
2174
2222
|
const expression = formulaExpression(item);
|
|
2175
2223
|
const formulaSummary = formulaDefinitionSummary(item);
|
|
2176
|
-
const filterSummary =
|
|
2224
|
+
const filterSummary = [
|
|
2225
|
+
...filterSummaryLines(item.filters, item.relation),
|
|
2226
|
+
...array(item.custom_filters).flatMap(filter => filterSummaryLines(filter).map(line => `Operand ${filter.index}: ${line}`)),
|
|
2227
|
+
];
|
|
2177
2228
|
const event = metricEventLabel(item) || 'unknown';
|
|
2178
2229
|
return `### ${index + 1}. ${heading(occurrenceLabel(item) || `Metric ${index + 1}`)}
|
|
2179
2230
|
|
|
@@ -2219,15 +2270,17 @@ function formulaDefinition(item) {
|
|
|
2219
2270
|
}
|
|
2220
2271
|
|
|
2221
2272
|
function formulaDependencies(item) {
|
|
2222
|
-
return array(formulaDefinition(item).formulationDeps).map((dependency) => {
|
|
2273
|
+
return array(formulaDefinition(item).formulationDeps).map((dependency, index) => {
|
|
2223
2274
|
const event = object(object(dependency).event);
|
|
2224
2275
|
const quota = object(object(dependency).quota);
|
|
2225
2276
|
const eventName = text(event.eventDesc || event.eventName) || '未知事件';
|
|
2226
2277
|
const aggregation = text(quota.quotaDesc || quota.quotaName) || '资产包未暴露聚合方式';
|
|
2227
|
-
const filters =
|
|
2228
|
-
|
|
2229
|
-
.filter(
|
|
2230
|
-
|
|
2278
|
+
const filters = [
|
|
2279
|
+
...filterSummaryLines(event.filter),
|
|
2280
|
+
...array(item.custom_filters).filter(filter => Number(filter.index) === index).flatMap(filter => filterSummaryLines(filter)),
|
|
2281
|
+
];
|
|
2282
|
+
const time = object(dependency).date ?? array(item.quota_time_ranges)[index];
|
|
2283
|
+
return `“${eventName}”的“${aggregation}”${filters.length ? `(过滤:${filters.join('、')})` : ''}${time != null ? ` [operand ${index}, time=${JSON.stringify(time)}]` : ''}`;
|
|
2231
2284
|
});
|
|
2232
2285
|
}
|
|
2233
2286
|
|
|
@@ -2249,6 +2302,10 @@ function formulaDefinitionSummary(item) {
|
|
|
2249
2302
|
}
|
|
2250
2303
|
|
|
2251
2304
|
function aggregationLabel(item) {
|
|
2305
|
+
const entity = array(item.entity_grain).map(row => object(row.taIdMeasure ?? row.ta_id_measure ?? row)).find(row => row.custom === true || text(row.entityType ?? row.entity_type).toUpperCase() === 'CUSTOM');
|
|
2306
|
+
if (text(item.aggregation_code) === 'A101' && entity) {
|
|
2307
|
+
return `Distinct entities (${text(entity.columnName ?? entity.column_name) || 'verify entity key'})`;
|
|
2308
|
+
}
|
|
2252
2309
|
const name = text(item.aggregation_name);
|
|
2253
2310
|
const code = text(item.aggregation_code);
|
|
2254
2311
|
const codeLabel = aggregationCodeLabel(code);
|
|
@@ -2312,31 +2369,19 @@ function isAggregationCode(value) {
|
|
|
2312
2369
|
return /^A\d{3}(?:_\d+)?$/.test(text(value));
|
|
2313
2370
|
}
|
|
2314
2371
|
|
|
2315
|
-
function filterSummaryLines(value) {
|
|
2316
|
-
|
|
2317
|
-
return [
|
|
2318
|
-
}
|
|
2319
|
-
|
|
2320
|
-
function collectFilterConditions(value) {
|
|
2321
|
-
if (Array.isArray(value)) return value.flatMap(collectFilterConditions);
|
|
2322
|
-
if (!value || typeof value !== 'object') return [];
|
|
2323
|
-
const source = object(value);
|
|
2324
|
-
const nested = Object.entries(source)
|
|
2325
|
-
.filter(([key]) => !['ftv', 'columnDesc', 'columnName', 'quotDesc', 'prop', 'calcuSymbol', 'calcu_symbol', 'relation', 'junctionKind', 'junction_kind'].includes(key))
|
|
2326
|
-
.flatMap(([, item]) => collectFilterConditions(item));
|
|
2327
|
-
const hasConditionIdentity = ['columnDesc', 'columnName', 'quotDesc', 'prop', 'propertyName', 'tagName'].some((key) => text(source[key]))
|
|
2328
|
-
|| array(source.ftv).some((item) => text(item));
|
|
2329
|
-
return hasConditionIdentity ? [source, ...nested] : nested;
|
|
2372
|
+
function filterSummaryLines(value, relation) {
|
|
2373
|
+
if (value == null || (Array.isArray(value) && !value.length) || (!Array.isArray(value) && !Object.keys(object(value)).length)) return [];
|
|
2374
|
+
return [renderConditionTree(conditionTree(value, relation), filterConditionSentence)];
|
|
2330
2375
|
}
|
|
2331
2376
|
|
|
2332
2377
|
function filterConditionSentence(condition) {
|
|
2333
2378
|
const field = text(condition.columnDesc || condition.quotDesc || condition.columnName || condition.prop || condition.propertyName || condition.tagName) || '未命名字段';
|
|
2334
|
-
const values = array(condition.ftv).map(
|
|
2379
|
+
const values = array(condition.ftv).map(value => typeof value === 'string' ? `“${value}”` : JSON.stringify(value));
|
|
2335
2380
|
const code = text(condition.calcuSymbol ?? condition.calcu_symbol);
|
|
2336
2381
|
const operator = filterOperatorLabel(code);
|
|
2337
2382
|
const relation = text(condition.relation || condition.junctionKind || condition.junction_kind);
|
|
2338
2383
|
const needsValue = filterOperatorNeedsValue(code);
|
|
2339
|
-
const valueText = values.length ? values.
|
|
2384
|
+
const valueText = values.length ? values.join('、') : needsValue ? '未给出固定值' : '';
|
|
2340
2385
|
const valuePart = valueText ? ` ${valueText}` : '';
|
|
2341
2386
|
const relationText = relation ? `(关系:${relation})` : '';
|
|
2342
2387
|
return `${field} ${operator}${valuePart}${relationText}`;
|
|
@@ -2381,6 +2426,7 @@ function filterOperatorNeedsValue(value) {
|
|
|
2381
2426
|
}
|
|
2382
2427
|
|
|
2383
2428
|
function occurrenceLabel(item) { return readableMetricText(item.display_name || item.event_title || item.event_name); }
|
|
2429
|
+
function groupLabel(group) { return text(object(group).columnDesc || object(group).columnName || group); }
|
|
2384
2430
|
|
|
2385
2431
|
function metricEventLabel(item) {
|
|
2386
2432
|
const title = readableMetricText(item.event_title);
|
|
@@ -2411,7 +2457,6 @@ function readableMetricText(value) {
|
|
|
2411
2457
|
.replace(/^、|、$/g, '')
|
|
2412
2458
|
.trim();
|
|
2413
2459
|
}
|
|
2414
|
-
function groupLabel(group) { return text(object(group).columnDesc || object(group).columnName || group); }
|
|
2415
2460
|
|
|
2416
2461
|
function renderSemanticConflictList() {
|
|
2417
2462
|
const reportLines = [...reportConflicts.entries()]
|
|
@@ -7,9 +7,9 @@ description: "Manage AE Agent and Agent team automations with ae-cli: create and
|
|
|
7
7
|
|
|
8
8
|
See [command risks](references/command_index.md). Read [the command contract](references/cli.md) before writing or executing an automation. Use `ae-cli automation --help` and leaf help to discover flags.
|
|
9
9
|
|
|
10
|
-
1. Resolve the host and space. A local user signs in with `ae-cli auth login`; a sandbox uses
|
|
10
|
+
1. Resolve the host and space. A local user signs in with `ae-cli auth login`; a sandbox uses the current conversation user's injected CLI token, never a shared sandbox's user binding. Pass `--agent-space-id` when the user identifies a space. Without an explicit space, the CLI inherits the current AgentSpace from its runtime context, working directory or verified current conversation; only a local invocation without space context uses the personal default. Show the resolved space when it matters.
|
|
11
11
|
2. Discover the automation with `automation list` and inspect `automation get`. Use the returned IDs; names are not unique.
|
|
12
|
-
3. For creation, select an Agent or team through the corresponding discovery commands. Match configuration to the target type. Preview writes with `--dry-run` when useful; the CLI shows a redacted request and asks the server to validate authentication, space, permissions, target state, fields, and idempotency without applying the operation.
|
|
12
|
+
3. For creation, select an Agent or team through the corresponding discovery commands. Match configuration to the target type. Use `--weekdays` or `--days-of-month` JSON arrays for multiple days with one time; skip dates absent in a month. For the actual last day of every month use `--days-of-month '["L"]'`, or combine it with dates, e.g. `'[1,15,"L"]'`; overlapping selections execute only once. Both Agent and Agent Team accept result notifications. Confirm the complete configuration before creation; save only the recurring task instructions as `--message`, never the automation-creation guide. Preview writes with `--dry-run` when useful; the CLI shows a redacted request and asks the server to validate authentication, space, permissions, target state, fields, and idempotency without applying the operation.
|
|
13
13
|
4. Execute the requested operation. Deleting an automation or stopping an execution requires confirmation; use `--yes` only when the user already authorized that action. In a non-interactive shell, these operations fail immediately unless global `--yes` is present.
|
|
14
14
|
5. Read the returned run ID, then `automation runs result`. Report acceptance separately from completion. Use `--wait` only when the user needs the final outcome now.
|
|
15
15
|
|
|
@@ -17,7 +17,7 @@ Exit condition: equivalent gateway capabilities preserve permissions, idempotenc
|
|
|
17
17
|
- `ae-cli automation enable|disable --id ID`
|
|
18
18
|
- `ae-cli automation delete --id ID --yes`
|
|
19
19
|
|
|
20
|
-
Scheduled creation needs `--schedule-kind` or `--cron-expression`. Frequencies: hourly with `--minute 0..59`; daily with `--time HH:mm`; weekly also `--
|
|
20
|
+
Scheduled creation needs `--schedule-kind` or `--cron-expression`. Frequencies: hourly with `--minute 0..59`; daily with `--time HH:mm`; weekly also `--weekdays '[1,3,5]'` (0=Sunday through 6=Saturday); monthly also `--days-of-month '[1,15,31]'` (integers 1–31 and `"L"` for the actual last day of the month). Existing single-day flags `--weekday 0..6` and `--day-of-month 1..31` remain supported; do not combine a single-day flag with its plural array. Arrays must be non-empty; duplicates are removed and dates sorted. Dates absent in the current month are skipped, never shifted into the next month. `"L"` follows the actual month end in the server timezone (including leap years). It can be selected alone or with numeric dates; overlapping selections at the same time run only once. Any minute 00–59 is valid in `--time`. Times use the server timezone. Manual creation does not require a schedule. `--trigger-immediately` executes after creation; `--wait` then waits for that execution. Creation and immediate execution have separate results: do not create another automation if only its execution failed.
|
|
21
21
|
|
|
22
22
|
On update, omitted fields retain their values. `--model null`, `--scope null`, destination IDs set to `null`, and `--save-to-knowledge-base-id null` clear nullable fields. `--channels '[]'` clears notifications; `--clear-message` clears instructions; `--reuse-conversation false` disables reuse. An automation's target is immutable on update. Team fields are rejected for Agent targets and vice versa.
|
|
23
23
|
|
|
@@ -51,3 +51,20 @@ Without `--wait`, success means accepted. With `--wait`, output includes complet
|
|
|
51
51
|
## Compatibility
|
|
52
52
|
|
|
53
53
|
The retired Agent-domain automation commands are unavailable. Use the `automation` domain for the complete lifecycle, including teams.
|
|
54
|
+
|
|
55
|
+
## Shared result notifications and identity
|
|
56
|
+
|
|
57
|
+
Agent and Agent Team both accept `--channels '["feishu","slack"]'`, `--feishu-chat-id`, `--lark-chat-id` and `--slack-channel-id`. Supported channels: `feishu`, `lark`, `slack`, `discord`, `dingtalk`, `wecom`, `mattermost`, `google_chat`, `zoom_chat`, `whatsapp`. Availability and destination permissions are validated by the server. Omitted update fields keep the existing values; `--channels '[]'` disables external delivery, and a receiver value of `null` clears that override. Results remain available in the application.
|
|
58
|
+
|
|
59
|
+
Automation requests use the current user's `cli-token`, including in a shared sandbox. They do not infer an owner from `SandboxUser` or fall back to a different member. `--agent-space-id` takes precedence over `TE_AGENT_CURRENT_AGENT_SPACE_ID` or the current AgentSpace working directory. When the shell has changed directories, `TE_AGENT_CONVERSATION_ID` resolves the space through the existing authenticated workspace target API; a resolution failure never falls back to the personal space. Membership and write permissions are enforced by the server.
|
|
60
|
+
|
|
61
|
+
For conversational creation, keep the supplied prompt editable until the user sends it. Resolve missing or ambiguous targets, summarize the configuration and obtain confirmation before creating. Agent runs inherit its model unless explicitly overridden; Agent Team retains its own configuration. Omit `--trigger-immediately` unless the user requests an immediate run.
|
|
62
|
+
|
|
63
|
+
## Multi-date examples
|
|
64
|
+
|
|
65
|
+
```bash
|
|
66
|
+
ae-cli automation create --target-type agent --agent-id ID --name "Weekly report" --message "Summarize the latest metrics" --schedule-kind weekly --weekdays '[1,3,5]' --time 09:17
|
|
67
|
+
ae-cli automation update --id ID --target-type team --schedule-kind monthly --days-of-month '[1,15,"L"]' --time 18:59
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
Each schedule has one execution time. Select `[1,2,3,4,5]` for Monday–Friday or `[0,6]` for weekends; these are weekdays, not a regional holiday calendar. Five-field cron accepts equivalent comma-separated weekday or month-day lists, such as `17 9 * * 1,3,5` and `59 18 1,15,L * *`. `L` is supported only in the month-day field. Ranges, steps and multiple execution times are not supported. `get` returns the saved `cronExpression` and notification configuration. On a scheduled active task, `nextRunAt`, `scheduleTimeZone` and `serverUtcOffset` describe the server's next execution.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: ae-capability
|
|
3
|
-
version: 1.3.
|
|
3
|
+
version: 1.3.15
|
|
4
4
|
description: "AE capability gateway discovery and generic invocation with ae-cli. Use when the user needs to list or search available capabilities, inspect an unknown capability schema/risk/auth contract, optionally validate complex input or dry-run before execute, or invoke a long-tail capability that has no curated ae-cli command. Always discover and inspect before composing input; never guess capability IDs or input fields. Prefer on-demand validate OR dry-run — do not stack both by default."
|
|
5
5
|
---
|
|
6
6
|
|
|
@@ -86,9 +86,13 @@ After running a gateway command, if you are asked whether intent telemetry was s
|
|
|
86
86
|
|
|
87
87
|
Fallbacks, in priority order: `.ta/cli-agent-context.json` / `.ae/cli-agent-context.json` > `--agent-context` for isolated no-write environments or local debugging > legacy `TA_CLI_*` variables > runtime environment auto-detection.
|
|
88
88
|
|
|
89
|
-
### Analysis
|
|
89
|
+
### Analysis outcome assessment and supplementary feedback
|
|
90
90
|
|
|
91
|
-
|
|
91
|
+
Scenario collection is project opt-in. Before preparing or uploading a scenario sample, read `ae-cli project info get --project-id <source_project_id>` for the current host, authenticated account, and source analysis project. Only `data.analysis_scenario_collection_enabled === true` permits collection. Recheck for each new scenario and before appending feedback or a revision; do not reuse a previous grant or the company configuration's 24-hour cache. A missing field (including older services), false, unreadable configuration, or unknown project means skip collection and answer normally. Never enable the project switch, choose a different project, or invent a project ID to make collection succeed. The curated collect/append commands also check immediately before submission, and the service is authoritative if the switch changes. This controls only scenario samples; existing invocation and intent logs stay unchanged. `accepted=false, skipped=true` is a normal skip, not a failed analysis or successful upload. Do not retry, backfill, or fall back to `analysis feedback submit` to bypass this switch.
|
|
92
|
+
|
|
93
|
+
Collect one sample for the whole user scenario, not one per CLI call. After forming the final answer, use `ae-cli analysis scenario collect` as specified in the `ae-cli analysis scenario collect --help` and `append --help` contracts, then deliver that exact answer text. Capture the original question/confirmed clarifications, actual final answer and the ordered query path (existing capability/call IDs and optional short purpose; prefer only meta.invocation_id, never confuse a business request_id with gateway client_request_id), never query result rows or error payloads; do not score correctness or generate a separate self-evaluation. The question_id is the original scenario's first user turn ID, stable across clarification, retries, multiple queries and corrections. Store/reuse the IDs returned by collection in conversation context. On later user objections, scope corrections or reported omissions targeting that answer, append the user's original text with `ae-cli analysis scenario append --entry-type user_feedback` and the original question_id/target_answer_id; do not decide whether the user is right. An ambiguous request for an explanation may be preserved as user text when clearly linked; do not invent a link. A normal follow-up or changed requirement is not automatically negative feedback. After a substantive follow-up answer to the same scenario (recheck, explanation, correction or scope clarification), append answer_revision with that exact answer/query path and a new answer_id, even when the conclusion stays the same; keep earlier answers. If the triggering user request was not already captured as user_feedback, include its exact text using --user-message on answer_revision so later scope changes are not lost. Do not duplicate feedback text in user_message. Acknowledgments such as thanks alone need no entry. A genuinely separate question gets a new question_id. Do not create another initial sample for an existing scenario. A separate evaluation task reads these samples later and can join existing call telemetry for parameters/status/timing. Path-only evidence cannot prove numeric answer correctness; unresolved correctness stays unknown. Only attempted analysis data queries qualify; metadata, discovery, help and asset writes alone do not. No routine satisfaction question. Collection failure must not block the answer or subsequent business work, trigger login or automatic retries, or be claimed as successful ingestion. For external Agents this is a Skill collection step, not a guaranteed post-answer hook; it has one collection tool call but no additional evaluation task on the answering path.
|
|
94
|
+
|
|
95
|
+
User feedback is evidence, not a correctness label. Preserve exact user words and their target answer even if you disagree. Do not routinely ask for a satisfaction rating. The existing `analysis feedback submit` contract remains available for explicit standalone user ratings without a collected scenario; when a scenario exists, use its append command instead of submitting duplicate feedback. No explicit feedback means no submission or new feedback event for `cli_analysis_feedback`. Do not upload the analysis answer through that legacy feedback command. All collection excludes credentials, full transcripts and private reasoning.
|
|
92
96
|
|
|
93
97
|
### On-demand pre-check (pick one)
|
|
94
98
|
|
|
@@ -0,0 +1,83 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: ae-evaluation
|
|
3
|
+
version: 2.4.5
|
|
4
|
+
description: "Operate the Agent Evaluation (agent-test) platform through ae-cli: manage evaluation assets, configure and run Agent or preset-output evaluations, and analyze results. Use for platform evaluation tasks and their model, credential, integration, or schedule setup."
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
# ae-evaluation
|
|
8
|
+
|
|
9
|
+
Use `ae-cli evaluation +<command> [options]` for existing operations and the resource commands below for new capabilities. All 121 commands target the Agent Evaluation platform.
|
|
10
|
+
AE Agent resource management belongs to `ae-agent`; this skill manages evaluation assets and agents under test.
|
|
11
|
+
|
|
12
|
+
## Read only the relevant reference
|
|
13
|
+
|
|
14
|
+
Before executing a command, read the reference for the operation. Do not load all references.
|
|
15
|
+
References contain command inputs, JSON contracts, and task-specific workflows; use command `--help` for additional flags.
|
|
16
|
+
|
|
17
|
+
| Task | Reference |
|
|
18
|
+
|---|---|
|
|
19
|
+
| Create/import datasets, manage fields and cases, publish versions | [datasets.md](references/datasets.md) |
|
|
20
|
+
| Configure LLM judges/templates, publish evaluator versions, calibrate or optimize prompts | [evaluators.md](references/evaluators.md) |
|
|
21
|
+
| Create/configure/start/retry/cancel runs, inspect results/traces, export results | [runs.md](references/runs.md) |
|
|
22
|
+
| Configure/test models and agents, manage credentials | [connections.md](references/connections.md) |
|
|
23
|
+
| Change resource visibility or administer project members | [access-and-members.md](references/access-and-members.md) |
|
|
24
|
+
| Download imported/exported files to disk | [files.md](references/files.md) |
|
|
25
|
+
| Create/update/delete recurring evaluation schedules | [schedules.md](references/schedules.md) |
|
|
26
|
+
| Connect/import AI Monitor data, configure TE reporting, discover TE Claude catalog | [integrations.md](references/integrations.md) |
|
|
27
|
+
|
|
28
|
+
## Endpoint and authentication
|
|
29
|
+
|
|
30
|
+
- In both personal terminals and AE sandboxes, evaluation commands default to the current AE Host and call `<current-host>/agenteval/api/v1/...` (an optional trailing `/agent` is removed from the host). Use `--host` to select another AE Host. Evaluation requests use the HTTP or HTTPS scheme configured for that AE Host, consistent with other AE commands. Legacy `--endpoint` and `AE_EVAL_BASE_URL` values may only match the current AE Host; other values fail before credentials are sent. Remove these settings and use `--host` to select a host.
|
|
31
|
+
- Session exchange and evaluation requests reject redirects; session exchange has a 30-second timeout.
|
|
32
|
+
- Path resource IDs must contain only letters, digits, hyphens, or underscores; separators and URL escape syntax are rejected locally.
|
|
33
|
+
- Use the online platform. Do not suggest localhost/loopback endpoints or obtain tokens from a local `agent-test/.env`.
|
|
34
|
+
- Authentication uses `AE_EVAL_ACCESS_TOKEN` when explicitly set; it is sent only as `Authorization: Bearer` and must never appear in command arguments or output.
|
|
35
|
+
- In an AE sandbox, when `AE_EVAL_ACCESS_TOKEN` is unset, ae-cli automatically exchanges the sandbox-provisioned `cli-token` through te-claude for a short-lived evaluation session. The CLI token is never sent to Agent Evaluation. If the token can access multiple TE projects, set `AE_EVAL_TE_PROJECT_ID` to the authorized TE project id.
|
|
36
|
+
- Outside a sandbox, obtain a session token through the platform web console (`sessionStorage.getItem('agent.evaluation.session-token')`) or `POST /agenteval/api/v1/auth/session`. An unset token produces `EVAL_ACCESS_TOKEN_MISSING`.
|
|
37
|
+
- Successful commands return JSON by default; use the actual error envelope to diagnose failures.
|
|
38
|
+
|
|
39
|
+
## IDs, versions, and mutations
|
|
40
|
+
|
|
41
|
+
- Discover real IDs with list/get commands. UUIDs are version-7 strings; never guess IDs, JSON payload shapes, frequency fields, or `lock_version` values.
|
|
42
|
+
- Published dataset, evaluator, and evaluation-task versions are immutable and separate from drafts. Runs and schedules pin versions; existing published versions cannot be overwritten. Publishing unchanged assets returns `ASSET_VERSION_NO_CHANGES`.
|
|
43
|
+
- Read operations can run once inputs are known. Writes require user intent; high-risk operations retain confirmation unless the user authorizes bypassing it with `--yes`.
|
|
44
|
+
- Credential enabling/disabling, run cancellations, deletes, ownership transfers, template disabling, access-scope changes, and member admission/permission changes are high-risk writes. Check command help for risk and role requirements; references identify ADMIN-only operations.
|
|
45
|
+
- Before each mutation requiring `--lock-version`, read the current resource. Every successful mutation increments its version; re-read before the next mutation.
|
|
46
|
+
- Use the corresponding get command for runs, datasets, evaluators, models, agents, credentials, dataset cases, TE report config, and AI Monitor connection. Schedules use `+list_schedules`; calibration cases use `+list_calibration_cases`.
|
|
47
|
+
- Field mutations use the dataset version; case mutations use the case version. First creation of TE report config or AI Monitor connection omits `--lock-version`; updates require the current value, including 0.
|
|
48
|
+
- Async dataset imports, evaluation-set imports, and result exports require a fresh unique `--idempotency-key` per attempt. Poll the relevant status until completion; resource creation alone does not establish readiness.
|
|
49
|
+
- Prefer `--dry-run` before broad/destructive writes whose targets have not been validated. It previews the request without executing it.
|
|
50
|
+
- Evaluation errors redact the submitted credential values and session tokens; server-provided error messages and hints are not persisted in CLI logs. Stable business error codes are preserved unchanged; non-sensitive diagnostics remain available in the JSON error envelope.
|
|
51
|
+
- Case data is redacted from command logs for both single-case and batch writes. Sensitive JSON parse failures report the flag and syntax guidance without echoing the input.
|
|
52
|
+
- Credential secrets are write-only. Never echo secrets, session claims, or sensitive trace evidence; use redacted dry-run output.
|
|
53
|
+
|
|
54
|
+
## Evaluation lifecycle
|
|
55
|
+
|
|
56
|
+
1. Select a published dataset version with `+list_dataset_versions`.
|
|
57
|
+
2. Choose `LIVE_AGENT` (default) to call an Agent/model, or `PRESET_RESULT` to score outputs stored in the dataset. PRESET_RESULT must omit agent/model config IDs. `IMPORTED_OUTPUT` is legacy data only; never recommend it.
|
|
58
|
+
3. Create a draft run, bind at least one ACTIVE evaluator (prefer pinned published versions), and configure complete field mapping plus result filter fields. An explicit empty result filter is required when no fields are selected.
|
|
59
|
+
4. Prefer `+validate_run` before starting. It runs the start preflight without a body or lock version. Read the latest run, then `+start_run` with its current lock version.
|
|
60
|
+
5. Inspect overview/distribution, drill into cases, and use the run reference for retry, trace, and export semantics.
|
|
61
|
+
|
|
62
|
+
Read [runs.md](references/runs.md) for the executable sequence and mapping examples. Setup of datasets, evaluators, or connections needs only the matching reference above.
|
|
63
|
+
|
|
64
|
+
## Transport transition
|
|
65
|
+
|
|
66
|
+
The existing platform contract exposes REST operations rather than evaluation
|
|
67
|
+
capabilities registered in this repository. New commands retain that transport
|
|
68
|
+
for typed input, permission confirmation, and file handling. They are Transitional;
|
|
69
|
+
see the matching references for migration records. Check deployment Gateway
|
|
70
|
+
coverage before extending these commands; prefer existing equivalent capabilities.
|
|
71
|
+
|
|
72
|
+
Transition status: transitional
|
|
73
|
+
Owning module: Agent Evaluation (agent-test)
|
|
74
|
+
Current transport: authenticated REST /agenteval/api/v1
|
|
75
|
+
Gateway target: TBD (evaluation capability registration)
|
|
76
|
+
Review after: 2027-01-21
|
|
77
|
+
Exit condition: once equivalent evaluation capabilities are available, migrate
|
|
78
|
+
selected typed/file commands to L2 or replace simple wrappers with L3; retain
|
|
79
|
+
existing + commands only where compatibility or orchestration justifies them.
|
|
80
|
+
|
|
81
|
+
This record also applies to the locale and process-evidence updates in the
|
|
82
|
+
existing evaluator/run references. It is a migration plan, not a claim that a
|
|
83
|
+
particular deployed Gateway has been inspected.
|
|
@@ -0,0 +1,55 @@
|
|
|
1
|
+
# Resource visibility and project members
|
|
2
|
+
|
|
3
|
+
Read [SKILL.md](../SKILL.md) for authentication and mutation rules. These commands
|
|
4
|
+
require administrator privileges (`root` or `agent_admin`). Permission changes
|
|
5
|
+
are high-risk writes; retain confirmation unless the user authorizes `--yes`.
|
|
6
|
+
|
|
7
|
+
## Resource visibility
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
ae-cli evaluation dataset access-scope --id <dataset-uuid> --access-scope COMPANY --lock-version <n>
|
|
11
|
+
ae-cli evaluation evaluator access-scope --id <evaluator-uuid> --access-scope COMPANY --lock-version <n>
|
|
12
|
+
ae-cli evaluation model-config access-scope --id <model-config-uuid> --access-scope PERSONAL --lock-version <n>
|
|
13
|
+
ae-cli evaluation agent access-scope --id <agent-uuid> --access-scope PERSONAL --lock-version <n>
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
Read the current resource with its `+get_*` command before changing it. Each
|
|
17
|
+
successful mutation increments `lock_version`. PERSONAL and COMPANY are mutable;
|
|
18
|
+
SYSTEM resources are read-only. COMPANY sharing stays within the authenticated
|
|
19
|
+
project/tenant. Shared assets cannot depend on personal configurations or
|
|
20
|
+
credentials (`ACCESS_SCOPE_DEPENDENCY_PRIVATE`); inspect dependencies rather
|
|
21
|
+
than silently broadening their visibility. Ordinary model/Agent creation produces
|
|
22
|
+
personal resources and is not restricted to administrators by its create route.
|
|
23
|
+
Credential operations and administrative routes remain administrator-only.
|
|
24
|
+
|
|
25
|
+
## Project membership
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
ae-cli evaluation +list_project_members
|
|
29
|
+
ae-cli evaluation project-member list-candidates
|
|
30
|
+
ae-cli evaluation project-member batch-add --members-json '[{"external_id":"<te-id>","display_name":"<name>"}]'
|
|
31
|
+
ae-cli evaluation project-member change-role --id <user-uuid> --role agent_admin
|
|
32
|
+
ae-cli evaluation project-member set-status --id <user-uuid> --status DISABLED
|
|
33
|
+
ae-cli evaluation project-member remove --id <user-uuid>
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
Discover candidates rather than guessing IDs. Admission takes TE `external_id`;
|
|
37
|
+
role/status/removal take the project member's `user_id` UUID from
|
|
38
|
+
`+list_project_members`. Batch admission returns added/skipped counts and skips
|
|
39
|
+
existing members. These routes do not accept `lock_version`.
|
|
40
|
+
|
|
41
|
+
Roles are `root`, `agent_admin`, and `member`; change-role accepts only
|
|
42
|
+
`agent_admin|member`. Root members are protected. `set-status` accepts
|
|
43
|
+
`ACTIVE|DISABLED` and affects login across all projects in the tenant. `remove`
|
|
44
|
+
revokes only the current project's membership, including subsequent TE login
|
|
45
|
+
and refresh, until explicitly re-added. Never describe disabling a login as
|
|
46
|
+
merely removing it from one project.
|
|
47
|
+
|
|
48
|
+
Transition status: transitional
|
|
49
|
+
Owning module: Agent Evaluation accessscope and membership
|
|
50
|
+
Current transport: authenticated REST /agenteval/api/v1
|
|
51
|
+
Gateway target: TBD (evaluation access-scope and member administration)
|
|
52
|
+
Review after: 2027-01-21
|
|
53
|
+
Exit condition: when equivalent capabilities exist, migrate typed permission
|
|
54
|
+
commands to L2 where confirmation and typed validation add value; use L3 for
|
|
55
|
+
simple discovery operations and remove duplicate Transitional routes.
|