@thinkingai/ae-cli 6.1.27 → 6.1.29
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +52 -6
- package/README.zh.md +52 -6
- package/bin/ae-cli.js +5 -0
- package/bin/runtime-capabilities.json +5 -0
- package/cli-installation-guide.md +24 -3
- package/dist/{auth-WA7YGGRT.js → auth-OBQ5KVQM.js} +175 -45
- package/dist/{automation-VAAQBM7Y.js → automation-NAR5DDGL.js} +94 -38
- package/dist/{capability-GAAYJ3IQ.js → capability-LFSVNI5B.js} +57 -28
- package/dist/chunk-2UW3Y463.js +2289 -0
- package/dist/{chunk-VZZNLV3I.js → chunk-3SDHJOCR.js} +68 -18
- package/dist/chunk-6HAUVWOQ.js +132 -0
- package/dist/{chunk-5XJYLEVC.js → chunk-72NJKS7O.js} +4 -4
- package/dist/{chunk-Q5US7JC5.js → chunk-7JAVT5TK.js} +3 -3
- package/dist/{chunk-7KU2WNQE.js → chunk-CVCTFLSO.js} +37 -15
- package/dist/{chunk-GHXZK54S.js → chunk-H3K5UGYC.js} +41 -13
- package/dist/{chunk-OP44TT4H.js → chunk-IN7KKL6K.js} +4 -2
- package/dist/{chunk-S6M42KND.js → chunk-JEUXDY3B.js} +1 -1
- package/dist/{chunk-ZL7SGNIA.js → chunk-JNVVJ273.js} +58 -46
- package/dist/chunk-N6XBLRB7.js +105 -0
- package/dist/{chunk-AP7YP2NX.js → chunk-OAWPT7VP.js} +2 -2
- package/dist/{chunk-AQJPNWZF.js → chunk-PWXDWMCW.js} +7 -7
- package/dist/{chunk-CT7QDJS6.js → chunk-QVRIDN42.js} +2 -2
- package/dist/chunk-QZLRHGVH.js +222 -0
- package/dist/{chunk-RPBOAQQQ.js → chunk-QZUVLFNR.js} +1 -1
- package/dist/{chunk-3YVEAZUN.js → chunk-RJHGDWSC.js} +32 -21
- package/dist/{chunk-MWIA43UJ.js → chunk-VCZE3IWO.js} +10 -8
- package/dist/chunk-YVG6QD3V.js +233 -0
- package/dist/{chunk-WNGFSIOB.js → chunk-Z3Q2AEIN.js} +237 -86
- package/dist/{community-report-client-PEERQCX2.js → community-report-client-TJVCLZ3K.js} +1 -1
- package/dist/compat-check-CII6B74E.js +32 -0
- package/dist/{config-JOD5DOPE.js → config-YZ6O2UQ2.js} +40 -7
- package/dist/{context-NGVFPCBK.js → context-F6FCGTF3.js} +4 -4
- package/dist/{data-integration-BCLQFTGU.js → data-integration-6UAZO4GG.js} +207 -24
- package/dist/evaluation-4LO2O4AF.js +3714 -0
- package/dist/index.js +352 -81
- package/dist/{local-data-upload-client-TWMHFLUA.js → local-data-upload-client-KCMG6J62.js} +1 -1
- package/dist/{memory-CQKH2JAV.js → memory-HHPX3NF2.js} +6 -6
- package/dist/{metadata-NR3PHR23.js → metadata-TIFY24LW.js} +17 -9
- package/dist/{model-VX5M2CJ6.js → model-7UHGWYVR.js} +4 -4
- package/dist/{personal-semantic-preference-2RAI36P3.js → personal-semantic-preference-WKJEGPTM.js} +7 -6
- package/dist/{project-semantic-3BUGP3ZM.js → project-semantic-EIQVZNHW.js} +11 -10
- package/dist/{sync-RZIAV37C.js → sync-RV3OPCTI.js} +8 -7
- package/dist/{te-agent-AORRE2FM.js → te-agent-5ENVZJUH.js} +757 -94
- package/dist/{te-analysis-GP7DSQ7Q.js → te-analysis-USIK6ZZG.js} +584 -100
- package/dist/{te-community-42M7ICDO.js → te-community-WE2UU4XV.js} +722 -228
- package/dist/{te-dataops-BNKNZ4F2.js → te-dataops-XNMO7MBF.js} +9 -6
- package/dist/{te-engage-2CO6LRCX.js → te-engage-6C3HCFQR.js} +13 -12
- package/dist/{te-experiment-652CLXMU.js → te-experiment-LVTRI3QK.js} +55 -21
- package/dist/{te-kb-ULATOTQ3.js → te-kb-TUPGMVBY.js} +409 -85
- package/dist/{te-system-UGHVKBVV.js → te-system-YM657FKO.js} +6 -6
- package/dist/{te-team-YA6SNOJ3.js → te-team-G4VEJJNL.js} +64 -41
- package/dist/{things-DTCOQIQB.js → things-22KQWC23.js} +7 -7
- package/dist/{update-ESNYKUVR.js → update-XX4ILU7X.js} +11 -7
- package/dist/{workspace-JEOHKFPX.js → workspace-LYDAP647.js} +6 -6
- package/package.json +26 -11
- package/skills/ae-agent/SKILL.md +22 -3
- package/skills/ae-agent/references/command_index.md +8 -0
- package/skills/ae-agent/references/resource-package.md +57 -0
- package/skills/ae-analysis/SKILL.md +23 -10
- package/skills/ae-analysis/references/adhoc_export.md +1 -1
- package/skills/ae-analysis/references/agent_review_preflight.md +55 -10
- package/skills/ae-analysis/references/agent_review_priorities_comparisons.md +5 -3
- package/skills/ae-analysis/references/agent_review_submit_to_page.md +60 -24
- package/skills/ae-analysis/references/ai_models/event.md +58 -2
- package/skills/ae-analysis/references/ai_models/rank_list.md +1 -0
- package/skills/ae-analysis/references/ai_models/retention.md +2 -0
- package/skills/ae-analysis/references/ai_models/revenue.md +2 -0
- package/skills/ae-analysis/references/analysis_data_export.md +11 -1
- package/skills/ae-analysis/references/asset_batch_info_export.md +2 -2
- package/skills/ae-analysis/references/asset_batch_sql_export.md +2 -2
- package/skills/ae-analysis/references/asset_export.md +1 -1
- package/skills/ae-analysis/references/bi_panel_page_data_export.md +1 -1
- package/skills/ae-analysis/references/check_export.md +1 -1
- package/skills/ae-analysis/references/command_index.md +39 -26
- package/skills/ae-analysis/references/dashboard_report_data_export.md +2 -0
- package/skills/ae-analysis/references/dashboard_report_get.md +6 -2
- package/skills/ae-analysis/references/dashboard_report_update.md +24 -5
- package/skills/ae-analysis/references/feedback_submit.md +1 -1
- package/skills/ae-analysis/references/governance_recommendation_export.md +40 -18
- package/skills/ae-analysis/references/governance_recommendation_run_update.md +13 -0
- package/skills/ae-analysis/references/live_data_export.md +1 -1
- package/skills/ae-analysis/references/metric_create.md +29 -0
- package/skills/ae-analysis/references/metric_get.md +2 -0
- package/skills/ae-analysis/references/metric_update.md +2 -0
- package/skills/ae-analysis/references/operation_record_export.md +1 -1
- package/skills/ae-analysis/references/plan_change_log_export.md +1 -1
- package/skills/ae-analysis/references/plan_export.md +1 -1
- package/skills/ae-analysis/references/project_info_get.md +4 -0
- package/skills/ae-analysis/references/project_member_group_add_members.md +35 -0
- package/skills/ae-analysis/references/project_member_group_create.md +25 -0
- package/skills/ae-analysis/references/project_member_group_delete.md +34 -0
- package/skills/ae-analysis/references/project_member_group_list.md +29 -0
- package/skills/ae-analysis/references/project_member_group_members.md +28 -0
- package/skills/ae-analysis/references/project_member_group_remove_members.md +35 -0
- package/skills/ae-analysis/references/project_member_group_rename.md +26 -0
- package/skills/ae-analysis/references/project_semantic_knowledge_wiki.md +9 -1
- package/skills/ae-analysis/references/project_semantic_knowledge_wiki_plan_schema.md +6 -1
- package/skills/ae-analysis/references/report_data_export.md +1 -1
- package/skills/ae-analysis/references/scenario_append.md +31 -0
- package/skills/ae-analysis/references/scenario_begin.md +11 -0
- package/skills/ae-analysis/references/scenario_collect.md +43 -0
- package/skills/ae-analysis/references/scenario_evaluation.md +18 -0
- package/skills/ae-analysis/references/scenario_get.md +15 -0
- package/skills/ae-analysis/references/scenario_list.md +64 -0
- package/skills/ae-analysis/scripts/project-semantic-knowledge-wiki/build-project-semantic-wiki.mjs +73 -4
- package/skills/ae-analysis/scripts/project-semantic-knowledge-wiki/company-kb.mjs +14 -1
- package/skills/ae-analysis/scripts/project-semantic-knowledge-wiki/default-compile-schema.md +5 -1
- package/skills/ae-analysis/scripts/project-semantic-knowledge-wiki/execution-contract.mjs +173 -0
- package/skills/ae-analysis/scripts/project-semantic-knowledge-wiki/generate-build-ir.mjs +109 -64
- package/skills/ae-automation/SKILL.md +2 -2
- package/skills/ae-automation/references/cli.md +18 -1
- package/skills/ae-capability/SKILL.md +7 -3
- package/skills/ae-community/SKILL.md +77 -27
- package/skills/ae-community/references/community-analyzing-theme-comment.md +10 -8
- package/skills/ae-community/references/community-data-quality.md +68 -0
- package/skills/ae-community/references/community-query-evidence.md +85 -0
- package/skills/ae-community/references/community-scenario-capabilities.md +137 -0
- package/skills/ae-community/references/community-weekly-report.md +6 -3
- package/skills/ae-community/references/get_channel_info.md +7 -4
- package/skills/ae-community/references/get_comment_tag_analysis.md +3 -3
- package/skills/ae-community/references/get_comments_summary.md +4 -4
- package/skills/ae-community/references/get_corpus_tags.md +3 -3
- package/skills/ae-community/references/get_daily_summary.md +3 -3
- package/skills/ae-community/references/get_hot_topics.md +3 -3
- package/skills/ae-community/references/get_livestream_analysis.md +3 -3
- package/skills/ae-community/references/get_livestream_detail.md +4 -4
- package/skills/ae-community/references/get_livestream_list.md +3 -3
- package/skills/ae-community/references/get_livestream_overview.md +3 -3
- package/skills/ae-community/references/get_livestream_room_metrics.md +3 -3
- package/skills/ae-community/references/get_livestream_rooms.md +3 -3
- package/skills/ae-community/references/get_overview_metrics.md +3 -3
- package/skills/ae-community/references/get_post_detail.md +4 -4
- package/skills/ae-community/references/get_risk_content.md +5 -5
- package/skills/ae-community/references/get_sentiment_overview.md +4 -4
- package/skills/ae-community/references/get_tag_trends.md +3 -3
- package/skills/ae-community/references/get_topic_detail.md +3 -3
- package/skills/ae-community/references/list_sources.md +28 -0
- package/skills/ae-community/references/list_spaces.md +39 -0
- package/skills/ae-community/references/search_posts.md +21 -7
- package/skills/ae-data-integration/SKILL.md +6 -4
- package/skills/ae-data-integration/references/error-handling.md +1 -0
- package/skills/ae-data-integration/references/handoff.md +1 -1
- package/skills/ae-data-integration/references/sink-upload.md +11 -0
- package/skills/ae-data-integration/references/source-inspect.md +2 -1
- package/skills/ae-data-integration/references/transform.md +1 -1
- package/skills/ae-data-integration/references/ue-mapping.md +1 -1
- package/skills/ae-evaluation/SKILL.md +83 -0
- package/skills/ae-evaluation/references/access-and-members.md +55 -0
- package/skills/ae-evaluation/references/connections.md +78 -0
- package/skills/ae-evaluation/references/datasets.md +57 -0
- package/skills/ae-evaluation/references/evaluators.md +81 -0
- package/skills/ae-evaluation/references/files.md +33 -0
- package/skills/ae-evaluation/references/integrations.md +39 -0
- package/skills/ae-evaluation/references/runs.md +160 -0
- package/skills/ae-evaluation/references/schedules.md +31 -0
- package/skills/ae-experiment/SKILL.md +7 -1
- package/skills/ae-experiment/references/manage_experiment.md +18 -1
- package/skills/ae-experiment/references/query_experiment_detail.md +7 -0
- package/skills/ae-experiment/references/save_experiment.md +76 -2
- package/skills/ae-experiment/references/save_submit_experiment.md +7 -0
- package/skills/ae-kb/SKILL.md +33 -7
- package/skills/ae-kb/references/publication.md +81 -0
- package/skills/ae-kb/references/query-workflow.md +14 -1
- package/skills/ae-metadata/SKILL.md +3 -2
- package/skills/ae-metadata/references/metadata_data_table_download.md +7 -1
- package/skills/ae-metadata/references/metadata_data_table_get.md +37 -0
- package/skills/ae-system/SKILL.md +1 -1
- package/skills/ae-team/SKILL.md +2 -2
- package/skills/ae-team/references/run-start.md +1 -0
- package/skills/ae-team/references/run-watch.md +7 -5
- package/skills/ae-things/SKILL.md +53 -51
- package/skills/ae-things/references/cli.md +56 -54
- package/skills/ae-things-progress/SKILL.md +20 -20
- package/skills/ae-things-review/SKILL.md +19 -19
- package/skills/ae-workspace/SKILL.md +1 -1
- package/dist/chunk-6VNIR4AP.js +0 -1422
- package/dist/chunk-CI24BDR2.js +0 -43
|
@@ -0,0 +1,160 @@
|
|
|
1
|
+
# Evaluation runs and results
|
|
2
|
+
|
|
3
|
+
Commands use `ae-cli evaluation +<command>`. Read the shared endpoint, authentication, and mutation rules in [SKILL.md](../SKILL.md). Use command `--help` for optional flags.
|
|
4
|
+
|
|
5
|
+
Run cancellation is a high-risk write and requires confirmation. Non-interactive cancellation requires explicit authorization and global `--yes`; use `--dry-run` to preview without cancelling.
|
|
6
|
+
|
|
7
|
+
### Evaluation runs
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
ae-cli evaluation +list_runs [--search <text>] [--status <status>] [--cursor <cursor>] [--limit 50]
|
|
11
|
+
ae-cli evaluation +get_run --id <run-uuid>
|
|
12
|
+
ae-cli evaluation +create_run --name "<name>" \
|
|
13
|
+
[--dataset-id <uuid>] --dataset-version-id <dataset-version-uuid> \
|
|
14
|
+
[--execution-mode LIVE_AGENT|PRESET_RESULT] [--subject-type AGENT|MODEL] \
|
|
15
|
+
[--agent-config-id <uuid>] [--model-config-id <uuid>] \
|
|
16
|
+
[--report-to-te true] \
|
|
17
|
+
[--field-mapping-json '<json>'] [--result-filter-fields-json '<json>'] \
|
|
18
|
+
[--pass-threshold 0.8] [--case-sample-size 100] [--idempotency-key "<key>"]
|
|
19
|
+
ae-cli evaluation +update_run --id <run-uuid> --name "<name>" [same optional fields as +create_run] --lock-version <n>
|
|
20
|
+
ae-cli evaluation +delete_run --id <run-uuid> --lock-version <n>
|
|
21
|
+
ae-cli evaluation +select_run_evaluators --id <run-uuid> --evaluator-ids '["<evaluator-uuid>"]' \
|
|
22
|
+
[--evaluator-versions-json '[{"evaluator_id":"<uuid>","version_id":"<uuid>"}]'] --lock-version <n>
|
|
23
|
+
ae-cli evaluation +list_run_evaluators --id <run-uuid>
|
|
24
|
+
ae-cli evaluation +update_run_field_mapping --id <run-uuid> --field-mapping-json '<json>' --lock-version <n>
|
|
25
|
+
ae-cli evaluation +update_run_result_filter_fields --id <run-uuid> --result-filter-fields-json '{"fields":[],"schema_version":"RESULT_FILTER_FIELDS_V1"}' --lock-version <n>
|
|
26
|
+
ae-cli evaluation +validate_run --id <run-uuid>
|
|
27
|
+
ae-cli evaluation +start_run --id <run-uuid> --lock-version <n>
|
|
28
|
+
ae-cli evaluation +cancel_run --id <run-uuid> --lock-version <n>
|
|
29
|
+
ae-cli evaluation +retry_failed_run_cases --id <run-uuid> --lock-version <n>
|
|
30
|
+
ae-cli evaluation +run_overview --id <run-uuid>
|
|
31
|
+
ae-cli evaluation +run_distribution --id <run-uuid> [--run-evaluator-id <uuid>]
|
|
32
|
+
ae-cli evaluation +run_cases --id <run-uuid> \
|
|
33
|
+
[--status <status>] [--evidence-completeness NONE|PARTIAL|COMPLETE|UNKNOWN] [--search <text>] \
|
|
34
|
+
[--evaluation-status <status>] [--score-status <status>] [--evaluator-id <uuid>] \
|
|
35
|
+
[--evaluator-ids '["<uuid>"]'] [--min-score 0.8] [--max-score 1] [--score-gte 0.2] [--score-lte 0.8] \
|
|
36
|
+
[--field-filters-json '[...]'] [--sort AVERAGE_SCORE] [--sort-evaluator-id <uuid>] [--order desc] \
|
|
37
|
+
[--cursor <cursor>] [--limit 50] [--offset 0]
|
|
38
|
+
ae-cli evaluation +get_run_case --id <run-uuid> --case-run-id <case-run-uuid>
|
|
39
|
+
ae-cli evaluation +get_run_case_trace --id <run-uuid> --case-run-id <case-run-uuid>
|
|
40
|
+
ae-cli evaluation +retry_run_case_trace --id <run-uuid> --case-run-id <case-run-uuid>
|
|
41
|
+
ae-cli evaluation +copy_run_case --id <run-uuid> --case-run-id <case-run-uuid>
|
|
42
|
+
ae-cli evaluation +transfer_run_owner --id <run-uuid> --owner-user-id <user-uuid> --lock-version <n>
|
|
43
|
+
ae-cli evaluation +list_project_members
|
|
44
|
+
ae-cli evaluation +create_next_task_version_draft --id <run-uuid>
|
|
45
|
+
ae-cli evaluation +list_evaluation_task_versions --task-id <task-uuid>
|
|
46
|
+
ae-cli evaluation +export_run_results --id <run-uuid> [--export-format CSV|JSONL] --idempotency-key "<fresh-key>"
|
|
47
|
+
ae-cli evaluation +export_run_low_scores --id <run-uuid> --max-score 0.5 [--min-score 0] [--status SUCCEEDED] [--export-format CSV|JSONL] --idempotency-key "<fresh-key>"
|
|
48
|
+
ae-cli evaluation file download --id <file-uuid> --output ./results.csv
|
|
49
|
+
ae-cli evaluation +get_file --id <file-uuid>
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
- Field mapping JSON follows the platform `FIELD_MAPPING_V*` contract (`FIELD_MAPPING_V5` adds `TEMPLATE` input and `AGENT_TOOL_SUMMARY` / `AGENT_ARTIFACT_SUMMARY` / `AGENT_EXECUTION_STATUS` evidence sources); result-filter-fields is `RESULT_FILTER_FIELDS_V1` with at most 5 fields. Do not guess shapes — inspect an existing run with `+get_run` for a working example.
|
|
53
|
+
- `--case-sample-size` randomly selects 1-1000 cases when the run starts; omit it to run every case. Start fails with `RUN_SAMPLE_SIZE_EXCEEDS_CASES` when the requested sample exceeds the selected dataset version's case count.
|
|
54
|
+
- Start prerequisites: bound evaluator(s) (`EVALUATOR_NOT_ACTIVE`), complete field mapping (`FIELD_MAPPING_INCOMPLETE`), and result filter fields — an explicit empty filter is required (`RESULT_FILTER_FIELD_UNAVAILABLE`).
|
|
55
|
+
- Retry semantics: `+retry_failed_run_cases` requeues only Agent-stage `FAILED|BLOCKED|UNKNOWN` cases in the same immutable snapshot, bumps the execution epoch and recomputes aggregates. `+retry_run_case_trace` is ADMIN-only and only for retryable AI Monitor-backed captures.
|
|
56
|
+
- `+create_next_task_version_draft` copies `case_sample_size` and selects the latest published evaluator versions for the new draft. Run and overview responses expose separate succeeded, failed, and cancelled case/score counts for progress tracking.
|
|
57
|
+
|
|
58
|
+
### Create and run an evaluation
|
|
59
|
+
|
|
60
|
+
1. Pick a published dataset version and create a draft run from it plus an agent or model:
|
|
61
|
+
|
|
62
|
+
```bash
|
|
63
|
+
ae-cli evaluation +list_dataset_versions --id <dataset-uuid>
|
|
64
|
+
ae-cli evaluation +create_run \
|
|
65
|
+
--name "release-checkout-run" \
|
|
66
|
+
--dataset-id <dataset-uuid> \
|
|
67
|
+
--dataset-version-id <dataset-version-uuid> \
|
|
68
|
+
--execution-mode LIVE_AGENT \
|
|
69
|
+
--subject-type AGENT \
|
|
70
|
+
--agent-config-id <agent-uuid>
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
For scoring pre-stored dataset fields without calling an Agent/Model, use `--execution-mode PRESET_RESULT` (then omit `--agent-config-id` / `--model-config-id`).
|
|
74
|
+
|
|
75
|
+
2. Configure the draft. Before `+start_run` the run needs: at least one ACTIVE evaluator (pin published evaluator versions when possible), a complete field mapping, and result filter fields (an explicit empty filter is fine). Each mutation consumes one lock_version — re-read with `+get_run` in between:
|
|
76
|
+
|
|
77
|
+
```bash
|
|
78
|
+
ae-cli evaluation +list_evaluator_versions --id <evaluator-uuid>
|
|
79
|
+
ae-cli evaluation +select_run_evaluators --id <run-uuid> \
|
|
80
|
+
--evaluator-ids '["<evaluator-uuid>"]' \
|
|
81
|
+
--evaluator-versions-json '[{"evaluator_id":"<evaluator-uuid>","version_id":"<evaluator-version-uuid>"}]' \
|
|
82
|
+
--lock-version <n>
|
|
83
|
+
ae-cli evaluation +get_run --id <run-uuid>
|
|
84
|
+
ae-cli evaluation +update_run_field_mapping --id <run-uuid> \
|
|
85
|
+
--field-mapping-json '{"schema_version":"FIELD_MAPPING_V5","agent":{"input":{"field_key":"input","source_type":"DATASET_FIELD"}},"evaluators":{"<evaluator-uuid>":{"input":{"field_key":"input","source_type":"DATASET_FIELD"},"output":{"source_type":"AGENT_OUTPUT"}}}}' \
|
|
86
|
+
--lock-version <n>
|
|
87
|
+
ae-cli evaluation +get_run --id <run-uuid>
|
|
88
|
+
ae-cli evaluation +update_run_result_filter_fields --id <run-uuid> \
|
|
89
|
+
--result-filter-fields-json '{"fields":[],"schema_version":"RESULT_FILTER_FIELDS_V1"}' \
|
|
90
|
+
--lock-version <n>
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
3. Validate (optional, same preflight as start, no lock_version needed), then start:
|
|
94
|
+
|
|
95
|
+
```bash
|
|
96
|
+
ae-cli evaluation +validate_run --id <run-uuid>
|
|
97
|
+
ae-cli evaluation +get_run --id <run-uuid>
|
|
98
|
+
ae-cli evaluation +start_run --id <run-uuid> --lock-version <n>
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
4. Watch the results and drill into failures:
|
|
102
|
+
|
|
103
|
+
```bash
|
|
104
|
+
ae-cli evaluation +run_overview --id <run-uuid>
|
|
105
|
+
ae-cli evaluation +run_distribution --id <run-uuid> --run-evaluator-id <run-evaluator-uuid>
|
|
106
|
+
ae-cli evaluation +run_cases --id <run-uuid> --status FAILED --evaluation-status FAILED
|
|
107
|
+
ae-cli evaluation +get_run_case --id <run-uuid> --case-run-id <case-run-uuid>
|
|
108
|
+
ae-cli evaluation +get_run_case_trace --id <run-uuid> --case-run-id <case-run-uuid>
|
|
109
|
+
ae-cli evaluation +retry_failed_run_cases --id <run-uuid> --lock-version <n>
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
### Export results and download
|
|
113
|
+
|
|
114
|
+
```bash
|
|
115
|
+
ae-cli evaluation +export_run_results --id <run-uuid> --export-format CSV --idempotency-key "export-$(date +%s)"
|
|
116
|
+
ae-cli evaluation +export_run_low_scores --id <run-uuid> --max-score 0.5 --idempotency-key "low-$(date +%s)"
|
|
117
|
+
# Poll the returned file until COMPLETED before downloading.
|
|
118
|
+
ae-cli evaluation +get_file --id <file-uuid>
|
|
119
|
+
ae-cli evaluation file download --id <file-uuid> --output ./results.csv
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
Traces are available only for LIVE_AGENT runs with subject AGENT. Capture status is independent of case/run success. Captures contain at most 500 spans and may be truncated; PARTIAL or reference-only evidence must not be described as complete. Never include hidden chain-of-thought, credentials, cookies, auth headers, or unredacted tool input/output in reports. Ownership transfer and project-member discovery require ADMIN.
|
|
123
|
+
|
|
124
|
+
### Process evidence mapping (FIELD_MAPPING_V6)
|
|
125
|
+
|
|
126
|
+
For LIVE_AGENT runs whose subject is AGENT, V6 adds `AGENT_EXECUTION_TRACE` and
|
|
127
|
+
`AGENT_TOOL_CALLS`. Keep the existing V5 mappings for final-output-only scoring.
|
|
128
|
+
Use real dataset field keys, evaluator IDs and a published evaluator version:
|
|
129
|
+
|
|
130
|
+
```json
|
|
131
|
+
{
|
|
132
|
+
"schema_version": "FIELD_MAPPING_V6",
|
|
133
|
+
"agent": {"input": {"source_type": "DATASET_FIELD", "field_key": "input"}},
|
|
134
|
+
"evaluators": {
|
|
135
|
+
"<evaluator-uuid>": {
|
|
136
|
+
"input": {"source_type": "DATASET_FIELD", "field_key": "input"},
|
|
137
|
+
"output": {"source_type": "AGENT_OUTPUT"},
|
|
138
|
+
"execution_trace": {"source_type": "AGENT_EXECUTION_TRACE"}
|
|
139
|
+
}
|
|
140
|
+
}
|
|
141
|
+
}
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
An evaluator declaring `tool_calls` may map it to `{ "source_type": "AGENT_TOOL_CALLS" }`.
|
|
145
|
+
Only map variables the evaluator declares. Send the complete mapping through
|
|
146
|
+
`+update_run_field_mapping` with the latest lock version, validate, then start.
|
|
147
|
+
|
|
148
|
+
The process projection is `PROCESS_EVIDENCE_V1`; TE Claude A2A tool artifacts use
|
|
149
|
+
`TE_CLAUDE_TOOL_CALLS_V1`. Scoring requires a READY/COMPLETE/non-truncated snapshot
|
|
150
|
+
from the current execution epoch and limits the projection to 64 KiB. Missing,
|
|
151
|
+
pending or unusable evidence fails only the dependent score with a null score;
|
|
152
|
+
Agent status stays independent and final-output evaluators can still score.
|
|
153
|
+
|
|
154
|
+
Score `input_variables` holds resource references (`content_location=CASE_EXECUTION_TRACE`),
|
|
155
|
+
not full tool payloads. Inspect `+get_run_case_trace` separately. TE Claude evidence
|
|
156
|
+
covers recorded tool/sub-Agent calls and explicit parent-child relationships,
|
|
157
|
+
not hidden model reasoning, delegated internal steps, timing or cost. Preserve
|
|
158
|
+
redaction and completeness limits when reporting results.
|
|
159
|
+
|
|
160
|
+
See [files.md](files.md) for authenticated downloads and file readiness checks.
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
# Scheduled evaluations
|
|
2
|
+
|
|
3
|
+
Commands use `ae-cli evaluation +<command>`. Read the shared endpoint, authentication, and mutation rules in [SKILL.md](../SKILL.md). Use command `--help` for optional flags.
|
|
4
|
+
|
|
5
|
+
### Schedules
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
ae-cli evaluation +list_schedules [--status PENDING|PROCESSING|COMPLETED|FAILED|CANCELLED] [--cursor <cursor>] [--limit 50]
|
|
9
|
+
ae-cli evaluation +create_schedule --run-id <run-uuid> --frequency HOURLY|DAILY|WEEKLY|MONTHLY --time-zone <iana> \
|
|
10
|
+
[--minute-of-hour 0] [--hour-of-day 9] [--day-of-week 1] [--day-of-month 1]
|
|
11
|
+
ae-cli evaluation +update_schedule --id <schedule-uuid> --frequency <freq> --time-zone <iana> \
|
|
12
|
+
[--minute-of-hour 0] [--hour-of-day 9] [--day-of-week 1] [--day-of-month 1] --lock-version <n>
|
|
13
|
+
ae-cli evaluation +delete_schedule --id <schedule-uuid> --lock-version <n>
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
HOURLY runs at minute 0 and must omit `--minute-of-hour` (or send 0). DAILY/WEEKLY/MONTHLY require `--minute-of-hour`. Schedules pin published task versions.
|
|
17
|
+
|
|
18
|
+
### Scheduled evaluations
|
|
19
|
+
|
|
20
|
+
```bash
|
|
21
|
+
ae-cli evaluation +list_schedules
|
|
22
|
+
ae-cli evaluation +create_schedule \
|
|
23
|
+
--run-id <run-uuid> \
|
|
24
|
+
--frequency DAILY \
|
|
25
|
+
--time-zone Asia/Shanghai \
|
|
26
|
+
--hour-of-day 9 \
|
|
27
|
+
--minute-of-hour 0
|
|
28
|
+
ae-cli evaluation +update_schedule --id <schedule-uuid> --frequency WEEKLY --time-zone Asia/Shanghai \
|
|
29
|
+
--day-of-week 1 --hour-of-day 9 --minute-of-hour 0 --lock-version <n>
|
|
30
|
+
ae-cli evaluation +delete_schedule --id <schedule-uuid> --lock-version <n>
|
|
31
|
+
```
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: ae-experiment
|
|
3
|
-
version: 1.0.
|
|
3
|
+
version: 1.0.5
|
|
4
4
|
description: "Use when managing Atlas AB or external experiments, traffic layers, Features, metrics, buckets, and experiment reports through ae-cli"
|
|
5
5
|
---
|
|
6
6
|
|
|
@@ -65,6 +65,12 @@ Naming and response boundary:
|
|
|
65
65
|
5. Check readiness with `experiment experiment ready-check`.
|
|
66
66
|
6. For a non-mutex traffic layer, run `experiment experiment conflict-check` before submit (needs `feature_key_list` from context or `experiment get`).
|
|
67
67
|
7. Move status with `experiment experiment manage`.
|
|
68
|
+
For a scheduled Feature start, save `startTime` (or use `--start-time`), then
|
|
69
|
+
submit with `--target-status pending`. Saving a draft does not submit it; `running`
|
|
70
|
+
starts immediately. Read [Feature timing](references/save_experiment.md#feature-timing)
|
|
71
|
+
before configuring a schedule or starting it early.
|
|
72
|
+
To resume `paused` to `running`, call `manage` directly and verify the status
|
|
73
|
+
with `get`; do not use the pre-launch `ready-check` as a resume prerequisite.
|
|
68
74
|
8. Query reports with `experiment report summary`, `experiment report sample-size`, and `experiment report metric-trend`.
|
|
69
75
|
|
|
70
76
|
If an experiment save returns `error_code: METRIC_NOT_FOUND`, list metrics for the same project. Create and verify the metric before retrying; never retry with another invented ID. Metric deletion returns `error_code: METRIC_IN_USE` while an active experiment binding exists.
|
|
@@ -12,6 +12,23 @@ Flags:
|
|
|
12
12
|
- `--target-status`: One of `draft`, `testing`, `pending`, `running`, `paused`, `ended`, `archive`.
|
|
13
13
|
- `--remark`: Optional operation remark.
|
|
14
14
|
|
|
15
|
-
When target status is `pending`, `testing`, or `running`, Hermes performs a ready
|
|
15
|
+
When target status is `pending`, `testing`, or `running`, Hermes performs a ready
|
|
16
|
+
check before changing status, except when resuming `paused` to `running`.
|
|
17
|
+
Resume uses the shared status transition service, which checks permissions,
|
|
18
|
+
deadline, and audience readiness and re-enables the Feature rules and RCC sync.
|
|
19
|
+
It preserves the first start time and the existing day-based or fixed deadline.
|
|
20
|
+
Do not run the standalone `ready-check` as a prerequisite for resuming a paused
|
|
21
|
+
experiment; that command accepts only pre-launch states.
|
|
22
|
+
|
|
23
|
+
For Feature scheduling, save `startTime` first and use `pending` to submit. The
|
|
24
|
+
server checks submitted experiments every half hour. `running` starts immediately;
|
|
25
|
+
read fresh detail and disclose `start_time_adjustment_required=true` before an
|
|
26
|
+
early manual start. No extra confirmation field is required. See
|
|
27
|
+
[Feature timing](save_experiment.md#feature-timing) for lifecycle and deadline rules.
|
|
16
28
|
|
|
17
29
|
Response shape: the operation or blocking readiness result is in `data.result`, with object keys recursively converted to snake_case.
|
|
30
|
+
|
|
31
|
+
An outer `ok=true` does not mean the status changed when `data.result.valid=false`.
|
|
32
|
+
Report the returned issues and read `experiment experiment get` to confirm the
|
|
33
|
+
actual status. A successful resume normally returns `data.result=true`; do not
|
|
34
|
+
report the experiment as running until an independent read confirms it.
|
|
@@ -12,6 +12,13 @@ Flags:
|
|
|
12
12
|
|
|
13
13
|
Response shape: the experiment is in `data.item`, with recursively snake_case keys such as `exp_id` and `feature_key_list`.
|
|
14
14
|
|
|
15
|
+
Feature timing is exposed as `start_time`, `end_time`, and `exp_cycle` (including
|
|
16
|
+
`cycle_type`, `fixed_end_time`, and planned `end_time`). The top-level `end_time`
|
|
17
|
+
records actual completion. Check `scheduled_start_error` for an expired pending
|
|
18
|
+
window and `start_time_adjustment_required` before starting a scheduled experiment
|
|
19
|
+
early. Use returned `status` to verify execution; a successful save or submission
|
|
20
|
+
does not mean the experiment is running.
|
|
21
|
+
|
|
15
22
|
When targeting contains reversible QP, it is returned as
|
|
16
23
|
`targeting.definition_request` with `targeting.definition_status=AVAILABLE`.
|
|
17
24
|
An absent definition is `NOT_APPLICABLE`; a historical unsupported definition is
|
|
@@ -3,12 +3,14 @@
|
|
|
3
3
|
Progressively create or patch an experiment draft.
|
|
4
4
|
|
|
5
5
|
```bash
|
|
6
|
-
ae-cli experiment experiment save --project-id <id> --req '<json>'
|
|
6
|
+
ae-cli experiment experiment save --project-id <id> --req '<json>' [--start-time <timestamp>] [--fixed-end-time <timestamp>]
|
|
7
7
|
```
|
|
8
8
|
|
|
9
9
|
Flags:
|
|
10
10
|
- `--project-id`, `-p`: Project ID.
|
|
11
11
|
- `--req`: Experiment save request JSON object.
|
|
12
|
+
- `--start-time`: Optional Feature start timestamp in ISO-8601 format with timezone offset. Overrides `req.startTime`.
|
|
13
|
+
- `--fixed-end-time`: Optional Feature deadline in the same format. Sets `req.expCycle.cycleType=time` and overrides `req.expCycle.fixedEndTime`, preserving other supplied cycle fields. These timing flags are for Feature experiments; external experiments use `external-experiment save-submit`.
|
|
12
14
|
|
|
13
15
|
## Contract
|
|
14
16
|
|
|
@@ -64,6 +66,7 @@ Optional while drafting:
|
|
|
64
66
|
- `isExpSharing`
|
|
65
67
|
- `bucketId`
|
|
66
68
|
- `expCycle`
|
|
69
|
+
- `startTime`
|
|
67
70
|
- `groups`
|
|
68
71
|
- `featureKeyList`
|
|
69
72
|
- `metrics`
|
|
@@ -101,7 +104,7 @@ ae-cli experiment experiment save --project-id 1 --req '{"expId":"exp_123","allo
|
|
|
101
104
|
- `isExpSharing=0` uses exclusive traffic with `bucketId`; select the bucket with `experiment bucket list`.
|
|
102
105
|
- `allocation` (experiment-level) is the experiment's share of the traffic layer. It is optional during draft construction, but before readiness it must be greater than `0` and no more than `100`. **Integer only** (for example `20`); decimals such as `20.5` are rejected at the CLI interface.
|
|
103
106
|
- `groups[].allocation` (group-level): **integer only; sum must equal `100` exactly**. Example for 3 groups: `34 + 33 + 33 = 100`. Decimals such as `33.33` are rejected at the CLI interface before save.
|
|
104
|
-
- `expCycle.cycleType` can be `day` or `
|
|
107
|
+
- `expCycle.cycleType` can be `day`, `sample`, or `time`; when `cycleType=day`, `dayNum` is `1..90`. `time` requires `fixedEndTime`. Blank cycle defaults to day + 30.
|
|
105
108
|
- Before readiness, `groups` must be non-empty, contain exactly one control group (`isControl=1`), group allocations must sum to exactly `100`, and each `expGroupValue` must be a non-empty JSON string array.
|
|
106
109
|
- Feature experiments must bind `featureKeyList` before readiness. `featureKeyList` contains Feature key strings, not Feature objects. The MCP currently supports one `featureKey` for feature experiments.
|
|
107
110
|
- Metrics use `metricRole=primary|secondary|guardrail`; the internal `observation` role is currently unavailable for saves. Before readiness at least one primary metric is required.
|
|
@@ -142,6 +145,61 @@ Semantic audience example:
|
|
|
142
145
|
}
|
|
143
146
|
```
|
|
144
147
|
|
|
148
|
+
## Feature Timing
|
|
149
|
+
|
|
150
|
+
Feature start time is optional. Use ISO-8601 timestamps with an explicit offset,
|
|
151
|
+
for example `2026-11-01T16:00:00+08:00`. Prefer minutes `00` or `30` to align
|
|
152
|
+
with the server's half-hour scheduling checks. The CLI forwards timestamps;
|
|
153
|
+
the server clears seconds and milliseconds on configured Feature start times.
|
|
154
|
+
|
|
155
|
+
Save timing on a new draft or patch an existing experiment:
|
|
156
|
+
|
|
157
|
+
```bash
|
|
158
|
+
ae-cli experiment experiment save --project-id 1 --req '{"expName":"Scheduled CTA"}' --start-time '2026-11-01T16:00:00+08:00' --fixed-end-time '2026-11-08T16:00:00+08:00' --dry-run
|
|
159
|
+
ae-cli experiment experiment save --project-id 1 --req '{"expId":"exp_123"}' --start-time '2026-11-01T16:00:00+08:00' --fixed-end-time '2026-11-08T16:00:00+08:00'
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
The second command produces the same native request as:
|
|
163
|
+
|
|
164
|
+
```json
|
|
165
|
+
{"expId":"exp_123","startTime":"2026-11-01T16:00:00+08:00","expCycle":{"cycleType":"time","fixedEndTime":"2026-11-08T16:00:00+08:00"}}
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
Saving does not submit or start the experiment. Complete the remaining Feature,
|
|
169
|
+
group, traffic, and metric configuration, check readiness, and check traffic
|
|
170
|
+
conflicts for a non-mutex layer before submitting:
|
|
171
|
+
|
|
172
|
+
```bash
|
|
173
|
+
ae-cli experiment experiment ready-check --project-id 1 --exp-id exp_123
|
|
174
|
+
ae-cli experiment experiment manage --project-id 1 --exp-id exp_123 --target-status pending
|
|
175
|
+
ae-cli experiment experiment get --project-id 1 --exp-id exp_123
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
Only submitted `pending` Feature experiments with a configured start time are
|
|
179
|
+
eligible for automatic start. A draft never starts automatically; a pending
|
|
180
|
+
experiment without a start time waits for manual start. The server checks every
|
|
181
|
+
half hour, so a successful submission is not evidence that the experiment is
|
|
182
|
+
already running. Verify `data.item.status`, `start_time`, and
|
|
183
|
+
`scheduled_start_error` with `get` (or `list`). An expired pending window remains
|
|
184
|
+
pending and reports an error; fix the timing instead of reporting a start.
|
|
185
|
+
|
|
186
|
+
Timing changes follow server lifecycle rules:
|
|
187
|
+
|
|
188
|
+
- Start time can change while draft or pending. Omission or `null` preserves a
|
|
189
|
+
saved start time; neither clears it. After the first run, start time is locked.
|
|
190
|
+
- `day` ends after start plus `dayNum` times 24 hours. `time` uses the absolute
|
|
191
|
+
`expCycle.fixedEndTime`, which must be later than start. The top-level `endTime`
|
|
192
|
+
is the actual end time, not the Feature deadline configuration.
|
|
193
|
+
- Submitted end conditions must keep their cycle type and may only be extended.
|
|
194
|
+
The server validates existing state and the effective time window.
|
|
195
|
+
- Manual `running` starts immediately. Read fresh detail first and disclose any
|
|
196
|
+
`start_time_adjustment_required=true` to the user: an early start uses current
|
|
197
|
+
server time, recalculates a day-based deadline, and keeps a fixed deadline.
|
|
198
|
+
Do not add an extra confirmation field to the request.
|
|
199
|
+
- Pause/resume keeps the original start and deadline. Running and paused
|
|
200
|
+
experiments are checked for expiry every half hour; deadlines between checks
|
|
201
|
+
are handled by a subsequent check.
|
|
202
|
+
|
|
145
203
|
## Payload Examples
|
|
146
204
|
|
|
147
205
|
Patch a Feature binding. `featureKeyList` is a string array:
|
|
@@ -168,6 +226,22 @@ Patch metrics. `metrics` is an object array:
|
|
|
168
226
|
ae-cli experiment experiment save --project-id 2 --req '{"expId":"e024","metrics":[{"metricId":"payment","metricRole":"primary"}]}'
|
|
169
227
|
```
|
|
170
228
|
|
|
229
|
+
## Timing Validation Errors
|
|
230
|
+
|
|
231
|
+
Atlas business time checks return `error.type=validation`, a specific code,
|
|
232
|
+
`error.details.field`, and a correction hint. Examples:
|
|
233
|
+
|
|
234
|
+
| Code | Field | Correction |
|
|
235
|
+
| --- | --- | --- |
|
|
236
|
+
| `EXPERIMENT_END_TIME_INVALID` | `req.expCycle.fixedEndTime` (related: `req.startTime`) | Make the deadline strictly later than the start; submission or start also requires an unexpired window. |
|
|
237
|
+
| `EXPERIMENT_END_TIME_REQUIRED` | `req.expCycle.fixedEndTime` | Supply a timestamp with timezone offset when `cycleType=time`. |
|
|
238
|
+
|
|
239
|
+
Fix the reported field before retrying. The error preserves `meta.invocation_id`
|
|
240
|
+
for investigation. These checks do not save the invalid configuration; use `get`
|
|
241
|
+
to verify the existing timing. Older backends may still return
|
|
242
|
+
`CAPABILITY_EXECUTION_FAILED`; changing CLI arguments cannot repair a missing
|
|
243
|
+
server-side exception translator.
|
|
244
|
+
|
|
171
245
|
## Dry-run Limitation
|
|
172
246
|
|
|
173
247
|
`--dry-run` calls the Capability Gateway preview endpoint. It validates the outer input contract but does not guarantee that every nested DTO business rule will pass execution.
|
|
@@ -14,3 +14,10 @@ internal `targetConfig` field or hand-write `Cxx`/`Axxx` codes.
|
|
|
14
14
|
Response shape: `data.result`, with object keys recursively converted to snake_case.
|
|
15
15
|
|
|
16
16
|
Prefer `experiment experiment save` for progressive draft construction.
|
|
17
|
+
|
|
18
|
+
Feature requests may include optional `req.startTime` and
|
|
19
|
+
`req.expCycle={"cycleType":"time","fixedEndTime":"2026-11-08T16:00:00+08:00"}`.
|
|
20
|
+
Use ISO-8601 timestamps with timezone offsets. Submitting a Feature draft
|
|
21
|
+
places it in `pending`; with a start time, the server checks it every half hour
|
|
22
|
+
for automatic start. See [Feature timing](save_experiment.md#feature-timing) for
|
|
23
|
+
the draft, submission, manual-start, and deadline boundaries.
|
package/skills/ae-kb/SKILL.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: ae-kb
|
|
3
|
-
version: 1.0.
|
|
3
|
+
version: 1.0.8
|
|
4
4
|
description: 'AE knowledge base CLI manual for creating, importing read-only compiled snapshots, querying, LLM-powered ask, listing accessible knowledge bases and their sources, deterministic index/grep/read retrieval, checking status, ZIP source upload and directory management, raw child-file reading, revision-checked child updates and deletion, compiling, schema generation, URL sources, source deletion, ordinary text source creation, full reads and revision-checked editing, file/ZIP source draft replacement and restoration, and knowledge base deletion. Use when the user asks to manage AE (Agentic Engine) knowledge bases, import a compiled Markdown ZIP snapshot, upload documents or URLs to a knowledge base, query knowledge, ask knowledge bases with an LLM, list accessible knowledge bases or source metadata, inspect knowledge base indexes, search knowledge base pages, read a specific knowledge base page, check knowledge base status, generate schemas, compile knowledge, remove sources, create or edit text sources, or delete a knowledge base. To choose which knowledge base is worth searching, use the ae-kb-discovery skill first; this skill runs the retrieval once a target is chosen. Must use ae-cli kb commands and must not guess knowledge base names, scopes, page paths, source IDs, source display names, JSON payload shapes, or URL formats.'
|
|
5
5
|
---
|
|
6
6
|
|
|
@@ -17,6 +17,10 @@ ae-cli kb +<command> [options]
|
|
|
17
17
|
ae-cli kb source <create-text|read-text|replace|restore|preview|commit|cancel|check-updates|check-url-updates> [options]
|
|
18
18
|
```
|
|
19
19
|
|
|
20
|
+
## Formal publication and pinned reads
|
|
21
|
+
|
|
22
|
+
Read [Publication and read contexts](references/publication.md) for company approval, personal auto-publication, candidate inspection and changed-only approval previews (`kb candidate changes`), explicit published queries, or consistent multi-command retrieval. Compile success can mean a candidate awaiting approval; use `kb publication get` to verify formal publication. Never treat the working build status as proof that the formal version changed.
|
|
23
|
+
|
|
20
24
|
## Global Rules
|
|
21
25
|
|
|
22
26
|
- Use this skill for AE knowledge base tasks: create, import a compiled snapshot, query, ask with LLM, list accessible knowledge bases and their sources, inspect indexes, grep pages, read pages, check status, upload sources, add URL sources, generate schema, compile, remove source files, and delete knowledge bases.
|
|
@@ -41,7 +45,8 @@ ae-cli kb source <create-text|read-text|replace|restore|preview|commit|cancel|ch
|
|
|
41
45
|
| `source check-url-updates` | write | Submit an asynchronous check for one compiled custom URL source; success means accepted only. |
|
|
42
46
|
| `source check-updates` | write | Check one compiled Feishu source with the scheduled detector; mark changes without compiling. |
|
|
43
47
|
| `+ask` | read | LLM-powered Q&A over knowledge bases; for multi-page synthesis or multi-hop questions. |
|
|
44
|
-
| `+ask-status` | read | Query
|
|
48
|
+
| `+ask-status` | read | Query an ask execution, or revisit its original versioned citation with `--source-index`. |
|
|
49
|
+
| `project-target get` | read | Resolve the authoritative company project semantic target by project ID, including tag conflicts, disabled and restricted states. |
|
|
45
50
|
| `+list` | read | List accessible knowledge bases filtered by buildStatus (default: compiled). |
|
|
46
51
|
| `+list-sources` | read | List source metadata for one knowledge base so exact source identifiers can be discovered safely. |
|
|
47
52
|
| `+index` | read | List accessible knowledge bases and their `index.md` navigation maps. |
|
|
@@ -59,6 +64,26 @@ ae-cli kb source <create-text|read-text|replace|restore|preview|commit|cancel|ch
|
|
|
59
64
|
| `+rm-source` | high-risk-write | Delete one source from a knowledge base by stable ID; exact display name is legacy compatibility only. |
|
|
60
65
|
| `+remove` | write | Delete an entire knowledge base. |
|
|
61
66
|
|
|
67
|
+
## Project Semantic Target
|
|
68
|
+
|
|
69
|
+
```bash
|
|
70
|
+
ae-cli kb project-target get --project-id <project_id> --host <host>
|
|
71
|
+
ae-cli kb +new --scope company --name <name> --project-id <project_id> --purpose project-semantic
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
Use `project-target get` for project semantic maintenance; `+list` remains a visible catalog and cannot prove that a project target is missing. The server verifies project access and considers all undeleted company candidates bound to that project, without filtering by enabled state or build state. One candidate is reused for legacy compatibility. With several candidates, exactly one fixed business tag `项目语义` selects the target; zero returns `multiple/untagged`, several returns `multiple/tag_conflict`. With no binding, the authoritative exact name `<project_name> 项目语义知识库` without another project binding is the legacy fallback, using the same tag rule.
|
|
75
|
+
|
|
76
|
+
`available` returns the exact target identity and access capabilities. `restricted`, `disabled`, and `multiple` stop maintenance; never select a different visible KB or create a replacement. Only `missing` permits creation. A query/auth failure or HTTP 404 from an older server is terminal; do not infer absence. Existing explicit target IDs, including stored automation targets, must not be silently switched.
|
|
77
|
+
|
|
78
|
+
`+new --purpose project-semantic` is optional and company-only, requires one valid project ID, presets the fixed tag on both client and server, and rechecks the target under the server KB write lock. User tags are preserved within the existing total limit of two; overflow fails explicitly. A 409 means refresh the target, not retry with a different name. Ordinary creation without `--purpose` retains its existing behavior. The tag classifies purpose and never grants access. Deploy the supporting Agent API before using the new CLI/Skill path; old CLI requests remain accepted by the new server.
|
|
79
|
+
|
|
80
|
+
Transition status: transitional
|
|
81
|
+
Owning module: te-claude Project Semantic Knowledge Base API
|
|
82
|
+
Current transport: authenticated Agent REST via `kbApi`
|
|
83
|
+
Gateway target: TBD (project semantic target resolution and purpose-aware creation)
|
|
84
|
+
Review after: 2027-01-27
|
|
85
|
+
Exit condition: migrate when the KB Gateway equivalent exists, preserving target identity, project isolation, tag and state semantics.
|
|
86
|
+
|
|
62
87
|
## Compilation Usage Logs
|
|
63
88
|
|
|
64
89
|
```bash
|
|
@@ -81,7 +106,7 @@ Exit condition: migrate to the Gateway equivalent when available, preserving typ
|
|
|
81
106
|
|
|
82
107
|
## Published Version History
|
|
83
108
|
|
|
84
|
-
Use the nine version commands described in [`references/versions.md`](references/versions.md) to inspect immutable published history, compare source changes, and explicitly roll back an earlier version as a new publication. These commands manage history;
|
|
109
|
+
Use the nine version commands described in [`references/versions.md`](references/versions.md) to inspect immutable published history, compare source changes, and explicitly roll back an earlier version as a new publication. These commands manage history. Ordinary queries resolve formal published content; Agent/Ask executions and explicit read contexts keep the formal versions captured at the start of that execution or context. Use the returned `readView` to identify the content actually read.
|
|
85
110
|
|
|
86
111
|
| Command | Risk | Purpose |
|
|
87
112
|
| --- | --- | --- |
|
|
@@ -250,7 +275,7 @@ ae-cli kb +schema --name engineering-handbook --scope company
|
|
|
250
275
|
|
|
251
276
|
Use `--force` only when `+status` reports `schema_generating` and the user explicitly wants to replace the current generation attempt. The replacement may consume additional tokens.
|
|
252
277
|
|
|
253
|
-
Schema and Compile accept the same optional model reference. Prefer the model record `id` returned by `ae-cli agent +list-models` (`Model.id`). Historical `modelId` and the unambiguous `modelId::scope` form remain compatible. A model `displayName` is presentation text, not a stable reference.
|
|
278
|
+
Schema and Compile accept the same optional model reference. By default, omit `--model` / `--model-id` and keep the existing KB default selection. Preserve an explicit choice from the user or task configuration. Never infer a model reference from the assistant's self-description. `company-auto` means the Company Auto routing policy, not a physical model; use the canonical reference `virtual:company-auto:v1` for compatibility with older servers. Prefer the model record `id` returned by `ae-cli agent +list-models` (`Model.id`). Historical `modelId` and the unambiguous `modelId::scope` form remain compatible. A model `displayName` is presentation text, not a stable reference.
|
|
254
279
|
|
|
255
280
|
To add one-time guidance for this generation without changing stored knowledge base metadata, pass `--custom-instructions`. The server trims the value, treats whitespace-only input as absent, and accepts up to 10,000 Unicode characters. Do not include secrets or credentials.
|
|
256
281
|
|
|
@@ -258,7 +283,6 @@ To add one-time guidance for this generation without changing stored knowledge b
|
|
|
258
283
|
ae-cli kb +schema \
|
|
259
284
|
--name engineering-handbook \
|
|
260
285
|
--scope company \
|
|
261
|
-
--model <model-ref> \
|
|
262
286
|
--custom-instructions "Prioritize troubleshooting workflows and preserve command examples"
|
|
263
287
|
```
|
|
264
288
|
|
|
@@ -267,7 +291,7 @@ Use `--dry-run` to inspect the request body before sending it. While generation
|
|
|
267
291
|
Compile after sources and schema are ready:
|
|
268
292
|
|
|
269
293
|
```bash
|
|
270
|
-
ae-cli kb +compile --name engineering-handbook --scope company --mode incremental
|
|
294
|
+
ae-cli kb +compile --name engineering-handbook --scope company --mode incremental
|
|
271
295
|
```
|
|
272
296
|
|
|
273
297
|
Valid compile modes are `incremental` and `full`; default is `incremental`.
|
|
@@ -440,6 +464,9 @@ ae-cli kb +ask-status --execution-id <id>
|
|
|
440
464
|
```
|
|
441
465
|
|
|
442
466
|
- `--execution-id`: required. The execution ID returned by `+ask` submission.
|
|
467
|
+
- `--source-index`: optional zero-based `sourceIndex` from `sources[].reference`. Returns that saved page's `content`, `path`, and public `readView`, at the original published version even after a new publication. Uses your current user permissions; query credentials are not revived. Legacy or old sources without an immutable version return `KB_REFERENCE_UNVERSIONED`; do not replace this with a fresh `+read` and call it the original citation.
|
|
468
|
+
- Completed Ask sources preserve `{knowledgeBaseKey, path, readView?, reference?}`. Keep `readView.versionId/versionNumber` when citing them. To revisit: `ae-cli kb +ask-status --execution-id <reference.executionId> --source-index <reference.sourceIndex>`.
|
|
469
|
+
- Historical source reads are unavailable inside an execution that requires a fixed KB context (`KB_READ_CONTEXT_REFERENCE_FORBIDDEN`). Do not bypass this by removing execution context or using an ordinary token. Revisit outside that execution with current user credentials. Ordinary `+ask-status` polling without `--source-index` remains available.
|
|
443
470
|
- Output: Returns one successful CLI envelope snapshot. For a failed execution, the stable server code remains at `data.error.code`: `{"ok":true,"data":{"executionId":"...","status":"failed","error":{"code":"retrieval_error","message":"..."}}}`. Does not poll.
|
|
444
471
|
|
|
445
472
|
### `+list`
|
|
@@ -610,7 +637,6 @@ Transport: `POST /agent/api/external/knowledge-bases/sources/check-updates`, bod
|
|
|
610
637
|
- Review after: 2026-10-16.
|
|
611
638
|
- Exit condition: once the Gateway exposes equivalent authorization and detector semantics, migrate the typed command to L2 or use L3 if it adds no value.
|
|
612
639
|
|
|
613
|
-
|
|
614
640
|
### Submit an ordinary URL update check
|
|
615
641
|
|
|
616
642
|
```bash
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
# Publication and read contexts
|
|
2
|
+
|
|
3
|
+
Source-managed personal KBs publish automatically after a successful compile. Company KBs produce an immutable candidate and automatically submit `kb.version.publish@1`; an authorized current content editor (including the requester) approves through the existing approval workflow. Editing working sources after compilation does not change the candidate. Approval publishes that frozen candidate without overwriting working edits. No feature switch is required.
|
|
4
|
+
|
|
5
|
+
## Inspect and manage
|
|
6
|
+
|
|
7
|
+
| Command | Purpose | Risk |
|
|
8
|
+
| --- | --- | --- |
|
|
9
|
+
| `ae-cli kb publication get` | Formal version and candidate summary | read |
|
|
10
|
+
| `ae-cli kb candidate get` | Candidate metadata | read |
|
|
11
|
+
| `ae-cli kb candidate changes` | Changed sources, Wiki and Index; optional bounded text diff | read |
|
|
12
|
+
| `ae-cli kb candidate read` | Immutable candidate file | read |
|
|
13
|
+
| `ae-cli kb candidate discard` | Close a confirmed failed candidate | high-risk-write |
|
|
14
|
+
| `ae-cli kb +rm-source` | Delete a source, optionally cancelling its pending candidate | high-risk-write |
|
|
15
|
+
| `ae-cli kb read-context create` | Pin authorized formal views | read |
|
|
16
|
+
| `ae-cli kb read-context release` | Release the current user's context | write |
|
|
17
|
+
|
|
18
|
+
- `ae-cli kb publication get --name <verified-name> [--scope personal|company]`: formal version and active candidate summary.
|
|
19
|
+
- `ae-cli kb candidate get --name <verified-name> [--scope ...] --candidate <id>`: candidate metadata.
|
|
20
|
+
- `ae-cli kb candidate read --name <verified-name> [--scope ...] --candidate <id> --path <verified-path>`: immutable candidate index.md, CLAUDE.md or wiki Markdown. Candidate preview requires current content-write permission; it is not a formal query.
|
|
21
|
+
- `ae-cli kb candidate discard --name <verified-name> [--scope ...] --candidate <id> --yes`: `high-risk-write`; discard a candidate after a confirmed retryable publication failure. Pending requests use approval cancel. A running or uncertain publication cannot be discarded.
|
|
22
|
+
- When source deletion conflicts with a pending candidate, explicitly combine `ae-cli kb +rm-source --name <verified-name> --id <source-id> --cancel-candidate --expected-candidate-id <id> --yes`. This is `high-risk-write`; both flags are required. A changed candidate causes a conflict; inspect again instead of silently deleting it.
|
|
23
|
+
|
|
24
|
+
## Review compilation changes
|
|
25
|
+
|
|
26
|
+
```bash
|
|
27
|
+
ae-cli kb candidate changes --name 'Verified KB' --scope company --candidate '<candidateId>'
|
|
28
|
+
ae-cli kb candidate changes --name 'Verified KB' --scope company --candidate '<candidateId>' --entry-id '<entryId>'
|
|
29
|
+
ae-cli kb candidate changes --name 'Verified KB' --scope company --candidate '<candidateId>' --entry-id '<entryId>' --file '<returned-member-path>' --context 3
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
The list returns `candidateId`, the fixed `baseline`, `status`, `firstPublication`, `entries` and `summary`. Only changed sources, Wiki pages and Index are included; compilation rules and historical comparison are outside this preview. Copy entry IDs and member paths from actual responses. `--file` requires `--entry-id`; `--context` is 0 (default, changes only) or 3 (adjacent unchanged lines).
|
|
33
|
+
|
|
34
|
+
Entry details include changed source `fields`, changed `files`, `selectedFile`, `preview` and bounded `hunks`. The first changed file is selected by default; choose other returned members explicitly. Binary files return metadata; text over 1 MB or a complex diff has an explicit preview status. These states are not proof of unchanged contents. A legacy publication without a reconstructible frozen baseline returns `status=baseline_unavailable` and `summary=null`; missing/corrupt snapshots, forbidden access and invalid selections fail explicitly. Never substitute current working files. Successful first publication is shown entirely as added. The endpoint requires the same current content-write permission as candidate reads. Existing `candidate get/read` calls remain unchanged. Unsupported older Hosts return an error; do not fall back to comparing mutable content.
|
|
35
|
+
|
|
36
|
+
`--dry-run` shows the GET path and validated selectors without remote access. This new command uses the same Transitional admission and migration exit below.
|
|
37
|
+
|
|
38
|
+
## Query a stable formal view
|
|
39
|
+
|
|
40
|
+
`ae-cli kb +list --view published` creates a temporary context, queries, then releases it. Old Hosts without context support fail explicitly. The legacy `kb +list` default remains `buildStatus=compiled`; published/contextual list omits only this implicit filter. Explicit `--build-status` still applies.
|
|
41
|
+
|
|
42
|
+
For several related reads:
|
|
43
|
+
|
|
44
|
+
```bash
|
|
45
|
+
ae-cli kb read-context create --sources '[{"scope":"company","name":"Verified KB"}]'
|
|
46
|
+
ae-cli kb +index --name 'Verified KB' --scope company --read-context '<readContextId>'
|
|
47
|
+
ae-cli kb +read --name 'Verified KB' --scope company --path 'wiki/verified.md' --read-context '<readContextId>'
|
|
48
|
+
ae-cli kb read-context release --id '<readContextId>'
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
Create accepts optional `--sources` array of `{scope,name}` and `--locale zh|en|ja|ko`. Omission discovers currently authorized formal KBs. Contexts last 15 minutes and bind list/index/grep/read to the selected formal versions. Sources, permissions, KB lifecycle and actor identity remain checked on every read. Release is owner-only and idempotent for an expired context. Do not manufacture version IDs, storage keys or context IDs.
|
|
52
|
+
|
|
53
|
+
For an expired or mismatched context, stop the sequence. A new explicit sequence may create a new context, but must not mix its results with prior context results. New snapshot-mode KBs with no published version are unpublished; an inconsistent formal pointer fails rather than falling back to working files.
|
|
54
|
+
|
|
55
|
+
Inside an Agent/Ask execution the runtime supplies a private execution context. The CLI automatically uses its scoped credential for the four query commands; do not read, print, copy or replace this credential. Missing/expired/mismatched runtime credentials fail closed, including on 401. Management commands continue using normal CLI authorization. Runtime capability negotiation is protocol validation, not a user feature toggle.
|
|
56
|
+
|
|
57
|
+
## Copy and export
|
|
58
|
+
|
|
59
|
+
A new company-copy approval freezes the current personal formal version when its Effect first executes. Retries reuse the same version and target identity. Later source changes do not alter the copy.
|
|
60
|
+
|
|
61
|
+
`ae-cli agent resource-package export --view working|published ...` defaults to working. Working exports preserve editable sources/schema and omit stale generated wiki. Published exports use one frozen formal snapshot and matching source metadata. Export retries reuse the frozen dependency closure. Editable imports start with pending sources and no published version; read-only compiled imports keep their established serving behavior.
|
|
62
|
+
|
|
63
|
+
## Command admission
|
|
64
|
+
|
|
65
|
+
- Classification: Transitional.
|
|
66
|
+
- Owning module: te-claude knowledge-base publication and query-context management; CLI owner: te-cli te-kb.
|
|
67
|
+
- Current transport: authenticated External REST through `kbApi`; runtime query calls additionally require the scoped context credential and header.
|
|
68
|
+
- Gateway target: an equivalent KB publication/candidate/read-context capability with the same current-editor checks, immutable candidates, idempotency, and fail-closed query binding; no equivalent is currently registered.
|
|
69
|
+
- Review after: 2026-11-25.
|
|
70
|
+
- Exit condition: when Gateway provides these semantics, migrate value-adding typed commands to L2 and simple wrappers to L3. Preserve existing command compatibility during migration.
|
|
71
|
+
- Write safety: candidate discard and combined source deletion require explicit `--yes`, current write permission and candidate identity checks. Context release is owner-scoped and idempotent; request preview is local and does not claim remote validation.
|
|
72
|
+
|
|
73
|
+
### Large change sets and first publication
|
|
74
|
+
|
|
75
|
+
Use `kb candidate changes --group sources|wiki|index --search <name-or-path> --change added|modified|deleted --offset 0 --limit 50` for a bounded directory page. `summary` and `groups` describe all changes; `page.total` describes entries matching the filters. Omitting these flags preserves the existing full-list call. A detail request uses `--entry-id`; `--file-search`, `--file-offset` and `--file-limit` page the changed members. `--file` selects an exact changed member independent of member pagination.
|
|
76
|
+
|
|
77
|
+
The first successful source-managed KB compilation now publishes automatically, including company KBs. Later company updates require approval. Empty version pointers alone do not prove a first publication: legacy formal content and historical versions still require review. Existing eligible first-publication pending requests are withdrawn by the server with a system audit and the same frozen candidate is published; do not approve or create another compilation to force this transition. Already decided requests are not converted. `publication get` and `candidate get` remain read-only and expose the current publication state; an automatically published candidate is still readable by its known ID.
|
|
78
|
+
|
|
79
|
+
Text and Markdown members return changed hunks. Small, unambiguous CSV/TSV changes may additionally include `table` containing only changed rows. ZIP and URL/Feishu sources expose changed frozen members; PDF, Office, images, invalid UTF-8 and oversized content explicitly return metadata/limit status rather than a fabricated textual diff. No source URL is fetched during preview.
|
|
80
|
+
|
|
81
|
+
Compatibility: deploy the matching server before using new paging flags. Old CLI requests continue to work on the new server. New flags against older servers fail explicitly with `KB_CANDIDATE_PAGING_UNSUPPORTED` when `page` or `filePage` is missing; the CLI never treats an unfiltered old response as a filtered result.
|
|
@@ -140,7 +140,20 @@ A revision or access-context change requires revalidation, not silent reuse.
|
|
|
140
140
|
5. **Assess coverage, then answer or iterate.** Map the user's question into
|
|
141
141
|
subquestions and check each one against the sections you actually read. If a
|
|
142
142
|
subquestion is covered, answer with citations (knowledge base + page path +
|
|
143
|
-
section
|
|
143
|
+
section, plus the returned `readView.versionId/versionNumber` for snapshots).
|
|
144
|
+
For `version_pinned` results, describe consistency within that captured view.
|
|
145
|
+
Unchanged `versionId/readIdentity` proves that the reads used the same snapshot;
|
|
146
|
+
it does not establish whether a newer publication or rollback occurred outside
|
|
147
|
+
the execution. Claims about the latest formal version require separate current
|
|
148
|
+
publication evidence obtained through an authorized management workflow.
|
|
149
|
+
Preserve version identity with the reference; a fresh read of the same path is
|
|
150
|
+
not a historical citation read. For completed Ask sources with `reference`, use
|
|
151
|
+
`ae-cli kb +ask-status --execution-id <reference.executionId> --source-index <reference.sourceIndex>`
|
|
152
|
+
to reopen the original version under current permissions. An unversioned source
|
|
153
|
+
has no guaranteed historical snapshot; do not label its current content historical.
|
|
154
|
+
Historical source reads are rejected inside a required fixed-context execution;
|
|
155
|
+
never remove that context or fall back to an ordinary token to bypass its KB set.
|
|
156
|
+
If a gap remains, go back to step 2 with another set of `--paths`
|
|
144
157
|
**copied** from current navigation, or return to discovery for the next candidate.
|
|
145
158
|
For a knowledge-only question, missing evidence after the allowed search
|
|
146
159
|
means "not covered by the knowledge base" — do not attach a confident fact
|