@thinkingai/ae-cli 6.1.27 → 6.1.28

package/skills/ae-evaluation/references/connections.md

@@ -0,0 +1,78 @@
+# Models, agents, and credentials
+
+Commands use `ae-cli evaluation +<command>`. Read the shared endpoint, authentication, and mutation rules in [SKILL.md](../SKILL.md). Use command `--help` for optional flags.
+
+### Model configs
+
+```bash
+ae-cli evaluation +list_model_configs
+ae-cli evaluation +get_model_config --id <model-config-uuid>
+ae-cli evaluation +create_model_config --name "<name>" --provider OPENAI_COMPATIBLE_CHAT|ANTHROPIC_CHAT|GEMINI_CHAT|AZURE_OPENAI_CHAT|TE_CLAUDE_WEB \
+  --model-name "<model>" --base-url "<url>" [--parameters-json '<json>'] [--capabilities-json '<json>'] [--credential-id <uuid>]
+ae-cli evaluation +update_model_config --id <model-config-uuid> [same fields as +create_model_config] --lock-version <n>
+ae-cli evaluation +delete_model_config --id <model-config-uuid> --lock-version <n>
+ae-cli evaluation +test_model_config_connectivity --id <model-config-uuid>
+ae-cli evaluation +transfer_model_config_owner --id <model-config-uuid> --owner-user-id <user-uuid> --lock-version <n>
+```
+
+- `--capabilities-json` must set `chat_completions: true` and `max_context_tokens >= 1`, otherwise `MODEL_CONFIG_INVALID`.
+- Use `MODEL_PARAMETERS_V2` for new model configs. It omits `max_output_tokens`; output limits are controlled by the provider. Historical frozen snapshots may still contain V1.
+- `+update_model_config` is full-replace: `+get_model_config` first and pass every field back.
+- The platform validates outbound reachability at create time; unreachable base URLs are rejected.
+
+### Agents
+
+```bash
+ae-cli evaluation +list_agents
+ae-cli evaluation +get_agent --id <agent-uuid>
+ae-cli evaluation +create_agent --name "<name>" --endpoint-url "<url>" \
+  --input-schema-json '{"schema_version":"AGENT_INPUT_SCHEMA_V1","type":"object","properties":{...},"required":[...]}' \
+  --config-json '<AGENT_CONFIG_V* json>' \
+  [--connector-type HTTP_JSON_V1|OPENCODE_SERVER|OPENCLAW_GATEWAY_V1|HERMES_RUNS_V1|TE_CLAUDE_WEB_V1] [--credential-id <uuid>]
+ae-cli evaluation +update_agent --id <agent-uuid> [same fields as +create_agent] --lock-version <n>
+ae-cli evaluation +delete_agent --id <agent-uuid> --lock-version <n>
+ae-cli evaluation +test_agent_connectivity --id <agent-uuid>
+ae-cli evaluation +agent_snapshot_preview --id <agent-uuid>
+ae-cli evaluation +transfer_agent_owner --id <agent-uuid> --owner-user-id <user-uuid> --lock-version <n>
+```
+
+- `--config-json` must match the connector type; use `+get_agent` on a working agent as a template. Hermes direct transport uses `AGENT_CONFIG_V2.config.hermes.transport_mode=DIRECT`; use `+list_te_claude_catalog` to discover TE Claude agent/model ids for `TE_CLAUDE_WEB` configs.
+- Under TE authentication, TE Claude model and agent configs bind the login credential automatically; omit `--credential-id`. The flag remains available for legacy non-TE authentication.
+
+### Credentials
+
+```bash
+ae-cli evaluation +list_credentials
+ae-cli evaluation +get_credential --id <credential-uuid>
+ae-cli evaluation +create_credential --name "<name>" --type BEARER_TOKEN|API_KEY_HEADER|AI_MONITOR_SESSION --secret "<secret>" [--header-name "<header>"] \
+  [--data-token "<token>"] [--report-source "<source>"]
+ae-cli evaluation +update_credential --id <credential-uuid> --name "<name>" --lock-version <n>
+ae-cli evaluation +rotate_credential --id <credential-uuid> --secret "<new-secret>" --lock-version <n>
+ae-cli evaluation +disable_credential --id <credential-uuid> --lock-version <n>
+ae-cli evaluation +enable_credential --id <credential-uuid> --lock-version <n>
+ae-cli evaluation +delete_credential --id <credential-uuid> --lock-version <n>
+ae-cli evaluation +transfer_credential_owner --id <credential-uuid> --owner-user-id <user-uuid> --lock-version <n>
+```
+
+- Credential enabling/disabling is high-risk and requires confirmation. Non-interactive calls require explicit authorization and global `--yes`; `--dry-run` remains a local preview.
+- Evaluation error output redacts submitted secrets and session tokens, including JSON-escaped and URL-encoded forms. Remote error messages and hints are not written to CLI logs.
+- Secrets are write-only: sent once on create/rotate, never returned or logged by the CLI.
+- `AI_MONITOR_SESSION` credentials additionally require `--data-token` (max 512) and `--report-source` (max 128); all three are sensitive — dry-run redacts them and they must never be logged.
+
+### Bring your own model and agent
+
+Credentials are prerequisites for model configs and agents:
+
+```bash
+ae-cli evaluation +create_credential --name "my-ark-token" --type BEARER_TOKEN --secret "<secret>"
+ae-cli evaluation +create_model_config --name "my-judge" --provider OPENAI_COMPATIBLE_CHAT \
+  --model-name "kimi-k3" --base-url "http://10.0.0.1:3003" \
+  --credential-id <credential-uuid> \
+  --parameters-json '{"schema_version":"MODEL_PARAMETERS_V2","temperature":0,"top_p":1,"seed":1,"timeout_ms":120000,"max_retries":2,"structured_output":"AUTO"}' \
+  --capabilities-json '{"schema_version":"MODEL_CAPABILITIES_V1","chat_completions":true,"json_schema":false,"seed":true,"usage":true,"max_context_tokens":8192}'
+ae-cli evaluation +test_model_config_connectivity --id <model-config-uuid>
+```
+
+To change one setting (e.g. timeout), `+update_model_config` is full-replace: `+get_model_config`, edit the JSON, pass every field back.
+
+Ownership transfers and credential deletion require ADMIN. TE Claude catalog discovery is documented in [integrations.md](integrations.md).

package/skills/ae-evaluation/references/datasets.md

@@ -0,0 +1,57 @@
+# Datasets
+
+Commands use `ae-cli evaluation +<command>`. Read the shared endpoint, authentication, and mutation rules in [SKILL.md](../SKILL.md). Use command `--help` for optional flags.
+
+### Datasets
+
+```bash
+ae-cli evaluation +list_datasets [--search <text>] [--status CREATING|READY|IMPORT_FAILED] [--cursor <cursor>] [--page-size 50]
+ae-cli evaluation +get_dataset --id <dataset-uuid>
+ae-cli evaluation +create_dataset --name "<name>" [--description "..."]
+ae-cli evaluation +import_dataset --name "<name>" --file <path> [--profile GENERIC|AGENT_TASK_CASES_V1] [--description "..."] --idempotency-key "<fresh-key>"
+ae-cli evaluation +dataset_import_status --id <dataset-uuid>
+ae-cli evaluation +dataset_import_errors --id <dataset-uuid>
+ae-cli evaluation +update_dataset --id <dataset-uuid> --name "<name>" [--description "..."] --lock-version <n>
+ae-cli evaluation +delete_dataset --id <dataset-uuid> --lock-version <n>
+ae-cli evaluation +transfer_dataset_owner --id <dataset-uuid> --owner-user-id <user-uuid> --lock-version <n>
+ae-cli evaluation +list_dataset_fields --id <dataset-uuid>
+ae-cli evaluation +create_dataset_field --id <dataset-uuid> --display-name "<name>" --data-type TEXT|NUMBER|BOOLEAN|JSON|DATE|TIME
+ae-cli evaluation +batch_create_dataset_fields --id <dataset-uuid> --fields-json '[{"display_name":"Question","data_type":"TEXT"}]'
+ae-cli evaluation +update_dataset_field --id <dataset-uuid> --field-id <field-uuid> --display-name "<name>" [--ordinal 0] --lock-version <n>
+ae-cli evaluation +delete_dataset_field --id <dataset-uuid> --field-id <field-uuid> --lock-version <n>
+ae-cli evaluation +reorder_dataset_fields --id <dataset-uuid> --field-ids '["<uuid-1>","<uuid-2>"]' --lock-version <n>
+ae-cli evaluation +list_dataset_cases --id <dataset-uuid> [--cursor <cursor>] [--page-size 50]
+ae-cli evaluation +get_dataset_case --id <dataset-uuid> --case-id <case-uuid>
+ae-cli evaluation +create_dataset_case --id <dataset-uuid> --data '{"input":"...","expected":"..."}'
+ae-cli evaluation +batch_create_dataset_cases --id <dataset-uuid> --cases-json '[{"data":{"input":"...","expected":"..."}}]'
+ae-cli evaluation +update_dataset_case --id <dataset-uuid> --case-id <case-uuid> --data '{...}' --lock-version <n>
+ae-cli evaluation +delete_dataset_case --id <dataset-uuid> --case-id <case-uuid> --lock-version <n>
+ae-cli evaluation +export_dataset --id <dataset-uuid> --export-format csv|jsonl
+ae-cli evaluation +list_dataset_versions --id <dataset-uuid>
+ae-cli evaluation +publish_dataset_version --id <dataset-uuid> --lock-version <n>
+ae-cli evaluation +get_dataset_version --id <dataset-uuid> --version-id <dataset-version-uuid>
+ae-cli evaluation +list_dataset_references --id <dataset-uuid>
+```
+
+- `--field-ids` must contain every field UUID exactly once, in the desired order.
+- Case `--data` is keyed by `field_key` from `+list_dataset_fields`. Single-case `--data` and batch `--cases-json` are redacted from command logs. Sensitive JSON parse errors omit input content; successful requests and explicit dry-run previews retain the supplied case data.
+- Batch field creation accepts 1-100 definitions and batch case creation accepts 1-1000 definitions. Each batch is transactional: either every item is created or none are.
+- Field mutations use the dataset lock_version; case mutations use the case lock_version from `+get_dataset_case`.
+
+### Import a dataset and wait for readiness
+
+Import requires a fresh Idempotency-Key and returns 202 with the created dataset; the pipeline is asynchronous:
+
+```bash
+ae-cli evaluation +import_dataset \
+  --name "release-checkout" \
+  --file ./cases.csv \
+  --profile GENERIC \
+  --idempotency-key "import-$(date +%s)"
+ae-cli evaluation +dataset_import_status --id <dataset-uuid>   # poll until READY
+ae-cli evaluation +dataset_import_errors --id <dataset-uuid>   # if IMPORT_FAILED
+ae-cli evaluation +list_dataset_fields --id <dataset-uuid>     # field keys for mapping/case data
+```
+
+Download exported files using [files.md](files.md): poll `+get_file`, then
+`ae-cli evaluation file download --id <file-uuid> --output ./dataset.csv`.

package/skills/ae-evaluation/references/evaluators.md

@@ -0,0 +1,81 @@
+# Evaluators and calibration
+
+Commands use `ae-cli evaluation +<command>`. Read the shared endpoint, authentication, and mutation rules in [SKILL.md](../SKILL.md). Use command `--help` for optional flags.
+
+### Evaluators and templates
+
+```bash
+ae-cli evaluation +list_evaluators [--search <text>] [--cursor <cursor>] [--page-size 50]
+ae-cli evaluation +get_evaluator --id <evaluator-uuid>
+ae-cli evaluation +create_evaluator --name "<name>" [--description "..."] [--category SAFETY] \
+  [--prompt "...{{input}} {{output}}..."] [--variables '["input","output"]'] \
+  [--model-config-id <uuid>] [--template-code <code>] [--template-locale zh|en]
+ae-cli evaluation +update_evaluator --id <evaluator-uuid> --name "<name>" --model-config-id <uuid> \
+  [--description "..."] [--category "..."] [--prompt "..."] [--variables '[...]'] --lock-version <n>
+ae-cli evaluation +delete_evaluator --id <evaluator-uuid> --lock-version <n>
+ae-cli evaluation +transfer_evaluator_owner --id <evaluator-uuid> --owner-user-id <user-uuid> --lock-version <n>
+ae-cli evaluation +list_evaluator_templates [--locale zh|en]
+ae-cli evaluation +create_evaluator_template --name "<name>" --category "<category>" --prompt "<prompt>" [--description "..."] [--variables '[...]']
+ae-cli evaluation +disable_evaluator_template --id <template-uuid>
+ae-cli evaluation evaluator-template enable --id <project-template-uuid>
+ae-cli evaluation +list_evaluator_versions --id <evaluator-uuid>
+ae-cli evaluation +publish_evaluator_version --id <evaluator-uuid> --lock-version <n>
+ae-cli evaluation +get_evaluator_health --id <evaluator-uuid>
+ae-cli evaluation +optimize_evaluator_prompt --model-config-id <uuid> --prompt "<prompt>"
+ae-cli evaluation +list_calibration_suites --id <evaluator-uuid>
+ae-cli evaluation +create_calibration_suite --id <evaluator-uuid> --name "<name>" [--description "..."]
+ae-cli evaluation +list_calibration_cases --id <suite-uuid>
+ae-cli evaluation +create_calibration_case --id <suite-uuid> --variables-json '{...}' --expected-score 0.8 --tolerance 0.1 --tags-json '["golden"]'
+ae-cli evaluation +delete_calibration_case --id <case-uuid> --lock-version <n>
+ae-cli evaluation +start_calibration_run --id <suite-uuid>
+ae-cli evaluation +get_calibration_run --id <calibration-run-uuid>
+ae-cli evaluation +list_calibration_results --id <calibration-run-uuid>
+```
+
+- `+create_evaluator` without `--model-config-id` creates a draft; binding a model config via `+update_evaluator` activates it.
+- Prompt placeholders `{{name}}` must match `--variables`.
+
+### Calibrate an evaluator
+
+```bash
+ae-cli evaluation +list_calibration_suites --id <evaluator-uuid>        # use the default suite
+ae-cli evaluation +create_calibration_case --id <suite-uuid> \
+  --variables-json '{"input":"...","expected":"..."}' \
+  --expected-score 0.8 --tolerance 0.1 \
+  --tags-json '["golden"]'
+ae-cli evaluation +start_calibration_run --id <suite-uuid>
+ae-cli evaluation +get_calibration_run --id <calibration-run-uuid>      # poll until terminal
+ae-cli evaluation +list_calibration_results --id <calibration-run-uuid>
+ae-cli evaluation +get_evaluator_health --id <evaluator-uuid>
+```
+
+Prompt optimization consumes model quota and returns a preview; it never overwrites the evaluator. Prefer the default calibration suite; create extra suites only for a concrete need. Template creation and disabling require ADMIN.
+
+### Template language and process scoring
+
+List built-ins with `+list_evaluator_templates --locale en` and create from them
+with `+create_evaluator --template-code <discovered-code> --template-locale en`.
+Both language options accept zh/en and default to zh. Project templates keep
+their original language; an omitted description uses the localized built-in
+description when creating from a built-in. Existing evaluators are unchanged.
+
+`TOOL_TRACE_REASONABLENESS` evaluates recorded tool choices, arguments, step
+relationships and evidence consistency using `input`, `output`, `execution_trace`.
+Discover the template and publish/pin its evaluator version. Map execution_trace
+using FIELD_MAPPING_V6 as shown in [runs.md](runs.md); pair it with a task-completion
+evaluator when final-answer quality also matters. `EVALUATOR_VARIABLES_V3` and
+`EVALUATOR_SNAPSHOT_V3` carry the new process-evidence source types. Create/update
+still use the public `--variables` array of names; do not send the versioned
+snapshot object as that flag.
+
+`evaluator-template enable` is administrator-only, bodyless and idempotent.
+Only project templates can be re-enabled; built-ins and out-of-scope IDs are not
+found. New evaluators use the latest enabled project template version.
+
+Transition status: transitional
+Owning module: Agent Evaluation evaluator templates
+Current transport: authenticated POST /agenteval/api/v1/evaluator-templates/{id}/enable
+Gateway target: TBD (evaluation template lifecycle)
+Review after: 2027-01-21
+Exit condition: replace with L3 when an equivalent capability exists, or retain
+L2 only if typed inputs and compatibility provide additional value.

package/skills/ae-evaluation/references/files.md

@@ -0,0 +1,33 @@
+# Download evaluation files
+
+Use `ae-cli evaluation file download --id <file-uuid> --output <new-path>`.
+Authentication and endpoint rules are in [SKILL.md](../SKILL.md).
+
+1. Queue an export using the dataset or run reference.
+2. Poll `+get_file --id <file-uuid>` until COMPLETED. Resource creation alone is
+   not evidence that the file is ready; failed/expired files cannot be downloaded.
+3. Save the attachment:
+
+```bash
+ae-cli evaluation file download --id <file-uuid> --output ./results.csv --dry-run
+ae-cli evaluation file download --id <file-uuid> --output ./results.csv
+```
+
+The authenticated `/files/{id}/content` route works for local and S3 storage.
+The parent directory must exist. The CLI refuses to overwrite an existing path,
+saves with owner-only permissions, removes failed partial output, and returns
+`{path, byte_size}` in its JSON envelope. Bytes are not printed to stdout.
+Dry-run previews the request and destination without opening a file or requesting
+a session. Downloads do not follow redirects or retry automatically.
+
+The older `+file_download_url` remains available for temporary URLs, but local
+storage returns a server-local `file://` path. Use `file download` for a remote
+client. Do not print signed URLs or private file content into reports.
+
+Transition status: transitional
+Owning module: Agent Evaluation files
+Current transport: authenticated GET /agenteval/api/v1/files/{id}/content
+Gateway target: TBD (evaluation file download capability)
+Review after: 2027-01-21
+Exit condition: migrate to Gateway when authenticated attachment delivery is
+available with equivalent binary output, redaction, and no-overwrite guarantees.

package/skills/ae-evaluation/references/integrations.md

@@ -0,0 +1,39 @@
+# AI Monitor and TE reporting
+
+Commands use `ae-cli evaluation +<command>`. Read the shared endpoint, authentication, and mutation rules in [SKILL.md](../SKILL.md). Use command `--help` for optional flags.
+
+```bash
+ae-cli evaluation +get_te_report_config
+ae-cli evaluation +save_te_report_config --base-url "<url>" --app-id "<id>" [--lock-version <n>]
+ae-cli evaluation +get_ai_monitor_connection
+ae-cli evaluation +save_ai_monitor_connection --base-url "<url>" --credential-id <uuid> --session-claims-json '{...}' [--lock-version <n>]
+ae-cli evaluation +verify_ai_monitor_connection
+ae-cli evaluation +import_ai_monitor_traces --name "<name>" --selection-json '{...}' [--idempotency-key "<key>"]
+ae-cli evaluation +list_te_claude_catalog --endpoint-url "<url>" [--credential-id <legacy-uuid>]
+```
+
+- `+save_te_report_config` / `+save_ai_monitor_connection`: omit `--lock-version` on first create; on update send the current version from the get command (0 is a valid version — never treat it as "unset").
+
+- `--session-claims-json` requires string `user_id`, `te_user_id`, `name`, `company_id` (optional `is_root`, `platform_access`); values are redacted in dry-run output and must not be logged.
+- `+list_te_claude_catalog` uses the login access token automatically under TE authentication. Supply `--credential-id` only for legacy non-TE authentication.
+
+### Import AI Monitor traces into a dataset
+
+The import is asynchronous: the 201 response returns the created dataset and import job — poll `+dataset_import_status` instead of treating creation as completion. `--session-claims-json` and selection content are identity/permission sensitive; never log or echo them.
+
+```bash
+ae-cli evaluation +get_ai_monitor_connection
+ae-cli evaluation +save_ai_monitor_connection \
+  --base-url "https://ai-monitor.example.com" \
+  --credential-id <ai-monitor-session-credential-uuid> \
+  --session-claims-json '{"user_id":"...","te_user_id":"...","name":"...","company_id":"..."}' \
+  --lock-version <n>
+ae-cli evaluation +verify_ai_monitor_connection
+ae-cli evaluation +import_ai_monitor_traces \
+  --name "ai-monitor-traces" \
+  --selection-json '{"mode":"TRACE_IDS","trace_ids":["trace-1"]}' \
+  --idempotency-key "aim-$(date +%s)"
+ae-cli evaluation +dataset_import_status --id <dataset-uuid>   # poll until READY
+```
+
+FILTER mode uses `{"mode":"FILTER","start_time":"<date>","end_time":"<date>"}` with optional `agent`, `channel`, `search_term`. Both dates must be valid calendar dates in `YYYY-MM-DD` format, with `end_time` not preceding `start_time`; impossible dates such as February 30 are rejected locally. TRACE_IDS carries 1-1000 ids.

package/skills/ae-evaluation/references/runs.md

@@ -0,0 +1,160 @@
+# Evaluation runs and results
+
+Commands use `ae-cli evaluation +<command>`. Read the shared endpoint, authentication, and mutation rules in [SKILL.md](../SKILL.md). Use command `--help` for optional flags.
+
+Run cancellation is a high-risk write and requires confirmation. Non-interactive cancellation requires explicit authorization and global `--yes`; use `--dry-run` to preview without cancelling.
+
+### Evaluation runs
+
+```bash
+ae-cli evaluation +list_runs [--search <text>] [--status <status>] [--cursor <cursor>] [--limit 50]
+ae-cli evaluation +get_run --id <run-uuid>
+ae-cli evaluation +create_run --name "<name>" \
+  [--dataset-id <uuid>] --dataset-version-id <dataset-version-uuid> \
+  [--execution-mode LIVE_AGENT|PRESET_RESULT] [--subject-type AGENT|MODEL] \
+  [--agent-config-id <uuid>] [--model-config-id <uuid>] \
+  [--report-to-te true] \
+  [--field-mapping-json '<json>'] [--result-filter-fields-json '<json>'] \
+  [--pass-threshold 0.8] [--case-sample-size 100] [--idempotency-key "<key>"]
+ae-cli evaluation +update_run --id <run-uuid> --name "<name>" [same optional fields as +create_run] --lock-version <n>
+ae-cli evaluation +delete_run --id <run-uuid> --lock-version <n>
+ae-cli evaluation +select_run_evaluators --id <run-uuid> --evaluator-ids '["<evaluator-uuid>"]' \
+  [--evaluator-versions-json '[{"evaluator_id":"<uuid>","version_id":"<uuid>"}]'] --lock-version <n>
+ae-cli evaluation +list_run_evaluators --id <run-uuid>
+ae-cli evaluation +update_run_field_mapping --id <run-uuid> --field-mapping-json '<json>' --lock-version <n>
+ae-cli evaluation +update_run_result_filter_fields --id <run-uuid> --result-filter-fields-json '{"fields":[],"schema_version":"RESULT_FILTER_FIELDS_V1"}' --lock-version <n>
+ae-cli evaluation +validate_run --id <run-uuid>
+ae-cli evaluation +start_run --id <run-uuid> --lock-version <n>
+ae-cli evaluation +cancel_run --id <run-uuid> --lock-version <n>
+ae-cli evaluation +retry_failed_run_cases --id <run-uuid> --lock-version <n>
+ae-cli evaluation +run_overview --id <run-uuid>
+ae-cli evaluation +run_distribution --id <run-uuid> [--run-evaluator-id <uuid>]
+ae-cli evaluation +run_cases --id <run-uuid> \
+  [--status <status>] [--evidence-completeness NONE|PARTIAL|COMPLETE|UNKNOWN] [--search <text>] \
+  [--evaluation-status <status>] [--score-status <status>] [--evaluator-id <uuid>] \
+  [--evaluator-ids '["<uuid>"]'] [--min-score 0.8] [--max-score 1] [--score-gte 0.2] [--score-lte 0.8] \
+  [--field-filters-json '[...]'] [--sort AVERAGE_SCORE] [--sort-evaluator-id <uuid>] [--order desc] \
+  [--cursor <cursor>] [--limit 50] [--offset 0]
+ae-cli evaluation +get_run_case --id <run-uuid> --case-run-id <case-run-uuid>
+ae-cli evaluation +get_run_case_trace --id <run-uuid> --case-run-id <case-run-uuid>
+ae-cli evaluation +retry_run_case_trace --id <run-uuid> --case-run-id <case-run-uuid>
+ae-cli evaluation +copy_run_case --id <run-uuid> --case-run-id <case-run-uuid>
+ae-cli evaluation +transfer_run_owner --id <run-uuid> --owner-user-id <user-uuid> --lock-version <n>
+ae-cli evaluation +list_project_members
+ae-cli evaluation +create_next_task_version_draft --id <run-uuid>
+ae-cli evaluation +list_evaluation_task_versions --task-id <task-uuid>
+ae-cli evaluation +export_run_results --id <run-uuid> [--export-format CSV|JSONL] --idempotency-key "<fresh-key>"
+ae-cli evaluation +export_run_low_scores --id <run-uuid> --max-score 0.5 [--min-score 0] [--status SUCCEEDED] [--export-format CSV|JSONL] --idempotency-key "<fresh-key>"
+ae-cli evaluation file download --id <file-uuid> --output ./results.csv
+ae-cli evaluation +get_file --id <file-uuid>
+```
+
+- Field mapping JSON follows the platform `FIELD_MAPPING_V*` contract (`FIELD_MAPPING_V5` adds `TEMPLATE` input and `AGENT_TOOL_SUMMARY` / `AGENT_ARTIFACT_SUMMARY` / `AGENT_EXECUTION_STATUS` evidence sources); result-filter-fields is `RESULT_FILTER_FIELDS_V1` with at most 5 fields. Do not guess shapes — inspect an existing run with `+get_run` for a working example.
+- `--case-sample-size` randomly selects 1-1000 cases when the run starts; omit it to run every case. Start fails with `RUN_SAMPLE_SIZE_EXCEEDS_CASES` when the requested sample exceeds the selected dataset version's case count.
+- Start prerequisites: bound evaluator(s) (`EVALUATOR_NOT_ACTIVE`), complete field mapping (`FIELD_MAPPING_INCOMPLETE`), and result filter fields — an explicit empty filter is required (`RESULT_FILTER_FIELD_UNAVAILABLE`).
+- Retry semantics: `+retry_failed_run_cases` requeues only Agent-stage `FAILED|BLOCKED|UNKNOWN` cases in the same immutable snapshot, bumps the execution epoch and recomputes aggregates. `+retry_run_case_trace` is ADMIN-only and only for retryable AI Monitor-backed captures.
+- `+create_next_task_version_draft` copies `case_sample_size` and selects the latest published evaluator versions for the new draft. Run and overview responses expose separate succeeded, failed, and cancelled case/score counts for progress tracking.
+
+### Create and run an evaluation
+
+1. Pick a published dataset version and create a draft run from it plus an agent or model:
+
+```bash
+ae-cli evaluation +list_dataset_versions --id <dataset-uuid>
+ae-cli evaluation +create_run \
+  --name "release-checkout-run" \
+  --dataset-id <dataset-uuid> \
+  --dataset-version-id <dataset-version-uuid> \
+  --execution-mode LIVE_AGENT \
+  --subject-type AGENT \
+  --agent-config-id <agent-uuid>
+```
+
+For scoring pre-stored dataset fields without calling an Agent/Model, use `--execution-mode PRESET_RESULT` (then omit `--agent-config-id` / `--model-config-id`).
+
+2. Configure the draft. Before `+start_run` the run needs: at least one ACTIVE evaluator (pin published evaluator versions when possible), a complete field mapping, and result filter fields (an explicit empty filter is fine). Each mutation consumes one lock_version — re-read with `+get_run` in between:
+
+```bash
+ae-cli evaluation +list_evaluator_versions --id <evaluator-uuid>
+ae-cli evaluation +select_run_evaluators --id <run-uuid> \
+  --evaluator-ids '["<evaluator-uuid>"]' \
+  --evaluator-versions-json '[{"evaluator_id":"<evaluator-uuid>","version_id":"<evaluator-version-uuid>"}]' \
+  --lock-version <n>
+ae-cli evaluation +get_run --id <run-uuid>
+ae-cli evaluation +update_run_field_mapping --id <run-uuid> \
+  --field-mapping-json '{"schema_version":"FIELD_MAPPING_V5","agent":{"input":{"field_key":"input","source_type":"DATASET_FIELD"}},"evaluators":{"<evaluator-uuid>":{"input":{"field_key":"input","source_type":"DATASET_FIELD"},"output":{"source_type":"AGENT_OUTPUT"}}}}' \
+  --lock-version <n>
+ae-cli evaluation +get_run --id <run-uuid>
+ae-cli evaluation +update_run_result_filter_fields --id <run-uuid> \
+  --result-filter-fields-json '{"fields":[],"schema_version":"RESULT_FILTER_FIELDS_V1"}' \
+  --lock-version <n>
+```
+
+3. Validate (optional, same preflight as start, no lock_version needed), then start:
+
+```bash
+ae-cli evaluation +validate_run --id <run-uuid>
+ae-cli evaluation +get_run --id <run-uuid>
+ae-cli evaluation +start_run --id <run-uuid> --lock-version <n>
+```
+
+4. Watch the results and drill into failures:
+
+```bash
+ae-cli evaluation +run_overview --id <run-uuid>
+ae-cli evaluation +run_distribution --id <run-uuid> --run-evaluator-id <run-evaluator-uuid>
+ae-cli evaluation +run_cases --id <run-uuid> --status FAILED --evaluation-status FAILED
+ae-cli evaluation +get_run_case --id <run-uuid> --case-run-id <case-run-uuid>
+ae-cli evaluation +get_run_case_trace --id <run-uuid> --case-run-id <case-run-uuid>
+ae-cli evaluation +retry_failed_run_cases --id <run-uuid> --lock-version <n>
+```
+
+### Export results and download
+
+```bash
+ae-cli evaluation +export_run_results --id <run-uuid> --export-format CSV --idempotency-key "export-$(date +%s)"
+ae-cli evaluation +export_run_low_scores --id <run-uuid> --max-score 0.5 --idempotency-key "low-$(date +%s)"
+# Poll the returned file until COMPLETED before downloading.
+ae-cli evaluation +get_file --id <file-uuid>
+ae-cli evaluation file download --id <file-uuid> --output ./results.csv
+```
+
+Traces are available only for LIVE_AGENT runs with subject AGENT. Capture status is independent of case/run success. Captures contain at most 500 spans and may be truncated; PARTIAL or reference-only evidence must not be described as complete. Never include hidden chain-of-thought, credentials, cookies, auth headers, or unredacted tool input/output in reports. Ownership transfer and project-member discovery require ADMIN.
+
+### Process evidence mapping (FIELD_MAPPING_V6)
+
+For LIVE_AGENT runs whose subject is AGENT, V6 adds `AGENT_EXECUTION_TRACE` and
+`AGENT_TOOL_CALLS`. Keep the existing V5 mappings for final-output-only scoring.
+Use real dataset field keys, evaluator IDs and a published evaluator version:
+
+```json
+{
+  "schema_version": "FIELD_MAPPING_V6",
+  "agent": {"input": {"source_type": "DATASET_FIELD", "field_key": "input"}},
+  "evaluators": {
+    "<evaluator-uuid>": {
+      "input": {"source_type": "DATASET_FIELD", "field_key": "input"},
+      "output": {"source_type": "AGENT_OUTPUT"},
+      "execution_trace": {"source_type": "AGENT_EXECUTION_TRACE"}
+    }
+  }
+}
+```
+
+An evaluator declaring `tool_calls` may map it to `{ "source_type": "AGENT_TOOL_CALLS" }`.
+Only map variables the evaluator declares. Send the complete mapping through
+`+update_run_field_mapping` with the latest lock version, validate, then start.
+
+The process projection is `PROCESS_EVIDENCE_V1`; TE Claude A2A tool artifacts use
+`TE_CLAUDE_TOOL_CALLS_V1`. Scoring requires a READY/COMPLETE/non-truncated snapshot
+from the current execution epoch and limits the projection to 64 KiB. Missing,
+pending or unusable evidence fails only the dependent score with a null score;
+Agent status stays independent and final-output evaluators can still score.
+
+Score `input_variables` holds resource references (`content_location=CASE_EXECUTION_TRACE`),
+not full tool payloads. Inspect `+get_run_case_trace` separately. TE Claude evidence
+covers recorded tool/sub-Agent calls and explicit parent-child relationships,
+not hidden model reasoning, delegated internal steps, timing or cost. Preserve
+redaction and completeness limits when reporting results.
+
+See [files.md](files.md) for authenticated downloads and file readiness checks.

package/skills/ae-evaluation/references/schedules.md

@@ -0,0 +1,31 @@
+# Scheduled evaluations
+
+Commands use `ae-cli evaluation +<command>`. Read the shared endpoint, authentication, and mutation rules in [SKILL.md](../SKILL.md). Use command `--help` for optional flags.
+
+### Schedules
+
+```bash
+ae-cli evaluation +list_schedules [--status PENDING|PROCESSING|COMPLETED|FAILED|CANCELLED] [--cursor <cursor>] [--limit 50]
+ae-cli evaluation +create_schedule --run-id <run-uuid> --frequency HOURLY|DAILY|WEEKLY|MONTHLY --time-zone <iana> \
+  [--minute-of-hour 0] [--hour-of-day 9] [--day-of-week 1] [--day-of-month 1]
+ae-cli evaluation +update_schedule --id <schedule-uuid> --frequency <freq> --time-zone <iana> \
+  [--minute-of-hour 0] [--hour-of-day 9] [--day-of-week 1] [--day-of-month 1] --lock-version <n>
+ae-cli evaluation +delete_schedule --id <schedule-uuid> --lock-version <n>
+```
+
+HOURLY runs at minute 0 and must omit `--minute-of-hour` (or send 0). DAILY/WEEKLY/MONTHLY require `--minute-of-hour`. Schedules pin published task versions.
+
+### Scheduled evaluations
+
+```bash
+ae-cli evaluation +list_schedules
+ae-cli evaluation +create_schedule \
+  --run-id <run-uuid> \
+  --frequency DAILY \
+  --time-zone Asia/Shanghai \
+  --hour-of-day 9 \
+  --minute-of-hour 0
+ae-cli evaluation +update_schedule --id <schedule-uuid> --frequency WEEKLY --time-zone Asia/Shanghai \
+  --day-of-week 1 --hour-of-day 9 --minute-of-hour 0 --lock-version <n>
+ae-cli evaluation +delete_schedule --id <schedule-uuid> --lock-version <n>
+```

package/skills/ae-kb/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: ae-kb
-version: 1.0.4
+version: 1.0.6
 description: 'AE knowledge base CLI manual for creating, importing read-only compiled snapshots, querying, LLM-powered ask, listing accessible knowledge bases and their sources, deterministic index/grep/read retrieval, checking status, ZIP source upload and directory management, raw child-file reading, revision-checked child updates and deletion, compiling, schema generation, URL sources, source deletion, ordinary text source creation, full reads and revision-checked editing, file/ZIP source draft replacement and restoration, and knowledge base deletion. Use when the user asks to manage AE (Agentic Engine) knowledge bases, import a compiled Markdown ZIP snapshot, upload documents or URLs to a knowledge base, query knowledge, ask knowledge bases with an LLM, list accessible knowledge bases or source metadata, inspect knowledge base indexes, search knowledge base pages, read a specific knowledge base page, check knowledge base status, generate schemas, compile knowledge, remove sources, create or edit text sources, or delete a knowledge base. To choose which knowledge base is worth searching, use the ae-kb-discovery skill first; this skill runs the retrieval once a target is chosen. Must use ae-cli kb commands and must not guess knowledge base names, scopes, page paths, source IDs, source display names, JSON payload shapes, or URL formats.'
 ---
 
@@ -610,7 +610,6 @@ Transport: `POST /agent/api/external/knowledge-bases/sources/check-updates`, bod
 - Review after: 2026-10-16.
 - Exit condition: once the Gateway exposes equivalent authorization and detector semantics, migrate the typed command to L2 or use L3 if it adds no value.
 
-
 ### Submit an ordinary URL update check
 
 ```bash

package/skills/ae-metadata/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: ae-metadata
-version: 1.0.2
+version: 1.0.4
 description: "AE metadata capability-gateway CLI: list, inspect, create, update, delete and download metadata data tables; bind an existing dimension table to a property or create and bind a CSV dimension table. Local file upload and event/property discovery are separate prerequisite capabilities."
 ---
 
@@ -43,6 +43,7 @@ Safety:
 - **Before any command**, read the matching `references/<name>.md` (filename = command with spaces → underscores, e.g. `metadata data-table list` → `metadata_data_table_list.md`).
 - Never invent `project_id`, event/property names, `input_file_id`, or `data_table_id`. Reuse verified names or discover a metadata lookup capability; find data table IDs via `metadata data-table list`.
 - `metadata data-table download` is an async artifact command: plain invocation submits, `--wait` waits, and `--output <file>` waits then streams atomically. Resume with `analysis run wait`; local interruption never cancels the remote run.
+- The member's natural-day quota counts every Gateway capability `execute` call eligible for usage statistics (an identifiable user and statistics enabled), including metadata-only reads, previews, table/binding writes, asset exports, and downloads that submit work. It does not depend on database engine or business purpose. Cache hits, repeated calls, reused runs, and failures within `execute` count, including schema errors, permission denials, and unknown capability IDs. Gateway capability discovery (list/inspect), validate/dry-run, and separate run-status/artifact-download HTTP endpoints do not count; pagination and cancellation through capability execution do count. On `QUERY_DAILY_LIMIT_EXCEEDED` (HTTP 429), wait until reset or contact an administrator; `QUERY_QUOTA_UNAVAILABLE` fails explicitly. Do not switch identities to bypass these errors. Omit download `--timeout-seconds` to use the server default; inspect the selected capability for its current policy.
 
 ## When to Use
 

package/skills/ae-metadata/references/metadata_data_table_download.md

@@ -13,7 +13,7 @@ ae-cli metadata data-table download --project-id <project_id> --data-table-id <i
 | `--project-id` / `-p` | Yes | Numeric project ID. |
 | `--data-table-id` | Yes | Data table ID. |
 | `--request-id` | No | Optional `cli_<32 lowercase hex>` request ID. |
-| `--timeout-seconds` | No | Remote runtime in seconds, 1 to 21600. |
+| `--timeout-seconds` | No | Omit to use the server default (normally 21600 seconds / 6 hours). This CLI accepts explicit values in `1..21600`. Inspect this capability for the server default/max; use `capability run` with `timeout_seconds` for larger values only when its schema allows them. |
 | `--wait` | No | Wait for run `SUCCEEDED` and artifact `COMPLETED`. |
 | `--output` | No | Wait, then stream and atomically publish the artifact to this file. |
 | `--force` | No | Replace an existing `--output` file atomically; invalid without `--output`. |
@@ -23,3 +23,9 @@ invocation submits only. `--output` implies `--wait`; resume an interrupted
 wait with `ae-cli analysis run wait --run-id <run_id> [--output <file>]`.
 Local interruption never cancels the remote run. Do not use this command for
 inline data-table metadata; use `metadata data-table get` instead.
+
+Read the returned `effective_timeout_seconds`, `timeout_source`, and `deadline_at` for the actual lifecycle deadline. Local waiting has its own bound and can be resumed without resubmitting.
+
+Each Gateway `execute` call consumes one unit of the member's natural-day quota when the user is identifiable and usage statistics are enabled, regardless of database engine or business purpose. Cache hits and failures within `execute` count, including invalid input schemas, permission denials, unknown capability IDs, and accepted work that later fails or times out. Repeating the same input counts again even if a handler reuses an existing run; Gateway has no general `client_request_id` idempotency guarantee. Resume with `analysis run wait` or use `analysis artifact download`: their separate status/download HTTP endpoints do not count. Gateway capability discovery (list/inspect) and validate/dry-run do not count either. Pagination and cancellation do count when invoked through capability execution, such as `analysis.query.cancel`.
+
+`QUERY_DAILY_LIMIT_EXCEEDED` (HTTP 429) requires waiting until reset or contacting an administrator; a blocked call does not increase the used quota but remains eligible for failed-call usage auditing. `QUERY_QUOTA_UNAVAILABLE` fails explicitly. Usage persistence is asynchronous and best effort, so persisted usage-row totals need not equal quota counters. Do not switch identities to bypass quota enforcement.

package/skills/ae-metadata/references/metadata_data_table_get.md

@@ -17,6 +17,8 @@ ae-cli metadata data-table get --project-id <project_id> --data-table-id <id> --
 | `--data-table-id` | Yes | Data table ID from `metadata data-table list`. |
 | `--include-preview` | No | Include preview rows when true. |
 
+Each call to this capability's Gateway `execute` endpoint consumes one unit of the member's natural-day quota when the user is identifiable and usage statistics are enabled. This includes metadata-only reads (`include_preview` omitted or false), previews of any table type, cache hits, repeated calls, and failures within `execute`, including schema errors or permission denials. The database engine does not affect counting. Gateway capability-discovery list/inspect and validate/dry-run endpoints do not count. `QUERY_DAILY_LIMIT_EXCEEDED` (HTTP 429) requires waiting until reset or contacting an administrator; a blocked call does not increase the used quota but remains eligible for failed-call usage auditing. `QUERY_QUOTA_UNAVAILABLE` fails explicitly. Usage persistence is asynchronous and best effort, so persisted usage-row totals need not equal quota counters. Do not change identities to bypass quota enforcement.
+
 ## Request Body
 
 ```json

package/skills/ae-system/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: ae-system
-version: 1.2.0
+version: 1.2.2
 description: "AE Agent system administration CLI for root and agent administrators. Use when the user asks to manage Agent members, sandboxes and shared tools, company model visibility/defaults/pricing, usage statistics and exports, cost quotas, balance alerts, IM channels, channel routing, WhatsApp Web linking, or Feishu user bindings. Must use ae-cli system commands, discover real IDs before writes, and never attempt to bypass a permission denial."
 ---
 

package/skills/ae-team/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: ae-team
-version: 1.0.0
+version: 1.0.2
 description: "ThinkingAI AE (Agentic Engine) ae-cli manual for AI Agent Team tasks: managing teams (list, create, update, delete, AI-generate, templates) and executing TeamRuns (start, chat, cancel, reply, result, artifacts). Use when the user asks to find a team, run a team task, check run status, retrieve results or artifacts, or set up multi-agent workflows. Must use ae-cli, read the matching references/<command>.md before composing commands, and never guess team IDs, run IDs, config structures, or parameter formats."
 ---
 
@@ -113,7 +113,7 @@ ae-cli team +run-start --team-id <team_id> --input "分析上周用户留存数
 # 4. Stream until done (blocks; no polling needed)
 ae-cli team +run-watch --id <run_id>
 # exit 0 → completed/partial_success → go to step 5
-# exit 1 → failed/cancelled → inspect errorMessage in output, report to user
+# exit 1 → inspect the terminal entity or error; for connection errors, use +run-result before considering a new run
 # exit 2 → waiting_user → go to step 4a
 
 # 4a. Handle waiting_user: read pendingQuestion from stdout, get user's answer, reply, re-watch

package/skills/ae-team/references/run-start.md

@@ -16,6 +16,7 @@ Domain: **TeamRun execution / write**
 - `--notification` must be valid JSON if provided, e.g. `'{"channels":["feishu"],"feishuChatId":"..."}'`.
 - This is an ordinary `write` operation and does not require CLI confirmation.
 - After starting, capture the returned `id` (run ID) and use it to poll `+run-result`.
+- `--save-to-kb-id` accepts a personal or company knowledge base ID and maps to the API field `saveToKnowledgeBaseId`. After completion, check `savedKnowledgeBase` and read back the saved source; a successful start alone does not confirm saving.
 
 ## Command
 ```bash