@thinkingai/ae-cli 6.0.55 → 6.0.57

package/skills/ae-data-integration/references/sink-upload.md

@@ -1,5 +1,16 @@
 # Sink — resolve destination and upload
 
+## Choose the sink path first
+
+Two sinks exist, and the choice is the user's — never a default:
+
+| Sink | When | Deliverable |
+| --- | --- | --- |
+| RESTful `/sync_json` | one-time / small load | uploads rows to the receiver directly |
+| File / LogBus2 | recurring / high-volume (daily/weekly exports) | `valid.ue.jsonl` written to a watched directory; **never uploads** |
+
+Ask the user which one applies before resolving any destination. The user supplying an endpoint/APPID does not settle it — a file that will recur usually belongs on LogBus2, and the same rows must not go through both paths. **File / LogBus2 mode skips the rest of this file:** run `convert` with `--output-dir` pointed at the directory LogBus2 watches, then stop — no endpoint, no APPID, no `upload`.
+
 ## Resolve the destination
 
 Use the current AE CLI project commands; never guess IDs:

package/skills/ae-data-integration/references/source-inspect.md

@@ -8,7 +8,7 @@ Confirm:
 2. The desired outcome: AE ingestion, local analysis, or help deciding.
 3. For ingestion, the target environment/project if already known.
 
-Accept one or more CSV, TSV, TXT, JSON, JSONL (NDJSON), XLS, or XLSX files. CSV/TSV/TXT/JSON/JSONL/XLSX have no hard size limit; files over 1 GB print a stderr warning with an estimated processing time and suggest splitting. XLS over 100 MB prints a memory-risk warning (the legacy parser loads the whole workbook, roughly 5-10x file size); XLS over 1 GB is still rejected — convert it to XLSX or split it first.
+Accept one or more CSV, TSV, TXT, JSON, JSONL (NDJSON), XLS, XLSX, or Parquet files. CSV/TSV/TXT/JSON/JSONL/XLSX/Parquet have no hard size limit; files over 1 GB print a stderr warning with an estimated processing time and suggest splitting. XLS over 100 MB prints a memory-risk warning (the legacy parser loads the whole workbook, roughly 5-10x file size); XLS over 1 GB is still rejected — convert it to XLSX or split it first.
 
 XLSX worksheet recognition matches `<sheet>` and namespaced `<x:sheet>` alike: some cleaning/export tools rewrite the default OOXML namespace as a prefix, and the workbook reads the same either way. The worksheet *row* nodes themselves are still read unprefixed, so a workbook whose row data is also namespaced falls back to CSV.
 
@@ -59,6 +59,7 @@ Summarize both — report the shape of the distribution and the names of the val
 - **Summary and total rows** — an exported report ends with a 合计 row, and a grouped one repeats 小计 after every group. Those rows are not observations: uploaded, each becomes an event that never happened whose amount is the whole group's revenue, and profiled, it doubles the column's `sum` and turns its `max` into the total. Nothing in the row itself says so, so inspect flags them under `summary_rows` and warns. Each entry carries `row` (the data-row ordinal, the same numbering `invalid.rows.jsonl` and `--salvage-from` use) and `signals`: `total_label` when a cell reads as a total label (`合计` / `总计` / `小计` / `汇总` as a prefix, `Total` / `Subtotal` / `Sum` as the whole cell), naming the column in `label_column` but never the cell text; `column_total` when a number on that row equals the total of its column's other rows, listing those columns in `total_columns`. A row can raise one signal or both — a labelled group subtotal holds its group's total, not the column's, so only the label fires. Nothing is removed and nothing is changed: `convert` writes these rows as records, and every number reported for their columns counts them, which is exactly why the finding has to be read. There is no flag that drops a data row, because a row labelled `合计` is sometimes a real business record; when the user confirms a row is a total, ask them to remove it from the source file or re-export without it, then inspect again. `convert` repeats the finding in `manifest.output.summary_rows`, since by upload time a subtotal row with a plausible identity and time is indistinguishable from data. This covers every format, `.xls` included — the check reads rows, not worksheet structure.
 - **Repeated business keys** — a customer re-exports a report whose range overlaps the last export, or pastes two sheets together, and the same observation arrives twice. Uploaded, each repeat is a second event: that user's revenue doubles, every funnel counts them twice, and AE appends accepted events with no way to un-send one — so the only place this is fixable is before the upload. Inspect compares each row's business key against the rows before it and reports repeats under `duplicate_keys`, plus a warning. The report carries `key_columns` (the columns actually compared — always read it, since the key is what the finding means), `checked_rows`, `duplicate_groups`, `extra_rows` (surplus records an upload would carry), and `groups`, each with `count`, the data-row `rows` (the numbering `invalid.rows.jsonl` and `--salvage-from` use), and a `key_hash` prefix that distinguishes groups without revealing values — the key's own text is never reported. `groups_truncated` / `rows_truncated` mean the list is bounded, not that the counts are; `tracking_truncated` means distinct keys outran the scan's budget and there may be more. The key comes from the mapping's identity, time, and event-name columns on `convert`, and from column-name matching on `inspect`; a single column is never a key, so a file with no recognizable time column is not scanned at all (identity alone would call every returning user's second row a repeat). Two limits to state when reporting: values are compared as written, so `2026-03-01 10:00:00` and `2026/03/01 10:00:00` are two different keys and a repeat spelled two ways is missed; and a source's own unique key (an order id) is not compared on unless the mapping names it as identity, time, or event. Nothing is removed — a repeat is sometimes a real pair of records, two order lines in the same checkout second — so ask the user whether the rows are separate observations, and if they are not, have them remove the rows from the source file and inspect again. `convert` repeats the finding in `manifest.output.duplicate_keys`, because by upload time both copies are ordinary valid records; it describes the whole source file even on a `--salvage-from` run, since every valid row of that file ends up in AE. This covers every format, `.xls` included — the check reads rows, not worksheet structure.
 - **TSV / TXT** — `.tsv` and `.tab` use a tab delimiter with no quoting convention; `.txt` and unknown extensions are content-sniffed into CSV, TSV, or NDJSON.
+- **Parquet** — `.parquet` / `.pq` files read through the schema (column names come from the schema, there is no header row), so header/title-row handling never applies. 64-bit integer columns arrive as decimal strings (values beyond 2^53 cannot be a JS number), decimals are rendered with their scale, and timestamp columns follow the file's own semantics: a UTC-adjusted timestamp (`isAdjustedToUTC=true`) is an absolute instant, a naive one (`isAdjustedToUTC=false`, e.g. Trino/Spark wall clocks) is read as a wall-clock string that `source_timezone` then applies. Nested list/struct/map columns are kept nested (inferred `list`/`object`), not flattened. **Directories are not accepted** — `--input-file` takes file paths only, so a partitioned dataset (a directory of `part-*.parquet`) must be expanded into its `.parquet` file list first, then passed as repeated `--input-file` values in a stable order (sort them) so row order is deterministic. **Many files, one profile:** when every input is Parquet and there is more than one file, inspect fully profiles only the first file and verifies the rest against it by footer (column names + Arrow types), returning a `schema_checked` shape with the representative profile plus each file's `row_count` / `size_bytes` — schemas are confirmed identical, not values, so a UTF8 column whose values look numeric/datetime/JSON can still infer differently in another file (rare for one export pipeline; a schema mismatch falls back to profiling every file). **Size:** a Parquet file over 1 GB warns about memory (the reader decodes the whole file), so large files should be split by row group or partition.
 - **Encoding** — text files are auto-detected (UTF-8, GBK, GB2312, Big5, and others); no flag is needed.
 - **Excel date cells** — a cell whose number format is a date or date+time is read as the wall-clock timestamp shown in Excel, not as the Excel serial number stored behind it, so the column infers as `datetime` and can serve as the time field. Inspect lists every such column in a warning. Treat that warning as a question to the user, not as a note: the same column profiled as `number` before this behavior existed, so if any part of this file was already sent to AE, the property may have been received as a number and its type is now locked — it cannot be changed to datetime, and the column has to be re-sent under a new property name. Ask whether the column was uploaded before, and only map it once the user answers. Elapsed-duration formats (`[h]:mm:ss` and the equivalent built-ins) are durations rather than points in time and stay `number`.
 - **Excel formula cells** — a spreadsheet stores a formula and, next to it, the result Excel last computed. That cached result is the value: it is read normally, including a result of `0` or `""`, which are real values and not blanks. This tool never evaluates a formula and never guesses a result, so a cell holding a formula the file never computed has nothing to upload; it is read as missing and counted, as is an Excel error value (`#N/A`, `#DIV/0!`, …). Inspect reports the counts per column in a warning and `convert` repeats them in `manifest.output.unreadable_cells`. Report them: the rows are kept and the record count is unchanged, so this is the only explanation for a column that is empty in AE while the spreadsheet looks full. When a column that matters reads as missing this way, ask the user to recalculate and re-export in Excel, or to export values instead of formulas, before uploading. This covers XLSX; a legacy `.xls` workbook goes through a different parser and is not counted here.

package/skills/ae-data-integration/references/transform.md

@@ -29,7 +29,7 @@ Walk the table item by item:
 
 Before saving, present the complete mapping for a final sign-off on one grouped page — system fields (`#type`/mode, `#account_id`, `#distinct_id`, `#time` + source timezone, `#zone_offset`, `#event_name`, `#ip`/`#uuid`), then the property table (source column → target AE name → type), then excluded columns — and wait for an explicit yes. Never dump the raw mapping JSON at the user.
 
-Save the reviewed `ae-data-integration-mapping/v1` JSON to `.ae-cli/data-integration/mapping.json` (project workspace). Convert into a new output directory:
+Save the reviewed `ae-data-integration-mapping/v1` JSON to `.ae-cli/data-integration/mapping.json` (project workspace). Ask the user where the output should land before converting — never pick it silently. Offer the default `.ae-cli/data-integration/runs/<run-id>`; for file / LogBus2 delivery the `--output-dir` must be the directory LogBus2 watches. Convert into a new output directory:
 
 ```bash
 ae-cli data-integration convert \

package/skills/ae-data-integration/references/ue-mapping.md

@@ -27,7 +27,7 @@ Required structure:
 }
 ```
 
-Use `distinct_id_field` when no account field exists. `mixed` requires `record_type_field`. Track requires `event_name_field` or a reviewed `default_event_name`. `source.format` is one of `csv`, `tsv`, `json`, `jsonl`, `xls`, `xlsx`. For a multi-file template mapping, `source.sha256` may be the wildcard `*`.
+Use `distinct_id_field` when no account field exists. `mixed` requires `record_type_field`. Track requires `event_name_field` or a reviewed `default_event_name`. `source.format` is one of `csv`, `tsv`, `json`, `jsonl`, `xls`, `xlsx`, `parquet`. For a multi-file template mapping, `source.sha256` may be the wildcard `*`.
 
 ## Record types
 

package/skills/ae-evaluation/SKILL.md

@@ -0,0 +1,83 @@
+---
+name: ae-evaluation
+version: 2.4.5
+description: "Operate the Agent Evaluation (agent-test) platform through ae-cli: manage evaluation assets, configure and run Agent or preset-output evaluations, and analyze results. Use for platform evaluation tasks and their model, credential, integration, or schedule setup."
+---
+
+# ae-evaluation
+
+Use `ae-cli evaluation +<command> [options]` for existing operations and the resource commands below for new capabilities. All 121 commands target the Agent Evaluation platform.
+AE Agent resource management belongs to `ae-agent`; this skill manages evaluation assets and agents under test.
+
+## Read only the relevant reference
+
+Before executing a command, read the reference for the operation. Do not load all references.
+References contain command inputs, JSON contracts, and task-specific workflows; use command `--help` for additional flags.
+
+| Task | Reference |
+|---|---|
+| Create/import datasets, manage fields and cases, publish versions | [datasets.md](references/datasets.md) |
+| Configure LLM judges/templates, publish evaluator versions, calibrate or optimize prompts | [evaluators.md](references/evaluators.md) |
+| Create/configure/start/retry/cancel runs, inspect results/traces, export results | [runs.md](references/runs.md) |
+| Configure/test models and agents, manage credentials | [connections.md](references/connections.md) |
+| Change resource visibility or administer project members | [access-and-members.md](references/access-and-members.md) |
+| Download imported/exported files to disk | [files.md](references/files.md) |
+| Create/update/delete recurring evaluation schedules | [schedules.md](references/schedules.md) |
+| Connect/import AI Monitor data, configure TE reporting, discover TE Claude catalog | [integrations.md](references/integrations.md) |
+
+## Endpoint and authentication
+
+- In both personal terminals and AE sandboxes, evaluation commands default to the current AE Host and call `<current-host>/agenteval/api/v1/...` (an optional trailing `/agent` is removed from the host). Use `--host` to select another AE Host. Evaluation requests use the HTTP or HTTPS scheme configured for that AE Host, consistent with other AE commands. Legacy `--endpoint` and `AE_EVAL_BASE_URL` values may only match the current AE Host; other values fail before credentials are sent. Remove these settings and use `--host` to select a host.
+- Session exchange and evaluation requests reject redirects; session exchange has a 30-second timeout.
+- Path resource IDs must contain only letters, digits, hyphens, or underscores; separators and URL escape syntax are rejected locally.
+- Use the online platform. Do not suggest localhost/loopback endpoints or obtain tokens from a local `agent-test/.env`.
+- Authentication uses `AE_EVAL_ACCESS_TOKEN` when explicitly set; it is sent only as `Authorization: Bearer` and must never appear in command arguments or output.
+- In an AE sandbox, when `AE_EVAL_ACCESS_TOKEN` is unset, ae-cli automatically exchanges the sandbox-provisioned `cli-token` through te-claude for a short-lived evaluation session. The CLI token is never sent to Agent Evaluation. If the token can access multiple TE projects, set `AE_EVAL_TE_PROJECT_ID` to the authorized TE project id.
+- Outside a sandbox, obtain a session token through the platform web console (`sessionStorage.getItem('agent.evaluation.session-token')`) or `POST /agenteval/api/v1/auth/session`. An unset token produces `EVAL_ACCESS_TOKEN_MISSING`.
+- Successful commands return JSON by default; use the actual error envelope to diagnose failures.
+
+## IDs, versions, and mutations
+
+- Discover real IDs with list/get commands. UUIDs are version-7 strings; never guess IDs, JSON payload shapes, frequency fields, or `lock_version` values.
+- Published dataset, evaluator, and evaluation-task versions are immutable and separate from drafts. Runs and schedules pin versions; existing published versions cannot be overwritten. Publishing unchanged assets returns `ASSET_VERSION_NO_CHANGES`.
+- Read operations can run once inputs are known. Writes require user intent; high-risk operations retain confirmation unless the user authorizes bypassing it with `--yes`.
+- Credential enabling/disabling, run cancellations, deletes, ownership transfers, template disabling, access-scope changes, and member admission/permission changes are high-risk writes. Check command help for risk and role requirements; references identify ADMIN-only operations.
+- Before each mutation requiring `--lock-version`, read the current resource. Every successful mutation increments its version; re-read before the next mutation.
+- Use the corresponding get command for runs, datasets, evaluators, models, agents, credentials, dataset cases, TE report config, and AI Monitor connection. Schedules use `+list_schedules`; calibration cases use `+list_calibration_cases`.
+- Field mutations use the dataset version; case mutations use the case version. First creation of TE report config or AI Monitor connection omits `--lock-version`; updates require the current value, including 0.
+- Async dataset imports, evaluation-set imports, and result exports require a fresh unique `--idempotency-key` per attempt. Poll the relevant status until completion; resource creation alone does not establish readiness.
+- Prefer `--dry-run` before broad/destructive writes whose targets have not been validated. It previews the request without executing it.
+- Evaluation errors redact the submitted credential values and session tokens; server-provided error messages and hints are not persisted in CLI logs. Stable business error codes are preserved unchanged; non-sensitive diagnostics remain available in the JSON error envelope.
+- Case data is redacted from command logs for both single-case and batch writes. Sensitive JSON parse failures report the flag and syntax guidance without echoing the input.
+- Credential secrets are write-only. Never echo secrets, session claims, or sensitive trace evidence; use redacted dry-run output.
+
+## Evaluation lifecycle
+
+1. Select a published dataset version with `+list_dataset_versions`.
+2. Choose `LIVE_AGENT` (default) to call an Agent/model, or `PRESET_RESULT` to score outputs stored in the dataset. PRESET_RESULT must omit agent/model config IDs. `IMPORTED_OUTPUT` is legacy data only; never recommend it.
+3. Create a draft run, bind at least one ACTIVE evaluator (prefer pinned published versions), and configure complete field mapping plus result filter fields. An explicit empty result filter is required when no fields are selected.
+4. Prefer `+validate_run` before starting. It runs the start preflight without a body or lock version. Read the latest run, then `+start_run` with its current lock version.
+5. Inspect overview/distribution, drill into cases, and use the run reference for retry, trace, and export semantics.
+
+Read [runs.md](references/runs.md) for the executable sequence and mapping examples. Setup of datasets, evaluators, or connections needs only the matching reference above.
+
+## Transport transition
+
+The existing platform contract exposes REST operations rather than evaluation
+capabilities registered in this repository. New commands retain that transport
+for typed input, permission confirmation, and file handling. They are Transitional;
+see the matching references for migration records. Check deployment Gateway
+coverage before extending these commands; prefer existing equivalent capabilities.
+
+Transition status: transitional
+Owning module: Agent Evaluation (agent-test)
+Current transport: authenticated REST /agenteval/api/v1
+Gateway target: TBD (evaluation capability registration)
+Review after: 2027-01-21
+Exit condition: once equivalent evaluation capabilities are available, migrate
+selected typed/file commands to L2 or replace simple wrappers with L3; retain
+existing + commands only where compatibility or orchestration justifies them.
+
+This record also applies to the locale and process-evidence updates in the
+existing evaluator/run references. It is a migration plan, not a claim that a
+particular deployed Gateway has been inspected.

package/skills/ae-evaluation/references/access-and-members.md

@@ -0,0 +1,55 @@
+# Resource visibility and project members
+
+Read [SKILL.md](../SKILL.md) for authentication and mutation rules. These commands
+require administrator privileges (`root` or `agent_admin`). Permission changes
+are high-risk writes; retain confirmation unless the user authorizes `--yes`.
+
+## Resource visibility
+
+```bash
+ae-cli evaluation dataset access-scope --id <dataset-uuid> --access-scope COMPANY --lock-version <n>
+ae-cli evaluation evaluator access-scope --id <evaluator-uuid> --access-scope COMPANY --lock-version <n>
+ae-cli evaluation model-config access-scope --id <model-config-uuid> --access-scope PERSONAL --lock-version <n>
+ae-cli evaluation agent access-scope --id <agent-uuid> --access-scope PERSONAL --lock-version <n>
+```
+
+Read the current resource with its `+get_*` command before changing it. Each
+successful mutation increments `lock_version`. PERSONAL and COMPANY are mutable;
+SYSTEM resources are read-only. COMPANY sharing stays within the authenticated
+project/tenant. Shared assets cannot depend on personal configurations or
+credentials (`ACCESS_SCOPE_DEPENDENCY_PRIVATE`); inspect dependencies rather
+than silently broadening their visibility. Ordinary model/Agent creation produces
+personal resources and is not restricted to administrators by its create route.
+Credential operations and administrative routes remain administrator-only.
+
+## Project membership
+
+```bash
+ae-cli evaluation +list_project_members
+ae-cli evaluation project-member list-candidates
+ae-cli evaluation project-member batch-add --members-json '[{"external_id":"<te-id>","display_name":"<name>"}]'
+ae-cli evaluation project-member change-role --id <user-uuid> --role agent_admin
+ae-cli evaluation project-member set-status --id <user-uuid> --status DISABLED
+ae-cli evaluation project-member remove --id <user-uuid>
+```
+
+Discover candidates rather than guessing IDs. Admission takes TE `external_id`;
+role/status/removal take the project member's `user_id` UUID from
+`+list_project_members`. Batch admission returns added/skipped counts and skips
+existing members. These routes do not accept `lock_version`.
+
+Roles are `root`, `agent_admin`, and `member`; change-role accepts only
+`agent_admin|member`. Root members are protected. `set-status` accepts
+`ACTIVE|DISABLED` and affects login across all projects in the tenant. `remove`
+revokes only the current project's membership, including subsequent TE login
+and refresh, until explicitly re-added. Never describe disabling a login as
+merely removing it from one project.
+
+Transition status: transitional
+Owning module: Agent Evaluation accessscope and membership
+Current transport: authenticated REST /agenteval/api/v1
+Gateway target: TBD (evaluation access-scope and member administration)
+Review after: 2027-01-21
+Exit condition: when equivalent capabilities exist, migrate typed permission
+commands to L2 where confirmation and typed validation add value; use L3 for
+simple discovery operations and remove duplicate Transitional routes.

package/skills/ae-evaluation/references/connections.md

@@ -0,0 +1,78 @@
+# Models, agents, and credentials
+
+Commands use `ae-cli evaluation +<command>`. Read the shared endpoint, authentication, and mutation rules in [SKILL.md](../SKILL.md). Use command `--help` for optional flags.
+
+### Model configs
+
+```bash
+ae-cli evaluation +list_model_configs
+ae-cli evaluation +get_model_config --id <model-config-uuid>
+ae-cli evaluation +create_model_config --name "<name>" --provider OPENAI_COMPATIBLE_CHAT|ANTHROPIC_CHAT|GEMINI_CHAT|AZURE_OPENAI_CHAT|TE_CLAUDE_WEB \
+  --model-name "<model>" --base-url "<url>" [--parameters-json '<json>'] [--capabilities-json '<json>'] [--credential-id <uuid>]
+ae-cli evaluation +update_model_config --id <model-config-uuid> [same fields as +create_model_config] --lock-version <n>
+ae-cli evaluation +delete_model_config --id <model-config-uuid> --lock-version <n>
+ae-cli evaluation +test_model_config_connectivity --id <model-config-uuid>
+ae-cli evaluation +transfer_model_config_owner --id <model-config-uuid> --owner-user-id <user-uuid> --lock-version <n>
+```
+
+- `--capabilities-json` must set `chat_completions: true` and `max_context_tokens >= 1`, otherwise `MODEL_CONFIG_INVALID`.
+- Use `MODEL_PARAMETERS_V2` for new model configs. It omits `max_output_tokens`; output limits are controlled by the provider. Historical frozen snapshots may still contain V1.
+- `+update_model_config` is full-replace: `+get_model_config` first and pass every field back.
+- The platform validates outbound reachability at create time; unreachable base URLs are rejected.
+
+### Agents
+
+```bash
+ae-cli evaluation +list_agents
+ae-cli evaluation +get_agent --id <agent-uuid>
+ae-cli evaluation +create_agent --name "<name>" --endpoint-url "<url>" \
+  --input-schema-json '{"schema_version":"AGENT_INPUT_SCHEMA_V1","type":"object","properties":{...},"required":[...]}' \
+  --config-json '<AGENT_CONFIG_V* json>' \
+  [--connector-type HTTP_JSON_V1|OPENCODE_SERVER|OPENCLAW_GATEWAY_V1|HERMES_RUNS_V1|TE_CLAUDE_WEB_V1] [--credential-id <uuid>]
+ae-cli evaluation +update_agent --id <agent-uuid> [same fields as +create_agent] --lock-version <n>
+ae-cli evaluation +delete_agent --id <agent-uuid> --lock-version <n>
+ae-cli evaluation +test_agent_connectivity --id <agent-uuid>
+ae-cli evaluation +agent_snapshot_preview --id <agent-uuid>
+ae-cli evaluation +transfer_agent_owner --id <agent-uuid> --owner-user-id <user-uuid> --lock-version <n>
+```
+
+- `--config-json` must match the connector type; use `+get_agent` on a working agent as a template. Hermes direct transport uses `AGENT_CONFIG_V2.config.hermes.transport_mode=DIRECT`; use `+list_te_claude_catalog` to discover TE Claude agent/model ids for `TE_CLAUDE_WEB` configs.
+- Under TE authentication, TE Claude model and agent configs bind the login credential automatically; omit `--credential-id`. The flag remains available for legacy non-TE authentication.
+
+### Credentials
+
+```bash
+ae-cli evaluation +list_credentials
+ae-cli evaluation +get_credential --id <credential-uuid>
+ae-cli evaluation +create_credential --name "<name>" --type BEARER_TOKEN|API_KEY_HEADER|AI_MONITOR_SESSION --secret "<secret>" [--header-name "<header>"] \
+  [--data-token "<token>"] [--report-source "<source>"]
+ae-cli evaluation +update_credential --id <credential-uuid> --name "<name>" --lock-version <n>
+ae-cli evaluation +rotate_credential --id <credential-uuid> --secret "<new-secret>" --lock-version <n>
+ae-cli evaluation +disable_credential --id <credential-uuid> --lock-version <n>
+ae-cli evaluation +enable_credential --id <credential-uuid> --lock-version <n>
+ae-cli evaluation +delete_credential --id <credential-uuid> --lock-version <n>
+ae-cli evaluation +transfer_credential_owner --id <credential-uuid> --owner-user-id <user-uuid> --lock-version <n>
+```
+
+- Credential enabling/disabling is high-risk and requires confirmation. Non-interactive calls require explicit authorization and global `--yes`; `--dry-run` remains a local preview.
+- Evaluation error output redacts submitted secrets and session tokens, including JSON-escaped and URL-encoded forms. Remote error messages and hints are not written to CLI logs.
+- Secrets are write-only: sent once on create/rotate, never returned or logged by the CLI.
+- `AI_MONITOR_SESSION` credentials additionally require `--data-token` (max 512) and `--report-source` (max 128); all three are sensitive — dry-run redacts them and they must never be logged.
+
+### Bring your own model and agent
+
+Credentials are prerequisites for model configs and agents:
+
+```bash
+ae-cli evaluation +create_credential --name "my-ark-token" --type BEARER_TOKEN --secret "<secret>"
+ae-cli evaluation +create_model_config --name "my-judge" --provider OPENAI_COMPATIBLE_CHAT \
+  --model-name "kimi-k3" --base-url "http://10.0.0.1:3003" \
+  --credential-id <credential-uuid> \
+  --parameters-json '{"schema_version":"MODEL_PARAMETERS_V2","temperature":0,"top_p":1,"seed":1,"timeout_ms":120000,"max_retries":2,"structured_output":"AUTO"}' \
+  --capabilities-json '{"schema_version":"MODEL_CAPABILITIES_V1","chat_completions":true,"json_schema":false,"seed":true,"usage":true,"max_context_tokens":8192}'
+ae-cli evaluation +test_model_config_connectivity --id <model-config-uuid>
+```
+
+To change one setting (e.g. timeout), `+update_model_config` is full-replace: `+get_model_config`, edit the JSON, pass every field back.
+
+Ownership transfers and credential deletion require ADMIN. TE Claude catalog discovery is documented in [integrations.md](integrations.md).

package/skills/ae-evaluation/references/datasets.md

@@ -0,0 +1,57 @@
+# Datasets
+
+Commands use `ae-cli evaluation +<command>`. Read the shared endpoint, authentication, and mutation rules in [SKILL.md](../SKILL.md). Use command `--help` for optional flags.
+
+### Datasets
+
+```bash
+ae-cli evaluation +list_datasets [--search <text>] [--status CREATING|READY|IMPORT_FAILED] [--cursor <cursor>] [--page-size 50]
+ae-cli evaluation +get_dataset --id <dataset-uuid>
+ae-cli evaluation +create_dataset --name "<name>" [--description "..."]
+ae-cli evaluation +import_dataset --name "<name>" --file <path> [--profile GENERIC|AGENT_TASK_CASES_V1] [--description "..."] --idempotency-key "<fresh-key>"
+ae-cli evaluation +dataset_import_status --id <dataset-uuid>
+ae-cli evaluation +dataset_import_errors --id <dataset-uuid>
+ae-cli evaluation +update_dataset --id <dataset-uuid> --name "<name>" [--description "..."] --lock-version <n>
+ae-cli evaluation +delete_dataset --id <dataset-uuid> --lock-version <n>
+ae-cli evaluation +transfer_dataset_owner --id <dataset-uuid> --owner-user-id <user-uuid> --lock-version <n>
+ae-cli evaluation +list_dataset_fields --id <dataset-uuid>
+ae-cli evaluation +create_dataset_field --id <dataset-uuid> --display-name "<name>" --data-type TEXT|NUMBER|BOOLEAN|JSON|DATE|TIME
+ae-cli evaluation +batch_create_dataset_fields --id <dataset-uuid> --fields-json '[{"display_name":"Question","data_type":"TEXT"}]'
+ae-cli evaluation +update_dataset_field --id <dataset-uuid> --field-id <field-uuid> --display-name "<name>" [--ordinal 0] --lock-version <n>
+ae-cli evaluation +delete_dataset_field --id <dataset-uuid> --field-id <field-uuid> --lock-version <n>
+ae-cli evaluation +reorder_dataset_fields --id <dataset-uuid> --field-ids '["<uuid-1>","<uuid-2>"]' --lock-version <n>
+ae-cli evaluation +list_dataset_cases --id <dataset-uuid> [--cursor <cursor>] [--page-size 50]
+ae-cli evaluation +get_dataset_case --id <dataset-uuid> --case-id <case-uuid>
+ae-cli evaluation +create_dataset_case --id <dataset-uuid> --data '{"input":"...","expected":"..."}'
+ae-cli evaluation +batch_create_dataset_cases --id <dataset-uuid> --cases-json '[{"data":{"input":"...","expected":"..."}}]'
+ae-cli evaluation +update_dataset_case --id <dataset-uuid> --case-id <case-uuid> --data '{...}' --lock-version <n>
+ae-cli evaluation +delete_dataset_case --id <dataset-uuid> --case-id <case-uuid> --lock-version <n>
+ae-cli evaluation +export_dataset --id <dataset-uuid> --export-format csv|jsonl
+ae-cli evaluation +list_dataset_versions --id <dataset-uuid>
+ae-cli evaluation +publish_dataset_version --id <dataset-uuid> --lock-version <n>
+ae-cli evaluation +get_dataset_version --id <dataset-uuid> --version-id <dataset-version-uuid>
+ae-cli evaluation +list_dataset_references --id <dataset-uuid>
+```
+
+- `--field-ids` must contain every field UUID exactly once, in the desired order.
+- Case `--data` is keyed by `field_key` from `+list_dataset_fields`. Single-case `--data` and batch `--cases-json` are redacted from command logs. Sensitive JSON parse errors omit input content; successful requests and explicit dry-run previews retain the supplied case data.
+- Batch field creation accepts 1-100 definitions and batch case creation accepts 1-1000 definitions. Each batch is transactional: either every item is created or none are.
+- Field mutations use the dataset lock_version; case mutations use the case lock_version from `+get_dataset_case`.
+
+### Import a dataset and wait for readiness
+
+Import requires a fresh Idempotency-Key and returns 202 with the created dataset; the pipeline is asynchronous:
+
+```bash
+ae-cli evaluation +import_dataset \
+  --name "release-checkout" \
+  --file ./cases.csv \
+  --profile GENERIC \
+  --idempotency-key "import-$(date +%s)"
+ae-cli evaluation +dataset_import_status --id <dataset-uuid>   # poll until READY
+ae-cli evaluation +dataset_import_errors --id <dataset-uuid>   # if IMPORT_FAILED
+ae-cli evaluation +list_dataset_fields --id <dataset-uuid>     # field keys for mapping/case data
+```
+
+Download exported files using [files.md](files.md): poll `+get_file`, then
+`ae-cli evaluation file download --id <file-uuid> --output ./dataset.csv`.

package/skills/ae-evaluation/references/evaluators.md

@@ -0,0 +1,81 @@
+# Evaluators and calibration
+
+Commands use `ae-cli evaluation +<command>`. Read the shared endpoint, authentication, and mutation rules in [SKILL.md](../SKILL.md). Use command `--help` for optional flags.
+
+### Evaluators and templates
+
+```bash
+ae-cli evaluation +list_evaluators [--search <text>] [--cursor <cursor>] [--page-size 50]
+ae-cli evaluation +get_evaluator --id <evaluator-uuid>
+ae-cli evaluation +create_evaluator --name "<name>" [--description "..."] [--category SAFETY] \
+  [--prompt "...{{input}} {{output}}..."] [--variables '["input","output"]'] \
+  [--model-config-id <uuid>] [--template-code <code>] [--template-locale zh|en]
+ae-cli evaluation +update_evaluator --id <evaluator-uuid> --name "<name>" --model-config-id <uuid> \
+  [--description "..."] [--category "..."] [--prompt "..."] [--variables '[...]'] --lock-version <n>
+ae-cli evaluation +delete_evaluator --id <evaluator-uuid> --lock-version <n>
+ae-cli evaluation +transfer_evaluator_owner --id <evaluator-uuid> --owner-user-id <user-uuid> --lock-version <n>
+ae-cli evaluation +list_evaluator_templates [--locale zh|en]
+ae-cli evaluation +create_evaluator_template --name "<name>" --category "<category>" --prompt "<prompt>" [--description "..."] [--variables '[...]']
+ae-cli evaluation +disable_evaluator_template --id <template-uuid>
+ae-cli evaluation evaluator-template enable --id <project-template-uuid>
+ae-cli evaluation +list_evaluator_versions --id <evaluator-uuid>
+ae-cli evaluation +publish_evaluator_version --id <evaluator-uuid> --lock-version <n>
+ae-cli evaluation +get_evaluator_health --id <evaluator-uuid>
+ae-cli evaluation +optimize_evaluator_prompt --model-config-id <uuid> --prompt "<prompt>"
+ae-cli evaluation +list_calibration_suites --id <evaluator-uuid>
+ae-cli evaluation +create_calibration_suite --id <evaluator-uuid> --name "<name>" [--description "..."]
+ae-cli evaluation +list_calibration_cases --id <suite-uuid>
+ae-cli evaluation +create_calibration_case --id <suite-uuid> --variables-json '{...}' --expected-score 0.8 --tolerance 0.1 --tags-json '["golden"]'
+ae-cli evaluation +delete_calibration_case --id <case-uuid> --lock-version <n>
+ae-cli evaluation +start_calibration_run --id <suite-uuid>
+ae-cli evaluation +get_calibration_run --id <calibration-run-uuid>
+ae-cli evaluation +list_calibration_results --id <calibration-run-uuid>
+```
+
+- `+create_evaluator` without `--model-config-id` creates a draft; binding a model config via `+update_evaluator` activates it.
+- Prompt placeholders `{{name}}` must match `--variables`.
+
+### Calibrate an evaluator
+
+```bash
+ae-cli evaluation +list_calibration_suites --id <evaluator-uuid>        # use the default suite
+ae-cli evaluation +create_calibration_case --id <suite-uuid> \
+  --variables-json '{"input":"...","expected":"..."}' \
+  --expected-score 0.8 --tolerance 0.1 \
+  --tags-json '["golden"]'
+ae-cli evaluation +start_calibration_run --id <suite-uuid>
+ae-cli evaluation +get_calibration_run --id <calibration-run-uuid>      # poll until terminal
+ae-cli evaluation +list_calibration_results --id <calibration-run-uuid>
+ae-cli evaluation +get_evaluator_health --id <evaluator-uuid>
+```
+
+Prompt optimization consumes model quota and returns a preview; it never overwrites the evaluator. Prefer the default calibration suite; create extra suites only for a concrete need. Template creation and disabling require ADMIN.
+
+### Template language and process scoring
+
+List built-ins with `+list_evaluator_templates --locale en` and create from them
+with `+create_evaluator --template-code <discovered-code> --template-locale en`.
+Both language options accept zh/en and default to zh. Project templates keep
+their original language; an omitted description uses the localized built-in
+description when creating from a built-in. Existing evaluators are unchanged.
+
+`TOOL_TRACE_REASONABLENESS` evaluates recorded tool choices, arguments, step
+relationships and evidence consistency using `input`, `output`, `execution_trace`.
+Discover the template and publish/pin its evaluator version. Map execution_trace
+using FIELD_MAPPING_V6 as shown in [runs.md](runs.md); pair it with a task-completion
+evaluator when final-answer quality also matters. `EVALUATOR_VARIABLES_V3` and
+`EVALUATOR_SNAPSHOT_V3` carry the new process-evidence source types. Create/update
+still use the public `--variables` array of names; do not send the versioned
+snapshot object as that flag.
+
+`evaluator-template enable` is administrator-only, bodyless and idempotent.
+Only project templates can be re-enabled; built-ins and out-of-scope IDs are not
+found. New evaluators use the latest enabled project template version.
+
+Transition status: transitional
+Owning module: Agent Evaluation evaluator templates
+Current transport: authenticated POST /agenteval/api/v1/evaluator-templates/{id}/enable
+Gateway target: TBD (evaluation template lifecycle)
+Review after: 2027-01-21
+Exit condition: replace with L3 when an equivalent capability exists, or retain
+L2 only if typed inputs and compatibility provide additional value.

package/skills/ae-evaluation/references/files.md

@@ -0,0 +1,33 @@
+# Download evaluation files
+
+Use `ae-cli evaluation file download --id <file-uuid> --output <new-path>`.
+Authentication and endpoint rules are in [SKILL.md](../SKILL.md).
+
+1. Queue an export using the dataset or run reference.
+2. Poll `+get_file --id <file-uuid>` until COMPLETED. Resource creation alone is
+   not evidence that the file is ready; failed/expired files cannot be downloaded.
+3. Save the attachment:
+
+```bash
+ae-cli evaluation file download --id <file-uuid> --output ./results.csv --dry-run
+ae-cli evaluation file download --id <file-uuid> --output ./results.csv
+```
+
+The authenticated `/files/{id}/content` route works for local and S3 storage.
+The parent directory must exist. The CLI refuses to overwrite an existing path,
+saves with owner-only permissions, removes failed partial output, and returns
+`{path, byte_size}` in its JSON envelope. Bytes are not printed to stdout.
+Dry-run previews the request and destination without opening a file or requesting
+a session. Downloads do not follow redirects or retry automatically.
+
+The older `+file_download_url` remains available for temporary URLs, but local
+storage returns a server-local `file://` path. Use `file download` for a remote
+client. Do not print signed URLs or private file content into reports.
+
+Transition status: transitional
+Owning module: Agent Evaluation files
+Current transport: authenticated GET /agenteval/api/v1/files/{id}/content
+Gateway target: TBD (evaluation file download capability)
+Review after: 2027-01-21
+Exit condition: migrate to Gateway when authenticated attachment delivery is
+available with equivalent binary output, redaction, and no-overwrite guarantees.

package/skills/ae-evaluation/references/integrations.md

@@ -0,0 +1,39 @@
+# AI Monitor and TE reporting
+
+Commands use `ae-cli evaluation +<command>`. Read the shared endpoint, authentication, and mutation rules in [SKILL.md](../SKILL.md). Use command `--help` for optional flags.
+
+```bash
+ae-cli evaluation +get_te_report_config
+ae-cli evaluation +save_te_report_config --base-url "<url>" --app-id "<id>" [--lock-version <n>]
+ae-cli evaluation +get_ai_monitor_connection
+ae-cli evaluation +save_ai_monitor_connection --base-url "<url>" --credential-id <uuid> --session-claims-json '{...}' [--lock-version <n>]
+ae-cli evaluation +verify_ai_monitor_connection
+ae-cli evaluation +import_ai_monitor_traces --name "<name>" --selection-json '{...}' [--idempotency-key "<key>"]
+ae-cli evaluation +list_te_claude_catalog --endpoint-url "<url>" [--credential-id <legacy-uuid>]
+```
+
+- `+save_te_report_config` / `+save_ai_monitor_connection`: omit `--lock-version` on first create; on update send the current version from the get command (0 is a valid version — never treat it as "unset").
+
+- `--session-claims-json` requires string `user_id`, `te_user_id`, `name`, `company_id` (optional `is_root`, `platform_access`); values are redacted in dry-run output and must not be logged.
+- `+list_te_claude_catalog` uses the login access token automatically under TE authentication. Supply `--credential-id` only for legacy non-TE authentication.
+
+### Import AI Monitor traces into a dataset
+
+The import is asynchronous: the 201 response returns the created dataset and import job — poll `+dataset_import_status` instead of treating creation as completion. `--session-claims-json` and selection content are identity/permission sensitive; never log or echo them.
+
+```bash
+ae-cli evaluation +get_ai_monitor_connection
+ae-cli evaluation +save_ai_monitor_connection \
+  --base-url "https://ai-monitor.example.com" \
+  --credential-id <ai-monitor-session-credential-uuid> \
+  --session-claims-json '{"user_id":"...","te_user_id":"...","name":"...","company_id":"..."}' \
+  --lock-version <n>
+ae-cli evaluation +verify_ai_monitor_connection
+ae-cli evaluation +import_ai_monitor_traces \
+  --name "ai-monitor-traces" \
+  --selection-json '{"mode":"TRACE_IDS","trace_ids":["trace-1"]}' \
+  --idempotency-key "aim-$(date +%s)"
+ae-cli evaluation +dataset_import_status --id <dataset-uuid>   # poll until READY
+```
+
+FILTER mode uses `{"mode":"FILTER","start_time":"<date>","end_time":"<date>"}` with optional `agent`, `channel`, `search_term`. Both dates must be valid calendar dates in `YYYY-MM-DD` format, with `end_time` not preceding `start_time`; impossible dates such as February 30 are rejected locally. TRACE_IDS carries 1-1000 ids.