@shulex/cxclawctl 0.1.2 → 0.1.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +25 -22
- package/capabilities.lock.json +535 -131
- package/dist/cli.js +2203 -127
- package/dist/cli.js.map +1 -1
- package/package.json +20 -6
- package/requirements-cli.txt +0 -2
- package/skills/__lib__/__init__.py +0 -0
- package/skills/__lib__/skill-runtime/bootstrap.py +0 -134
- package/skills/__lib__/solvea-api/__init__.py +0 -0
- package/skills/__lib__/solvea-api/client.py +0 -1944
- package/skills/__lib__/solvea-api/dashboard.py +0 -100
- package/skills/__lib__/solvea-api/solvea_api.py +0 -1341
- package/skills/agent-analytics/SKILL.md +0 -74
- package/skills/agent-analytics/references/chat-report-fields.md +0 -36
- package/skills/agent-analytics/references/report-fields.md +0 -130
- package/skills/agent-analytics/references/transcript.md +0 -110
- package/skills/agent-builder/SKILL.md +0 -42
- package/skills/agent-builder/interfaces/fallback-chatflow.md +0 -47
- package/skills/agent-builder/references/chatflow/graph-spec.md +0 -259
- package/skills/agent-builder/references/chatflow/node-selection.md +0 -89
- package/skills/agent-builder/references/chatflow/nodes/agent.md +0 -350
- package/skills/agent-builder/references/chatflow/nodes/business-track.md +0 -43
- package/skills/agent-builder/references/chatflow/nodes/code.md +0 -157
- package/skills/agent-builder/references/chatflow/nodes/collect-order.md +0 -47
- package/skills/agent-builder/references/chatflow/nodes/finish.md +0 -34
- package/skills/agent-builder/references/chatflow/nodes/human-review.md +0 -60
- package/skills/agent-builder/references/chatflow/nodes/if-else.md +0 -100
- package/skills/agent-builder/references/chatflow/nodes/intent.md +0 -79
- package/skills/agent-builder/references/chatflow/nodes/knowledge-reply.md +0 -43
- package/skills/agent-builder/references/chatflow/nodes/llm.md +0 -117
- package/skills/agent-builder/references/chatflow/nodes/note.md +0 -38
- package/skills/agent-builder/references/chatflow/nodes/question.md +0 -97
- package/skills/agent-builder/references/chatflow/nodes/rag-reply.md +0 -68
- package/skills/agent-builder/references/chatflow/nodes/reply.md +0 -97
- package/skills/agent-builder/references/chatflow/nodes/start.md +0 -35
- package/skills/agent-builder/references/chatflow/nodes/sub-graph.md +0 -48
- package/skills/agent-builder/references/chatflow/nodes/tool.md +0 -138
- package/skills/agent-builder/references/chatflow/nodes/transfer-to-human.md +0 -63
- package/skills/agent-builder/references/chatflow/nodes/variable-assignment.md +0 -52
- package/skills/agent-builder/references/chatflow/nodes/variable-extract.md +0 -50
- package/skills/agent-builder/references/chatflow/nodes/variable-process.md +0 -70
- package/skills/agent-builder/references/chatflow/overview.md +0 -53
- package/skills/agent-builder/references/chatflow/visual-translation.md +0 -64
- package/skills/agent-builder/references/chatflow/workflow.md +0 -134
- package/skills/agent-builder/references/multi-agent/create-update.md +0 -185
- package/skills/agent-builder/references/multi-agent/graph-spec.md +0 -246
- package/skills/agent-builder/references/multi-agent/graph-structure.md +0 -12
- package/skills/agent-builder/references/multi-agent/migrate-to-chatflow.md +0 -13
- package/skills/agent-builder/references/multi-agent/migration/large.md +0 -554
- package/skills/agent-builder/references/multi-agent/migration/principles.md +0 -446
- package/skills/agent-builder/references/multi-agent/migration/small.md +0 -225
- package/skills/agent-builder/references/multi-agent/orchestration.md +0 -11
- package/skills/agent-builder/references/multi-agent/overview.md +0 -48
- package/skills/agent-builder/references/multi-agent/update.md +0 -128
- package/skills/agent-builder/references/multi-agent/workflow.md +0 -87
- package/skills/agent-builder/references/single-agent/overview.md +0 -48
- package/skills/agent-builder/references/single-agent/prompt.md +0 -16
- package/skills/agent-builder/references/single-agent/tools.md +0 -36
- package/skills/agent-builder/references/single-agent/voice.md +0 -46
- package/skills/agent-builder/references/tools/http-tool.md +0 -315
- package/skills/agent-builder/references/tools/mock-tool.md +0 -279
- package/skills/agent-builder/references/tools/overview.md +0 -46
- package/skills/agent-builder/references/tools/tool-config.md +0 -43
- package/skills/agent-builder/references/tools/x-token.md +0 -34
- package/skills/agent-builder/scripts/chatflow/chatflow_builder.py +0 -3458
- package/skills/agent-builder/scripts/chatflow/validate_graph.py +0 -1697
- package/skills/agent-builder/scripts/copy_marketplace_agent.py +0 -61
- package/skills/agent-builder/scripts/multi-agent/legacy_graph_analyzer.py +0 -5974
- package/skills/agent-builder/scripts/multi-agent/replay_compare.py +0 -679
- package/skills/agent-builder/scripts/patch_agent_tools.py +0 -135
- package/skills/agent-builder/scripts/tools/validate_tool_config.py +0 -384
- package/skills/agent-tester/SKILL.md +0 -35
- package/skills/agent-tester/batch-test/overview.md +0 -86
- package/skills/agent-tester/scripts/batch_test_sheet.py +0 -213
- package/skills/agent-tester/single-test/badcase-retest.md +0 -85
- package/skills/agent-tester/single-test/direct-test.md +0 -24
- package/skills/agent-tester/single-test/overview.md +0 -142
- package/skills/delivery/SKILL.md +0 -112
- package/skills/delivery/interfaces/classify.md +0 -38
- package/skills/delivery/interfaces/create-agents.md +0 -89
- package/skills/delivery/interfaces/delivery-test.md +0 -67
- package/skills/delivery/references/flow.md +0 -40
- package/skills/delivery/references/validation.md +0 -15
- package/skills/delivery/scripts/batch_test.py +0 -704
- package/skills/delivery/scripts/classify.py +0 -22
- package/skills/delivery/scripts/create_agents.py +0 -695
- package/skills/delivery/scripts/gen_delivery_report.py +0 -335
- package/skills/delivery/scripts/gen_excel.py +0 -234
- package/skills/delivery/scripts/gen_report.py +0 -208
- package/skills/delivery/scripts/plan.py +0 -163
- package/skills/delivery/scripts/render.py +0 -713
- package/skills/delivery/scripts/render_summary.py +0 -38
- package/skills/delivery/scripts/sample_history_cases.py +0 -109
- package/skills/kb-builder/SKILL.md +0 -80
- package/skills/kb-builder/interfaces/qa-upload.md +0 -55
- package/skills/kb-builder/interfaces/web-kb.md +0 -84
- package/skills/kb-builder/references/directory.md +0 -373
- package/skills/kb-builder/references/export.md +0 -282
- package/skills/kb-builder/references/isolation-keywords.md +0 -113
- package/skills/kb-builder/references/knowledge.md +0 -153
- package/skills/kb-builder/references/product.md +0 -52
- package/skills/kb-builder/references/qa-from-help-center.md +0 -58
- package/skills/kb-builder/references/qa-from-pdf.md +0 -94
- package/skills/kb-builder/references/qa-from-tickets.md +0 -168
- package/skills/kb-builder/references/qa-from-web.md +0 -98
- package/skills/kb-builder/references/qa-quality.md +0 -73
- package/skills/kb-builder/references/reply-template.md +0 -237
- package/skills/kb-builder/references/tag-isolation-migration.md +0 -61
- package/skills/kb-builder/references/tags.md +0 -45
- package/skills/kb-builder/references/upload.md +0 -130
- package/skills/kb-builder/scripts/batch_extract_ticket_qa.py +0 -333
- package/skills/kb-builder/scripts/build_directory_plan.py +0 -144
- package/skills/kb-builder/scripts/crawl_site.py +0 -433
- package/skills/kb-builder/scripts/create_dirs.py +0 -146
- package/skills/kb-builder/scripts/export_knowledge.py +0 -454
- package/skills/kb-builder/scripts/extract_ticket_qa.py +0 -441
- package/skills/kb-builder/scripts/extract_web_qa.py +0 -118
- package/skills/kb-builder/scripts/import_qa.py +0 -219
- package/skills/kb-builder/scripts/import_source_documents.py +0 -143
- package/skills/kb-builder/scripts/import_templates.py +0 -409
- package/skills/poc/SKILL.md +0 -125
- package/skills/poc/assets/default_manual.css +0 -279
- package/skills/poc/assets/notice_escalate.png +0 -0
- package/skills/poc/assets/notice_escalate2.png +0 -0
- package/skills/poc/assets/notice_new_chat.png +0 -0
- package/skills/poc/assets/poc_test_template.html +0 -645
- package/skills/poc/assets/poc_test_template.md +0 -104
- package/skills/poc/assets/step3_preview.png +0 -0
- package/skills/poc/assets/step4_order_note.png +0 -0
- package/skills/poc/interfaces/classify.md +0 -38
- package/skills/poc/interfaces/create-agents.md +0 -108
- package/skills/poc/interfaces/poc-test.md +0 -94
- package/skills/poc/references/flow.md +0 -47
- package/skills/poc/references/kb-directory.md +0 -22
- package/skills/poc/references/validation.md +0 -17
- package/skills/poc/scripts/classify.py +0 -22
- package/skills/poc/scripts/create_agents.py +0 -818
- package/skills/poc/scripts/gen_excel.py +0 -234
- package/skills/poc/scripts/gen_poc_manual.py +0 -249
- package/skills/poc/scripts/gen_report.py +0 -208
- package/skills/poc/scripts/inspect_chatflow.py +0 -209
- package/skills/poc/scripts/plan.py +0 -178
- package/skills/poc/scripts/poc_test.py +0 -495
- package/skills/poc/scripts/render.py +0 -713
- package/skills/poc/scripts/render_summary.py +0 -41
- package/skills/runtime-inspector/SKILL.md +0 -51
- package/skills/runtime-inspector/references/frontend-rendering.md +0 -126
- package/skills/runtime-inspector/references/model.md +0 -59
- package/skills/runtime-inspector/references/sandbox.md +0 -131
- package/skills/runtime-inspector/scripts/model_inspect.py +0 -68
- package/skills/runtime-inspector/scripts/prepare.py +0 -98
- package/skills/runtime-inspector/scripts/process.py +0 -84
- package/skills/runtime-inspector/scripts/read_state.py +0 -32
- package/skills/runtime-inspector/scripts/sandbox_inspect.py +0 -463
- package/skills/runtime-inspector/scripts/test_claude_proxy.py +0 -49
- package/skills/runtime-inspector/scripts/test_openai_proxy.py +0 -41
- package/skills/runtime-inspector/scripts/verify_persistence.py +0 -101
- package/skills/runtime-inspector/scripts/write_test.py +0 -41
- package/skills/setup/SKILL.md +0 -17
- package/skills/shulex-support/SKILL.md +0 -39
- package/skills/shulex-support/references/badcase-analysis/overview.md +0 -34
- package/skills/shulex-support/references/badcase-analysis/text/conversation-record.md +0 -105
- package/skills/shulex-support/references/badcase-analysis/text/core-flow.md +0 -89
- package/skills/shulex-support/references/badcase-analysis/text/log-query-rules.md +0 -36
- package/skills/shulex-support/references/badcase-analysis/text/m1-entry.md +0 -53
- package/skills/shulex-support/references/badcase-analysis/text/m10-chatflow-service.md +0 -259
- package/skills/shulex-support/references/badcase-analysis/text/m11-algorithm-service.md +0 -148
- package/skills/shulex-support/references/badcase-analysis/text/m12-quality-inspection.md +0 -57
- package/skills/shulex-support/references/badcase-analysis/text/m2-intent.md +0 -120
- package/skills/shulex-support/references/badcase-analysis/text/m3-strategy.md +0 -126
- package/skills/shulex-support/references/badcase-analysis/text/m4-aifield.md +0 -121
- package/skills/shulex-support/references/badcase-analysis/text/m5-agent.md +0 -208
- package/skills/shulex-support/references/badcase-analysis/text/m6-tool.md +0 -115
- package/skills/shulex-support/references/badcase-analysis/text/m7-reply.md +0 -72
- package/skills/shulex-support/references/badcase-analysis/text/m8-error.md +0 -92
- package/skills/shulex-support/references/badcase-analysis/text/m9-knowledge-search.md +0 -147
- package/skills/shulex-support/references/badcase-analysis/text/sls-query.md +0 -283
- package/skills/shulex-support/references/badcase-analysis/voice/core-flow.md +0 -77
- package/skills/shulex-support/references/badcase-analysis/voice/dependencies.md +0 -19
- package/skills/shulex-support/references/badcase-analysis/voice/platform-api.md +0 -23
- package/skills/shulex-support/references/badcase-analysis/voice/sls-playbook.md +0 -31
- package/skills/shulex-support/references/product-docs.md +0 -70
- package/skills/shulex-support/references/tickets.md +0 -184
- package/skills/shulex-support/scripts/support.py +0 -427
- package/skills/solvea-desk/SKILL.md +0 -39
- package/skills/solvea-desk/references/report/fields.md +0 -82
- package/skills/solvea-desk/references/report/report-design.md +0 -200
- package/skills/solvea-desk/references/report/report-template.html +0 -44
- package/skills/solvea-desk/references/report/report.css +0 -517
- package/skills/solvea-desk/references/report/workflow.md +0 -464
- package/skills/solvea-desk/references/tickets/query.md +0 -103
- package/skills/solvea-desk/references/tickets/tags.md +0 -83
- package/skills/solvea-desk/scripts/report/fetch_sources.py +0 -580
- package/skills/solvea-desk/scripts/report/render_report.py +0 -763
- package/skills/solvea-desk/scripts/tickets/desk_api.py +0 -73
- package/skills/solvea-desk/scripts/tickets/fetch_tickets.py +0 -403
- package/skills/solvea-desk/scripts/tickets/ticket_tags.py +0 -305
- package/skills/solvea-kit/SKILL.md +0 -84
- package/skills/solvea-kit/references/agent.md +0 -145
- package/skills/solvea-kit/references/chat.md +0 -244
- package/skills/solvea-kit/references/intent.md +0 -69
- package/skills/solvea-kit/references/plugin.md +0 -53
- package/skills/solvea-kit/references/quality-inspection.md +0 -75
- package/skills/solvea-kit/references/strategy.md +0 -62
- package/skills/solvea-kit/references/tagging.md +0 -231
- package/skills/solvea-kit/references/variables.md +0 -53
- package/skills/solvea-kit/scripts/generate_label_import_file.py +0 -151
- package/skills/solvea-kit/scripts/intent_classify.py +0 -843
|
@@ -1,135 +0,0 @@
|
|
|
1
|
-
#!/usr/bin/env python3
|
|
2
|
-
"""Patch headers on tools attached to one or more agents.
|
|
3
|
-
|
|
4
|
-
Usage:
|
|
5
|
-
python3 /skills/agent-builder/scripts/patch_agent_tools.py \
|
|
6
|
-
--agent-id 123 --tool-name new_Order_information_ship --patch-x-token --dry-run --pretty
|
|
7
|
-
"""
|
|
8
|
-
from __future__ import annotations
|
|
9
|
-
|
|
10
|
-
import argparse
|
|
11
|
-
import json
|
|
12
|
-
import os
|
|
13
|
-
import sys
|
|
14
|
-
|
|
15
|
-
sys.path.insert(0, os.path.join(os.environ.get("SOLVEA_LIB_DIR", ""), "skill-runtime"))
|
|
16
|
-
try:
|
|
17
|
-
from bootstrap import * # noqa: F401,F403
|
|
18
|
-
except Exception:
|
|
19
|
-
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "..", "__lib__", "skill-runtime"))
|
|
20
|
-
from bootstrap import * # noqa: F401,F403
|
|
21
|
-
|
|
22
|
-
from client import SolveaClient
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
def _token_value(raw) -> str:
|
|
26
|
-
if isinstance(raw, dict):
|
|
27
|
-
return str(raw.get("apiKey") or raw.get("key") or raw.get("value") or raw.get("data") or "")
|
|
28
|
-
return str(raw or "")
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
def _tool_items(response):
|
|
32
|
-
if isinstance(response, list):
|
|
33
|
-
return response
|
|
34
|
-
if isinstance(response, dict):
|
|
35
|
-
for key in ("list", "data", "records"):
|
|
36
|
-
if isinstance(response.get(key), list):
|
|
37
|
-
return response[key]
|
|
38
|
-
return []
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
def _patch_headers(config: dict, headers: dict[str, str]) -> dict:
|
|
42
|
-
updated = dict(config or {})
|
|
43
|
-
existing = dict(updated.get("headers") or {})
|
|
44
|
-
existing.update(headers)
|
|
45
|
-
updated["headers"] = existing
|
|
46
|
-
|
|
47
|
-
external = updated.get("externalHttpCallConfig")
|
|
48
|
-
if isinstance(external, dict) and isinstance(external.get("headerParams"), list):
|
|
49
|
-
by_name = {h.get("name"): h for h in external["headerParams"] if isinstance(h, dict)}
|
|
50
|
-
for name, value in headers.items():
|
|
51
|
-
if name in by_name:
|
|
52
|
-
by_name[name]["value"] = value
|
|
53
|
-
else:
|
|
54
|
-
external["headerParams"].append({"name": name, "value": value, "description": ""})
|
|
55
|
-
return updated
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
def patch_agent_tools(
|
|
59
|
-
client: SolveaClient,
|
|
60
|
-
agent_id: int,
|
|
61
|
-
tool_names: set[str],
|
|
62
|
-
headers: dict[str, str],
|
|
63
|
-
dry_run: bool = False,
|
|
64
|
-
) -> list[dict]:
|
|
65
|
-
response = client._curl("GET", f"/api_v2/gpt/ap/agent/{agent_id}/tools", {})
|
|
66
|
-
results = []
|
|
67
|
-
for tool in _tool_items(response):
|
|
68
|
-
name = tool.get("name")
|
|
69
|
-
if tool_names and name not in tool_names:
|
|
70
|
-
continue
|
|
71
|
-
tool_id = tool.get("id")
|
|
72
|
-
if not tool_id:
|
|
73
|
-
continue
|
|
74
|
-
config = _patch_headers(tool.get("config") or {}, headers)
|
|
75
|
-
item = {"agent_id": agent_id, "tool_id": tool_id, "tool_name": name, "dry_run": dry_run}
|
|
76
|
-
if not dry_run:
|
|
77
|
-
client._curl("PUT", f"/api_v2/gpt/ap/tools/{tool_id}", {"config": config})
|
|
78
|
-
item["updated"] = True
|
|
79
|
-
results.append(item)
|
|
80
|
-
return results
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
def main() -> None:
|
|
84
|
-
parser = argparse.ArgumentParser(description="Patch headers for tools attached to agents")
|
|
85
|
-
parser.add_argument("--agent-id", type=int, action="append", required=True,
|
|
86
|
-
help="Agent id. Can be repeated.")
|
|
87
|
-
parser.add_argument("--tool-name", action="append", default=[],
|
|
88
|
-
help="Only patch matching tool name. Can be repeated. Omit to patch all tools.")
|
|
89
|
-
parser.add_argument("--header", action="append", default=[],
|
|
90
|
-
help="Header assignment, e.g. X-Token=abc. Can be repeated.")
|
|
91
|
-
parser.add_argument("--patch-x-token", action="store_true",
|
|
92
|
-
help="Fetch current project API key and patch X-Token.")
|
|
93
|
-
parser.add_argument("--env", choices=["staging", "prod"], default=os.environ.get("SOLVEA_ENV", "prod"))
|
|
94
|
-
parser.add_argument("--project-id", default=None)
|
|
95
|
-
parser.add_argument("--dry-run", action="store_true")
|
|
96
|
-
parser.add_argument("--pretty", action="store_true")
|
|
97
|
-
args = parser.parse_args()
|
|
98
|
-
|
|
99
|
-
headers = {}
|
|
100
|
-
for item in args.header:
|
|
101
|
-
if "=" not in item:
|
|
102
|
-
raise ValueError(f"--header must be NAME=VALUE, got {item!r}")
|
|
103
|
-
name, value = item.split("=", 1)
|
|
104
|
-
headers[name] = value
|
|
105
|
-
|
|
106
|
-
client = SolveaClient(env=args.env, project_id=args.project_id)
|
|
107
|
-
if args.patch_x_token:
|
|
108
|
-
x_token = _token_value(client.get_api_key())
|
|
109
|
-
if not x_token:
|
|
110
|
-
raise RuntimeError("get_api_key returned no token value")
|
|
111
|
-
headers["X-Token"] = x_token
|
|
112
|
-
if not headers:
|
|
113
|
-
raise ValueError("provide --header NAME=VALUE or --patch-x-token")
|
|
114
|
-
|
|
115
|
-
patched = []
|
|
116
|
-
for agent_id in args.agent_id:
|
|
117
|
-
patched.extend(
|
|
118
|
-
patch_agent_tools(
|
|
119
|
-
client,
|
|
120
|
-
agent_id,
|
|
121
|
-
set(args.tool_name),
|
|
122
|
-
headers,
|
|
123
|
-
dry_run=args.dry_run,
|
|
124
|
-
)
|
|
125
|
-
)
|
|
126
|
-
output = {
|
|
127
|
-
"patched": patched,
|
|
128
|
-
"matched_count": len(patched),
|
|
129
|
-
"headers": sorted(headers),
|
|
130
|
-
}
|
|
131
|
-
print(json.dumps(output, ensure_ascii=False, indent=2 if args.pretty else None))
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
if __name__ == "__main__":
|
|
135
|
-
main()
|
|
@@ -1,384 +0,0 @@
|
|
|
1
|
-
#!/usr/bin/env python3
|
|
2
|
-
"""Validate a local Agent Builder HTTP tool JSON payload."""
|
|
3
|
-
from __future__ import annotations
|
|
4
|
-
|
|
5
|
-
import argparse
|
|
6
|
-
import json
|
|
7
|
-
import math
|
|
8
|
-
from pathlib import Path
|
|
9
|
-
from typing import Any
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
PARAMETER_TYPES = {"string", "integer", "number", "date", "boolean", "array", "object"}
|
|
13
|
-
API_TYPES_BY_PARAMETER_TYPE = {
|
|
14
|
-
"string": {"TEXT"},
|
|
15
|
-
"integer": {"NUMBER"},
|
|
16
|
-
"number": {"NUMBER"},
|
|
17
|
-
"date": {"DATE_TO_NOW", "NOW_TO_DATE"},
|
|
18
|
-
"boolean": {"BOOLEAN"},
|
|
19
|
-
"array": {"ARRAY_SIZE"},
|
|
20
|
-
"object": set(),
|
|
21
|
-
}
|
|
22
|
-
RELATIONS_BY_API_TYPE = {
|
|
23
|
-
"TEXT": {"CONTAIN", "NOT_CONTAIN", "IS_EMPTY", "NOT_EMPTY", "EQUAL", "NOT_EQUAL"},
|
|
24
|
-
"NUMBER": {
|
|
25
|
-
"GREATER_THAN", "GREATER_THAN_OR_EQUAL", "LESS_THAN", "LESS_THAN_OR_EQUAL",
|
|
26
|
-
"BETWEEN", "EQUAL", "NOT_EQUAL", "IS_EMPTY", "NOT_EMPTY",
|
|
27
|
-
},
|
|
28
|
-
"DATE_TO_NOW": {
|
|
29
|
-
"GREATER_THAN", "GREATER_THAN_OR_EQUAL", "LESS_THAN", "LESS_THAN_OR_EQUAL",
|
|
30
|
-
"BETWEEN", "EQUAL", "NOT_EQUAL", "IS_EMPTY", "NOT_EMPTY",
|
|
31
|
-
},
|
|
32
|
-
"NOW_TO_DATE": {
|
|
33
|
-
"GREATER_THAN", "GREATER_THAN_OR_EQUAL", "LESS_THAN", "LESS_THAN_OR_EQUAL",
|
|
34
|
-
"BETWEEN", "EQUAL", "NOT_EQUAL", "IS_EMPTY", "NOT_EMPTY",
|
|
35
|
-
},
|
|
36
|
-
"ARRAY_SIZE": {
|
|
37
|
-
"GREATER_THAN", "GREATER_THAN_OR_EQUAL", "LESS_THAN", "LESS_THAN_OR_EQUAL",
|
|
38
|
-
"BETWEEN", "EQUAL", "NOT_EQUAL", "IS_EMPTY", "NOT_EMPTY",
|
|
39
|
-
},
|
|
40
|
-
"BOOLEAN": {"EQUAL", "NOT_EQUAL", "IS_EMPTY", "NOT_EMPTY"},
|
|
41
|
-
}
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
def reject_non_finite_json_number(value: str) -> None:
|
|
45
|
-
raise ValueError(f"non-finite JSON number {value} is not allowed")
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
def parse_finite_json_float(value: str) -> float:
|
|
49
|
-
parsed = float(value)
|
|
50
|
-
if not math.isfinite(parsed):
|
|
51
|
-
reject_non_finite_json_number(value)
|
|
52
|
-
return parsed
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
def is_finite_number_value(value: Any) -> bool:
|
|
56
|
-
if isinstance(value, bool):
|
|
57
|
-
return False
|
|
58
|
-
if isinstance(value, (int, float)):
|
|
59
|
-
return math.isfinite(value)
|
|
60
|
-
if isinstance(value, str) and value.strip():
|
|
61
|
-
try:
|
|
62
|
-
return math.isfinite(float(value.strip()))
|
|
63
|
-
except ValueError:
|
|
64
|
-
return False
|
|
65
|
-
return False
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
def validate_tool_config(data: Any) -> dict[str, Any]:
|
|
69
|
-
errors: list[dict[str, str]] = []
|
|
70
|
-
|
|
71
|
-
def add_error(path: str, code: str, message: str) -> None:
|
|
72
|
-
errors.append({"path": path, "code": code, "message": message})
|
|
73
|
-
|
|
74
|
-
if not isinstance(data, dict):
|
|
75
|
-
add_error("$", "invalid_tool_config", "tool config must be a JSON object")
|
|
76
|
-
return {"valid": False, "errors": errors}
|
|
77
|
-
|
|
78
|
-
if data.get("type") != "EXTERNAL_HTTP_CALL":
|
|
79
|
-
add_error("type", "invalid_tool_type", "type must be EXTERNAL_HTTP_CALL")
|
|
80
|
-
|
|
81
|
-
meta = data.get("meta")
|
|
82
|
-
if not isinstance(meta, dict):
|
|
83
|
-
add_error("meta", "missing_meta", "meta must be an object")
|
|
84
|
-
meta = {}
|
|
85
|
-
external = meta.get("externalHttpCallConfig")
|
|
86
|
-
if not isinstance(external, dict):
|
|
87
|
-
add_error(
|
|
88
|
-
"meta.externalHttpCallConfig",
|
|
89
|
-
"missing_external_http_call_config",
|
|
90
|
-
"externalHttpCallConfig must be an object",
|
|
91
|
-
)
|
|
92
|
-
external = {}
|
|
93
|
-
|
|
94
|
-
category = meta.get("category")
|
|
95
|
-
is_manual = category == "MANUAL" or (category is None and data.get("source") == "MANUAL")
|
|
96
|
-
if is_manual and not str(external.get("url") or "").strip():
|
|
97
|
-
add_error(
|
|
98
|
-
"meta.externalHttpCallConfig.url",
|
|
99
|
-
"missing_manual_url",
|
|
100
|
-
"MANUAL tools require a non-empty URL",
|
|
101
|
-
)
|
|
102
|
-
if meta.get("category") == "EXCEL" and not str(meta.get("fileName") or "").strip():
|
|
103
|
-
add_error("meta.fileName", "missing_excel_file_name", "EXCEL tools require fileName")
|
|
104
|
-
|
|
105
|
-
headers = external.get("headerParams", [])
|
|
106
|
-
if not isinstance(headers, list):
|
|
107
|
-
add_error(
|
|
108
|
-
"meta.externalHttpCallConfig.headerParams",
|
|
109
|
-
"invalid_headers",
|
|
110
|
-
"headerParams must be an array",
|
|
111
|
-
)
|
|
112
|
-
else:
|
|
113
|
-
for index, header in enumerate(headers):
|
|
114
|
-
header_path = f"meta.externalHttpCallConfig.headerParams[{index}]"
|
|
115
|
-
if not isinstance(header, dict):
|
|
116
|
-
add_error(header_path, "invalid_header", "header must be an object")
|
|
117
|
-
continue
|
|
118
|
-
name = header.get("name")
|
|
119
|
-
if not isinstance(name, str) or not name.strip():
|
|
120
|
-
add_error(
|
|
121
|
-
f"{header_path}.name",
|
|
122
|
-
"invalid_header_name",
|
|
123
|
-
"header name must not be empty",
|
|
124
|
-
)
|
|
125
|
-
|
|
126
|
-
parameter_count = {"request": 0, "response": 0}
|
|
127
|
-
response_paths: dict[str, str] = {}
|
|
128
|
-
|
|
129
|
-
def validate_items(
|
|
130
|
-
items: Any,
|
|
131
|
-
schema_path: str,
|
|
132
|
-
kind: str,
|
|
133
|
-
api_parent: tuple[str, ...] = (),
|
|
134
|
-
include_api_name: bool = True,
|
|
135
|
-
) -> None:
|
|
136
|
-
if not isinstance(items, list):
|
|
137
|
-
add_error(schema_path, "invalid_parameter_items", "items must be an array")
|
|
138
|
-
return
|
|
139
|
-
|
|
140
|
-
names: set[str] = set()
|
|
141
|
-
for index, item in enumerate(items):
|
|
142
|
-
item_path = f"{schema_path}[{index}]"
|
|
143
|
-
if not isinstance(item, dict):
|
|
144
|
-
add_error(item_path, "invalid_parameter", "parameter must be an object")
|
|
145
|
-
continue
|
|
146
|
-
parameter_count[kind] += 1
|
|
147
|
-
name = item.get("name")
|
|
148
|
-
clean_name = name.strip() if isinstance(name, str) else ""
|
|
149
|
-
if not clean_name:
|
|
150
|
-
add_error(f"{item_path}.name", "invalid_parameter_name", "parameter name must not be empty")
|
|
151
|
-
elif "." in clean_name:
|
|
152
|
-
add_error(f"{item_path}.name", "invalid_parameter_name", "parameter name must not contain dots")
|
|
153
|
-
if clean_name in names:
|
|
154
|
-
add_error(
|
|
155
|
-
f"{item_path}.name",
|
|
156
|
-
"duplicate_parameter_name",
|
|
157
|
-
f"duplicate parameter name {clean_name!r} in the same scope",
|
|
158
|
-
)
|
|
159
|
-
elif clean_name:
|
|
160
|
-
names.add(clean_name)
|
|
161
|
-
|
|
162
|
-
parameter_type = item.get("type")
|
|
163
|
-
valid_parameter_type = isinstance(parameter_type, str) and parameter_type in PARAMETER_TYPES
|
|
164
|
-
if not valid_parameter_type:
|
|
165
|
-
add_error(
|
|
166
|
-
f"{item_path}.type",
|
|
167
|
-
"invalid_parameter_type",
|
|
168
|
-
f"type must be one of {sorted(PARAMETER_TYPES)}",
|
|
169
|
-
)
|
|
170
|
-
|
|
171
|
-
api_path = api_parent + ((clean_name,) if clean_name and include_api_name else ())
|
|
172
|
-
if kind == "response" and clean_name and include_api_name and valid_parameter_type:
|
|
173
|
-
response_paths[".".join(api_path)] = parameter_type
|
|
174
|
-
|
|
175
|
-
if parameter_type == "array":
|
|
176
|
-
child = item.get("listSubItem")
|
|
177
|
-
if not isinstance(child, dict):
|
|
178
|
-
add_error(f"{item_path}.listSubItem", "missing_list_sub_item", "array requires listSubItem")
|
|
179
|
-
else:
|
|
180
|
-
validate_items([child], f"{item_path}.listSubItem", kind, api_path, False)
|
|
181
|
-
elif parameter_type == "object":
|
|
182
|
-
children = item.get("objectItems")
|
|
183
|
-
if not isinstance(children, list) or not children:
|
|
184
|
-
add_error(f"{item_path}.objectItems", "missing_object_items", "object requires non-empty objectItems")
|
|
185
|
-
else:
|
|
186
|
-
validate_items(children, f"{item_path}.objectItems", kind, api_path)
|
|
187
|
-
|
|
188
|
-
for kind, key in (("request", "requestParam"), ("response", "responseParam")):
|
|
189
|
-
container_path = f"meta.externalHttpCallConfig.{key}"
|
|
190
|
-
container = external.get(key, {})
|
|
191
|
-
if not isinstance(container, dict):
|
|
192
|
-
add_error(container_path, "invalid_parameter_config", f"{key} must be an object")
|
|
193
|
-
continue
|
|
194
|
-
validate_items(container.get("items", []), f"{container_path}.items", kind)
|
|
195
|
-
|
|
196
|
-
statuses = external.get("actionStatuses", [])
|
|
197
|
-
if not isinstance(statuses, list):
|
|
198
|
-
add_error(
|
|
199
|
-
"meta.externalHttpCallConfig.actionStatuses",
|
|
200
|
-
"invalid_action_statuses",
|
|
201
|
-
"actionStatuses must be an array",
|
|
202
|
-
)
|
|
203
|
-
statuses = []
|
|
204
|
-
|
|
205
|
-
status_names: set[str] = set()
|
|
206
|
-
for status_index, status in enumerate(statuses):
|
|
207
|
-
status_path = f"meta.externalHttpCallConfig.actionStatuses[{status_index}]"
|
|
208
|
-
if not isinstance(status, dict):
|
|
209
|
-
add_error(status_path, "invalid_action_status", "action status must be an object")
|
|
210
|
-
continue
|
|
211
|
-
name = status.get("name")
|
|
212
|
-
clean_name = name.strip() if isinstance(name, str) else ""
|
|
213
|
-
if not clean_name:
|
|
214
|
-
add_error(f"{status_path}.name", "invalid_status_name", "action status name must not be empty")
|
|
215
|
-
elif clean_name in status_names:
|
|
216
|
-
add_error(f"{status_path}.name", "duplicate_status_name", "action status names must be unique")
|
|
217
|
-
else:
|
|
218
|
-
status_names.add(clean_name)
|
|
219
|
-
|
|
220
|
-
condition = status.get("condition")
|
|
221
|
-
is_default = condition == {} or condition == {"or": []}
|
|
222
|
-
if status_index == len(statuses) - 1:
|
|
223
|
-
if not is_default:
|
|
224
|
-
add_error(
|
|
225
|
-
f"{status_path}.condition",
|
|
226
|
-
"invalid_default_status",
|
|
227
|
-
"the last action status must have condition: {}",
|
|
228
|
-
)
|
|
229
|
-
elif is_default:
|
|
230
|
-
add_error(
|
|
231
|
-
f"{status_path}.condition",
|
|
232
|
-
"invalid_default_status",
|
|
233
|
-
"only the last action status may be the default",
|
|
234
|
-
)
|
|
235
|
-
|
|
236
|
-
if is_default:
|
|
237
|
-
continue
|
|
238
|
-
groups = condition.get("or") if isinstance(condition, dict) else None
|
|
239
|
-
all_rules: list[tuple[dict[str, Any], str]] = []
|
|
240
|
-
if isinstance(groups, list):
|
|
241
|
-
for group_index, group in enumerate(groups):
|
|
242
|
-
group_path = f"{status_path}.condition.or[{group_index}]"
|
|
243
|
-
if not isinstance(group, dict):
|
|
244
|
-
add_error(group_path, "invalid_condition_group", "condition group must be an object")
|
|
245
|
-
continue
|
|
246
|
-
rules = group.get("rules")
|
|
247
|
-
if not isinstance(rules, list) or not rules:
|
|
248
|
-
add_error(
|
|
249
|
-
f"{group_path}.rules",
|
|
250
|
-
"missing_group_rules",
|
|
251
|
-
"each condition group requires a non-empty rules array",
|
|
252
|
-
)
|
|
253
|
-
continue
|
|
254
|
-
for rule_index, rule in enumerate(rules):
|
|
255
|
-
rule_path = f"{group_path}.rules[{rule_index}]"
|
|
256
|
-
if isinstance(rule, dict):
|
|
257
|
-
all_rules.append((rule, rule_path))
|
|
258
|
-
else:
|
|
259
|
-
add_error(rule_path, "invalid_status_rule", "rule must be an object")
|
|
260
|
-
if not all_rules:
|
|
261
|
-
add_error(
|
|
262
|
-
f"{status_path}.condition",
|
|
263
|
-
"missing_status_rules",
|
|
264
|
-
"non-default action statuses require at least one rule",
|
|
265
|
-
)
|
|
266
|
-
continue
|
|
267
|
-
|
|
268
|
-
for rule, rule_path in all_rules:
|
|
269
|
-
if rule.get("type") != "GENERIC":
|
|
270
|
-
add_error(f"{rule_path}.type", "invalid_rule_type", "rule type must be GENERIC")
|
|
271
|
-
api_name = rule.get("apiName")
|
|
272
|
-
if not isinstance(api_name, str) or not api_name.strip():
|
|
273
|
-
add_error(f"{rule_path}.apiName", "missing_api_name", "apiName must not be empty")
|
|
274
|
-
response_type = None
|
|
275
|
-
else:
|
|
276
|
-
response_type = response_paths.get(api_name)
|
|
277
|
-
if response_type is None:
|
|
278
|
-
add_error(
|
|
279
|
-
f"{rule_path}.apiName",
|
|
280
|
-
"unknown_response_path",
|
|
281
|
-
f"apiName {api_name!r} is not an exact response parameter path",
|
|
282
|
-
)
|
|
283
|
-
elif response_type == "object":
|
|
284
|
-
add_error(
|
|
285
|
-
f"{rule_path}.apiName",
|
|
286
|
-
"object_status_not_supported",
|
|
287
|
-
"object response parameters cannot be used in action status rules",
|
|
288
|
-
)
|
|
289
|
-
|
|
290
|
-
api_type = rule.get("apiType")
|
|
291
|
-
if api_type is None or api_type == "":
|
|
292
|
-
add_error(f"{rule_path}.apiType", "missing_api_type", "apiType is required")
|
|
293
|
-
valid_api_type = None
|
|
294
|
-
elif not isinstance(api_type, str) or api_type not in RELATIONS_BY_API_TYPE:
|
|
295
|
-
add_error(
|
|
296
|
-
f"{rule_path}.apiType",
|
|
297
|
-
"invalid_api_type",
|
|
298
|
-
f"apiType must be one of {sorted(RELATIONS_BY_API_TYPE)}",
|
|
299
|
-
)
|
|
300
|
-
valid_api_type = None
|
|
301
|
-
else:
|
|
302
|
-
valid_api_type = api_type
|
|
303
|
-
if (
|
|
304
|
-
response_type
|
|
305
|
-
and valid_api_type is not None
|
|
306
|
-
and valid_api_type not in API_TYPES_BY_PARAMETER_TYPE.get(response_type, set())
|
|
307
|
-
):
|
|
308
|
-
add_error(
|
|
309
|
-
f"{rule_path}.apiType",
|
|
310
|
-
"api_type_mismatch",
|
|
311
|
-
f"apiType {api_type!r} is incompatible with response type {response_type!r}",
|
|
312
|
-
)
|
|
313
|
-
|
|
314
|
-
relation = rule.get("relation")
|
|
315
|
-
valid_relation = relation if isinstance(relation, str) else None
|
|
316
|
-
if valid_relation not in RELATIONS_BY_API_TYPE.get(valid_api_type, set()):
|
|
317
|
-
add_error(
|
|
318
|
-
f"{rule_path}.relation",
|
|
319
|
-
"invalid_relation",
|
|
320
|
-
f"relation {relation!r} is not supported for apiType {api_type!r}",
|
|
321
|
-
)
|
|
322
|
-
|
|
323
|
-
values = rule.get("values")
|
|
324
|
-
values_valid = isinstance(values, list)
|
|
325
|
-
if values_valid and valid_relation in {"IS_EMPTY", "NOT_EMPTY"}:
|
|
326
|
-
values_valid = values == []
|
|
327
|
-
elif values_valid and valid_relation == "BETWEEN":
|
|
328
|
-
values_valid = len(values) == 2
|
|
329
|
-
elif values_valid and valid_api_type == "TEXT" and valid_relation in {"CONTAIN", "NOT_CONTAIN"}:
|
|
330
|
-
values_valid = len(values) >= 1
|
|
331
|
-
elif values_valid:
|
|
332
|
-
values_valid = len(values) == 1
|
|
333
|
-
if values_valid and valid_api_type == "BOOLEAN" and valid_relation in {"EQUAL", "NOT_EQUAL"}:
|
|
334
|
-
boolean_value = values[0]
|
|
335
|
-
values_valid = isinstance(boolean_value, bool) or (
|
|
336
|
-
isinstance(boolean_value, str)
|
|
337
|
-
and boolean_value.strip().lower() in {"true", "false"}
|
|
338
|
-
)
|
|
339
|
-
if values_valid and valid_api_type == "TEXT" and values:
|
|
340
|
-
values_valid = all(isinstance(value, str) for value in values)
|
|
341
|
-
if values_valid and valid_api_type in {"NUMBER", "DATE_TO_NOW", "NOW_TO_DATE", "ARRAY_SIZE"} and values:
|
|
342
|
-
values_valid = all(is_finite_number_value(value) for value in values)
|
|
343
|
-
if not values_valid:
|
|
344
|
-
add_error(
|
|
345
|
-
f"{rule_path}.values",
|
|
346
|
-
"invalid_rule_values",
|
|
347
|
-
"values has an invalid count or value type for the selected relation and apiType",
|
|
348
|
-
)
|
|
349
|
-
|
|
350
|
-
if errors:
|
|
351
|
-
return {"valid": False, "errors": errors}
|
|
352
|
-
return {
|
|
353
|
-
"valid": True,
|
|
354
|
-
"request_parameter_count": parameter_count["request"],
|
|
355
|
-
"response_parameter_count": parameter_count["response"],
|
|
356
|
-
"action_status_count": len(statuses),
|
|
357
|
-
}
|
|
358
|
-
|
|
359
|
-
|
|
360
|
-
def main() -> int:
|
|
361
|
-
parser = argparse.ArgumentParser(description="Validate an Agent Builder tool JSON payload")
|
|
362
|
-
parser.add_argument("--data-file", required=True, help="Path to a local JSON file")
|
|
363
|
-
parser.add_argument("--pretty", action="store_true")
|
|
364
|
-
args = parser.parse_args()
|
|
365
|
-
|
|
366
|
-
try:
|
|
367
|
-
with Path(args.data_file).open(encoding="utf-8") as file:
|
|
368
|
-
data = json.load(
|
|
369
|
-
file,
|
|
370
|
-
parse_constant=reject_non_finite_json_number,
|
|
371
|
-
parse_float=parse_finite_json_float,
|
|
372
|
-
)
|
|
373
|
-
output = validate_tool_config(data)
|
|
374
|
-
except (OSError, UnicodeDecodeError, ValueError) as exc:
|
|
375
|
-
output = {
|
|
376
|
-
"valid": False,
|
|
377
|
-
"errors": [{"path": "$", "code": "invalid_data_file", "message": str(exc)}],
|
|
378
|
-
}
|
|
379
|
-
print(json.dumps(output, ensure_ascii=False, indent=2 if args.pretty else None))
|
|
380
|
-
return 0 if output["valid"] else 1
|
|
381
|
-
|
|
382
|
-
|
|
383
|
-
if __name__ == "__main__":
|
|
384
|
-
raise SystemExit(main())
|
|
@@ -1,35 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: agent-tester
|
|
3
|
-
description: Use when directly testing, regression-testing, batch-testing, or checking Download Center progress for one Shulex Agent. Supports live_test single cases and system-native batch tests; CHAT_FLOW Agents use the canvas multi-turn batch-test path and other Agents use the menu batch-test path.
|
|
4
|
-
requires:
|
|
5
|
-
libs: [solvea-api]
|
|
6
|
-
---
|
|
7
|
-
|
|
8
|
-
# Agent Tester
|
|
9
|
-
|
|
10
|
-
本文件是完整文档目录。先根据请求定位到唯一工作流,然后**直接读取该工作流指定的最终文档**;不要逐层读取无关文档。
|
|
11
|
-
|
|
12
|
-
## 文档目录
|
|
13
|
-
|
|
14
|
-
| 用户场景 | 直接读取 | 说明 |
|
|
15
|
-
|---|---|---|
|
|
16
|
-
| 指定 Agent、单个问题、“随便测一下”、继续探索 | `single-test/direct-test.md` | 使用 `live_test` 运行固定 case 或探索式单轮 |
|
|
17
|
-
| 历史 ChatId、badcase 复测、验证修复或比较变化 | `single-test/badcase-retest.md` | 读取原会话,重放 USER 消息并比较 AgentAction |
|
|
18
|
-
| 多个 case、测试集、Excel、批测、下载中心进度 | `batch-test/overview.md` | 系统 xlsx 批测:Chatflow 多轮或普通 Agent 菜单批测 |
|
|
19
|
-
|
|
20
|
-
## 目录层级
|
|
21
|
-
|
|
22
|
-
```text
|
|
23
|
-
agent-tester/
|
|
24
|
-
├── SKILL.md
|
|
25
|
-
├── single-test/
|
|
26
|
-
│ ├── overview.md # 单测公共 live_test 契约(仅 direct/badcase 文档需要时读取)
|
|
27
|
-
│ ├── direct-test.md # 单问题与探索式测试
|
|
28
|
-
│ └── badcase-retest.md # 历史 ChatId 复测
|
|
29
|
-
├── batch-test/
|
|
30
|
-
│ └── overview.md # 系统批测工作流
|
|
31
|
-
└── scripts/
|
|
32
|
-
└── batch_test_sheet.py # 生成 xlsx、上传、提交、查询下载中心
|
|
33
|
-
```
|
|
34
|
-
|
|
35
|
-
批测必须提交系统下载中心任务,不要调用或复用 POC / Delivery 的本地并发脚本、状态、cases 协议或报告模板。
|
|
@@ -1,86 +0,0 @@
|
|
|
1
|
-
# 系统批量测试
|
|
2
|
-
|
|
3
|
-
批测必须对齐系统入口:生成并填充系统模板兼容 xlsx,提交下载中心任务,再按需查询进度。不要使用 POC / Delivery 的本地并发脚本、状态或报告协议。
|
|
4
|
-
|
|
5
|
-
## 入口路由
|
|
6
|
-
|
|
7
|
-
先执行 `npx --yes @shulex/cxclawctl@latest api get-agent --id <agentId> --pretty`:
|
|
8
|
-
|
|
9
|
-
| 类型 | 系统入口 | task type | 数据 | 上限 |
|
|
10
|
-
|---|---|---|---|---|
|
|
11
|
-
| `CHAT_FLOW` | Chatflow 画布“批量多轮测试” | `MULTI_TURN_REPLY` | 一个 case 多 turn | 600 case |
|
|
12
|
-
| 其他 Agent | 菜单“批量测试” | `REPLY` | 一个 case 一个 question | 200 case |
|
|
13
|
-
|
|
14
|
-
不要把 Chatflow 降级为菜单批测,也不要将普通 Agent 当作多轮 Chatflow 任务。
|
|
15
|
-
|
|
16
|
-
### case 数量建议
|
|
17
|
-
|
|
18
|
-
- 1-10 个 case:如果需要即时查看每轮回复、节点和调试信息,可使用 `live_test`。
|
|
19
|
-
- **超过 10 个 case:优先生成系统模板并提交下载中心批测任务**,不要把每个 case 拆成多个 `live_test`。
|
|
20
|
-
- Chatflow 批测使用 `MULTI_TURN_REPLY`;普通 Agent 批测使用 `REPLY`。
|
|
21
|
-
|
|
22
|
-
## 创建测试表
|
|
23
|
-
|
|
24
|
-
用户可以给问题列表/文件、ChatId 列表,或要求按条件找一批历史对话。JSON 仅是 skill 在后台归一化输入给脚本的中间产物,**不要要求用户创建、上传或确认 JSON**。
|
|
25
|
-
|
|
26
|
-
先向用户展示可读的测试集表格并确认,再在后台生成 JSON 和 xlsx。表格至少展示:case 名称、来源(手工问题/ChatId/历史筛选)、轮次、用户问题、固定 metadata、目标模式;订单号、邮箱等 metadata 不得编造。用户确认后才执行生成和提交。
|
|
27
|
-
|
|
28
|
-
脚本内部使用 case JSON 生成第 1 行说明、第 2 行表头的系统兼容 xlsx:
|
|
29
|
-
|
|
30
|
-
```bash
|
|
31
|
-
npx --yes @shulex/cxclawctl@latest script skills/agent-tester/scripts/batch_test_sheet.py create-sheet \
|
|
32
|
-
--mode chatflow --cases-file /workspace/outputs/agent-tester/chatflow_cases.json
|
|
33
|
-
|
|
34
|
-
npx --yes @shulex/cxclawctl@latest script skills/agent-tester/scripts/batch_test_sheet.py create-sheet \
|
|
35
|
-
--mode agent --cases-file /workspace/outputs/agent-tester/agent_cases.json
|
|
36
|
-
```
|
|
37
|
-
|
|
38
|
-
Chatflow JSON:
|
|
39
|
-
|
|
40
|
-
```json
|
|
41
|
-
[{"case_name":"refund-follow-up","turns":[{"question":"我要退款","metadata":{"order_number":"A-1001"}},{"question":"订单号就是 A-1001"}]}]
|
|
42
|
-
```
|
|
43
|
-
|
|
44
|
-
普通 Agent JSON:
|
|
45
|
-
|
|
46
|
-
```json
|
|
47
|
-
[{"case_name":"shipping","question":"我的订单什么时候发货?","metadata":{"order_number":"A-1001"}}]
|
|
48
|
-
```
|
|
49
|
-
|
|
50
|
-
### `REPLY` 模板的 `meta_data` 格式
|
|
51
|
-
|
|
52
|
-
普通 Agent(`REPLY`)批测模板的 `meta_data` 列目前必须填写为**纯文本键值对**,多个字段用英文或中文分号分隔:
|
|
53
|
-
|
|
54
|
-
```text
|
|
55
|
-
touchPoint: LIVE_CHAT; order_number: A-1001
|
|
56
|
-
```
|
|
57
|
-
|
|
58
|
-
请按上述格式填写,`LIVE_CHAT`、`EMAIL` 等渠道值使用系统枚举值。
|
|
59
|
-
|
|
60
|
-
Chatflow 表头是 `case_name, chat_id, metadata_1, question_1, ai_answer_1, ...`;普通 Agent 是 `question, origin_answer, ai_answer, similarity, judgement, reason, chat_id, meta_data, fail_reason`。所有系统输出列保持空白。xlsx 输出到 `/workspace/outputs/agent-tester/`,状态写到 `/workspace/state/agent-tester/latest_batch_test.json`。
|
|
61
|
-
|
|
62
|
-
## 提交和查询
|
|
63
|
-
|
|
64
|
-
默认不指定 `specified_agent_id` 或 `specified_flow_version`,让任务从最外层意图识别和系统策略路由开始执行。只有用户明确要求锁定某个 Chatflow Agent(例如验证该 Agent 的草稿或已发布版本)时,才先用 `get-chatflow-graph --agent-id <agentId>` 取得实际 version,并同时传入 Agent 和版本;不要只传其中一个。历史预览或调试面板打开时,画布不允许提交。
|
|
65
|
-
|
|
66
|
-
```bash
|
|
67
|
-
npx --yes @shulex/cxclawctl@latest script skills/agent-tester/scripts/batch_test_sheet.py submit \
|
|
68
|
-
--mode chatflow --file <generated.xlsx> --bot-id <botId>
|
|
69
|
-
|
|
70
|
-
# 只有明确指定 Chatflow Agent / 版本时追加:
|
|
71
|
-
npx --yes @shulex/cxclawctl@latest script skills/agent-tester/scripts/batch_test_sheet.py submit \
|
|
72
|
-
--mode chatflow --file <generated.xlsx> --bot-id <botId> \
|
|
73
|
-
--agent-id <agentId> --flow-version <currentVersion>
|
|
74
|
-
|
|
75
|
-
npx --yes @shulex/cxclawctl@latest script skills/agent-tester/scripts/batch_test_sheet.py submit \
|
|
76
|
-
--mode agent --file <generated.xlsx> --bot-id <botId>
|
|
77
|
-
```
|
|
78
|
-
|
|
79
|
-
脚本获取临时 S3 凭证并上传 xlsx。默认 Chatflow 请求不带路由 metadata;仅在指定模式下才携带 `specified_agent_id` 和 `specified_flow_version`。提交接口只返回 `success`,不返回 taskId;随后按类型查询下载中心,选本次时间相符的最新任务:
|
|
80
|
-
|
|
81
|
-
```bash
|
|
82
|
-
npx --yes @shulex/cxclawctl@latest script skills/agent-tester/scripts/batch_test_sheet.py get-tasks --mode chatflow
|
|
83
|
-
npx --yes @shulex/cxclawctl@latest script skills/agent-tester/scripts/batch_test_sheet.py get-tasks --mode agent
|
|
84
|
-
```
|
|
85
|
-
|
|
86
|
-
`WAIT_TAG` 是等待中,`TAGGING` 是执行中,`SUCCESS` 完成且 `file` 可下载,`FAIL` 是失败;`successCount / totalCount` 为当前进度。除非用户明确要求,不要阻塞式轮询。
|