solace-agent-mesh 1.0.5__py3-none-any.whl → 1.0.7__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Potentially problematic release.
This version of solace-agent-mesh might be problematic. Click here for more details.
- solace_agent_mesh/agent/adk/artifacts/__init__.py +1 -0
- solace_agent_mesh/agent/adk/{filesystem_artifact_service.py → artifacts/filesystem_artifact_service.py} +14 -15
- solace_agent_mesh/agent/adk/artifacts/s3_artifact_service.py +440 -0
- solace_agent_mesh/agent/adk/callbacks.py +123 -159
- solace_agent_mesh/agent/adk/embed_resolving_mcp_toolset.py +316 -0
- solace_agent_mesh/agent/adk/intelligent_mcp_callbacks.py +414 -0
- solace_agent_mesh/agent/adk/mcp_content_processor.py +665 -0
- solace_agent_mesh/agent/adk/services.py +35 -1
- solace_agent_mesh/agent/adk/setup.py +85 -45
- solace_agent_mesh/agent/adk/tool_wrapper.py +19 -3
- solace_agent_mesh/agent/protocol/event_handlers.py +1 -1
- solace_agent_mesh/agent/sac/app.py +67 -0
- solace_agent_mesh/agent/sac/component.py +14 -86
- solace_agent_mesh/assets/docs/404.html +3 -3
- solace_agent_mesh/assets/docs/assets/js/04989206.b9dfe831.js +1 -0
- solace_agent_mesh/assets/docs/assets/js/0e682baa.b3bbde9a.js +1 -0
- solace_agent_mesh/assets/docs/assets/js/1023fc19.364235d5.js +1 -0
- solace_agent_mesh/assets/docs/assets/js/1523c6b4.1b0ec6f9.js +1 -0
- solace_agent_mesh/assets/docs/assets/js/166ab619.e8f3a7c7.js +1 -0
- solace_agent_mesh/assets/docs/assets/js/21ceee5f.3bf39250.js +1 -0
- solace_agent_mesh/assets/docs/assets/js/3d406171.7d02a73b.js +1 -0
- solace_agent_mesh/assets/docs/assets/js/42b3f8d8.8ccb9901.js +1 -0
- solace_agent_mesh/assets/docs/assets/js/442a8107.b3159bb2.js +1 -0
- solace_agent_mesh/assets/docs/assets/js/4c2787c2.fc6804f2.js +1 -0
- solace_agent_mesh/assets/docs/assets/js/5b4258a4.0d080cd9.js +1 -0
- solace_agent_mesh/assets/docs/assets/js/75384d09.ccd480c4.js +1 -0
- solace_agent_mesh/assets/docs/assets/js/768e31b0.8b51cd70.js +1 -0
- solace_agent_mesh/assets/docs/assets/js/945fb41e.c63791d1.js +1 -0
- solace_agent_mesh/assets/docs/assets/js/{9eff14a2.036c35ea.js → 9eff14a2.472b0310.js} +1 -1
- solace_agent_mesh/assets/docs/assets/js/a3a92b25.4b7fa6a2.js +1 -0
- solace_agent_mesh/assets/docs/assets/js/aba87c2f.76376d7c.js +1 -0
- solace_agent_mesh/assets/docs/assets/js/ae4415af.7a2f0bbf.js +1 -0
- solace_agent_mesh/assets/docs/assets/js/b7006a3a.73a79653.js +1 -0
- solace_agent_mesh/assets/docs/assets/js/beecea0d.ae31f6a7.js +1 -0
- solace_agent_mesh/assets/docs/assets/js/c2c06897.587b4af5.js +1 -0
- solace_agent_mesh/assets/docs/assets/js/{cd3d4052.ca6eed8c.js → cd3d4052.b6535013.js} +1 -1
- solace_agent_mesh/assets/docs/assets/js/f284c35a.731836ad.js +1 -0
- solace_agent_mesh/assets/docs/assets/js/f897a61a.0aa29dbb.js +1 -0
- solace_agent_mesh/assets/docs/assets/js/main.d79f063b.js +2 -0
- solace_agent_mesh/assets/docs/assets/js/runtime~main.6415ad00.js +1 -0
- solace_agent_mesh/assets/docs/docs/documentation/concepts/agents/index.html +28 -4
- solace_agent_mesh/assets/docs/docs/documentation/concepts/architecture/index.html +6 -6
- solace_agent_mesh/assets/docs/docs/documentation/concepts/cli/index.html +8 -8
- solace_agent_mesh/assets/docs/docs/documentation/concepts/gateways/index.html +5 -5
- solace_agent_mesh/assets/docs/docs/documentation/concepts/orchestrator/index.html +5 -5
- solace_agent_mesh/assets/docs/docs/documentation/concepts/plugins/index.html +34 -5
- solace_agent_mesh/assets/docs/docs/documentation/deployment/debugging/index.html +4 -4
- solace_agent_mesh/assets/docs/docs/documentation/deployment/deploy/index.html +5 -5
- solace_agent_mesh/assets/docs/docs/documentation/deployment/observability/index.html +4 -4
- solace_agent_mesh/assets/docs/docs/documentation/getting-started/component-overview/index.html +6 -6
- solace_agent_mesh/assets/docs/docs/documentation/getting-started/configurations/index.html +72 -0
- solace_agent_mesh/assets/docs/docs/documentation/getting-started/installation/index.html +5 -5
- solace_agent_mesh/assets/docs/docs/documentation/getting-started/introduction/index.html +7 -7
- solace_agent_mesh/assets/docs/docs/documentation/getting-started/quick-start/index.html +35 -16
- solace_agent_mesh/assets/docs/docs/documentation/tutorials/bedrock-agents/index.html +4 -4
- solace_agent_mesh/assets/docs/docs/documentation/tutorials/custom-agent/index.html +17 -11
- solace_agent_mesh/assets/docs/docs/documentation/tutorials/event-mesh-gateway/index.html +4 -4
- solace_agent_mesh/assets/docs/docs/documentation/tutorials/mcp-integration/index.html +5 -5
- solace_agent_mesh/assets/docs/docs/documentation/tutorials/mongodb-integration/index.html +4 -4
- solace_agent_mesh/assets/docs/docs/documentation/tutorials/rag-integration/index.html +6 -6
- solace_agent_mesh/assets/docs/docs/documentation/tutorials/rest-gateway/index.html +6 -6
- solace_agent_mesh/assets/docs/docs/documentation/tutorials/slack-integration/index.html +6 -6
- solace_agent_mesh/assets/docs/docs/documentation/tutorials/sql-database/index.html +4 -4
- solace_agent_mesh/assets/docs/docs/documentation/user-guide/builtin-tools/artifact-management/index.html +8 -8
- solace_agent_mesh/assets/docs/docs/documentation/user-guide/builtin-tools/audio-tools/index.html +14 -14
- solace_agent_mesh/assets/docs/docs/documentation/user-guide/builtin-tools/data-analysis-tools/index.html +8 -8
- solace_agent_mesh/assets/docs/docs/documentation/user-guide/builtin-tools/embeds/index.html +4 -4
- solace_agent_mesh/assets/docs/docs/documentation/user-guide/builtin-tools/index.html +6 -6
- solace_agent_mesh/assets/docs/docs/documentation/user-guide/create-agents/index.html +35 -23
- solace_agent_mesh/assets/docs/docs/documentation/user-guide/create-gateways/index.html +4 -4
- solace_agent_mesh/assets/docs/docs/documentation/user-guide/creating-service-providers/index.html +6 -6
- solace_agent_mesh/assets/docs/docs/documentation/user-guide/solace-ai-connector/index.html +4 -4
- solace_agent_mesh/assets/docs/docs/documentation/user-guide/structure/index.html +4 -4
- solace_agent_mesh/assets/docs/lunr-index-1756146501924.json +1 -0
- solace_agent_mesh/assets/docs/lunr-index.json +1 -1
- solace_agent_mesh/assets/docs/search-doc-1756146501924.json +1 -0
- solace_agent_mesh/assets/docs/search-doc.json +1 -1
- solace_agent_mesh/assets/docs/sitemap.xml +1 -1
- solace_agent_mesh/cli/__init__.py +1 -1
- solace_agent_mesh/cli/commands/add_cmd/add_cmd_llm.txt +1 -1
- solace_agent_mesh/cli/commands/add_cmd/agent_cmd.py +67 -10
- solace_agent_mesh/cli/commands/add_cmd/gateway_cmd.py +2 -2
- solace_agent_mesh/cli/commands/eval_cmd.py +8 -2
- solace_agent_mesh/cli/commands/init_cmd/__init__.py +20 -2
- solace_agent_mesh/cli/commands/init_cmd/env_step.py +25 -1
- solace_agent_mesh/cli/commands/init_cmd/orchestrator_step.py +45 -1
- solace_agent_mesh/cli/utils.py +21 -12
- solace_agent_mesh/client/webui/frontend/static/assets/main-BucUdn9m.js +673 -0
- solace_agent_mesh/client/webui/frontend/static/index.html +1 -1
- solace_agent_mesh/common/a2a_protocol.py +1 -1
- solace_agent_mesh/common/utils/initializer.py +4 -6
- solace_agent_mesh/common/utils/mime_helpers.py +60 -1
- solace_agent_mesh/config_portal/backend/server.py +1 -1
- solace_agent_mesh/config_portal/frontend/static/client/assets/{_index-xSu2leR8.js → _index-MqsrTd6g.js} +9 -9
- solace_agent_mesh/config_portal/frontend/static/client/assets/{manifest-950eb3be.js → manifest-28271392.js} +1 -1
- solace_agent_mesh/config_portal/frontend/static/client/index.html +1 -1
- solace_agent_mesh/core_a2a/core_a2a_llm.txt +1 -1
- solace_agent_mesh/core_a2a/service.py +1 -1
- solace_agent_mesh/evaluation/run.py +149 -15
- solace_agent_mesh/evaluation/summary_builder.py +5 -3
- solace_agent_mesh/gateway/http_sse/dependencies.py +1 -1
- solace_agent_mesh/gateway/http_sse/http_sse_llm.txt +1 -1
- solace_agent_mesh/gateway/http_sse/services/task_service.py +1 -1
- solace_agent_mesh/llm_detail.txt +2 -2
- solace_agent_mesh/templates/agent_template.yaml +1 -1
- solace_agent_mesh/templates/plugin_agent_config_template.yaml +3 -3
- solace_agent_mesh/templates/plugin_readme_template.md +1 -1
- solace_agent_mesh/templates/shared_config.yaml +8 -1
- {solace_agent_mesh-1.0.5.dist-info → solace_agent_mesh-1.0.7.dist-info}/METADATA +5 -2
- {solace_agent_mesh-1.0.5.dist-info → solace_agent_mesh-1.0.7.dist-info}/RECORD +114 -109
- solace_agent_mesh/assets/docs/assets/js/04989206.da8246cd.js +0 -1
- solace_agent_mesh/assets/docs/assets/js/0e682baa.79f0ab22.js +0 -1
- solace_agent_mesh/assets/docs/assets/js/1023fc19.8e6d174c.js +0 -1
- solace_agent_mesh/assets/docs/assets/js/1523c6b4.91c7bc01.js +0 -1
- solace_agent_mesh/assets/docs/assets/js/166ab619.7d97ccaf.js +0 -1
- solace_agent_mesh/assets/docs/assets/js/21ceee5f.614fa8dd.js +0 -1
- solace_agent_mesh/assets/docs/assets/js/3d406171.9b081d5f.js +0 -1
- solace_agent_mesh/assets/docs/assets/js/42b3f8d8.36090198.js +0 -1
- solace_agent_mesh/assets/docs/assets/js/442a8107.5ba94b65.js +0 -1
- solace_agent_mesh/assets/docs/assets/js/4c2787c2.66ee00e9.js +0 -1
- solace_agent_mesh/assets/docs/assets/js/5b4258a4.bda20761.js +0 -1
- solace_agent_mesh/assets/docs/assets/js/75384d09.c3991823.js +0 -1
- solace_agent_mesh/assets/docs/assets/js/768e31b0.a12673db.js +0 -1
- solace_agent_mesh/assets/docs/assets/js/945fb41e.74d728aa.js +0 -1
- solace_agent_mesh/assets/docs/assets/js/a3a92b25.26ca071f.js +0 -1
- solace_agent_mesh/assets/docs/assets/js/aba87c2f.a6b84da6.js +0 -1
- solace_agent_mesh/assets/docs/assets/js/ae4415af.96189a93.js +0 -1
- solace_agent_mesh/assets/docs/assets/js/b7006a3a.38c0cf3d.js +0 -1
- solace_agent_mesh/assets/docs/assets/js/bb2ef573.56931473.js +0 -1
- solace_agent_mesh/assets/docs/assets/js/c2c06897.63b76e9e.js +0 -1
- solace_agent_mesh/assets/docs/assets/js/f284c35a.5aff74ab.js +0 -1
- solace_agent_mesh/assets/docs/assets/js/f897a61a.862b0514.js +0 -1
- solace_agent_mesh/assets/docs/assets/js/main.946fa17b.js +0 -2
- solace_agent_mesh/assets/docs/assets/js/runtime~main.aa687c82.js +0 -1
- solace_agent_mesh/assets/docs/docs/documentation/enterprise/index.html +0 -17
- solace_agent_mesh/assets/docs/lunr-index-1755275703209.json +0 -1
- solace_agent_mesh/assets/docs/search-doc-1755275703209.json +0 -1
- solace_agent_mesh/client/webui/frontend/static/assets/main-DzKPMTRs.js +0 -673
- /solace_agent_mesh/assets/docs/assets/js/{main.946fa17b.js.LICENSE.txt → main.d79f063b.js.LICENSE.txt} +0 -0
- {solace_agent_mesh-1.0.5.dist-info → solace_agent_mesh-1.0.7.dist-info}/WHEEL +0 -0
- {solace_agent_mesh-1.0.5.dist-info → solace_agent_mesh-1.0.7.dist-info}/entry_points.txt +0 -0
- {solace_agent_mesh-1.0.5.dist-info → solace_agent_mesh-1.0.7.dist-info}/licenses/LICENSE +0 -0
|
@@ -1 +1 @@
|
|
|
1
|
-
window.__remixManifest={"entry":{"module":"/assets/entry.client-CEumGClk.js","imports":["/assets/index-DSo1AH_7.js","/assets/components-B7lKcHVY.js"],"css":[]},"routes":{"root":{"id":"root","path":"","hasAction":false,"hasLoader":false,"hasClientAction":false,"hasClientLoader":false,"hasErrorBoundary":false,"module":"/assets/root-C4XmHinv.js","imports":["/assets/index-DSo1AH_7.js","/assets/components-B7lKcHVY.js"],"css":["/assets/root-DxRwaWiE.css"]},"routes/_index":{"id":"routes/_index","parentId":"root","index":true,"hasAction":false,"hasLoader":false,"hasClientAction":false,"hasClientLoader":false,"hasErrorBoundary":false,"module":"/assets/_index-
|
|
1
|
+
window.__remixManifest={"entry":{"module":"/assets/entry.client-CEumGClk.js","imports":["/assets/index-DSo1AH_7.js","/assets/components-B7lKcHVY.js"],"css":[]},"routes":{"root":{"id":"root","path":"","hasAction":false,"hasLoader":false,"hasClientAction":false,"hasClientLoader":false,"hasErrorBoundary":false,"module":"/assets/root-C4XmHinv.js","imports":["/assets/index-DSo1AH_7.js","/assets/components-B7lKcHVY.js"],"css":["/assets/root-DxRwaWiE.css"]},"routes/_index":{"id":"routes/_index","parentId":"root","index":true,"hasAction":false,"hasLoader":false,"hasClientAction":false,"hasClientLoader":false,"hasErrorBoundary":false,"module":"/assets/_index-MqsrTd6g.js","imports":["/assets/index-DSo1AH_7.js"],"css":[]}},"url":"/assets/manifest-28271392.js","version":"28271392"};
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
<!DOCTYPE html>
|
|
2
|
-
<html lang="en"><head><meta charSet="utf-8"/><meta name="viewport" content="width=device-width, initial-scale=1"/><link rel="stylesheet" href="/assets/root-DxRwaWiE.css"/><link rel="preconnect" href="https://fonts.googleapis.com"/><link rel="preconnect" href="https://fonts.gstatic.com" crossorigin="anonymous"/><link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Inter:ital,opsz,wght@0,14..32,100..900;1,14..32,100..900&display=swap"/></head><body><p>Loading...</p><link rel="modulepreload" href="/assets/manifest-
|
|
2
|
+
<html lang="en"><head><meta charSet="utf-8"/><meta name="viewport" content="width=device-width, initial-scale=1"/><link rel="stylesheet" href="/assets/root-DxRwaWiE.css"/><link rel="preconnect" href="https://fonts.googleapis.com"/><link rel="preconnect" href="https://fonts.gstatic.com" crossorigin="anonymous"/><link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Inter:ital,opsz,wght@0,14..32,100..900;1,14..32,100..900&display=swap"/></head><body><p>Loading...</p><link rel="modulepreload" href="/assets/manifest-28271392.js"/><link rel="modulepreload" href="/assets/entry.client-CEumGClk.js"/><link rel="modulepreload" href="/assets/index-DSo1AH_7.js"/><link rel="modulepreload" href="/assets/components-B7lKcHVY.js"/><link rel="modulepreload" href="/assets/root-C4XmHinv.js"/><script>window.__remixContext = {"basename":"/","future":{"v3_fetcherPersist":false,"v3_relativeSplatPath":false,"v3_throwAbortReason":false,"v3_routeConfig":false,"v3_singleFetch":false,"v3_lazyRouteDiscovery":false,"unstable_optimizeDeps":false},"isSpaMode":true,"state":{"loaderData":{"root":null,"routes/_index":null},"actionData":null,"errors":null}};</script><script type="module" async="">import "/assets/manifest-28271392.js";
|
|
3
3
|
import * as route0 from "/assets/root-C4XmHinv.js";
|
|
4
4
|
|
|
5
5
|
window.__remixRouteModules = {"root":route0};
|
|
@@ -28,7 +28,7 @@ No public classes, functions, or constants defined.
|
|
|
28
28
|
- `get_all_agents() -> List[AgentCard]` - Retrieves all currently discovered agent cards from the registry
|
|
29
29
|
- `process_discovery_message(agent_card: AgentCard)` - Processes an incoming agent card discovery message
|
|
30
30
|
- `agent_registry: AgentRegistry` - The shared agent registry instance
|
|
31
|
-
- `namespace: str` - The
|
|
31
|
+
- `namespace: str` - The namespace string
|
|
32
32
|
- `log_identifier: str` - Identifier used for logging
|
|
33
33
|
|
|
34
34
|
**Functions:**
|
|
@@ -33,7 +33,7 @@ class CoreA2AService:
|
|
|
33
33
|
|
|
34
34
|
Args:
|
|
35
35
|
agent_registry: An instance of the shared AgentRegistry.
|
|
36
|
-
namespace: The
|
|
36
|
+
namespace: The namespace string.
|
|
37
37
|
"""
|
|
38
38
|
if not isinstance(agent_registry, AgentRegistry):
|
|
39
39
|
raise TypeError("agent_registry must be an instance of AgentRegistry")
|
|
@@ -85,10 +85,11 @@ class TestRun:
|
|
|
85
85
|
class ProcessManager:
|
|
86
86
|
"""Manages subprocess lifecycle for the Solace AI Connector."""
|
|
87
87
|
|
|
88
|
-
def __init__(self, config: EvaluationConfig):
|
|
88
|
+
def __init__(self, config: EvaluationConfig, verbose: bool = False):
|
|
89
89
|
self.config = config
|
|
90
90
|
self.process: Optional[subprocess.Popen] = None
|
|
91
91
|
self.namespace: Optional[str] = None
|
|
92
|
+
self.verbose = verbose
|
|
92
93
|
|
|
93
94
|
def start_services(self) -> Tuple[subprocess.Popen, str]:
|
|
94
95
|
"""Start the Solace AI Connector and return process and namespace."""
|
|
@@ -138,9 +139,10 @@ class ProcessManager:
|
|
|
138
139
|
class TaskService:
|
|
139
140
|
"""Handles task submission and tracking."""
|
|
140
141
|
|
|
141
|
-
def __init__(self, config: EvaluationConfig):
|
|
142
|
+
def __init__(self, config: EvaluationConfig, verbose: bool = False):
|
|
142
143
|
self.config = config
|
|
143
144
|
self.base_url = config.API_BASE_URL
|
|
145
|
+
self.verbose = verbose
|
|
144
146
|
|
|
145
147
|
def submit_task(
|
|
146
148
|
self, agent_name: str, message: str, artifact_paths: Optional[List[str]] = None
|
|
@@ -262,9 +264,10 @@ class TestRunBuilder:
|
|
|
262
264
|
class TestExecutor:
|
|
263
265
|
"""Executes individual test runs."""
|
|
264
266
|
|
|
265
|
-
def __init__(self, task_service: TaskService, file_service: FileService):
|
|
267
|
+
def __init__(self, task_service: TaskService, file_service: FileService, verbose: bool = False):
|
|
266
268
|
self.task_service = task_service
|
|
267
269
|
self.file_service = file_service
|
|
270
|
+
self.verbose = verbose
|
|
268
271
|
|
|
269
272
|
def execute_test(
|
|
270
273
|
self,
|
|
@@ -331,13 +334,14 @@ class TestExecutor:
|
|
|
331
334
|
class ModelEvaluator:
|
|
332
335
|
"""Handles the evaluation of a single model."""
|
|
333
336
|
|
|
334
|
-
def __init__(self, config: EvaluationConfig):
|
|
337
|
+
def __init__(self, config: EvaluationConfig, verbose: bool = False):
|
|
335
338
|
self.config = config
|
|
336
|
-
self.process_manager = ProcessManager(config)
|
|
337
|
-
self.task_service = TaskService(config)
|
|
339
|
+
self.process_manager = ProcessManager(config, verbose=verbose)
|
|
340
|
+
self.task_service = TaskService(config, verbose=verbose)
|
|
338
341
|
self.file_service = FileService()
|
|
339
342
|
self.test_builder = TestRunBuilder(config)
|
|
340
|
-
self.test_executor = TestExecutor(self.task_service, self.file_service)
|
|
343
|
+
self.test_executor = TestExecutor(self.task_service, self.file_service, verbose=verbose)
|
|
344
|
+
self.verbose = verbose
|
|
341
345
|
|
|
342
346
|
def evaluate_model(
|
|
343
347
|
self, model_config: Dict[str, Any], base_results_path: str
|
|
@@ -444,9 +448,10 @@ class ModelEvaluator:
|
|
|
444
448
|
class ResultsProcessor:
|
|
445
449
|
"""Handles post-processing of evaluation results."""
|
|
446
450
|
|
|
447
|
-
def __init__(self, file_service: FileService):
|
|
451
|
+
def __init__(self, file_service: FileService, verbose: bool = False):
|
|
448
452
|
self.file_service = file_service
|
|
449
453
|
self.summary_builder = SummaryBuilder()
|
|
454
|
+
self.verbose = verbose
|
|
450
455
|
|
|
451
456
|
def summarize_results(self, base_results_path: str):
|
|
452
457
|
"""Generate summaries for all test results."""
|
|
@@ -486,11 +491,12 @@ class ResultsProcessor:
|
|
|
486
491
|
class EvaluationRunner:
|
|
487
492
|
"""Main orchestrator that coordinates the entire evaluation process."""
|
|
488
493
|
|
|
489
|
-
def __init__(self):
|
|
494
|
+
def __init__(self, verbose: bool = False):
|
|
490
495
|
self.config: Optional[EvaluationConfig] = None
|
|
491
496
|
self.file_service = FileService()
|
|
492
|
-
self.results_processor = ResultsProcessor(self.file_service)
|
|
497
|
+
self.results_processor = ResultsProcessor(self.file_service, verbose=verbose)
|
|
493
498
|
self.report_generator: Optional[ReportGenerator] = None
|
|
499
|
+
self.verbose = verbose
|
|
494
500
|
|
|
495
501
|
def run_evaluation(self, config_path: str):
|
|
496
502
|
"""Main entry point for the evaluation process."""
|
|
@@ -518,6 +524,10 @@ class EvaluationRunner:
|
|
|
518
524
|
# Generate reports
|
|
519
525
|
self._generate_reports(config_path, base_results_path)
|
|
520
526
|
|
|
527
|
+
# Display verbose summary if enabled
|
|
528
|
+
if self.verbose:
|
|
529
|
+
self._display_verbose_summary(base_results_path)
|
|
530
|
+
|
|
521
531
|
except Exception as e:
|
|
522
532
|
print(f"Evaluation failed: {e}")
|
|
523
533
|
raise
|
|
@@ -543,7 +553,7 @@ class EvaluationRunner:
|
|
|
543
553
|
model_execution_times = {}
|
|
544
554
|
|
|
545
555
|
for model_config in self.config.llm_models:
|
|
546
|
-
model_evaluator = ModelEvaluator(self.config)
|
|
556
|
+
model_evaluator = ModelEvaluator(self.config, verbose=self.verbose)
|
|
547
557
|
execution_time = model_evaluator.evaluate_model(
|
|
548
558
|
model_config, base_results_path
|
|
549
559
|
)
|
|
@@ -580,12 +590,130 @@ class EvaluationRunner:
|
|
|
580
590
|
if self.report_generator:
|
|
581
591
|
self.report_generator.generate_report(base_results_path)
|
|
582
592
|
|
|
593
|
+
def _display_verbose_summary(self, base_results_path: Path):
|
|
594
|
+
"""Display a verbose summary of the evaluation results in the terminal."""
|
|
595
|
+
|
|
596
|
+
# Pre-process data to find column widths
|
|
597
|
+
summary_data = []
|
|
598
|
+
max_model_len = 0
|
|
599
|
+
max_test_case_len = 0
|
|
600
|
+
|
|
601
|
+
for model_dir in sorted(base_results_path.iterdir()):
|
|
602
|
+
if not model_dir.is_dir():
|
|
603
|
+
continue
|
|
604
|
+
|
|
605
|
+
results_file = model_dir / "results.json"
|
|
606
|
+
if not results_file.exists():
|
|
607
|
+
continue
|
|
608
|
+
|
|
609
|
+
try:
|
|
610
|
+
results_data = self.file_service.load_json(str(results_file))
|
|
611
|
+
model_name = results_data.get("model_name", model_dir.name)
|
|
612
|
+
max_model_len = max(max_model_len, len(model_name))
|
|
613
|
+
|
|
614
|
+
for test_case in results_data.get("test_cases", []):
|
|
615
|
+
test_case_id = test_case.get("test_case_id")
|
|
616
|
+
if not test_case_id:
|
|
617
|
+
continue
|
|
618
|
+
|
|
619
|
+
max_test_case_len = max(max_test_case_len, len(test_case_id))
|
|
620
|
+
|
|
621
|
+
scores = {}
|
|
622
|
+
tool_match = test_case.get("tool_match_scores", {}).get("average")
|
|
623
|
+
if tool_match is not None:
|
|
624
|
+
scores["Tool Match"] = f"{tool_match:.2f}"
|
|
625
|
+
|
|
626
|
+
response_match = test_case.get("response_match_scores", {}).get("average")
|
|
627
|
+
if response_match is not None:
|
|
628
|
+
scores["Response Match"] = f"{response_match:.2f}"
|
|
629
|
+
|
|
630
|
+
llm_eval = test_case.get("llm_eval_scores", {}).get("average")
|
|
631
|
+
if llm_eval is not None:
|
|
632
|
+
scores["LLM Eval"] = f"{llm_eval:.2f}"
|
|
633
|
+
|
|
634
|
+
if scores:
|
|
635
|
+
summary_data.append((model_name, test_case_id, scores))
|
|
636
|
+
|
|
637
|
+
except Exception as e:
|
|
638
|
+
print(f"Error processing results for {model_dir.name}: {e}")
|
|
639
|
+
|
|
640
|
+
# Print formatted output
|
|
641
|
+
if not summary_data:
|
|
642
|
+
print("No summary data to display.")
|
|
643
|
+
return
|
|
644
|
+
|
|
645
|
+
# Define headers and find max score lengths
|
|
646
|
+
headers = ["Tool Match", "Response Match", "LLM Eval"]
|
|
647
|
+
|
|
648
|
+
# Print header
|
|
649
|
+
header_line = (
|
|
650
|
+
f"{'Model':<{max_model_len}} | {'Test Case':<{max_test_case_len}} | "
|
|
651
|
+
f"{'Tool Match':<12} | {'Response Match':<16} | {'LLM Eval':<10}"
|
|
652
|
+
)
|
|
653
|
+
print(header_line)
|
|
654
|
+
print("-" * len(header_line))
|
|
655
|
+
|
|
656
|
+
# Print data rows
|
|
657
|
+
for model_name, test_case_id, scores in summary_data:
|
|
658
|
+
tool_score = scores.get("Tool Match", "N/A")
|
|
659
|
+
response_score = scores.get("Response Match", "N/A")
|
|
660
|
+
llm_score = scores.get("LLM Eval", "N/A")
|
|
661
|
+
|
|
662
|
+
print(
|
|
663
|
+
f"{model_name:<{max_model_len}} | {test_case_id:<{max_test_case_len}} | "
|
|
664
|
+
f"{tool_score:<12} | {response_score:<16} | {llm_score:<10}"
|
|
665
|
+
)
|
|
666
|
+
|
|
667
|
+
def _get_model_stats(self, model_path: str) -> Dict[str, Any]:
|
|
668
|
+
"""Process results for a single model and return stats."""
|
|
669
|
+
model_stats = {}
|
|
670
|
+
results_file = os.path.join(model_path, "results.json")
|
|
671
|
+
if not os.path.exists(results_file):
|
|
672
|
+
return model_stats
|
|
673
|
+
|
|
674
|
+
results_data = self.file_service.load_json(results_file)
|
|
675
|
+
model_name = results_data.get("model_name", os.path.basename(model_path))
|
|
676
|
+
model_stats[model_name] = {}
|
|
677
|
+
|
|
678
|
+
for test_case in results_data.get("test_cases", []):
|
|
679
|
+
test_case_id = test_case.get("test_case_id")
|
|
680
|
+
if not test_case_id:
|
|
681
|
+
continue
|
|
682
|
+
|
|
683
|
+
scores = {}
|
|
684
|
+
tool_match = test_case.get("tool_match_scores", {}).get("average")
|
|
685
|
+
if tool_match is not None:
|
|
686
|
+
scores["avg_tool_match"] = tool_match
|
|
687
|
+
|
|
688
|
+
response_match = test_case.get("response_match_scores", {}).get("average")
|
|
689
|
+
if response_match is not None:
|
|
690
|
+
scores["avg_response_match"] = response_match
|
|
691
|
+
|
|
692
|
+
llm_eval = test_case.get("llm_eval_scores", {}).get("average")
|
|
693
|
+
if llm_eval is not None:
|
|
694
|
+
scores["avg_llm_eval"] = llm_eval
|
|
695
|
+
|
|
696
|
+
if scores:
|
|
697
|
+
model_stats[model_name][test_case_id] = scores
|
|
698
|
+
return model_stats
|
|
699
|
+
|
|
583
700
|
def _save_execution_stats(self, base_results_path: str, start_time: float):
|
|
584
701
|
"""Save overall execution statistics."""
|
|
585
702
|
end_time = time.time()
|
|
586
703
|
total_execution_time = end_time - start_time
|
|
704
|
+
stats = {"total_execution_time": total_execution_time, "models": {}}
|
|
705
|
+
|
|
706
|
+
try:
|
|
707
|
+
for model_dir in os.listdir(base_results_path):
|
|
708
|
+
model_path = os.path.join(base_results_path, model_dir)
|
|
709
|
+
if not os.path.isdir(model_path):
|
|
710
|
+
continue
|
|
711
|
+
model_stats = self._get_model_stats(model_path)
|
|
712
|
+
stats["models"].update(model_stats)
|
|
713
|
+
|
|
714
|
+
except Exception as e:
|
|
715
|
+
print(f"Error processing results for stats: {e}")
|
|
587
716
|
|
|
588
|
-
stats = {"total_execution_time": total_execution_time}
|
|
589
717
|
stats_path = os.path.join(base_results_path, "stats.json")
|
|
590
718
|
self.file_service.save_json(stats, stats_path)
|
|
591
719
|
|
|
@@ -593,9 +721,9 @@ class EvaluationRunner:
|
|
|
593
721
|
print(f"Total execution time: {total_execution_time:.2f} seconds")
|
|
594
722
|
|
|
595
723
|
|
|
596
|
-
def main(config_path: str):
|
|
724
|
+
def main(config_path: str, verbose: bool = False):
|
|
597
725
|
"""Main entry point for the evaluation script."""
|
|
598
|
-
orchestrator = EvaluationRunner()
|
|
726
|
+
orchestrator = EvaluationRunner(verbose=verbose)
|
|
599
727
|
orchestrator.run_evaluation(config_path)
|
|
600
728
|
|
|
601
729
|
|
|
@@ -609,5 +737,11 @@ if __name__ == "__main__":
|
|
|
609
737
|
type=str,
|
|
610
738
|
help="Path to the evaluation test_suite_config.json file.",
|
|
611
739
|
)
|
|
740
|
+
parser.add_argument(
|
|
741
|
+
"-v",
|
|
742
|
+
"--verbose",
|
|
743
|
+
action="store_true",
|
|
744
|
+
help="Enable verbose output.",
|
|
745
|
+
)
|
|
612
746
|
args = parser.parse_args()
|
|
613
|
-
main(args.test_suite_config_path)
|
|
747
|
+
main(args.test_suite_config_path, args.verbose)
|
|
@@ -362,10 +362,12 @@ class MessageProcessor:
|
|
|
362
362
|
content = data.get("content", {})
|
|
363
363
|
parts = content.get("parts", [])
|
|
364
364
|
|
|
365
|
-
|
|
366
|
-
function_call =
|
|
367
|
-
|
|
365
|
+
for part in parts:
|
|
366
|
+
function_call = part.get("function_call", {})
|
|
367
|
+
if not function_call:
|
|
368
|
+
continue
|
|
368
369
|
|
|
370
|
+
call_id = function_call.get("id")
|
|
369
371
|
if call_id and call_id not in processed_tool_calls:
|
|
370
372
|
tool_call = ToolCall(
|
|
371
373
|
call_id=call_id,
|
|
@@ -189,7 +189,7 @@ def get_publish_a2a_func(
|
|
|
189
189
|
def get_namespace(
|
|
190
190
|
component: "WebUIBackendComponent" = Depends(get_sac_component),
|
|
191
191
|
) -> str:
|
|
192
|
-
"""FastAPI dependency to get the
|
|
192
|
+
"""FastAPI dependency to get the namespace."""
|
|
193
193
|
log.debug("[Dependencies] get_namespace called")
|
|
194
194
|
return component.get_namespace()
|
|
195
195
|
|
|
@@ -51,7 +51,7 @@ This design creates a clean separation of concerns, where the web layer (FastAPI
|
|
|
51
51
|
- `get_agent_registry() -> AgentRegistry`: Returns the shared `AgentRegistry` instance.
|
|
52
52
|
- `get_core_a2a_service() -> CoreA2AService`: Returns the core service for creating A2A messages.
|
|
53
53
|
- `get_shared_artifact_service() -> Optional[BaseArtifactService]`: Returns the service for artifact storage.
|
|
54
|
-
- `get_namespace() -> str`: Returns the configured
|
|
54
|
+
- `get_namespace() -> str`: Returns the configured namespace.
|
|
55
55
|
- `get_gateway_id() -> str`: Returns the unique ID of this gateway.
|
|
56
56
|
- **Core Logic Methods:**
|
|
57
57
|
- `publish_a2a(topic: str, payload: Dict, user_properties: Optional[Dict] = None)`: Publishes a message onto the A2A messaging fabric. This is the primary method for sending data to agents.
|
|
@@ -44,7 +44,7 @@ class TaskService:
|
|
|
44
44
|
publish_func: A callable function (provided by WebUIBackendComponent)
|
|
45
45
|
to publish messages to the A2A messaging layer.
|
|
46
46
|
Expected signature: publish_func(topic: str, payload: Dict, user_properties: Optional[Dict])
|
|
47
|
-
namespace: The
|
|
47
|
+
namespace: The namespace string.
|
|
48
48
|
gateway_id: The unique ID of this gateway instance.
|
|
49
49
|
sse_manager: An instance of the SSEManager.
|
|
50
50
|
task_context_map: Shared dictionary to store task context.
|
solace_agent_mesh/llm_detail.txt
CHANGED
|
@@ -1907,7 +1907,7 @@ No public classes, functions, or constants defined.
|
|
|
1907
1907
|
- `get_all_agents() -> List[AgentCard]` - Retrieves all currently discovered agent cards from the registry
|
|
1908
1908
|
- `process_discovery_message(agent_card: AgentCard)` - Processes an incoming agent card discovery message
|
|
1909
1909
|
- `agent_registry: AgentRegistry` - The shared agent registry instance
|
|
1910
|
-
- `namespace: str` - The
|
|
1910
|
+
- `namespace: str` - The namespace string
|
|
1911
1911
|
- `log_identifier: str` - Identifier used for logging
|
|
1912
1912
|
|
|
1913
1913
|
**Functions:**
|
|
@@ -2355,7 +2355,7 @@ This design creates a clean separation of concerns, where the web layer (FastAPI
|
|
|
2355
2355
|
- `get_agent_registry() -> AgentRegistry`: Returns the shared `AgentRegistry` instance.
|
|
2356
2356
|
- `get_core_a2a_service() -> CoreA2AService`: Returns the core service for creating A2A messages.
|
|
2357
2357
|
- `get_shared_artifact_service() -> Optional[BaseArtifactService]`: Returns the service for artifact storage.
|
|
2358
|
-
- `get_namespace() -> str`: Returns the configured
|
|
2358
|
+
- `get_namespace() -> str`: Returns the configured namespace.
|
|
2359
2359
|
- `get_gateway_id() -> str`: Returns the unique ID of this gateway.
|
|
2360
2360
|
- **Core Logic Methods:**
|
|
2361
2361
|
- `publish_a2a(topic: str, payload: Dict, user_properties: Optional[Dict] = None)`: Publishes a message onto the A2A messaging fabric. This is the primary method for sending data to agents.
|
|
@@ -61,11 +61,11 @@ apps:
|
|
|
61
61
|
namespace: ${NAMESPACE}
|
|
62
62
|
supports_streaming: true
|
|
63
63
|
agent_name: "__COMPONENT_PASCAL_CASE_NAME__"
|
|
64
|
-
display_name: "
|
|
64
|
+
display_name: "__COMPONENT_SPACED_CAPITALIZED_NAME__ Agent"
|
|
65
65
|
model: *general_model
|
|
66
66
|
|
|
67
67
|
instruction: |
|
|
68
|
-
|
|
68
|
+
You're __COMPONENT_SPACED_CAPITALIZED_NAME__ agent.
|
|
69
69
|
Configure its specific instructions here.
|
|
70
70
|
|
|
71
71
|
tools:
|
|
@@ -110,7 +110,7 @@ apps:
|
|
|
110
110
|
description: "__PLUGIN_DESCRIPTION__"
|
|
111
111
|
defaultInputModes: ["text"]
|
|
112
112
|
defaultOutputModes: ["text"]
|
|
113
|
-
skills:
|
|
113
|
+
skills: [] # Remove brackets after adding entries
|
|
114
114
|
# - id: "example_tool"
|
|
115
115
|
# name: "Example Tool"
|
|
116
116
|
# description: "An example tool provided by the __PLUGIN_PASCAL_CASE_NAME__ plugin."
|
|
@@ -6,6 +6,8 @@ shared_config:
|
|
|
6
6
|
broker_password: ${SOLACE_BROKER_PASSWORD, default}
|
|
7
7
|
broker_vpn: ${SOLACE_BROKER_VPN, default}
|
|
8
8
|
temporary_queue: ${USE_TEMPORARY_QUEUES, true}
|
|
9
|
+
# Ensure high enough limits if many agents are running
|
|
10
|
+
# max_connection_retries: -1 # Retry forever
|
|
9
11
|
|
|
10
12
|
- models:
|
|
11
13
|
planning: &planning_model
|
|
@@ -16,6 +18,12 @@ shared_config:
|
|
|
16
18
|
api_base: ${LLM_SERVICE_ENDPOINT} # Use env var for endpoint URL
|
|
17
19
|
# 'api_key' provides authentication.
|
|
18
20
|
api_key: ${LLM_SERVICE_API_KEY} # Use env var for API key
|
|
21
|
+
# Enable parallel tool calls for planning model
|
|
22
|
+
parallel_tool_calls: true
|
|
23
|
+
|
|
24
|
+
# max_tokens: ${MAX_TOKENS, 16000} # Set a reasonable max token limit for planning
|
|
25
|
+
# temperature: 0.1 # Lower temperature for more deterministic planning
|
|
26
|
+
|
|
19
27
|
|
|
20
28
|
general: &general_model
|
|
21
29
|
# This dictionary structure tells ADK to use the LiteLlm wrapper.
|
|
@@ -45,7 +53,6 @@ shared_config:
|
|
|
45
53
|
api_key: ${LLM_SERVICE_API_KEY} # Use env var for API key
|
|
46
54
|
|
|
47
55
|
multimodal: &multimodal_model "gemini-2.5-flash-preview-04-17"
|
|
48
|
-
gemini_pro: &gemini_pro_model "gemini-2.5-pro-exp-03-25"
|
|
49
56
|
|
|
50
57
|
- services:
|
|
51
58
|
# Default session service configuration
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: solace-agent-mesh
|
|
3
|
-
Version: 1.0.
|
|
3
|
+
Version: 1.0.7
|
|
4
4
|
Summary: Solace Agent Mesh is an open-source framework for building event-driven, multi-agent AI systems where specialized agents collaborate on complex tasks.
|
|
5
5
|
Project-URL: Homepage, https://github.com/SolaceLabs/solace-agent-mesh
|
|
6
6
|
Project-URL: Repository, https://github.com/SolaceLabs/solace-agent-mesh
|
|
@@ -236,6 +236,7 @@ Requires-Dist: markdownify>=1.1.0
|
|
|
236
236
|
Requires-Dist: markitdown[all]==0.1.1
|
|
237
237
|
Requires-Dist: mermaid-cli
|
|
238
238
|
Requires-Dist: numpy>=2.2.5
|
|
239
|
+
Requires-Dist: openai==1.99.9
|
|
239
240
|
Requires-Dist: pandas>=2.2.3
|
|
240
241
|
Requires-Dist: playwright>=1.54.0
|
|
241
242
|
Requires-Dist: plotly>=6.0.1
|
|
@@ -261,6 +262,8 @@ Provides-Extra: employee-tools
|
|
|
261
262
|
Requires-Dist: holidays>=0.24; extra == 'employee-tools'
|
|
262
263
|
Provides-Extra: gcs
|
|
263
264
|
Requires-Dist: google-cloud-storage>=2.0; extra == 'gcs'
|
|
265
|
+
Provides-Extra: s3
|
|
266
|
+
Requires-Dist: boto3<2.0,>=1.35.0; extra == 's3'
|
|
264
267
|
Provides-Extra: vertex
|
|
265
268
|
Requires-Dist: google-cloud-aiplatform>=1.38; extra == 'vertex'
|
|
266
269
|
Description-Content-Type: text/markdown
|
|
@@ -310,7 +313,7 @@ SAM creates a standardized communication layer where AI agents can:
|
|
|
310
313
|
|
|
311
314
|
SAM is built on top of the Solace AI Connector (SAC) which allows Solace Platform Event Brokers to connect to AI models and services and Google's Agent Development Kit (ADK) for AI logic and tool integrations.
|
|
312
315
|
|
|
313
|
-
The result? A fully asynchronous, event-driven and decoupled AI agent architecture ready for production deployment. It is robust, reliable and easy to maintain.
|
|
316
|
+
The result? A fully asynchronous, event-driven and decoupled AI agent architecture ready for production deployment. It is robust, reliable and easy to maintain.
|
|
314
317
|
|
|
315
318
|
---
|
|
316
319
|
|