ibm-watsonx-orchestrate-evaluation-framework 1.0.0__py3-none-any.whl

ibm_watsonx_orchestrate_evaluation_framework-1.0.0.dist-info/METADATA

@@ -0,0 +1,322 @@
+Metadata-Version: 2.4
+Name: ibm-watsonx-orchestrate-evaluation-framework
+Version: 1.0.0
+Summary: The WxO evaluation framework
+Author-email: Haode Qi <Haode.Qi@ibm.com>
+License: MIT
+Requires-Python: <3.14,>=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: rich~=13.9.4
+Requires-Dist: ibm-watsonx-ai~=1.3.6
+Requires-Dist: pydantic~=2.10.6
+Requires-Dist: pyyaml~=6.0.2
+Requires-Dist: jinja2~=3.1.5
+Requires-Dist: python-dotenv~=1.0.1
+Requires-Dist: dataclasses-json~=0.6.7
+Requires-Dist: jsonargparse~=4.37.0
+Requires-Dist: networkx~=3.4.2
+Requires-Dist: matplotlib~=3.10.1
+Requires-Dist: numpy~=1.26.4
+Requires-Dist: langchain-openai~=0.3.23
+Provides-Extra: dev
+Requires-Dist: setuptools~=70.3.0; extra == "dev"
+Requires-Dist: pytest<9.0.0,>=8.3.4; extra == "dev"
+Requires-Dist: pytest-cov==6.0.0; extra == "dev"
+Requires-Dist: pytest-mock==3.14.0; extra == "dev"
+Requires-Dist: pytest-asyncio==0.25.1; extra == "dev"
+Requires-Dist: coverage[toml]>=6.5; extra == "dev"
+Requires-Dist: black~=22.3.0; extra == "dev"
+Requires-Dist: pylint~=2.16.4; extra == "dev"
+Provides-Extra: rag-eval
+Requires-Dist: tqdm~=4.67.1; extra == "rag-eval"
+Requires-Dist: sentence-transformers~=3.3.1; extra == "rag-eval"
+Requires-Dist: scikit-learn~=1.6.1; extra == "rag-eval"
+Requires-Dist: pandas~=2.1.4; extra == "rag-eval"
+Requires-Dist: notebook~=7.4.1; extra == "rag-eval"
+Requires-Dist: ipywidgets~=8.1.6; extra == "rag-eval"
+Requires-Dist: jupyter_contrib_nbextensions; extra == "rag-eval"
+Requires-Dist: jupyter~=1.1.1; extra == "rag-eval"
+Dynamic: license-file
+
+# WXO-agent evaluation framework
+
+- This framework is designed to test a tool-calling agent's ability to make real API calls against a `wxo-dev` testing tenant on your local wxo-lite server instance.
+
+- As an LLM-as-agent evaluation framework, we aim to test the agent's ability to do the following:
+    - We use a ground truth to evaluate our conversation against after inference. The process of inference is manifested through a user-LLM and agent simulation. Please set `enable_verbose_logging: True` in your configuration.
+    - Make real API calls correctly and efficiently. We provide metrics which measure the number of bad tool calls made by the agent, normalized against the number of ground truth calls made. 
+
+- The `benchmarks/` folder contains test-cases for the different agents we have evaluated so far. They are segmented by release versions of the `wxo-domains` repository.
+- The agent calls the `runs/` endpoint of the wxo-lite server instance, and the actual tool code is executed on the server side. The server database is not visible to our framework. 
+
+## prerequisite
+Follow the [SDK setup guide](https://github.ibm.com/WatsonOrchestrate/wxo-clients/tree/main) to install the SDK. Make sure you are using version 1.2.0 of the SDK as this is the version this framework requires.
+
+## setup for evaluation framework
+Run the following command to install evaluation framework in the same env:
+``` 
+pip install -e .
+```
+
+
+## quick experiment against the default wxo-dev env
+```bash
+orchestrate server start
+export WATSONX_SPACE_ID=""
+export WATSONX_API_KEY=""
+```
+
+Import sample hr tools and agent into your default `wxo-dev` env:
+```bash
+orchestrate tools import -f benchmarks/hr_sample/tools.py -k python
+orchestrate agents import -f benchmarks/hr_sample/hr_agent.json
+```
+
+Run the main script:
+```bash
+python -m wxo_agentic_evaluation.main --config benchmarks/hr_sample/config.yaml --output_dir=results/test --num_workers=2
+```
+Note:  
+1. This approach uses the default `wxo-dev` tenant already available in your orchestrate env if you have used wxo-lite before.
+2. ADK also reads the env environments variable. If you have an env conflict, start the wxo-lite server before exporting the envs.
+
+
+## run against a deployed local env
+
+1. start the orchestrated server: `orchestrate server start`
+2. create a simple test case like the following save in a folder like `benchmarks/TEST_CASE_NAME`:
+```JSON
+{
+  "agent": "NAME_OF_THE_AGENT",
+  "goals": {
+    "summarize": []
+  },
+  "goal_details": [
+    {
+      "type": "text",
+      "name": "summarize",
+      "response": "Your timeoff schedule for 20250101 to 20250303 is: 20250105",
+      "keywords": [
+        "20250105"
+      ]
+    }
+  ],
+  "mine_fields": [],
+  "story": "Your username is nwaters and you want to find out timeoff schedule from 20250101 to 20250303."
+}
+```
+Note:
+- The target agent name can be found `orchestrate agents list`
+- the example shown only evaluate the final response for the agent. For more sophisticated examples, follow `benchmarks/hr_sample/data_simple.json` or `benchmarks/hr_sample/data_complex.json`.
+
+
+3. create a test config yaml like the following:
+```YAML
+test_paths:
+  - benchmarks/TEST_CASE_NAME
+
+auth_config:
+  url: http://localhost:4321
+  tenant_name: wxo-dev
+
+output_dir: "results/TEST_CASE_NAME/MODEL_NAME"
+```
+
+
+NOTE: run `orchestrate env list` to find the name of the active tenant. for default `local` tenant, the name should be `wxo-dev`
+
+4. Run the test:
+```bash
+export WATSONX_SPACE_ID=""
+export WATSONX_API_KEY=""
+python -m wxo_agentic_evaluation.main --config benchmarks/hr_sample/config.yaml
+```
+
+NOTE: if your run fails for any reason and doesn't cover all the test cases, you can re-run the main script with `--skip_available_results=True` to skip the test cases that are already completed.
+
+## analyze error
+```bash
+python -m wxo_agentic_evaluation.analyze_run --input_data results/hr_sample/llama_3_2_90b/messages/data_simple.messages.json --ground_truth benchmarks/hr_sample/data_simple.json --enable_verbose_logging False
+```
+You can also run the analyze script on a batch of test cases in a folder
+```bash
+python -m wxo_agentic_evaluation.analyze_run --input_data results/hr_sample/llama_3_2_90b/messages/ --ground_truth benchmarks/hr_sample/ --enable_verbose_logging False --enable_verbose_logging False
+```
+
+
+## Run Against a SaaS Tenant (Orchestrate SDK ≥ 1.2)
+
+This section describes how to run benchmark tests using a **SaaS-based Orchestrate tenant**. The rest of the setup (test case creation, config structure, etc.) is similar to the [local setup](#run-against-a-deployed-local-env) and can be referred to as needed.
+
+### Prerequisites
+
+- **Orchestrate SDK version ≥ 1.2** is required.
+- Access to the **production SaaS Orchestrate instance** or **staging SaaS Orchestrate instance**.
+
+---
+
+### 1. Get Authentication Details
+
+1. Visit the Orchestrate UI [ Prod /staging]: 
+
+- **AWS Production us-east-1:** [https://dl.watson-orchestrate.ibm.com](https://dl.watson-orchestrate.ibm.com)
+For other locations, please use the designated url for your data center.
+- **AWS Staging:** [https://staging-wa.watson-orchestrate.ibm.com](https://staging-wa.watson-orchestrate.ibm.com)
+- **IBM Cloud Production us-south:** [https://us-south.watson-orchestrate.cloud.ibm.com](https://us-south.watson-orchestrate.cloud.ibm.com)
+
+2. Log in and click the **Settings** button (top-right corner).
+
+3. Open the **API details** tab, then copy the **Instance URL** and generate an **API Key**.
+
+4. For more detailed instructions, refer to this guide:  
+   https://developer.ibm.com/apis/catalog/watsonorchestrate--custom-assistants/Getting+the+API+endpoint
+
+---
+
+### 2. Add the SaaS Tenant
+
+Run the following command:
+
+```bash
+orchestrate env add -n saas \
+  -u [INSTANCE_URL] \
+  -t mcsp \
+  -a
+```
+if using stagging setup then pass the --iam-url argument as follow:
+- For AWS:
+```bash
+orchestrate env add -n saas \
+  -u [INSTANCE_URL] \
+  --iam-url https://iam.platform.test.saas.ibm.com \
+  -a
+```
+
+ - For IBM Cloud:
+ ```bash
+ orchestrate env add -n saas \
+  -u [INSTANCE_URL] \
+  --iam-url https://iam.test.cloud.ibm.com \
+  -a
+ ```
+
+> When prompted, paste the API key generated above.
+
+---
+
+### 3. Set the IAM API Key Environment Variable
+
+```bash
+export WATSONX_IAM_SAAS_APIKEY=[your_generated_api_key]
+```
+
+---
+
+### 4. Update Your Test Config YAML
+
+Make sure your YAML config includes the correct SaaS tenant name:
+
+```yaml
+test_paths:
+  - benchmarks/TEST_CASE_NAME
+
+auth_config:
+  url: [INSTANCE_URL]
+  tenant_name: saas
+
+output_dir: "results/TEST_CASE_NAME/MODEL_NAME"
+```
+- Use staging url if using the staging set-up.
+---
+
+###  5. Run the Simulation in SaaS Mode
+
+```bash
+python -m wxo_agentic_evaluation.main --config benchmarks/hr_sample/config.yaml 
+```
+
+---
+
+### Batch Test case Generation
+
+For full instructions on setting up tools, writing stories, configuring the pipeline, and generating batch test cases, see the [Batch Test case Generation Guide](./benchmarks/batch_sample/README.MD).
+
+## Workflow diagram
+
+To help better understand the workflow, this is a diagram of how this repo works together with wxO lite python SDK and a wxO runtime.
+
+![Alt text](./doc/assets/workflow.png "Workflow")
+
+Inputs:
+- [a test config yaml](benchmarks/hr_sample/config.yaml)
+- a json file containing test cases, see [example 1](benchmarks/hr_sample/data_complex.json) or [example 2](benchmarks/hr_sample/data_simple.json) as a reference
+- optionally, a `tools.py` file for tools definition and one or more agent definitions e.g. `benchmarks/hr_sample/hr_agent.json`. Alternatively, these files are not needed if you have a tenant already set up with such tools and agents
+
+Steps:
+1. (optional) this repo will call wxO Lite python SDK (which calls several endpoints exposed by the wxO runtime) to set up the environment needed for the evaluation run, tools and agents will be imported
+2. Create test cases by following the sample instructions at [benchmarks/sap_successfactors_sample/annotation/README.md](benchmarks/sap_successfactors_sample/annotation/README.md)
+3. Start the evaluation run by calling the `wxo_agentic_evaluation.main` script of this repo, which will invoke the `/runs` endpoint of the wxO runtime to simulate conversations with the agent
+4. Reports and metrics will be generated by this repo
+5. (optional) this repo will call wxO Lite python SDK (which calls several endpoints exposed by the wxO runtime) to clean up the environment, to avoid the tools and agents affecting subsequent runs with the same tenant
+6. (optional) You can generate further error analysis by using the `wxo_agentic_evaluation.analyze_run` script from this repo
+
+
+## results
+### workday
+
+| Model                         | User Setting | Total Step  | Agent Step | Journey Success  | Wrong Function Calls  | Bad Calls | Wrong Parameters  | Wrong Routing Calls  | Text Match | Test Cases | WXO Avg. Response Time  |
+|-------------------------------|------------- |-------------|------------|------------------|-----------------------|-----------|-------------------|----------------------|------------|------------|-------------------------|
+| llama-3-2-90b-vision-instruct | normal       | 8.13        | 4.21       | 0.87             | 0.01                  | 0.0       | 0.20              | 0.00                 | 0.95       | 38         | 15.09                   |
+| llama-3-2-90b-vision-instruct | verbose      | 11.76       | 6.11       | 0.79             | 0.02                  | 0.0       | 0.19              | 0.00                 | 0.86       | 38         | 14.32                   |
+| llama-3-405b-instruct         | normal       | 9.66        | 5.03       | 0.82             | 0.02                  | 0.0       | 0.47              | 0.04                 | 0.89       | 38         | 13.36                   |
+| llama-3-405b-instruct         | verbose      | 11.76       | 6.11       | 0.84             | 0.05                  | 0.0       | 0.70              | 0.04                 | 0.92       | 38         | 12.21                   |
+
+You can find the detailed results under [results/workday](results/workday)
+
+### sap successfactor (rel-1.7)
+
+| Model                         | User Setting | Total Step  | Agent Step | Journey Success  | Wrong Function Calls  | Bad Calls | Wrong Parameters  | Wrong Routing Calls  | Text Match | Test Cases | WXO Avg. Response Time  |
+|------------------------------ |--------------|-------------|------------|------------------|-----------------------|-----------|-------------------|----------------------|------------|------------|-------------------------|
+| llama-3-2-90b-vision-instruct | normal       | 10.32       | 5.84       | 0.73             | 0.04                  | 0.0       | 0.06              | 0.08                 | 0.84       | 38         | -                       |
+| llama-3-2-90b-vision-instruct | verbose      | 11.19       | 6.35       | 0.68             | 0.04                  | 0.0       | 0.08              | 0.16                 | 0.81       | 38         | -                       |
+| llama-3-405b-instruct         | normal       | 11.41       | 6.24       | 0.46             | 0.01                  | 0.0       | 0.23              | 0.02                 | 0.62       | 38         | -                       |
+| llama-3-405b-instruct         | verbose      | 15.32       | 8.38       | 0.46             | 0.04                  | 0.0       | 0.40              | 0.06                 | 0.62       | 38         | -                       |
+
+You can find the detailed results under [results/sap_successfactor_4](results/sap_successfactor_4)
+
+
+## METRICS KEY
+
+| Metric | Description | Calculation | Range/Type |
+|--------|-------------|-------------|------------|
+| **Total Step** | Total number of messages/steps in the conversation | Count of all messages in the conversation | Integer ≥ 0 |
+| **Agent Step** | Number of assistant responses (text or tool calls) | Count of messages where `role == "assistant"` and `type` is text or tool_call | Integer ≥ 0 |
+| **Ground Truth Calls** | Expected number of tool calls based on ground truth | Count of goal_details with `type == ContentType.tool_call` | Integer ≥ 0 |
+| **Journey Success** | Whether the agent completed tasks in the correct order | `is_topological_sort(ground_truth.goals, labelled_messages)` | Boolean |
+| **Wrong Function Calls** | Number of calls to non-existent or unexpected functions | Count of labelled_messages containing "_WRONG_FUNCTION_CALL" | Integer ≥ 0 |
+| **Bad Calls** | Reserved metric for future use | Currently hardcoded to 0 | Integer (0) |
+| **Wrong Parameters** | Number of tool calls with incorrect parameters | Count of labelled_messages containing "_WRONG_PARAMETERS" | Integer ≥ 0 |
+| **Wrong Routing Calls** | Number of incorrect agent routing calls | Count of labelled_messages containing "_WRONG_ROUTING_CALL" | Integer ≥ 0 |
+| **Text Match** | Quality of final text summary | "Keyword Mismatch" \| "Semantic Mismatch" \| "Summary Matched" | Categorical |
+| **Tool Call Accuracy** | Percentage of non-routing tool calls that were executed correctly | `correct_tool_calls / non_transfer_tool_calls` | Float 0.0-1.0 |
+| **Tool Call Relevancy** | Percentage of non-routing tool calls that were relevant to the task | `(relevant_tool_calls - expected_routing_calls) / non_transfer_tool_calls` | Float 0.0-1.0 |
+| **Agent Routing Accuracy** | Percentage of routing calls that were executed correctly | `expected_routing_calls / total_routing_calls` | Float 0.0-1.0 |
+| **WXO Average Response Time (Secs)** | Average response time for agent responses | Mean response time across all agent interactions | Float ≥ 0.0 |
+
+### Key Definitions
+
+- **Relevant Tool Call**: A tool call whose name matches one of the expected tool names defined in the ground truth
+- **Correct Tool Call**: A relevant tool call that also has the correct parameters/arguments
+- **Routing Call**: A tool call whose name starts with "transfer_" (used for agent-to-agent routing)
+- **Non-Transfer Tool Call**: Regular tool calls excluding routing calls (`total_tool_calls - total_routing_calls`)
+- **Expected Routing Call**: A routing call that was both expected and executed correctly
+
+### Averaging Behavior
+
+- **Per Test Case Average**: Total Step, Agent Step, Tool Call Accuracy, Tool Call Relevancy, Agent Routing Accuracy, WXO Average Response Time
+- **Per Ground Truth Calls Average**: Wrong Function Calls, Bad Calls, Wrong Parameters, Wrong Routing Calls
+- **Special Calculations**: 
+  - Journey Success: Proportion of test cases that succeeded (0.0-1.0)
+  - Text Match: Proportion of test cases with "Summary Matched" (0.0-1.0)
+

ibm_watsonx_orchestrate_evaluation_framework-1.0.0.dist-info/RECORD

@@ -0,0 +1,46 @@
+ibm_watsonx_orchestrate_evaluation_framework-1.0.0.dist-info/licenses/LICENSE,sha256=Shgxx7hTdCOkiVRmfGgp_1ISISrwQD7m2f0y8Hsapl4,1083
+wxo_agentic_evaluation/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+wxo_agentic_evaluation/analyze_run.py,sha256=qcdew4htpIg0sxCXXX3QS_XhoPOGg4_CEPYFjZiMsnA,4343
+wxo_agentic_evaluation/annotate.py,sha256=nxYMc6gwfQ-GuNjCPFtbX_-Es5-9XDdbXpMH89yRDdc,1228
+wxo_agentic_evaluation/arg_configs.py,sha256=_ws5GX43rG8HdIZe5JAgb3heQubzpfOGWsvzT9Zfs2A,2016
+wxo_agentic_evaluation/batch_annotate.py,sha256=5e-1FpqjSdk3EaGELHhj493fcJKY3_gcv7NfFXxl3pY,6511
+wxo_agentic_evaluation/data_annotator.py,sha256=to8FfIYMx-JzJ5aRmpMb1SiFS1KTXgdZU2qwowdn6BU,7823
+wxo_agentic_evaluation/evaluation_package.py,sha256=RFo5oC2Gydc7wQ28bDSs5nisnRj22GCnjjrFrn4O2L4,21031
+wxo_agentic_evaluation/inference_backend.py,sha256=8nW3LZg6dLTemrHBmDBx8b2NUvBjvSC4bLLxJX9yPiY,25754
+wxo_agentic_evaluation/llm_matching.py,sha256=ISRMvZq-oC0amVBCPNlghPpMDCPxIjVGgm9DtZWTU40,1501
+wxo_agentic_evaluation/llm_rag_eval.py,sha256=n1e0nffYlgxqu7i4Ef_j1zmo9zXcqc2zfTfNYuMsopc,1675
+wxo_agentic_evaluation/llm_user.py,sha256=Ppc-iKtPGaf6tPoHGYFLqXX-vy8LoQneLTQBVOXLHiA,1422
+wxo_agentic_evaluation/main.py,sha256=H6sLZrCUBfqjh9gPoQN6BHxSby-UKrea705cuIiyknc,8042
+wxo_agentic_evaluation/record_chat.py,sha256=Q5w9ouvVfikms_kYyQ6wgqvNN_DxV400I2HriTOdMfg,5969
+wxo_agentic_evaluation/service_instance.py,sha256=yt7XpwheaRRG8Ri4TFIS5G2p5mnCwvNgj6T7bDF5uTU,6494
+wxo_agentic_evaluation/tool_planner.py,sha256=nOwoq_RMBO5ISRFwrKeblgdbMz50qfmVqHkZRAXjP3s,8075
+wxo_agentic_evaluation/type.py,sha256=QbwEedAYnot9WBVIJVSP23s1KHJc7uFQyOhL_MYEdmI,4832
+wxo_agentic_evaluation/watsonx_provider.py,sha256=GH4PhHIZbSRsiQ29CsZmu8wSVt0KX4htNNQKnSltmfA,5983
+wxo_agentic_evaluation/analytics/tools/analyzer.py,sha256=qvkhoyY4JgneE03cmo-KuMAaQB7iM_Lm0C1sexyPFwY,18056
+wxo_agentic_evaluation/analytics/tools/main.py,sha256=ocwPUlEjyK7PMdXBg5OM2DVDQBcaHT4UjR4ZmEhR0C4,6567
+wxo_agentic_evaluation/analytics/tools/types.py,sha256=IFLKI1CCQwPR2iWjif8AqL_TEq--VbLwdwnMqfJujBw,4461
+wxo_agentic_evaluation/analytics/tools/ux.py,sha256=CcnaefAZwzPx3FW0BzlOx7OBPwNfmA5yVtYB-gYci9w,18324
+wxo_agentic_evaluation/external_agent/external_validate.py,sha256=peW89lsf8u8DXmzmRe9z2BWwPsmhUaNcNpVCtM8tkCM,2629
+wxo_agentic_evaluation/external_agent/types.py,sha256=6WmDGetJGSg92HqPW_Q9K7AEorivTraiw8HgdxaiGxs,1481
+wxo_agentic_evaluation/metrics/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+wxo_agentic_evaluation/metrics/llm_as_judge.py,sha256=bybJQfVWiVh3BoFEZjdBmU9EQO9Ukheu3YWmkI9b1ks,1218
+wxo_agentic_evaluation/metrics/metrics.py,sha256=SdDBWdo3KFycmupQ6mtjwp6WKKNJxGTeM20I_FV9Da0,3913
+wxo_agentic_evaluation/prompt/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+wxo_agentic_evaluation/prompt/answer_relevancy_prompt.jinja2,sha256=vLrMWce-5HlvniCQdtifnl-YdbJfT8-oixzfwulZs98,3839
+wxo_agentic_evaluation/prompt/batch_testcase_prompt.jinja2,sha256=PPnfczM_HjCjho8UKFTL9OYRYshpwqkBKBas8C1jMHY,1807
+wxo_agentic_evaluation/prompt/faithfulness_prompt.jinja2,sha256=DW9OdjeZJbOWrngRqTAVD4w0va_HtA2FR4G1POIIamM,2524
+wxo_agentic_evaluation/prompt/keyword_matching_prompt.jinja2,sha256=7mTkSrppjgPluUAIMTWaT30K7M4J4hyR_LjSjW1Ofq0,1290
+wxo_agentic_evaluation/prompt/keywords_generation_prompt.jinja2,sha256=PiCjr1ag44Jk5xD3F24fLD_bOGYh2sF0i5miY4OrVlc,1890
+wxo_agentic_evaluation/prompt/llama_user_prompt.jinja2,sha256=nDfCD0o9cRYmsgIjzD-RZNQxotlvuqrzdsZIY-vT794,684
+wxo_agentic_evaluation/prompt/semantic_matching_prompt.jinja2,sha256=MltPfEXYyOwEC2xNLl7UsFTxNbr8CwHaEcPqtvKE2r8,2749
+wxo_agentic_evaluation/prompt/template_render.py,sha256=arMtlNcbRIczRwpfZWlITrgDlDsaDCbwGaNPGaJyBko,3080
+wxo_agentic_evaluation/prompt/tool_chain_agent.jinja2,sha256=9RcIjLYoOvtFsf-RgyMfMcj2Fe8fq1wGkE4nG1zamYY,297
+wxo_agentic_evaluation/prompt/tool_planner.jinja2,sha256=gpI7wrcIL6ytCcllr_ZdWwjVLtC_06at3-JkAcAl2HE,1243
+wxo_agentic_evaluation/prompt/examples/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+wxo_agentic_evaluation/prompt/examples/data_simple.json,sha256=O0GeKBf2s9d-7TxkDFFmCEV2sl3e3HcpT11cN0DYFjw,2354
+wxo_agentic_evaluation/utils/__init__.py,sha256=QMxk6hx1CDvCBLFh40WpPZmqFNJtDqwXP7S7cXD6NQE,145
+wxo_agentic_evaluation/utils/utils.py,sha256=8mD6_L_qP-2jQtRkA3Njtg2HFCSQ4FX2NgO4oZq-gow,7994
+ibm_watsonx_orchestrate_evaluation_framework-1.0.0.dist-info/METADATA,sha256=Nx-ZE-egxcobYevowbspUR0hLyo0RKo4UQ-Bz0F5dD8,16276
+ibm_watsonx_orchestrate_evaluation_framework-1.0.0.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
+ibm_watsonx_orchestrate_evaluation_framework-1.0.0.dist-info/top_level.txt,sha256=2okpqtpxyqHoLyb2msio4pzqSg7yPSzwI7ekks96wYE,23
+ibm_watsonx_orchestrate_evaluation_framework-1.0.0.dist-info/RECORD,,

ibm_watsonx_orchestrate_evaluation_framework-1.0.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (80.9.0)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

ibm_watsonx_orchestrate_evaluation_framework-1.0.0.dist-info/licenses/LICENSE

@@ -0,0 +1,22 @@
+(The MIT License)
+
+Copyright (c) 2024, 2025 IBM Corporation
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

ibm_watsonx_orchestrate_evaluation_framework-1.0.0.dist-info/top_level.txt

@@ -0,0 +1 @@
+wxo_agentic_evaluation