@synsci/cli-darwin-arm64 1.1.49

package/bin/skills/nemo-evaluator/references/adapter-system.md

@@ -0,0 +1,340 @@
+# Adapter and Interceptor System
+
+NeMo Evaluator uses an adapter system to process requests and responses between the evaluation engine and model endpoints. The `nemo-evaluator` core library provides built-in interceptors for common use cases.
+
+## Architecture Overview
+
+```
+┌───────────────────────────────────────────────────────────────┐
+│                     Adapter Pipeline                           │
+│                                                                │
+│  Request  ───►  [Interceptor 1]  ───►  [Interceptor 2]  ───►  │
+│                                                                │
+│                              │                                 │
+│                              ▼                                 │
+│                  ┌───────────────────────────────────┐         │
+│                  │      Endpoint Interceptor          │        │
+│                  │   (HTTP call to Model API)         │        │
+│                  └───────────────────────────────────┘         │
+│                              │                                 │
+│                              ▼                                 │
+│  Response  ◄───  [Interceptor 3]  ◄───  [Interceptor 4]  ◄─── │
+│                                                                │
+└───────────────────────────────────────────────────────────────┘
+```
+
+Interceptors execute in order for requests, and in reverse order for responses.
+
+## Configuring Adapters
+
+The adapter configuration is specified in the `target.api_endpoint.adapter_config` section:
+
+```yaml
+target:
+  api_endpoint:
+    model_id: meta/llama-3.1-8b-instruct
+    url: https://integrate.api.nvidia.com/v1/chat/completions
+    api_key_name: NGC_API_KEY
+    adapter_config:
+      interceptors:
+        - name: system_message
+          config:
+            system_message: "You are a helpful assistant."
+        - name: caching
+          config:
+            cache_dir: "./cache"
+        - name: endpoint
+        - name: reasoning
+          config:
+            start_reasoning_token: "<think>"
+            end_reasoning_token: "</think>"
+```
+
+## Available Interceptors
+
+### System Message Interceptor
+
+Injects a system prompt into chat requests.
+
+```yaml
+- name: system_message
+  config:
+    system_message: "You are a helpful AI assistant. Think step by step."
+```
+
+**Effect**: Prepends a system message to the messages array.
+
+### Request Logging Interceptor
+
+Logs outbound API requests for debugging and analysis.
+
+```yaml
+- name: request_logging
+  config:
+    max_requests: 1000
+```
+
+### Caching Interceptor
+
+Caches responses to avoid repeated API calls for identical requests.
+
+```yaml
+- name: caching
+  config:
+    cache_dir: "./evaluation_cache"
+    reuse_cached_responses: true
+    save_requests: true
+    save_responses: true
+    max_saved_requests: 1000
+    max_saved_responses: 1000
+```
+
+### Endpoint Interceptor
+
+Performs the actual HTTP communication with the model endpoint. This is typically added automatically and has no configuration parameters.
+
+```yaml
+- name: endpoint
+```
+
+### Reasoning Interceptor
+
+Extracts and removes reasoning tokens (e.g., `<think>` tags) from model responses.
+
+```yaml
+- name: reasoning
+  config:
+    start_reasoning_token: "<think>"
+    end_reasoning_token: "</think>"
+    enable_reasoning_tracking: true
+```
+
+**Effect**: Strips reasoning content from the response and tracks it separately.
+
+### Response Logging Interceptor
+
+Logs API responses.
+
+```yaml
+- name: response_logging
+  config:
+    max_responses: 1000
+```
+
+### Progress Tracking Interceptor
+
+Reports evaluation progress to an external URL.
+
+```yaml
+- name: progress_tracking
+  config:
+    progress_tracking_url: "http://localhost:3828/progress"
+    progress_tracking_interval: 10
+```
+
+### Additional Interceptors
+
+Other available interceptors include:
+- `payload_modifier`: Transforms request parameters
+- `response_stats`: Collects aggregated statistics from responses
+- `raise_client_errors`: Handles and raises exceptions for client errors (4xx)
+
+## Interceptor Chain Example
+
+A typical interceptor chain for evaluation:
+
+```yaml
+adapter_config:
+  interceptors:
+    # Pre-endpoint (request processing)
+    - name: system_message
+      config:
+        system_message: "You are a helpful AI assistant."
+    - name: request_logging
+      config:
+        max_requests: 50
+    - name: caching
+      config:
+        cache_dir: "./evaluation_cache"
+        reuse_cached_responses: true
+
+    # Endpoint (HTTP call)
+    - name: endpoint
+
+    # Post-endpoint (response processing)
+    - name: response_logging
+      config:
+        max_responses: 50
+    - name: reasoning
+      config:
+        start_reasoning_token: "<think>"
+        end_reasoning_token: "</think>"
+```
+
+## Python API Usage
+
+You can also configure adapters programmatically:
+
+```python
+from nemo_evaluator.adapters.adapter_config import AdapterConfig, InterceptorConfig
+from nemo_evaluator.api.api_dataclasses import ApiEndpoint, EndpointType
+
+adapter_config = AdapterConfig(
+    interceptors=[
+        InterceptorConfig(
+            name="system_message",
+            config={"system_message": "You are a helpful assistant."}
+        ),
+        InterceptorConfig(
+            name="caching",
+            config={
+                "cache_dir": "./cache",
+                "reuse_cached_responses": True
+            }
+        ),
+        InterceptorConfig(name="endpoint"),
+        InterceptorConfig(
+            name="reasoning",
+            config={
+                "start_reasoning_token": "<think>",
+                "end_reasoning_token": "</think>"
+            }
+        )
+    ]
+)
+
+api_endpoint = ApiEndpoint(
+    url="http://localhost:8080/v1/chat/completions",
+    type=EndpointType.CHAT,
+    model_id="my_model",
+    adapter_config=adapter_config
+)
+```
+
+## OpenAI API Compatibility
+
+NeMo Evaluator supports OpenAI-compatible endpoints with different endpoint types:
+
+### Chat Completions
+
+```yaml
+target:
+  api_endpoint:
+    type: chat  # or omit, chat is default
+    url: http://endpoint/v1/chat/completions
+```
+
+### Text Completions
+
+```yaml
+target:
+  api_endpoint:
+    type: completions
+    url: http://endpoint/v1/completions
+```
+
+### Vision-Language Models
+
+```yaml
+target:
+  api_endpoint:
+    type: vlm
+    url: http://endpoint/v1/chat/completions
+```
+
+## Error Handling
+
+Configure error handling via the `log_failed_requests` option:
+
+```yaml
+adapter_config:
+  log_failed_requests: true
+  interceptors:
+    - name: raise_client_errors
+    # ... other interceptors
+```
+
+## Debugging
+
+### Enable Logging Interceptors
+
+Add request and response logging to debug issues:
+
+```yaml
+adapter_config:
+  interceptors:
+    - name: request_logging
+      config:
+        max_requests: 100
+    - name: endpoint
+    - name: response_logging
+      config:
+        max_responses: 100
+```
+
+### Common Issues
+
+**Issue: System message not applied**
+
+Ensure the `system_message` interceptor is listed before the `endpoint` interceptor.
+
+**Issue: Cache not being used**
+
+Check that `reuse_cached_responses: true` is set and the cache directory exists:
+```yaml
+- name: caching
+  config:
+    cache_dir: "./cache"
+    reuse_cached_responses: true
+```
+
+**Issue: Reasoning tokens not extracted**
+
+Verify the token patterns match your model's output format:
+```yaml
+- name: reasoning
+  config:
+    start_reasoning_token: "<think>"  # Must match model output exactly
+    end_reasoning_token: "</think>"
+```
+
+## Custom Interceptor Discovery
+
+NeMo Evaluator supports discovering custom interceptors via the `DiscoveryConfig` within `AdapterConfig`. You can specify modules or directories where your custom interceptors are located:
+
+```yaml
+adapter_config:
+  discovery:
+    modules:
+      - "my_custom.interceptors"
+      - "my_package.adapters"
+    dirs:
+      - "/path/to/custom/interceptors"
+  interceptors:
+    - name: my_custom_interceptor
+      config:
+        custom_option: value
+```
+
+Custom interceptors must implement the standard interceptor interface expected by `nemo-evaluator`.
+
+## Additional AdapterConfig Options
+
+Beyond interceptors, `AdapterConfig` supports these additional fields:
+
+| Field | Description |
+|-------|-------------|
+| `discovery` | Configure custom interceptor discovery |
+| `post_eval_hooks` | List of hooks to run after evaluation |
+| `endpoint_type` | Default endpoint type (e.g., "chat") |
+| `caching_dir` | Legacy option for response caching |
+| `generate_html_report` | Generate HTML report of results |
+| `log_failed_requests` | Log requests that fail |
+| `tracking_requests_stats` | Enable request statistics |
+| `html_report_size` | Number of request-response pairs in report |
+
+## Notes
+
+- The interceptor chain order matters - request interceptors run in order, response interceptors run in reverse
+- Interceptors can be enabled/disabled via the `enabled` field in `InterceptorConfig`
+- For complex custom logic, consider packaging as a custom container with your interceptors pre-installed