@bluefly/openstandardagents 0.3.3 → 0.3.6

package/dist/spec/v0.3/conformance/README.md

@@ -0,0 +1,263 @@
+# OSSA Conformance Testing
+
+Conformance testing validates that OSSA agent manifests meet specific requirements defined in conformance profiles.
+
+## Overview
+
+The conformance suite provides:
+- **Profiles**: Predefined sets of requirements for different use cases
+- **Feature Detection**: Automatic detection of features in manifests
+- **Scoring**: Weighted scoring based on required and optional features
+- **Constraint Validation**: Enforce specific value constraints
+- **Recommendations**: Actionable guidance for improving conformance
+
+## Available Profiles
+
+### Baseline Profile
+**ID**: `baseline`
+**Purpose**: Minimal OSSA compliance for basic agent manifests
+
+**Required Features** (70% weight):
+- `apiVersion`: Must follow `ossa/vX.Y.Z` format
+- `kind`: Must be Agent, AgentTeam, or AgentWorkflow
+- `metadata.name`: Agent name
+- `spec.type`: Agent type (service, workflow, etc.)
+- `spec.identity.id`: Unique agent identifier
+
+**Optional Features** (30% weight):
+- `metadata.description`: Agent description
+- `metadata.labels`: Metadata labels
+- `spec.capabilities`: Agent capabilities
+- `spec.llm`: LLM configuration
+- `spec.instructions`: Agent instructions
+
+**Pass Threshold**: 70%
+
+### Enterprise Profile
+**ID**: `enterprise`
+**Purpose**: Production-ready requirements for enterprise deployments
+
+Extends baseline profile with additional requirements:
+
+**Required Features** (60% weight):
+- All baseline required features
+- `metadata.version`: Semantic version
+- `spec.llm.provider` & `spec.llm.model`: LLM configuration
+- `spec.autonomy.level` & `spec.autonomy.approval_required`: Autonomy settings
+- `spec.constraints.cost` & `spec.constraints.performance`: Resource limits
+- `spec.observability.*`: Tracing, metrics, logging
+
+**Optional Features** (40% weight):
+- `spec.security.*`: Authentication, authorization, data privacy
+- `spec.resilience.*`: Retry policies, timeouts, circuit breakers
+- `spec.deployment.*`: Environment and scaling configuration
+
+**Pass Threshold**: 80%
+
+### GitLab Kagent Profile
+**ID**: `gitlab-kagent`
+**Purpose**: Requirements for GitLab Kagent extension compatibility
+
+Extends enterprise profile with Kagent-specific requirements:
+
+**Required Features** (50% weight):
+- All enterprise required features
+- `extensions.kagent.enabled`: Kagent extension enabled
+- `extensions.kagent.gitlab_integration`: GitLab integration config
+- `extensions.kagent.kubernetes_native`: Kubernetes native support
+
+**Optional Features** (50% weight):
+- `extensions.kagent.gitlab_integration.*`: CI/CD, issues, MRs, pipelines
+- `extensions.kagent.kubernetes_native.*`: Deployment, scaling, monitoring
+- `extensions.kagent.observability.*`: Traces and metrics
+- `extensions.kagent.security.*`: RBAC and secrets management
+
+**Pass Threshold**: 75%
+
+## Usage
+
+### CLI Commands
+
+#### Run Conformance Test
+```bash
+# Test against baseline profile
+ossa conformance run manifest.yaml --profile baseline
+
+# Test against enterprise profile with verbose output
+ossa conformance run manifest.yaml --profile enterprise --verbose
+
+# Test with strict validation (fail on validation errors)
+ossa conformance run manifest.yaml --profile enterprise --strict
+
+# JSON output for CI/CD integration
+ossa conformance run manifest.yaml --profile baseline --output json
+```
+
+#### List Available Profiles
+```bash
+ossa conformance list
+```
+
+#### Show Profile Details
+```bash
+ossa conformance profile baseline
+ossa conformance profile enterprise --output json
+```
+
+### Programmatic Usage
+
+```typescript
+import { ConformanceService } from '@ossa/openstandardagents';
+import { container } from '@ossa/openstandardagents/di-container';
+
+const conformanceService = container.get(ConformanceService);
+
+// Run conformance test
+const result = await conformanceService.runConformanceTest(
+  manifest,
+  'enterprise',
+  false // strict mode
+);
+
+// Generate report
+const report = await conformanceService.generateReport(
+  manifest,
+  'baseline'
+);
+
+// Batch test multiple manifests
+const results = await conformanceService.batchTest(
+  manifests,
+  'enterprise'
+);
+
+// Get summary statistics
+const stats = conformanceService.getSummaryStatistics(results);
+```
+
+## Test Fixtures
+
+Test fixtures are organized by profile:
+
+```
+spec/v0.3/conformance/tests/
+├── baseline/
+│   ├── valid/
+│   │   ├── minimal-agent.yaml
+│   │   └── basic-agent.yaml
+│   └── invalid/
+│       ├── missing-kind.yaml
+│       └── missing-identity.yaml
+├── enterprise/
+│   ├── valid/
+│   │   ├── production-agent.yaml
+│   │   └── enterprise-agent.yaml
+└── gitlab-kagent/
+    └── valid/
+        ├── kagent-agent.yaml
+        └── kagent-full.yaml
+```
+
+## Scoring Algorithm
+
+```
+score = (required_present / required_total) * required_weight +
+        (optional_present / optional_total) * optional_weight
+```
+
+- **Required features** have higher weight (typically 60-70%)
+- **Optional features** have lower weight (typically 30-40%)
+- **Pass threshold** varies by profile (70-80%)
+- **Constraint violations** fail the test regardless of score
+
+## CI/CD Integration
+
+### GitLab CI
+
+The conformance suite includes GitLab CI jobs:
+
+```yaml
+# .gitlab-ci.yml
+include:
+  - local: .gitlab/ci/conformance-jobs.yml
+
+# Runs on MR and main branches
+conformance:baseline:
+  script:
+    - ossa conformance run manifest.yaml --profile baseline
+```
+
+See `.gitlab/ci/conformance-jobs.yml` for complete CI configuration.
+
+### GitHub Actions
+
+```yaml
+- name: Run conformance tests
+  run: |
+    npm run ossa -- conformance run manifest.yaml --profile enterprise --output json
+```
+
+## Creating Custom Profiles
+
+Create a JSON file in `spec/v0.3/conformance/profiles/`:
+
+```json
+{
+  "id": "custom",
+  "name": "Custom Profile",
+  "version": "0.3.5",
+  "description": "Custom conformance requirements",
+  "extends": "baseline",
+  "required": {
+    "features": ["apiVersion", "kind"],
+    "weight": 0.7
+  },
+  "optional": {
+    "features": ["metadata.description"],
+    "weight": 0.3
+  },
+  "constraints": {
+    "apiVersion": {
+      "pattern": "^ossa/v[0-9]+\\.[0-9]+\\.[0-9]+$",
+      "description": "Must follow ossa/vX.Y.Z format"
+    }
+  },
+  "scoring": {
+    "pass_threshold": 0.75,
+    "warn_threshold": 0.85
+  }
+}
+```
+
+## Best Practices
+
+1. **Start with baseline**: Ensure baseline conformance before testing higher profiles
+2. **Use verbose mode**: Get detailed feature detection results and recommendations
+3. **Enable strict mode**: Fail fast on validation errors in production pipelines
+4. **Review recommendations**: Follow actionable guidance to improve conformance
+5. **Test early**: Run conformance tests during development, not just in CI
+6. **Track scores**: Monitor conformance scores over time to ensure quality
+
+## Troubleshooting
+
+### Score too low
+- Use `--verbose` flag to see missing features
+- Check recommendations section for guidance
+- Review profile requirements with `ossa conformance profile <name>`
+
+### Constraint violations
+- Check expected vs actual values in error messages
+- Review constraint descriptions in profile
+- Validate manifest structure with `ossa validate`
+
+### Profile not found
+- List available profiles: `ossa conformance list`
+- Check profile ID matches exactly (case-sensitive)
+- Ensure profiles directory exists and is readable
+
+## Resources
+
+- [OSSA Specification](../../README.md)
+- [Example Manifests](../../examples/)
+- [Validation Guide](../../../docs/validation.md)
+- [CI/CD Integration](../.gitlab/ci/conformance-jobs.yml)

package/dist/spec/v0.3/conformance/profiles/baseline.json

@@ -0,0 +1,27 @@
+{
+  "$schema": "https://openstandardagents.org/schemas/conformance-profile.json",
+  "id": "baseline",
+  "name": "Baseline",
+  "version": "0.3.5",
+  "description": "Minimal OSSA v0.3 compliance - Linux Foundation adoption ready",
+  "required": {
+    "features": [
+      "metadata.name",
+      "metadata.version",
+      "spec.identity.id",
+      "spec.role"
+    ],
+    "weight": 100
+  },
+  "optional": {
+    "features": [
+      "metadata.labels",
+      "metadata.annotations"
+    ],
+    "weight": 0
+  },
+  "scoring": {
+    "pass_threshold": 100,
+    "warn_threshold": 80
+  }
+}

package/dist/spec/v0.3/conformance/profiles/enterprise.json

@@ -0,0 +1,33 @@
+{
+  "$schema": "https://openstandardagents.org/schemas/conformance-profile.json",
+  "id": "enterprise",
+  "name": "Enterprise",
+  "extends": "baseline",
+  "version": "0.3.5",
+  "description": "Production-ready features for enterprise deployment",
+  "required": {
+    "features": [
+      "metadata.name",
+      "metadata.version",
+      "spec.identity.id",
+      "spec.role",
+      "spec.completion",
+      "spec.checkpointing",
+      "spec.monitoring",
+      "spec.security"
+    ],
+    "weight": 70
+  },
+  "optional": {
+    "features": [
+      "spec.extensions.moe",
+      "spec.extensions.feedback_loops",
+      "spec.extensions.bat"
+    ],
+    "weight": 30
+  },
+  "scoring": {
+    "pass_threshold": 85,
+    "warn_threshold": 70
+  }
+}

package/dist/spec/v0.3/conformance/profiles/gitlab-kagent.json

@@ -0,0 +1,36 @@
+{
+  "$schema": "https://openstandardagents.org/schemas/conformance-profile.json",
+  "id": "gitlab-kagent",
+  "name": "GitLab Kagent",
+  "extends": "enterprise",
+  "version": "0.3.5",
+  "description": "GitLab Kagent extension compliance",
+  "required": {
+    "features": [
+      "metadata.name",
+      "metadata.version",
+      "spec.identity.id",
+      "spec.role",
+      "spec.completion",
+      "spec.checkpointing",
+      "spec.monitoring",
+      "spec.security",
+      "spec.extensions.kagent",
+      "spec.extensions.kagent.kubernetes.namespace",
+      "spec.extensions.kagent.kubernetes.resourceLimits"
+    ],
+    "weight": 80
+  },
+  "optional": {
+    "features": [
+      "spec.extensions.moe",
+      "spec.extensions.feedback_loops",
+      "spec.extensions.bat"
+    ],
+    "weight": 20
+  },
+  "scoring": {
+    "pass_threshold": 90,
+    "warn_threshold": 75
+  }
+}

package/dist/spec/v0.3/conformance/tests/baseline/invalid/missing-identity.yaml

@@ -0,0 +1,6 @@
+apiVersion: ossa/v0.3.5
+kind: Agent
+metadata:
+  name: invalid-agent-no-identity
+spec:
+  type: service

package/dist/spec/v0.3/conformance/tests/baseline/invalid/missing-kind.yaml

@@ -0,0 +1,7 @@
+apiVersion: ossa/v0.3.5
+metadata:
+  name: invalid-agent-no-kind
+spec:
+  type: service
+  identity:
+    id: invalid-agent

package/dist/spec/v0.3/conformance/tests/baseline/valid/basic-agent.yaml

@@ -0,0 +1,22 @@
+apiVersion: ossa/v0.3.5
+kind: Agent
+metadata:
+  name: basic-test-agent
+  version: "1.0.0"
+  description: Basic agent with optional features for baseline testing
+  labels:
+    environment: test
+    category: conformance
+spec:
+  role: assistant
+  type: service
+  identity:
+    id: basic-test-agent
+    version: "1.0.0"
+  instructions: "You are a test agent for conformance testing"
+  capabilities:
+    - name: echo
+      description: Echo back user input
+  llm:
+    provider: openai
+    model: gpt-4

package/dist/spec/v0.3/conformance/tests/baseline/valid/minimal-agent.yaml

@@ -0,0 +1,15 @@
+apiVersion: ossa/v0.3.5
+kind: Agent
+metadata:
+  name: minimal-test-agent
+  version: "1.0.0"
+  description: Minimal valid agent for baseline conformance testing
+spec:
+  role: assistant
+  type: service
+  identity:
+    id: minimal-test-agent
+    version: "1.0.0"
+  capabilities:
+    - name: test_capability
+      description: Test capability

package/dist/spec/v0.3/conformance/tests/enterprise/valid/enterprise-agent.yaml

@@ -0,0 +1,88 @@
+apiVersion: ossa/v0.3.5
+kind: Agent
+metadata:
+  name: enterprise-test-agent
+  description: Enterprise agent with full configuration
+  version: "2.1.0"
+  labels:
+    environment: staging
+    tier: enterprise
+  annotations:
+    owner: platform-team
+    compliance: SOC2
+spec:
+  role: assistant
+  type: service
+  identity:
+    id: enterprise-test-agent
+    version: "2.1.0"
+  completion:
+    criteria:
+      - type: task_completed
+        description: Task is successfully completed
+    timeout_ms: 300000
+  checkpointing:
+    enabled: true
+    interval_ms: 30000
+    strategy: incremental
+  monitoring:
+    metrics:
+      enabled: true
+      interval_ms: 60000
+    alerts:
+      enabled: true
+  security:
+    authentication:
+      required: true
+      method: api_key
+    authorization:
+      rbac: true
+  capabilities:
+    - name: analysis
+      description: Analyze data
+  llm:
+    provider: openai
+    model: gpt-4-turbo
+  autonomy:
+    level: semi_autonomous
+    approval_required: false
+  constraints:
+    cost:
+      max_cost_per_task: 1.0
+      currency: USD
+    performance:
+      max_execution_time_ms: 60000
+  observability:
+    tracing:
+      enabled: true
+    metrics:
+      enabled: true
+    logging:
+      enabled: true
+  extensions:
+    moe:
+      enabled: true
+      routing_strategy: least_cost
+      models:
+        - provider: openai
+          model: gpt-4o-mini
+          use_case: simple_queries
+        - provider: openai
+          model: gpt-4-turbo
+          use_case: complex_analysis
+    bat:
+      enabled: true
+      criteria:
+        - type: cost
+          weight: 0.4
+        - type: performance
+          weight: 0.3
+        - type: quality
+          weight: 0.3
+    feedback_loops:
+      enabled: true
+      mechanisms:
+        - type: user_feedback
+          collection_method: inline
+        - type: performance_metrics
+          interval_ms: 60000

package/dist/spec/v0.3/conformance/tests/enterprise/valid/production-agent.yaml

@@ -0,0 +1,112 @@
+apiVersion: ossa/v0.3.5
+kind: Agent
+metadata:
+  name: production-test-agent
+  description: Production-ready agent for enterprise conformance testing
+  version: "1.0.0"
+  labels:
+    environment: production
+    tier: enterprise
+    category: conformance
+spec:
+  role: assistant
+  type: service
+  identity:
+    id: production-test-agent
+    version: "1.0.0"
+  completion:
+    criteria:
+      - type: success_criteria
+        description: Task completed successfully
+    timeout_ms: 180000
+  checkpointing:
+    enabled: true
+    interval_ms: 60000
+    strategy: full
+  monitoring:
+    metrics:
+      enabled: true
+      interval_ms: 30000
+    alerts:
+      enabled: true
+      channels: ["email", "slack"]
+  capabilities:
+    - name: data_processing
+      description: Process and analyze data
+    - name: reporting
+      description: Generate reports
+  llm:
+    provider: anthropic
+    model: claude-3-5-sonnet-20241022
+    temperature: 0.7
+  autonomy:
+    level: monitored
+    approval_required: true
+  constraints:
+    cost:
+      max_cost_per_task: 0.50
+      currency: USD
+    performance:
+      max_execution_time_ms: 30000
+      max_memory_mb: 512
+  observability:
+    tracing:
+      enabled: true
+      provider: opentelemetry
+    metrics:
+      enabled: true
+      provider: prometheus
+    logging:
+      enabled: true
+      level: info
+  security:
+    authentication:
+      required: true
+      methods:
+        - oauth2
+    authorization:
+      enabled: true
+      model: rbac
+    data_privacy:
+      enabled: true
+      pii_handling: redact
+  resilience:
+    retry_policy:
+      max_attempts: 3
+      backoff_multiplier: 2
+    timeout_ms: 30000
+    circuit_breaker:
+      enabled: true
+      failure_threshold: 5
+  deployment:
+    environment: production
+    scaling:
+      min_instances: 2
+      max_instances: 10
+  extensions:
+    moe:
+      enabled: true
+      routing_strategy: least_cost
+      models:
+        - provider: anthropic
+          model: claude-3-5-haiku-20241022
+          use_case: simple_queries
+        - provider: anthropic
+          model: claude-3-5-sonnet-20241022
+          use_case: complex_analysis
+    bat:
+      enabled: true
+      criteria:
+        - type: cost
+          weight: 0.4
+        - type: performance
+          weight: 0.3
+        - type: quality
+          weight: 0.3
+    feedback_loops:
+      enabled: true
+      mechanisms:
+        - type: user_feedback
+          collection_method: inline
+        - type: performance_metrics
+          interval_ms: 60000