predicate-claw 0.1.0

package/docs/OPERATIONAL_RUNBOOK.md

@@ -0,0 +1,389 @@
+# Operational Runbook
+
+This runbook provides step-by-step procedures for operating and troubleshooting
+the OpenClaw Predicate Provider in production environments.
+
+## Quick Reference
+
+| Incident Type | Severity | First Response |
+|---------------|----------|----------------|
+| Circuit breaker open | P1 | Check sidecar health |
+| Elevated deny rate | P2 | Compare to policy changes |
+| High latency | P3 | Check sidecar resources |
+| Audit export failures | P4 | Check control plane connectivity |
+
+## Prerequisites
+
+Before using this runbook, ensure you have:
+
+- Access to provider logs and metrics dashboards
+- Access to sidecar logs (`predicate-authorityd`)
+- Ability to restart provider/sidecar processes
+- Contact information for on-call escalation
+
+## Incident Response Procedures
+
+### P1: Circuit Breaker Stuck Open
+
+**Symptoms:**
+- All authorization requests failing immediately
+- `CircuitOpenError` in provider logs
+- Metrics showing `predicate_circuit_state = open`
+
+**Diagnosis Steps:**
+
+1. **Check sidecar health**
+   ```bash
+   curl -s http://localhost:8787/health | jq .
+   ```
+   Expected: `{"status": "healthy"}`
+
+2. **Check sidecar logs for errors**
+   ```bash
+   journalctl -u predicate-authorityd -n 100 --no-pager
+   # or
+   docker logs predicate-authorityd --tail 100
+   ```
+
+3. **Verify network connectivity**
+   ```bash
+   curl -w "@curl-format.txt" -s -o /dev/null http://localhost:8787/health
+   ```
+
+4. **Check control plane sync status**
+   ```bash
+   curl -s http://localhost:8787/v1/sync/status | jq .
+   ```
+
+**Resolution Steps:**
+
+1. **If sidecar is unhealthy:**
+   ```bash
+   # Restart sidecar
+   systemctl restart predicate-authorityd
+   # or
+   docker restart predicate-authorityd
+   ```
+
+2. **If sidecar is healthy but circuit is still open:**
+   - Circuit will auto-recover after `resetTimeoutMs` (default: 30s)
+   - For immediate recovery, restart the provider process
+
+3. **If control plane sync is failing:**
+   - Check control plane endpoint accessibility
+   - Verify API credentials are valid
+   - Check for control plane service incidents
+
+**Escalation:**
+- If not resolved in 5 minutes, page on-call engineer
+- If sidecar restart doesn't help, escalate to platform team
+
+---
+
+### P2: Elevated Deny Rate
+
+**Symptoms:**
+- Sudden increase in deny decisions (>2x baseline)
+- User reports of blocked actions
+- `denied_by_policy` reason code spike
+
+**Diagnosis Steps:**
+
+1. **Check deny rate trend**
+   ```bash
+   # Query recent deny events
+   curl -s "http://localhost:8787/v1/audit/decisions?outcome=deny&limit=50" | jq .
+   ```
+
+2. **Compare to recent policy changes**
+   - Check control plane for recent policy deployments
+   - Review policy version in metrics
+
+3. **Identify affected actions/resources**
+   ```bash
+   # Group denials by action
+   curl -s "http://localhost:8787/v1/audit/decisions?outcome=deny" | \
+     jq -r '.items | group_by(.action) | map({action: .[0].action, count: length})'
+   ```
+
+4. **Check for attack patterns**
+   - Look for repeated denials from same principal
+   - Check for unusual resource patterns (path traversal, etc.)
+
+**Resolution Steps:**
+
+1. **If caused by policy change:**
+   - Rollback to previous policy version via control plane
+   - Or fix policy and redeploy
+
+2. **If attack attempt:**
+   - Document attack patterns
+   - Consider adding rate limiting
+   - Report to security team
+
+3. **If false positives:**
+   - Review policy rules for overly broad denials
+   - Add specific allow rules for legitimate use cases
+
+**Escalation:**
+- If attack suspected, notify security team immediately
+- If policy rollback needed, coordinate with policy owners
+
+---
+
+### P3: High Authorization Latency
+
+**Symptoms:**
+- p95 latency > 150ms
+- Slow tool execution reported by users
+- Timeout errors in logs
+
+**Diagnosis Steps:**
+
+1. **Check current latency percentiles**
+   ```bash
+   curl -s http://localhost:8787/metrics | grep predicate_auth_latency
+   ```
+
+2. **Check sidecar resource usage**
+   ```bash
+   # CPU and memory
+   top -p $(pgrep predicate-authorityd)
+   # or
+   docker stats predicate-authorityd --no-stream
+   ```
+
+3. **Check control plane sync load**
+   ```bash
+   curl -s http://localhost:8787/v1/sync/status | jq '.last_sync_duration_ms'
+   ```
+
+4. **Check concurrent request volume**
+   ```bash
+   curl -s http://localhost:8787/metrics | grep predicate_auth_concurrent
+   ```
+
+**Resolution Steps:**
+
+1. **If sidecar CPU is high:**
+   - Check for runaway policy evaluation
+   - Consider scaling sidecar resources
+   - Review policy complexity
+
+2. **If sync is slow:**
+   - Check control plane latency
+   - Consider increasing sync interval
+   - Review policy size
+
+3. **If high concurrent load:**
+   - Consider horizontal scaling
+   - Review request batching options
+   - Check for retry storms
+
+**Escalation:**
+- If resources are maxed, request capacity increase
+- If policy is too complex, work with policy team to optimize
+
+---
+
+### P4: Audit Export Failures
+
+**Symptoms:**
+- Missing audit events in control plane
+- `audit_export_failure` in logs
+- Non-zero `predicate_audit_failures` counter
+
+**Diagnosis Steps:**
+
+1. **Check export error logs**
+   ```bash
+   grep "audit.*error" /var/log/provider.log | tail -20
+   ```
+
+2. **Verify control plane connectivity**
+   ```bash
+   curl -s https://control-plane.example.com/health
+   ```
+
+3. **Check export queue depth**
+   ```bash
+   curl -s http://localhost:8787/metrics | grep predicate_audit_queue
+   ```
+
+**Resolution Steps:**
+
+1. **If control plane unreachable:**
+   - Check network/firewall rules
+   - Verify TLS certificates
+   - Check for control plane incidents
+
+2. **If queue is backed up:**
+   - Audit export is best-effort; auth continues working
+   - Events will retry automatically
+   - Check disk space for local buffer
+
+3. **If credentials expired:**
+   - Rotate API credentials
+   - Update provider configuration
+   - Restart provider
+
+**Escalation:**
+- Audit failures are P4 (non-blocking)
+- Escalate only if prolonged (>1 hour) or compliance-critical
+
+---
+
+## Routine Operations
+
+### Restarting the Provider
+
+```bash
+# Graceful restart (allows in-flight requests to complete)
+systemctl reload openclaw-provider
+
+# Full restart
+systemctl restart openclaw-provider
+```
+
+### Rotating Credentials
+
+1. Generate new credentials in control plane
+2. Update provider configuration
+3. Restart provider
+4. Verify connectivity
+5. Revoke old credentials
+
+### Updating Policy
+
+1. Deploy new policy to control plane
+2. Monitor sync status on sidecars
+3. Watch deny rate for anomalies
+4. Rollback if issues detected
+
+### Scaling Sidecars
+
+For high-load environments:
+
+1. Deploy additional sidecar instances
+2. Configure load balancer
+3. Update provider `baseUrl` to load balancer
+4. Verify even distribution
+
+---
+
+## Health Checks
+
+### Provider Health
+
+```bash
+# Local provider health
+curl -s http://localhost:3000/health
+
+# Expected response
+{
+  "status": "healthy",
+  "sidecar": "connected",
+  "circuit": "closed"
+}
+```
+
+### Sidecar Health
+
+```bash
+# Sidecar health
+curl -s http://localhost:8787/health
+
+# Expected response
+{
+  "status": "healthy",
+  "policy_version": "v1.2.3",
+  "last_sync": "2026-02-20T12:00:00Z"
+}
+```
+
+### End-to-End Check
+
+```bash
+# Test authorization flow
+curl -X POST http://localhost:8787/v1/authorize \
+  -H "Content-Type: application/json" \
+  -d '{
+    "principal": "test:health-check",
+    "action": "health.check",
+    "resource": "system"
+  }'
+
+# Expected: allow decision for health check action
+```
+
+---
+
+## Monitoring Checklist
+
+### Daily
+
+- [ ] Review deny rate trends
+- [ ] Check circuit breaker state
+- [ ] Verify audit export completeness
+
+### Weekly
+
+- [ ] Review latency percentiles
+- [ ] Check policy sync freshness
+- [ ] Audit access logs
+
+### Monthly
+
+- [ ] Review and update SLO thresholds
+- [ ] Test incident response procedures
+- [ ] Update runbook with learnings
+
+---
+
+## Contact Information
+
+| Role | Contact |
+|------|---------|
+| On-call engineer | PagerDuty: `predicate-oncall` |
+| Platform team | Slack: `#predicate-platform` |
+| Security team | Slack: `#security-incidents` |
+| Control plane status | https://status.predicatesystems.ai |
+
+---
+
+## Appendix
+
+### Useful Commands
+
+```bash
+# View real-time logs
+journalctl -u predicate-authorityd -f
+
+# Check process status
+systemctl status predicate-authorityd
+
+# View metrics
+curl -s http://localhost:8787/metrics
+
+# Force policy sync
+curl -X POST http://localhost:8787/v1/sync/trigger
+
+# Get current policy version
+curl -s http://localhost:8787/v1/policy/version
+```
+
+### Log Locations
+
+| Component | Log Path |
+|-----------|----------|
+| Provider | `/var/log/openclaw-provider/provider.log` |
+| Sidecar | `/var/log/predicate-authorityd/sidecar.log` |
+| Audit events | `/var/log/predicate-authorityd/audit.jsonl` |
+
+### Configuration Files
+
+| Component | Config Path |
+|-----------|-------------|
+| Provider | `/etc/openclaw-provider/config.yaml` |
+| Sidecar | `/etc/predicate-authorityd/config.yaml` |
+| Policy | Managed via control plane |

package/docs/PRODUCTION_READINESS.md

@@ -0,0 +1,134 @@
+# Production Readiness Checklist
+
+This document tracks production readiness criteria for the OpenClaw Predicate
+Provider. All items must be verified before GA release.
+
+**Status:** Ready for review
+**Last Updated:** 2026-02-20
+**Owner:** Platform Security
+
+## 1. Security Posture
+
+| Criteria | Status | Evidence |
+|----------|--------|----------|
+| Fail-closed default for high-risk actions | ✓ | `provider.ts` throws on sidecar errors |
+| No embedded signing keys in plugin | ✓ | Keys remain in sidecar/control plane |
+| Log redaction for sensitive values | ✓ | Tests in `audit-event-e2e.test.ts` |
+| SecurityError returns redacted reasons | ✓ | `errors.ts` implementation |
+| Path traversal protection | ✓ | Tests in `hack-vs-fix-demo.test.ts` |
+| Prompt injection blocking | ✓ | Tests in `hack-vs-fix-demo.test.ts` |
+
+**Security Signoff:** _________________ Date: _________
+
+## 2. Reliability
+
+| Criteria | Status | Evidence |
+|----------|--------|----------|
+| Circuit breaker for sidecar outages | ✓ | `circuit-breaker.ts` |
+| Exponential backoff with jitter | ✓ | `calculateBackoff()` function |
+| Configurable timeouts | ✓ | `config.ts` (300ms default) |
+| Graceful degradation on sync failure | ✓ | Local policy evaluation continues |
+| Load tested (100 sequential, 50 concurrent) | ✓ | `load-latency.test.ts` |
+
+## 3. Observability
+
+| Criteria | Status | Evidence |
+|----------|--------|----------|
+| Decision telemetry (allow/deny/error) | ✓ | `provider.ts` telemetry hooks |
+| Latency metrics | ✓ | `load-latency.test.ts` p50/p95 |
+| Circuit breaker state metrics | ✓ | `CircuitBreaker.getMetrics()` |
+| Audit export integration | ✓ | `audit-event-e2e.test.ts` |
+| Correlation IDs (session, tenant, trace) | ✓ | `multi-tenant-isolation.test.ts` |
+
+## 4. SLOs and Alerting
+
+| Criteria | Status | Evidence |
+|----------|--------|----------|
+| Latency SLOs defined (p50 <25ms, p95 <75ms) | ✓ | `docs/SLO_THRESHOLDS.md` |
+| Availability SLOs defined (99.9%) | ✓ | `docs/SLO_THRESHOLDS.md` |
+| Alert thresholds documented | ✓ | `docs/SLO_THRESHOLDS.md` |
+| Circuit breaker alert thresholds | ✓ | `docs/SLO_THRESHOLDS.md` |
+| Deny spike detection criteria | ✓ | `docs/SLO_THRESHOLDS.md` |
+
+## 5. Operations
+
+| Criteria | Status | Evidence |
+|----------|--------|----------|
+| Operational runbook | ✓ | `docs/OPERATIONAL_RUNBOOK.md` |
+| P1-P4 incident procedures | ✓ | `docs/OPERATIONAL_RUNBOOK.md` |
+| Health check endpoints | ✓ | Documented in runbook |
+| Restart/recovery procedures | ✓ | `docs/OPERATIONAL_RUNBOOK.md` |
+| Credential rotation procedures | ✓ | `docs/OPERATIONAL_RUNBOOK.md` |
+| Scaling guidance | ✓ | `docs/OPERATIONAL_RUNBOOK.md` |
+
+## 6. Testing
+
+| Criteria | Status | Evidence |
+|----------|--------|----------|
+| Unit tests | ✓ | 15 test files, all passing |
+| Integration tests (sidecar wire format) | ✓ | `provider.test.ts` |
+| Load/latency tests | ✓ | `load-latency.test.ts` |
+| Multi-tenant isolation tests | ✓ | `multi-tenant-isolation.test.ts` |
+| JWKS/key rotation tests | ✓ | `jwks-rotation.test.ts` |
+| Adversarial/security tests | ✓ | `hack-vs-fix-demo.test.ts` |
+| CI pipeline (Node 20/22) | ✓ | `.github/workflows/tests.yml` |
+
+## 7. Documentation
+
+| Criteria | Status | Evidence |
+|----------|--------|----------|
+| API contract documented | ✓ | Design doc action/resource mapping |
+| Fail-open/fail-closed policy table | ✓ | Design doc |
+| SLO documentation | ✓ | `docs/SLO_THRESHOLDS.md` |
+| Operational runbook | ✓ | `docs/OPERATIONAL_RUNBOOK.md` |
+| Docker adversarial test guide | ✓ | `examples/README.md` |
+
+## 8. Control Plane Integration
+
+| Criteria | Status | Evidence |
+|----------|--------|----------|
+| Policy sync client | ✓ | `control-plane-sync.ts` |
+| Revocation propagation | ✓ | `control-plane-sync.ts` |
+| Stale-sync observability | ✓ | `ControlPlaneSyncStatusTracker` |
+| Audit export wiring | ✓ | `audit-event-e2e.test.ts` |
+
+## 9. Known Limitations
+
+| Limitation | Impact | Planned Fix |
+|------------|--------|-------------|
+| `state_hash` not integrated into auth flow | Limits pre-execution state verification | Post-Phase 4 |
+| No automatic sidecar discovery | Requires manual `baseUrl` config | Future enhancement |
+
+## Sign-off
+
+### Engineering Review
+
+- [ ] All test suites passing
+- [ ] Code review completed
+- [ ] Performance benchmarks acceptable
+
+**Engineering Lead:** _________________ Date: _________
+
+### Security Review
+
+- [ ] Fail-closed behavior verified
+- [ ] Log redaction verified
+- [ ] No credential exposure risks
+
+**Security Lead:** _________________ Date: _________
+
+### Operations Review
+
+- [ ] Runbook reviewed and validated
+- [ ] Alerting configured
+- [ ] On-call procedures documented
+
+**Operations Lead:** _________________ Date: _________
+
+### Final Approval
+
+- [ ] All sections signed off
+- [ ] No blocking issues
+- [ ] Ready for GA release
+
+**Release Manager:** _________________ Date: _________

package/docs/SLO_THRESHOLDS.md

@@ -0,0 +1,193 @@
+# SLOs and Alert Thresholds
+
+This document defines Service Level Objectives (SLOs) and alert thresholds for
+the OpenClaw Predicate Provider in production deployments.
+
+## Latency SLOs
+
+### Authorization Call Latency
+
+| Percentile | Target | Alert Threshold |
+|------------|--------|-----------------|
+| p50 | < 25 ms | > 50 ms |
+| p95 | < 75 ms | > 150 ms |
+| p99 | < 150 ms | > 300 ms |
+
+These targets assume local sidecar deployment. For remote sidecar deployments,
+add network RTT to each target.
+
+### Sidecar Timeout
+
+- **Default timeout:** 300 ms
+- **Hard timeout (fail-closed):** 500 ms
+
+If the sidecar does not respond within the timeout, the provider fails closed
+(denies the action) for high-risk operations.
+
+## Availability SLOs
+
+### Provider Availability
+
+| Metric | Target | Alert Threshold |
+|--------|--------|-----------------|
+| Uptime | 99.9% | < 99.5% |
+| Error rate | < 0.1% | > 1% |
+
+### Sidecar Availability
+
+| Metric | Target | Alert Threshold |
+|--------|--------|-----------------|
+| Uptime | 99.95% | < 99.9% |
+| Circuit breaker open rate | < 0.5% | > 2% |
+
+## Decision Quality SLOs
+
+### Deny Spike Detection
+
+| Metric | Baseline | Alert Threshold |
+|--------|----------|-----------------|
+| Deny rate | ~5% (varies by policy) | > 2x baseline over 5 min |
+| Deny rate spike | N/A | > 50% increase in 1 min |
+
+A sudden spike in deny rates may indicate:
+- Misconfigured policy rollout
+- Attack attempt (should trigger investigation)
+- Sidecar sync failure
+
+### Reason Code Distribution
+
+Monitor reason code distribution for anomalies:
+
+| Reason Code | Expected Range | Alert if |
+|-------------|----------------|----------|
+| `denied_by_policy` | 80-95% of denials | < 70% |
+| `sidecar_timeout` | < 1% | > 5% |
+| `circuit_open` | < 0.5% | > 2% |
+| `missing_context` | < 0.1% | > 1% |
+
+## Circuit Breaker Thresholds
+
+### Default Configuration
+
+```typescript
+{
+  failureThreshold: 5,      // Opens after 5 consecutive failures
+  resetTimeoutMs: 30_000,   // Attempts recovery after 30 seconds
+  successThreshold: 2,      // Closes after 2 successful calls in half-open
+}
+```
+
+### Alert Thresholds
+
+| Event | Alert Level | Action |
+|-------|-------------|--------|
+| Circuit opened | Warning | Investigate sidecar health |
+| Circuit open > 1 min | Critical | Page on-call |
+| Circuit open > 5 min | Critical | Consider manual intervention |
+
+## Telemetry and Audit SLOs
+
+### Audit Export Latency
+
+| Metric | Target | Alert Threshold |
+|--------|--------|-----------------|
+| Export delay (best-effort) | < 5 seconds | > 30 seconds |
+| Export failure rate | < 1% | > 5% |
+
+Note: Audit export is best-effort and should never block the authorization path.
+
+### Telemetry Completeness
+
+| Metric | Target | Alert Threshold |
+|--------|--------|-----------------|
+| Decision events captured | > 99.9% | < 99% |
+| Context fields present | > 99% | < 95% |
+
+## Control Plane Sync SLOs
+
+### Policy Sync
+
+| Metric | Target | Alert Threshold |
+|--------|--------|-----------------|
+| Sync interval | < 60 seconds | > 5 minutes |
+| Stale policy age | < 5 minutes | > 15 minutes |
+
+### Revocation Propagation
+
+| Metric | Target | Alert Threshold |
+|--------|--------|-----------------|
+| Revocation latency | < 30 seconds | > 2 minutes |
+| Revocation completeness | 100% | Any missed revocation |
+
+## Monitoring Implementation
+
+### Required Metrics
+
+```typescript
+// Authorization metrics
+counter("predicate_auth_total", { outcome: "allow" | "deny" | "error" });
+histogram("predicate_auth_latency_ms", { action: string });
+
+// Circuit breaker metrics
+gauge("predicate_circuit_state", { state: "closed" | "open" | "half_open" });
+counter("predicate_circuit_transitions", { from: string, to: string });
+
+// Sync metrics
+gauge("predicate_policy_age_seconds");
+counter("predicate_sync_failures");
+
+// Audit metrics
+counter("predicate_audit_exports", { status: "success" | "failure" });
+histogram("predicate_audit_latency_ms");
+```
+
+### Dashboard Panels
+
+1. **Authorization Overview**
+   - Request rate by action
+   - Allow/deny/error distribution
+   - p50/p95/p99 latency
+
+2. **Circuit Breaker Status**
+   - Current state per sidecar
+   - Transition history
+   - Recovery time
+
+3. **Sync Health**
+   - Policy version timeline
+   - Sync lag
+   - Revocation propagation
+
+4. **Deny Analysis**
+   - Deny rate over time
+   - Top deny reasons
+   - Deny by tenant/action
+
+## Incident Response
+
+### P1: Circuit Breaker Stuck Open
+
+1. Check sidecar health and logs
+2. Verify network connectivity
+3. Check control plane status
+4. Consider manual circuit reset if sidecar is healthy
+
+### P2: Elevated Deny Rate
+
+1. Compare to policy change timeline
+2. Check for attack patterns
+3. Review deny reasons distribution
+4. Validate policy sync status
+
+### P3: Elevated Latency
+
+1. Check sidecar resource usage
+2. Review concurrent request load
+3. Check control plane sync load
+4. Consider scaling sidecars
+
+## Review Cadence
+
+- **Weekly:** Review latency percentiles and deny trends
+- **Monthly:** Audit SLO compliance and adjust thresholds
+- **Quarterly:** Review and update SLO targets based on operational learnings