mcp-wordpress 1.5.2 → 2.0.0

package/docs/DOCKER.md

@@ -1,10 +1,10 @@
 # Docker Deployment Guide (Legacy)
 
-![Docker](https://img.shields.io/badge/Docker-ready-blue)
-![Version](https://img.shields.io/badge/version-1.2.0-green)
+![Docker](https://img.shields.io/badge/Docker-ready-blue) ![Version](https://img.shields.io/badge/version-1.2.0-green)
 ![Security](https://img.shields.io/badge/security-hardened-brightgreen)
 
-> **📖 New Users**: For Claude Desktop MCP integration, see the **[Docker Setup Guide](user-guides/DOCKER_SETUP.md)** instead.
+> **📖 New Users**: For Claude Desktop MCP integration, see the **[Docker Setup Guide](user-guides/DOCKER_SETUP.md)**
+> instead.
 
 This guide covers advanced Docker deployment scenarios for production and development environments.
 
@@ -23,7 +23,8 @@ docker run -d \
   docdyhr/mcp-wordpress:latest
 ```
 
-**⚠️ Claude Desktop Users**: Do NOT use `-d` flag with Claude Desktop. See [Docker Setup Guide](user-guides/DOCKER_SETUP.md) for MCP integration.
+**⚠️ Claude Desktop Users**: Do NOT use `-d` flag with Claude Desktop. See
+[Docker Setup Guide](user-guides/DOCKER_SETUP.md) for MCP integration.
 
 ### Option 2: Docker Compose
 
@@ -54,15 +55,15 @@ docker run -d --name mcp-wordpress mcp-wordpress
 
 ### Environment Variables
 
-| Variable | Required | Description | Example |
-|----------|----------|-------------|---------|
-| `WORDPRESS_SITE_URL` | ✅ | WordPress site URL | `https://example.com` |
-| `WORDPRESS_USERNAME` | ✅ | WordPress username | `admin` |
-| `WORDPRESS_APP_PASSWORD` | ✅ | Application password | `xxxx xxxx xxxx xxxx xxxx xxxx` |
-| `WORDPRESS_AUTH_METHOD` | ❌ | Authentication method | `app-password` (default) |
-| `NODE_ENV` | ❌ | Environment mode | `production` |
-| `DEBUG` | ❌ | Enable debug logging | `false` |
-| `DISABLE_CACHE` | ❌ | Disable caching system | `false` |
+| Variable                 | Required | Description            | Example                         |
+| ------------------------ | -------- | ---------------------- | ------------------------------- |
+| `WORDPRESS_SITE_URL`     | ✅       | WordPress site URL     | `https://example.com`           |
+| `WORDPRESS_USERNAME`     | ✅       | WordPress username     | `admin`                         |
+| `WORDPRESS_APP_PASSWORD` | ✅       | Application password   | `xxxx xxxx xxxx xxxx xxxx xxxx` |
+| `WORDPRESS_AUTH_METHOD`  | ❌       | Authentication method  | `app-password` (default)        |
+| `NODE_ENV`               | ❌       | Environment mode       | `production`                    |
+| `DEBUG`                  | ❌       | Enable debug logging   | `false`                         |
+| `DISABLE_CACHE`          | ❌       | Disable caching system | `false`                         |
 
 ### Multi-Site Configuration
 
@@ -111,7 +112,7 @@ docker run -d \
 **docker-compose.yml:**
 
 ```yaml
-version: '3.8'
+version: "3.8"
 
 services:
   mcp-wordpress:
@@ -157,7 +158,7 @@ docker-compose up --profile dev
 ### Docker Swarm
 
 ```yaml
-version: '3.8'
+version: "3.8"
 
 services:
   mcp-wordpress:
@@ -209,51 +210,51 @@ spec:
         app: mcp-wordpress
     spec:
       containers:
-      - name: mcp-wordpress
-        image: docdyhr/mcp-wordpress:latest
-        ports:
-        - containerPort: 3000
-        env:
-        - name: NODE_ENV
-          value: \"production\"
-        - name: WORDPRESS_SITE_URL
-          valueFrom:
-            secretKeyRef:
-              name: wordpress-secrets
-              key: site-url
-        - name: WORDPRESS_USERNAME
-          valueFrom:
-            secretKeyRef:
-              name: wordpress-secrets
-              key: username
-        - name: WORDPRESS_APP_PASSWORD
-          valueFrom:
-            secretKeyRef:
-              name: wordpress-secrets
-              key: app-password
-        resources:
-          requests:
-            memory: \"256Mi\"
-            cpu: \"250m\"
-          limits:
-            memory: \"512Mi\"
-            cpu: \"500m\"
-        livenessProbe:
-          exec:
-            command:
-            - node
-            - dist/index.js
-            - --health-check
-          initialDelaySeconds: 30
-          periodSeconds: 30
-        readinessProbe:
-          exec:
-            command:
-            - node
-            - dist/index.js
-            - --health-check
-          initialDelaySeconds: 10
-          periodSeconds: 10
+        - name: mcp-wordpress
+          image: docdyhr/mcp-wordpress:latest
+          ports:
+            - containerPort: 3000
+          env:
+            - name: NODE_ENV
+              value: \"production\"
+            - name: WORDPRESS_SITE_URL
+              valueFrom:
+                secretKeyRef:
+                  name: wordpress-secrets
+                  key: site-url
+            - name: WORDPRESS_USERNAME
+              valueFrom:
+                secretKeyRef:
+                  name: wordpress-secrets
+                  key: username
+            - name: WORDPRESS_APP_PASSWORD
+              valueFrom:
+                secretKeyRef:
+                  name: wordpress-secrets
+                  key: app-password
+          resources:
+            requests:
+              memory: \"256Mi\"
+              cpu: \"250m\"
+            limits:
+              memory: \"512Mi\"
+              cpu: \"500m\"
+          livenessProbe:
+            exec:
+              command:
+                - node
+                - dist/index.js
+                - --health-check
+            initialDelaySeconds: 30
+            periodSeconds: 30
+          readinessProbe:
+            exec:
+              command:
+                - node
+                - dist/index.js
+                - --health-check
+            initialDelaySeconds: 10
+            periodSeconds: 10
 ```
 
 ## 🔧 Management Commands

package/docs/EVALUATION.md

@@ -0,0 +1,397 @@
+# MCP WordPress Tools Evaluation Guide
+
+This guide covers the comprehensive evaluation system for MCP WordPress tools using [mcp-evals](https://github.com/mclenhard/mcp-evals).
+
+## Overview
+
+The evaluation system provides automated testing and scoring of WordPress MCP tools using LLM-based evaluation to ensure:
+
+- **Tool Reliability**: Consistent performance across different scenarios
+- **Quality Assurance**: Comprehensive testing of all 59 WordPress tools
+- **Performance Monitoring**: Track tool performance over time
+- **Regression Detection**: Identify when changes affect tool quality
+
+## Quick Start
+
+### Prerequisites
+
+- Node.js 20+
+- OpenAI API key (configured in GitHub secrets)
+- WordPress test site credentials
+
+### Running Evaluations
+
+```bash
+# Run all evaluations
+npm run eval
+
+# Run quick evaluation (critical tools only)
+npm run eval:quick
+
+# Run critical tools evaluation
+npm run eval:critical
+
+# Generate evaluation report
+npm run eval:report
+
+# Watch mode for development
+npm run eval:watch
+```
+
+## Evaluation Configurations
+
+### 1. Comprehensive Evaluation (`wordpress-tools-eval.yaml`)
+
+Tests all 59 WordPress tools across multiple categories:
+
+- **Post Management**: Create, read, update, delete posts
+- **Media Management**: Upload, manage media files
+- **User Management**: User creation, role management
+- **Comment Management**: Comment moderation workflows
+- **Taxonomy Management**: Categories and tags
+- **Site Management**: Settings, health checks
+- **Performance**: Cache management, optimization
+- **Authentication**: Security and permissions
+- **Error Handling**: Edge cases and failures
+
+### 2. Critical Tools Evaluation (`critical-tools-eval.yaml`)
+
+Focused evaluation of the most important tools with stricter scoring:
+
+- Higher pass threshold (4.0/5.0 vs 3.5/5.0)
+- Essential functionality only
+- Faster execution for CI/CD pipelines
+
+### 3. TypeScript Evaluations (`critical-tools.eval.ts`)
+
+Advanced evaluations using TypeScript for complex scenarios:
+
+- Multi-tool workflows
+- Error recovery scenarios
+- Performance benchmarks
+- Security testing
+
+## Evaluation Scoring
+
+### Scoring Criteria
+
+Each evaluation is scored on five dimensions (1-5 scale):
+
+1. **Accuracy** (25%): How accurately the tool performs its function
+2. **Completeness** (20%): How thoroughly it completes the task
+3. **Relevance** (20%): How relevant the response is to the request
+4. **Clarity** (20%): How clear and understandable the output is
+5. **Reasoning** (15%): How well it handles edge cases and errors
+
+### Scoring Thresholds
+
+- **Pass**: 3.5/5.0 (Acceptable performance)
+- **Good**: 4.0/5.0 (Solid performance)
+- **Excellent**: 4.5/5.0 (Outstanding performance)
+
+### Example Evaluation
+
+```yaml
+- name: create_post_basic
+  description: Test basic post creation functionality
+  prompt: "Create a new blog post titled 'AI Trends in 2025' with content about emerging AI technologies"
+  expected_tools:
+    - wp_create_post
+  success_criteria:
+    - Post is created successfully
+    - Title matches the request
+    - Content is relevant to AI trends
+```
+
+## GitHub Actions Integration
+
+### Automated Evaluation Pipeline
+
+The evaluation system runs automatically on:
+
+- **Pull Requests**: When tools are modified
+- **Push to Main**: After merging changes
+- **Weekly Schedule**: Every Monday at 2 AM UTC
+- **Manual Trigger**: Via GitHub Actions UI
+
+### Workflow Features
+
+- **PR Comments**: Automatic evaluation results in PR comments
+- **Artifacts**: Evaluation results saved for 30 days
+- **Regression Detection**: Compares with previous results
+- **Performance Tracking**: Trends over time
+- **Failure Notifications**: Creates issues for significant regressions
+
+### Environment Variables
+
+Required secrets in GitHub repository:
+
+```bash
+OPENAI_API_KEY=your_openai_api_key
+TEST_WORDPRESS_URL=https://test-site.com
+TEST_WORDPRESS_USER=testuser
+TEST_WORDPRESS_PASSWORD=app_password
+```
+
+## Writing Custom Evaluations
+
+### YAML Configuration
+
+```yaml
+model:
+  provider: openai
+  name: gpt-4o
+  temperature: 0.3
+
+evals:
+  - name: custom_evaluation
+    description: Description of what this tests
+    prompt: "Test prompt for the LLM"
+    expected_tools:
+      - wp_tool_name
+    success_criteria:
+      - What constitutes success
+      - Additional criteria
+```
+
+### TypeScript Evaluation
+
+```typescript
+import { EvalFunction, grade } from 'mcp-evals';
+import { openai } from 'mcp-evals/models';
+
+export const customEval: EvalFunction = {
+  name: 'custom_evaluation',
+  description: 'Test custom functionality',
+  run: async () => {
+    const result = await grade(
+      openai("gpt-4o"),
+      "Your test prompt here",
+      {
+        systemPrompt: "Evaluation criteria...",
+        responseFormat: { type: "json_object" }
+      }
+    );
+    return JSON.parse(result);
+  }
+};
+```
+
+## Evaluation Categories
+
+### 1. Functional Testing
+
+Tests basic tool functionality:
+
+```yaml
+- name: create_post_basic
+  prompt: "Create a new blog post with title and content"
+  expected_tools: [wp_create_post]
+```
+
+### 2. Integration Testing
+
+Tests multiple tools working together:
+
+```yaml
+- name: content_publishing_workflow
+  prompt: "Create a post with images and publish it"
+  expected_tools: [wp_create_post, wp_upload_media, wp_update_post]
+```
+
+### 3. Error Handling Testing
+
+Tests edge cases and error scenarios:
+
+```yaml
+- name: handle_invalid_post
+  prompt: "Try to update a non-existent post"
+  expected_tools: [wp_update_post]
+```
+
+### 4. Performance Testing
+
+Tests tool performance and efficiency:
+
+```yaml
+- name: high_volume_operations
+  prompt: "List 100 most recent posts and generate summary"
+  expected_tools: [wp_list_posts, wp_get_post]
+```
+
+## Best Practices
+
+### Writing Effective Evaluations
+
+1. **Clear Prompts**: Write specific, actionable prompts
+2. **Realistic Scenarios**: Test real-world use cases
+3. **Success Criteria**: Define clear success metrics
+4. **Error Cases**: Include failure scenarios
+5. **Performance**: Consider timeout and efficiency
+
+### Evaluation Maintenance
+
+1. **Regular Updates**: Keep evaluations current with tool changes
+2. **Threshold Tuning**: Adjust scoring thresholds based on results
+3. **Coverage Analysis**: Ensure all tools are tested
+4. **Performance Monitoring**: Track evaluation trends
+
+### CI/CD Integration
+
+1. **Fast Feedback**: Use `eval:quick` for PR checks
+2. **Comprehensive Testing**: Full evaluations on main branch
+3. **Regression Detection**: Compare with previous results
+4. **Performance Gates**: Fail builds on significant regressions
+
+## Troubleshooting
+
+### Common Issues
+
+1. **API Key Missing**: Ensure `OPENAI_API_KEY` is set
+2. **WordPress Connection**: Verify test site credentials
+3. **Timeout Errors**: Increase timeout or reduce evaluation scope
+4. **Rate Limiting**: Add delays between evaluations
+
+### Debug Mode
+
+```bash
+# Run with debug output
+DEBUG=true npm run eval
+
+# Run single evaluation
+npm run eval:quick -- --filter create_post_basic
+```
+
+### Local Testing
+
+```bash
+# Set up local environment
+export OPENAI_API_KEY=your_key
+export WORDPRESS_SITE_URL=http://localhost/wp
+export WORDPRESS_USERNAME=admin
+export WORDPRESS_APP_PASSWORD=your_password
+
+# Run evaluations
+npm run eval:quick
+```
+
+## Reporting
+
+### Automatic Reports
+
+Evaluation results are automatically processed into:
+
+- **JSON Summary**: Machine-readable results
+- **HTML Report**: Visual dashboard
+- **Markdown Report**: Documentation-friendly format
+
+### Manual Report Generation
+
+```bash
+# Generate comprehensive report
+npm run eval:report
+
+# View results
+open evaluations/reports/evaluation-report.html
+```
+
+### Example Report Structure
+
+```json
+{
+  "overall_score": 4.2,
+  "tests_passed": 23,
+  "total_tests": 25,
+  "status": "good",
+  "categories": {
+    "post": { "passed": 5, "total": 6, "avg_score": 4.1 },
+    "media": { "passed": 3, "total": 3, "avg_score": 4.5 }
+  },
+  "failed_tests": [
+    {
+      "name": "handle_invalid_post",
+      "score": 3.2,
+      "reason": "Error handling could be more graceful"
+    }
+  ],
+  "recommendations": [
+    "Improve error handling for edge cases",
+    "Add more comprehensive validation"
+  ]
+}
+```
+
+## Advanced Features
+
+### Performance Tracking
+
+Track evaluation performance over time:
+
+```bash
+# Generate performance trends
+node evaluations/scripts/track-performance.js
+
+# Compare with previous results
+node evaluations/scripts/compare-performance.js
+```
+
+### Custom Scoring
+
+Override default scoring with custom logic:
+
+```typescript
+const customScoring = {
+  accuracy: { weight: 0.3, min: 4.0 },
+  completeness: { weight: 0.3, min: 3.5 },
+  relevance: { weight: 0.2, min: 3.0 },
+  clarity: { weight: 0.1, min: 3.0 },
+  reasoning: { weight: 0.1, min: 3.0 }
+};
+```
+
+### Integration with Monitoring
+
+Connect evaluations to monitoring systems:
+
+```bash
+# Export metrics to Prometheus
+npm run eval:metrics
+
+# Send results to monitoring dashboard
+npm run eval:monitor
+```
+
+## Contributing
+
+### Adding New Evaluations
+
+1. Create evaluation in `evaluations/config/`
+2. Add to appropriate category
+3. Test locally with `npm run eval:quick`
+4. Submit PR with evaluation results
+
+### Improving Existing Evaluations
+
+1. Review current evaluation scores
+2. Identify areas for improvement
+3. Update prompts and success criteria
+4. Test changes thoroughly
+
+### Reporting Issues
+
+- Use GitHub Issues for evaluation problems
+- Include full evaluation results
+- Provide reproduction steps
+- Tag with `evaluation` label
+
+## Resources
+
+- [mcp-evals Documentation](https://github.com/mclenhard/mcp-evals)
+- [OpenAI API Documentation](https://platform.openai.com/docs)
+- [WordPress REST API](https://developer.wordpress.org/rest-api/)
+- [GitHub Actions Documentation](https://docs.github.com/en/actions)
+
+---
+
+**Ready to improve tool quality?** Start by running `npm run eval:quick` to see current performance, then dive into
+writing custom evaluations for your specific use cases!