specweave 0.3.13 → 0.4.0

package/src/skills/ml-pipeline-workflow/SKILL.md

@@ -0,0 +1,245 @@
+---
+name: ml-pipeline-workflow
+description: Build end-to-end MLOps pipelines from data preparation through model training, validation, and production deployment. Use when creating ML pipelines, implementing MLOps practices, or automating model training and deployment workflows.
+---
+
+# ML Pipeline Workflow
+
+Complete end-to-end MLOps pipeline orchestration from data preparation through model deployment.
+
+## Overview
+
+This skill provides comprehensive guidance for building production ML pipelines that handle the full lifecycle: data ingestion → preparation → training → validation → deployment → monitoring.
+
+## When to Use This Skill
+
+- Building new ML pipelines from scratch
+- Designing workflow orchestration for ML systems
+- Implementing data → model → deployment automation
+- Setting up reproducible training workflows
+- Creating DAG-based ML orchestration
+- Integrating ML components into production systems
+
+## What This Skill Provides
+
+### Core Capabilities
+
+1. **Pipeline Architecture**
+   - End-to-end workflow design
+   - DAG orchestration patterns (Airflow, Dagster, Kubeflow)
+   - Component dependencies and data flow
+   - Error handling and retry strategies
+
+2. **Data Preparation**
+   - Data validation and quality checks
+   - Feature engineering pipelines
+   - Data versioning and lineage
+   - Train/validation/test splitting strategies
+
+3. **Model Training**
+   - Training job orchestration
+   - Hyperparameter management
+   - Experiment tracking integration
+   - Distributed training patterns
+
+4. **Model Validation**
+   - Validation frameworks and metrics
+   - A/B testing infrastructure
+   - Performance regression detection
+   - Model comparison workflows
+
+5. **Deployment Automation**
+   - Model serving patterns
+   - Canary deployments
+   - Blue-green deployment strategies
+   - Rollback mechanisms
+
+### Reference Documentation
+
+See the `references/` directory for detailed guides:
+- **data-preparation.md** - Data cleaning, validation, and feature engineering
+- **model-training.md** - Training workflows and best practices
+- **model-validation.md** - Validation strategies and metrics
+- **model-deployment.md** - Deployment patterns and serving architectures
+
+### Assets and Templates
+
+The `assets/` directory contains:
+- **pipeline-dag.yaml.template** - DAG template for workflow orchestration
+- **training-config.yaml** - Training configuration template
+- **validation-checklist.md** - Pre-deployment validation checklist
+
+## Usage Patterns
+
+### Basic Pipeline Setup
+
+```python
+# 1. Define pipeline stages
+stages = [
+    "data_ingestion",
+    "data_validation",
+    "feature_engineering",
+    "model_training",
+    "model_validation",
+    "model_deployment"
+]
+
+# 2. Configure dependencies
+# See assets/pipeline-dag.yaml.template for full example
+```
+
+### Production Workflow
+
+1. **Data Preparation Phase**
+   - Ingest raw data from sources
+   - Run data quality checks
+   - Apply feature transformations
+   - Version processed datasets
+
+2. **Training Phase**
+   - Load versioned training data
+   - Execute training jobs
+   - Track experiments and metrics
+   - Save trained models
+
+3. **Validation Phase**
+   - Run validation test suite
+   - Compare against baseline
+   - Generate performance reports
+   - Approve for deployment
+
+4. **Deployment Phase**
+   - Package model artifacts
+   - Deploy to serving infrastructure
+   - Configure monitoring
+   - Validate production traffic
+
+## Best Practices
+
+### Pipeline Design
+
+- **Modularity**: Each stage should be independently testable
+- **Idempotency**: Re-running stages should be safe
+- **Observability**: Log metrics at every stage
+- **Versioning**: Track data, code, and model versions
+- **Failure Handling**: Implement retry logic and alerting
+
+### Data Management
+
+- Use data validation libraries (Great Expectations, TFX)
+- Version datasets with DVC or similar tools
+- Document feature engineering transformations
+- Maintain data lineage tracking
+
+### Model Operations
+
+- Separate training and serving infrastructure
+- Use model registries (MLflow, Weights & Biases)
+- Implement gradual rollouts for new models
+- Monitor model performance drift
+- Maintain rollback capabilities
+
+### Deployment Strategies
+
+- Start with shadow deployments
+- Use canary releases for validation
+- Implement A/B testing infrastructure
+- Set up automated rollback triggers
+- Monitor latency and throughput
+
+## Integration Points
+
+### Orchestration Tools
+
+- **Apache Airflow**: DAG-based workflow orchestration
+- **Dagster**: Asset-based pipeline orchestration
+- **Kubeflow Pipelines**: Kubernetes-native ML workflows
+- **Prefect**: Modern dataflow automation
+
+### Experiment Tracking
+
+- MLflow for experiment tracking and model registry
+- Weights & Biases for visualization and collaboration
+- TensorBoard for training metrics
+
+### Deployment Platforms
+
+- AWS SageMaker for managed ML infrastructure
+- Google Vertex AI for GCP deployments
+- Azure ML for Azure cloud
+- Kubernetes + KServe for cloud-agnostic serving
+
+## Progressive Disclosure
+
+Start with the basics and gradually add complexity:
+
+1. **Level 1**: Simple linear pipeline (data → train → deploy)
+2. **Level 2**: Add validation and monitoring stages
+3. **Level 3**: Implement hyperparameter tuning
+4. **Level 4**: Add A/B testing and gradual rollouts
+5. **Level 5**: Multi-model pipelines with ensemble strategies
+
+## Common Patterns
+
+### Batch Training Pipeline
+
+```yaml
+# See assets/pipeline-dag.yaml.template
+stages:
+  - name: data_preparation
+    dependencies: []
+  - name: model_training
+    dependencies: [data_preparation]
+  - name: model_evaluation
+    dependencies: [model_training]
+  - name: model_deployment
+    dependencies: [model_evaluation]
+```
+
+### Real-time Feature Pipeline
+
+```python
+# Stream processing for real-time features
+# Combined with batch training
+# See references/data-preparation.md
+```
+
+### Continuous Training
+
+```python
+# Automated retraining on schedule
+# Triggered by data drift detection
+# See references/model-training.md
+```
+
+## Troubleshooting
+
+### Common Issues
+
+- **Pipeline failures**: Check dependencies and data availability
+- **Training instability**: Review hyperparameters and data quality
+- **Deployment issues**: Validate model artifacts and serving config
+- **Performance degradation**: Monitor data drift and model metrics
+
+### Debugging Steps
+
+1. Check pipeline logs for each stage
+2. Validate input/output data at boundaries
+3. Test components in isolation
+4. Review experiment tracking metrics
+5. Inspect model artifacts and metadata
+
+## Next Steps
+
+After setting up your pipeline:
+
+1. Explore **hyperparameter-tuning** skill for optimization
+2. Learn **experiment-tracking-setup** for MLflow/W&B
+3. Review **model-deployment-patterns** for serving strategies
+4. Implement monitoring with observability tools
+
+## Related Skills
+
+- **experiment-tracking-setup**: MLflow and Weights & Biases integration
+- **hyperparameter-tuning**: Automated hyperparameter optimization
+- **model-deployment-patterns**: Advanced deployment strategies

package/src/skills/paypal-integration/SKILL.md

@@ -0,0 +1,467 @@
+---
+name: paypal-integration
+description: Integrate PayPal payment processing with support for express checkout, subscriptions, and refund management. Use when implementing PayPal payments, processing online transactions, or building e-commerce checkout flows.
+---
+
+# PayPal Integration
+
+Master PayPal payment integration including Express Checkout, IPN handling, recurring billing, and refund workflows.
+
+## When to Use This Skill
+
+- Integrating PayPal as a payment option
+- Implementing express checkout flows
+- Setting up recurring billing with PayPal
+- Processing refunds and payment disputes
+- Handling PayPal webhooks (IPN)
+- Supporting international payments
+- Implementing PayPal subscriptions
+
+## Core Concepts
+
+### 1. Payment Products
+**PayPal Checkout**
+- One-time payments
+- Express checkout experience
+- Guest and PayPal account payments
+
+**PayPal Subscriptions**
+- Recurring billing
+- Subscription plans
+- Automatic renewals
+
+**PayPal Payouts**
+- Send money to multiple recipients
+- Marketplace and platform payments
+
+### 2. Integration Methods
+**Client-Side (JavaScript SDK)**
+- Smart Payment Buttons
+- Hosted payment flow
+- Minimal backend code
+
+**Server-Side (REST API)**
+- Full control over payment flow
+- Custom checkout UI
+- Advanced features
+
+### 3. IPN (Instant Payment Notification)
+- Webhook-like payment notifications
+- Asynchronous payment updates
+- Verification required
+
+## Quick Start
+
+```javascript
+// Frontend - PayPal Smart Buttons
+<div id="paypal-button-container"></div>
+
+<script src="https://www.paypal.com/sdk/js?client-id=YOUR_CLIENT_ID&currency=USD"></script>
+<script>
+  paypal.Buttons({
+    createOrder: function(data, actions) {
+      return actions.order.create({
+        purchase_units: [{
+          amount: {
+            value: '25.00'
+          }
+        }]
+      });
+    },
+    onApprove: function(data, actions) {
+      return actions.order.capture().then(function(details) {
+        // Payment successful
+        console.log('Transaction completed by ' + details.payer.name.given_name);
+
+        // Send to backend for verification
+        fetch('/api/paypal/capture', {
+          method: 'POST',
+          headers: {'Content-Type': 'application/json'},
+          body: JSON.stringify({orderID: data.orderID})
+        });
+      });
+    }
+  }).render('#paypal-button-container');
+</script>
+```
+
+```python
+# Backend - Verify and capture order
+from paypalrestsdk import Payment
+import paypalrestsdk
+
+paypalrestsdk.configure({
+    "mode": "sandbox",  # or "live"
+    "client_id": "YOUR_CLIENT_ID",
+    "client_secret": "YOUR_CLIENT_SECRET"
+})
+
+def capture_paypal_order(order_id):
+    """Capture a PayPal order."""
+    payment = Payment.find(order_id)
+
+    if payment.execute({"payer_id": payment.payer.payer_info.payer_id}):
+        # Payment successful
+        return {
+            'status': 'success',
+            'transaction_id': payment.id,
+            'amount': payment.transactions[0].amount.total
+        }
+    else:
+        # Payment failed
+        return {
+            'status': 'failed',
+            'error': payment.error
+        }
+```
+
+## Express Checkout Implementation
+
+### Server-Side Order Creation
+```python
+import requests
+import json
+
+class PayPalClient:
+    def __init__(self, client_id, client_secret, mode='sandbox'):
+        self.client_id = client_id
+        self.client_secret = client_secret
+        self.base_url = 'https://api-m.sandbox.paypal.com' if mode == 'sandbox' else 'https://api-m.paypal.com'
+        self.access_token = self.get_access_token()
+
+    def get_access_token(self):
+        """Get OAuth access token."""
+        url = f"{self.base_url}/v1/oauth2/token"
+        headers = {"Accept": "application/json", "Accept-Language": "en_US"}
+
+        response = requests.post(
+            url,
+            headers=headers,
+            data={"grant_type": "client_credentials"},
+            auth=(self.client_id, self.client_secret)
+        )
+
+        return response.json()['access_token']
+
+    def create_order(self, amount, currency='USD'):
+        """Create a PayPal order."""
+        url = f"{self.base_url}/v2/checkout/orders"
+        headers = {
+            "Content-Type": "application/json",
+            "Authorization": f"Bearer {self.access_token}"
+        }
+
+        payload = {
+            "intent": "CAPTURE",
+            "purchase_units": [{
+                "amount": {
+                    "currency_code": currency,
+                    "value": str(amount)
+                }
+            }]
+        }
+
+        response = requests.post(url, headers=headers, json=payload)
+        return response.json()
+
+    def capture_order(self, order_id):
+        """Capture payment for an order."""
+        url = f"{self.base_url}/v2/checkout/orders/{order_id}/capture"
+        headers = {
+            "Content-Type": "application/json",
+            "Authorization": f"Bearer {self.access_token}"
+        }
+
+        response = requests.post(url, headers=headers)
+        return response.json()
+
+    def get_order_details(self, order_id):
+        """Get order details."""
+        url = f"{self.base_url}/v2/checkout/orders/{order_id}"
+        headers = {
+            "Authorization": f"Bearer {self.access_token}"
+        }
+
+        response = requests.get(url, headers=headers)
+        return response.json()
+```
+
+## IPN (Instant Payment Notification) Handling
+
+### IPN Verification and Processing
+```python
+from flask import Flask, request
+import requests
+from urllib.parse import parse_qs
+
+app = Flask(__name__)
+
+@app.route('/ipn', methods=['POST'])
+def handle_ipn():
+    """Handle PayPal IPN notifications."""
+    # Get IPN message
+    ipn_data = request.form.to_dict()
+
+    # Verify IPN with PayPal
+    if not verify_ipn(ipn_data):
+        return 'IPN verification failed', 400
+
+    # Process IPN based on transaction type
+    payment_status = ipn_data.get('payment_status')
+    txn_type = ipn_data.get('txn_type')
+
+    if payment_status == 'Completed':
+        handle_payment_completed(ipn_data)
+    elif payment_status == 'Refunded':
+        handle_refund(ipn_data)
+    elif payment_status == 'Reversed':
+        handle_chargeback(ipn_data)
+
+    return 'IPN processed', 200
+
+def verify_ipn(ipn_data):
+    """Verify IPN message authenticity."""
+    # Add 'cmd' parameter
+    verify_data = ipn_data.copy()
+    verify_data['cmd'] = '_notify-validate'
+
+    # Send back to PayPal for verification
+    paypal_url = 'https://ipnpb.sandbox.paypal.com/cgi-bin/webscr'  # or production URL
+
+    response = requests.post(paypal_url, data=verify_data)
+
+    return response.text == 'VERIFIED'
+
+def handle_payment_completed(ipn_data):
+    """Process completed payment."""
+    txn_id = ipn_data.get('txn_id')
+    payer_email = ipn_data.get('payer_email')
+    mc_gross = ipn_data.get('mc_gross')
+    item_name = ipn_data.get('item_name')
+
+    # Check if already processed (prevent duplicates)
+    if is_transaction_processed(txn_id):
+        return
+
+    # Update database
+    # Send confirmation email
+    # Fulfill order
+    print(f"Payment completed: {txn_id}, Amount: ${mc_gross}")
+
+def handle_refund(ipn_data):
+    """Handle refund."""
+    parent_txn_id = ipn_data.get('parent_txn_id')
+    mc_gross = ipn_data.get('mc_gross')
+
+    # Process refund in your system
+    print(f"Refund processed: {parent_txn_id}, Amount: ${mc_gross}")
+
+def handle_chargeback(ipn_data):
+    """Handle payment reversal/chargeback."""
+    txn_id = ipn_data.get('txn_id')
+    reason_code = ipn_data.get('reason_code')
+
+    # Handle chargeback
+    print(f"Chargeback: {txn_id}, Reason: {reason_code}")
+```
+
+## Subscription/Recurring Billing
+
+### Create Subscription Plan
+```python
+def create_subscription_plan(name, amount, interval='MONTH'):
+    """Create a subscription plan."""
+    client = PayPalClient(CLIENT_ID, CLIENT_SECRET)
+
+    url = f"{client.base_url}/v1/billing/plans"
+    headers = {
+        "Content-Type": "application/json",
+        "Authorization": f"Bearer {client.access_token}"
+    }
+
+    payload = {
+        "product_id": "PRODUCT_ID",  # Create product first
+        "name": name,
+        "billing_cycles": [{
+            "frequency": {
+                "interval_unit": interval,
+                "interval_count": 1
+            },
+            "tenure_type": "REGULAR",
+            "sequence": 1,
+            "total_cycles": 0,  # Infinite
+            "pricing_scheme": {
+                "fixed_price": {
+                    "value": str(amount),
+                    "currency_code": "USD"
+                }
+            }
+        }],
+        "payment_preferences": {
+            "auto_bill_outstanding": True,
+            "setup_fee": {
+                "value": "0",
+                "currency_code": "USD"
+            },
+            "setup_fee_failure_action": "CONTINUE",
+            "payment_failure_threshold": 3
+        }
+    }
+
+    response = requests.post(url, headers=headers, json=payload)
+    return response.json()
+
+def create_subscription(plan_id, subscriber_email):
+    """Create a subscription for a customer."""
+    client = PayPalClient(CLIENT_ID, CLIENT_SECRET)
+
+    url = f"{client.base_url}/v1/billing/subscriptions"
+    headers = {
+        "Content-Type": "application/json",
+        "Authorization": f"Bearer {client.access_token}"
+    }
+
+    payload = {
+        "plan_id": plan_id,
+        "subscriber": {
+            "email_address": subscriber_email
+        },
+        "application_context": {
+            "return_url": "https://yourdomain.com/subscription/success",
+            "cancel_url": "https://yourdomain.com/subscription/cancel"
+        }
+    }
+
+    response = requests.post(url, headers=headers, json=payload)
+    subscription = response.json()
+
+    # Get approval URL
+    for link in subscription.get('links', []):
+        if link['rel'] == 'approve':
+            return {
+                'subscription_id': subscription['id'],
+                'approval_url': link['href']
+            }
+```
+
+## Refund Workflows
+
+```python
+def create_refund(capture_id, amount=None, note=None):
+    """Create a refund for a captured payment."""
+    client = PayPalClient(CLIENT_ID, CLIENT_SECRET)
+
+    url = f"{client.base_url}/v2/payments/captures/{capture_id}/refund"
+    headers = {
+        "Content-Type": "application/json",
+        "Authorization": f"Bearer {client.access_token}"
+    }
+
+    payload = {}
+    if amount:
+        payload["amount"] = {
+            "value": str(amount),
+            "currency_code": "USD"
+        }
+
+    if note:
+        payload["note_to_payer"] = note
+
+    response = requests.post(url, headers=headers, json=payload)
+    return response.json()
+
+def get_refund_details(refund_id):
+    """Get refund details."""
+    client = PayPalClient(CLIENT_ID, CLIENT_SECRET)
+
+    url = f"{client.base_url}/v2/payments/refunds/{refund_id}"
+    headers = {
+        "Authorization": f"Bearer {client.access_token}"
+    }
+
+    response = requests.get(url, headers=headers)
+    return response.json()
+```
+
+## Error Handling
+
+```python
+class PayPalError(Exception):
+    """Custom PayPal error."""
+    pass
+
+def handle_paypal_api_call(api_function):
+    """Wrapper for PayPal API calls with error handling."""
+    try:
+        result = api_function()
+        return result
+    except requests.exceptions.RequestException as e:
+        # Network error
+        raise PayPalError(f"Network error: {str(e)}")
+    except Exception as e:
+        # Other errors
+        raise PayPalError(f"PayPal API error: {str(e)}")
+
+# Usage
+try:
+    order = handle_paypal_api_call(lambda: client.create_order(25.00))
+except PayPalError as e:
+    # Handle error appropriately
+    log_error(e)
+```
+
+## Testing
+
+```python
+# Use sandbox credentials
+SANDBOX_CLIENT_ID = "..."
+SANDBOX_SECRET = "..."
+
+# Test accounts
+# Create test buyer and seller accounts at developer.paypal.com
+
+def test_payment_flow():
+    """Test complete payment flow."""
+    client = PayPalClient(SANDBOX_CLIENT_ID, SANDBOX_SECRET, mode='sandbox')
+
+    # Create order
+    order = client.create_order(10.00)
+    assert 'id' in order
+
+    # Get approval URL
+    approval_url = next((link['href'] for link in order['links'] if link['rel'] == 'approve'), None)
+    assert approval_url is not None
+
+    # After approval (manual step with test account)
+    # Capture order
+    # captured = client.capture_order(order['id'])
+    # assert captured['status'] == 'COMPLETED'
+```
+
+## Resources
+
+- **references/express-checkout.md**: Express Checkout implementation guide
+- **references/ipn-handling.md**: IPN verification and processing
+- **references/refund-workflows.md**: Refund handling patterns
+- **references/billing-agreements.md**: Recurring billing setup
+- **assets/paypal-client.py**: Production PayPal client
+- **assets/ipn-processor.py**: IPN webhook processor
+- **assets/recurring-billing.py**: Subscription management
+
+## Best Practices
+
+1. **Always Verify IPN**: Never trust IPN without verification
+2. **Idempotent Processing**: Handle duplicate IPN notifications
+3. **Error Handling**: Implement robust error handling
+4. **Logging**: Log all transactions and errors
+5. **Test Thoroughly**: Use sandbox extensively
+6. **Webhook Backup**: Don't rely solely on client-side callbacks
+7. **Currency Handling**: Always specify currency explicitly
+
+## Common Pitfalls
+
+- **Not Verifying IPN**: Accepting IPN without verification
+- **Duplicate Processing**: Not checking for duplicate transactions
+- **Wrong Environment**: Mixing sandbox and production URLs/credentials
+- **Missing Webhooks**: Not handling all payment states
+- **Hardcoded Values**: Not making configurable for different environments