rapitapir 0.1.2 → 2.0.0

data/docs/STRICT_VALIDATION.md

@@ -0,0 +1,229 @@
+# Strict Validation by Default - Implementation Summary
+
+## Overview
+
+This document outlines the implementation of strict validation as the default behavior for RapiTapir v2.0 hash schemas, addressing security concerns about unexpected fields in API payloads.
+
+## Problem Statement
+
+**Before**: RapiTapir allowed additional properties in hash schemas by default:
+```ruby
+# This would succeed even with extra fields
+BOOK_SCHEMA = RapiTapir::Types.hash({
+  'title' => RapiTapir::Types.string,
+  'author' => RapiTapir::Types.string
+})
+
+# Request with extra fields would be accepted
+{
+  "title": "Book Title",
+  "author": "Author Name", 
+  "malicious_field": "unexpected_data"  # ❌ Should be rejected
+}
+```
+
+**Security Issues**:
+- Data leakage through unvalidated fields
+- Potential injection attacks via unexpected parameters
+- API contract violations going unnoticed
+- Difficulty in maintaining data integrity
+
+## Solution Implemented
+
+### 1. Changed Default Behavior
+
+**File**: `lib/rapitapir/types/hash.rb`
+
+**Change**: Modified default `additional_properties` from `true` to `false`
+
+```ruby
+# Before
+def initialize(field_types = {}, additional_properties: true, **options)
+
+# After  
+def initialize(field_types = {}, additional_properties: false, **options)
+```
+
+### 2. Added Coercion-Time Validation
+
+**Added Method**: `validate_no_unexpected_fields`
+
+```ruby
+def validate_no_unexpected_fields(value)
+  expected_keys = field_types.keys.map { |k| [k, k.to_s, k.to_sym] }.flatten.uniq
+  unexpected_keys = value.keys - expected_keys
+  return if unexpected_keys.empty?
+
+  unexpected_list = unexpected_keys.map(&:inspect).join(', ')
+  raise CoercionError.new(value, 'Hash', "Unexpected fields in hash: #{unexpected_list}. Only these fields are allowed: #{field_types.keys.join(', ')}")
+end
+```
+
+**Integration**: Added check in `coerce_hash_value` method:
+
+```ruby
+def coerce_hash_value(value)
+  coerced = {}
+
+  # Check for unexpected fields if additional properties are not allowed
+  validate_no_unexpected_fields(value) unless constraints[:additional_properties]
+  
+  # ... rest of coercion
+end
+```
+
+### 3. Added Explicit Opt-in for Flexible Schemas
+
+**File**: `lib/rapitapir/types.rb`
+
+**New Method**: `open_hash` for when additional properties are needed
+
+```ruby
+def self.open_hash(field_types = {}, **options)
+  Hash.new(field_types, additional_properties: true, **options)
+end
+```
+
+## Usage Examples
+
+### Strict Validation (Default)
+
+```ruby
+# Strict by default - rejects unexpected fields
+STRICT_SCHEMA = RapiTapir::Types.hash({
+  'name' => RapiTapir::Types.string,
+  'email' => RapiTapir::Types.email
+})
+
+# ✅ Valid request
+{ "name": "John", "email": "john@example.com" }
+
+# ❌ Rejected request
+{ 
+  "name": "John", 
+  "email": "john@example.com",
+  "extra_field": "unexpected"  # Causes validation error
+}
+```
+
+### Open Validation (Explicit Opt-in)
+
+```ruby
+# Explicitly allow additional properties
+FLEXIBLE_SCHEMA = RapiTapir::Types.open_hash({
+  'name' => RapiTapir::Types.string,
+  'email' => RapiTapir::Types.email
+})
+
+# ✅ Valid request with extra fields
+{ 
+  "name": "John", 
+  "email": "john@example.com",
+  "custom_field": "allowed"  # Accepted
+}
+```
+
+## Error Messages
+
+### Before
+```json
+{
+  "error": "Internal Server Error",
+  "message": "Generic error message"
+}
+```
+
+### After
+```json
+{
+  "error": "Validation Error",
+  "message": "Unexpected fields in hash: \"extra_field\", \"another_field\". Only these fields are allowed: title, author, isbn, published",
+  "field": null,
+  "value": {...},
+  "expected_type": "Hash"
+}
+```
+
+## Security Benefits
+
+### 🔒 **Enhanced Security**
+- **Prevents data injection**: Unexpected fields are rejected at the validation layer
+- **Reduces attack surface**: Limits what data can be passed to application logic
+- **Contract enforcement**: Ensures API only accepts explicitly defined data
+
+### 🛡️ **Data Integrity**
+- **Schema compliance**: Guarantees data matches expected structure
+- **Prevents pollution**: Stops unvalidated data from entering the system
+- **Clear boundaries**: Explicit definition of acceptable input
+
+### 📋 **Developer Experience**
+- **Clear errors**: Specific messages about which fields are unexpected
+- **Explicit intent**: Developers must consciously choose to allow extra fields
+- **Better debugging**: Easier to track down validation issues
+
+## When to Use Each Approach
+
+### Use Strict Validation (Default) For:
+- ✅ **Production APIs** - Maximum security and data integrity
+- ✅ **User input forms** - Prevent form tampering
+- ✅ **Payment/financial data** - Critical data validation
+- ✅ **Authentication payloads** - Security-sensitive endpoints
+- ✅ **Most API endpoints** - Default choice for better security
+
+### Use Open Validation (`open_hash`) For:
+- 🌐 **Webhook payloads** - Third-party services with varying data
+- 🔧 **Configuration objects** - User-defined custom fields
+- 📦 **Migration endpoints** - Backward compatibility needs
+- 🔄 **Proxy/transformation APIs** - Pass-through scenarios
+- 📊 **Analytics events** - Variable event properties
+
+## Backward Compatibility
+
+### Breaking Change Considerations
+- **Default behavior changed**: Existing code may need updates
+- **Migration path**: Replace `Types.hash()` with `Types.open_hash()` where needed
+- **Gradual adoption**: Can be implemented endpoint by endpoint
+
+### Recommended Migration Strategy
+1. **Audit existing schemas**: Identify which endpoints need flexible validation
+2. **Update specific cases**: Change to `open_hash()` only where required
+3. **Test thoroughly**: Verify all endpoints work with strict validation
+4. **Monitor errors**: Watch for unexpected field rejections in production
+
+## Testing
+
+All implementations have been tested with:
+- ✅ Unit tests for strict validation behavior
+- ✅ Integration tests with Sinatra adapter
+- ✅ Error message format verification
+- ✅ Backward compatibility checks
+- ✅ Performance impact assessment
+
+## Files Modified
+
+1. **`lib/rapitapir/types/hash.rb`**
+   - Changed default `additional_properties: false`
+   - Added `validate_no_unexpected_fields` method
+   - Enhanced coercion-time validation
+
+2. **`lib/rapitapir/types.rb`**
+   - Added `open_hash()` factory method
+   - Maintained existing `hash()` method with new defaults
+
+3. **`examples/strict_validation_examples.rb`**
+   - Comprehensive usage examples
+   - Security benefit demonstrations
+
+## Impact Assessment
+
+### Performance
+- **Minimal overhead**: Additional validation only when needed
+- **Early rejection**: Fails fast on invalid data
+- **Efficient checks**: Simple key comparison operations
+
+### Security Posture
+- **Significantly improved**: Default-secure behavior
+- **Reduced risk**: Fewer attack vectors through unexpected data
+- **Compliance friendly**: Better for regulatory requirements
+
+This implementation makes RapiTapir more secure by default while maintaining flexibility for specific use cases that require it.

data/docs/VALIDATION_IMPROVEMENTS.md

@@ -0,0 +1,218 @@
+# Validation Error Improvements Summary
+
+## Overview
+
+This document summarizes the comprehensive validation error improvements made to RapiTapir v2.0, addressing the user feedback about unclear error messages when API validation fails.
+
+## Problem Statement
+
+**Before**: When a required field was missing from an API request, RapiTapir returned a generic error:
+```json
+{
+  "error": "Internal Server Error",
+  "message": "Cannot coerce nil to RapiTapir::Types::String: Required value cannot be nil"
+}
+```
+
+This error message didn't tell developers:
+- Which field was missing
+- What the expected format was
+- How to fix the issue
+
+## Solution Implemented
+
+### 1. Enhanced Hash Type Validation
+
+**File**: `lib/rapitapir/types/hash.rb`
+
+**Changes**:
+- Added specific missing field detection in `coerce_defined_fields`
+- Enhanced error messages with field context in validation
+- Wrapped field coercion errors with field names
+
+**Before**:
+```ruby
+def coerce_defined_fields(value, coerced)
+  field_types.each do |field_name, field_type|
+    field_value = find_field_value(value, field_name)
+    coerced[field_name] = field_type.coerce(field_value) if field_value || !field_type.optional?
+  end
+end
+```
+
+**After**:
+```ruby
+def coerce_defined_fields(value, coerced)
+  field_types.each do |field_name, field_type|
+    field_value = find_field_value(value, field_name)
+    
+    # Check for missing required fields
+    if field_value.nil? && !field_type.optional?
+      raise CoercionError.new(nil, field_type.class.name, "Required field '#{field_name}' is missing from hash")
+    end
+    
+    # Only coerce if we have a value or field is optional
+    if field_value || !field_type.optional?
+      begin
+        coerced[field_name] = field_type.coerce(field_value)
+      rescue CoercionError => e
+        # Re-raise with field context
+        raise CoercionError.new(e.value, e.type, "Field '#{field_name}': #{e.reason}")
+      end
+    end
+  end
+end
+```
+
+### 2. Improved Sinatra Error Handling
+
+**File**: `lib/rapitapir/server/sinatra_adapter.rb`
+
+**Changes**:
+- Added specific catch blocks for `CoercionError` and `ValidationError`
+- Enhanced error response format with field context
+- Added detailed error information for better debugging
+
+**Before**:
+```ruby
+rescue ArgumentError => e
+  error_response(400, e.message)
+rescue StandardError => e
+  error_response(500, 'Internal Server Error', e.message)
+```
+
+**After**:
+```ruby
+rescue RapiTapir::Types::CoercionError => e
+  detailed_error_response(400, 'Validation Error', e.reason, {
+    field: extract_field_from_error(e),
+    value: e.value,
+    expected_type: e.type
+  })
+rescue RapiTapir::Types::ValidationError => e
+  detailed_error_response(400, 'Validation Error', e.message, {
+    errors: e.errors,
+    value: e.value,
+    expected_type: e.type.to_s
+  })
+```
+
+### 3. Enhanced Error Response Format
+
+**New Response Structure**:
+```json
+{
+  "error": "Validation Error",
+  "message": "Required field 'isbn' is missing from hash",
+  "field": "isbn",
+  "value": null,
+  "expected_type": "RapiTapir::Types::String"
+}
+```
+
+## Results
+
+### Before vs After Comparison
+
+**Scenario**: Missing `isbn` field in book creation request
+
+**Before**:
+```json
+{
+  "error": "Internal Server Error",
+  "message": "Cannot coerce nil to RapiTapir::Types::String: Required value cannot be nil"
+}
+```
+
+**After**:
+```json
+{
+  "error": "Validation Error",
+  "message": "Required field 'isbn' is missing from hash",
+  "field": "isbn",
+  "value": null,
+  "expected_type": "RapiTapir::Types::String"
+}
+```
+
+### Types of Improved Error Messages
+
+1. **Missing Required Fields**:
+   ```json
+   {
+     "error": "Validation Error",
+     "message": "Required field 'email' is missing from hash",
+     "field": "email",
+     "value": null,
+     "expected_type": "RapiTapir::Types::Email"
+   }
+   ```
+
+2. **Type Coercion Failures**:
+   ```json
+   {
+     "error": "Validation Error", 
+     "message": "Field 'published': Cannot convert 'not a boolean' to boolean",
+     "field": "published",
+     "value": "not a boolean",
+     "expected_type": "Boolean"
+   }
+   ```
+
+3. **Nested Field Errors**:
+   ```json
+   {
+     "error": "Validation Error",
+     "message": "Field 'profile': Required field 'newsletter' is missing from hash",
+     "field": "profile",
+     "value": null,
+     "expected_type": "RapiTapir::Types::Boolean"
+   }
+   ```
+
+## Benefits
+
+### 🎯 **Developer Experience**
+- **Specific field identification**: Developers immediately know which field has the issue
+- **Clear error context**: Understanding whether field is missing vs. invalid format
+- **Actionable feedback**: Error messages suggest what needs to be fixed
+
+### 🔧 **API Debugging**
+- **Faster development cycles**: Less time spent debugging validation issues
+- **Better error logs**: More informative server logs for troubleshooting
+- **Client-side handling**: Frontend can display field-specific error messages
+
+### 🚀 **Production Benefits**
+- **Better monitoring**: Easier to track which API fields cause the most validation errors
+- **User-friendly errors**: Can be directly displayed to end users (with proper sanitization)
+- **Reduced support tickets**: Clearer errors mean fewer developer questions
+
+## Backward Compatibility
+
+✅ **Fully backward compatible**
+- All existing error handling continues to work
+- Enhanced error handling is additive
+- No breaking changes to API contracts
+- Existing tests continue to pass
+
+## Usage Examples
+
+See `examples/validation_error_examples.rb` for comprehensive examples showing:
+- Missing required fields
+- Invalid email formats
+- Age constraint violations
+- Invalid nested fields
+- String length validation
+- Successful validation cases
+
+## Testing
+
+All improvements have been tested with:
+- ✅ Unit tests for type validation
+- ✅ Integration tests for Sinatra adapter
+- ✅ Real API scenario testing
+- ✅ Backward compatibility verification
+
+## Impact
+
+This improvement significantly enhances the **ergonomics** of the RapiTapir library, making it much more developer-friendly and suitable for production use where clear validation feedback is essential for good API design.

data/docs/ai-integration-plan.md

@@ -0,0 +1,112 @@
+# RapiTapir AI Integration: Full Implementation Plan
+
+## 🧠 Vision
+
+RapiTapir aims to become the first Ruby API framework with native support for AI-driven workflows, agent orchestration, and LLM-powered developer experience. This plan details the steps to integrate Model Context Protocol (MCP), Retrieval-Augmented Generation (RAG), and LLM/agent orchestration into the core and ecosystem.
+
+---
+
+## Phase A: Core AI Integration
+
+### 1. Model Context Protocol (MCP) Support
+- Add endpoint(s) to expose API schemas, example requests/responses, and documentation in MCP-compatible JSON format.
+- Implement a DSL helper (`.mcp_export`) to mark endpoints as context providers for LLMs/agents.
+- CLI: `rapitapir export mcp` to generate MCP context files for agent toolchains.
+- Documentation: Add a section on “Using RapiTapir APIs as LLM Tools (MCP)”.
+
+### 2. Retrieval-Augmented Generation (RAG) Pipelines
+- Extend the endpoint DSL with `.rag_inference(llm:, retrieval:, context_fields:)`.
+- Provide built-in RAG pipeline support:
+  - Accept user query, retrieve relevant data (DB/API), pass to LLM, return result.
+  - Support for OpenAI, local LLMs, and pluggable retrieval backends.
+- Example endpoint generator for RAG:
+  ```ruby
+  endpoint = RapiTapir.post("/ask")
+    .in(json_body(:question => :string))
+    .rag_inference(llm: :openai, retrieval: :postgres, context_fields: [:user_id])
+    .out(json_body(:answer => :string))
+  ```
+- Add test coverage for RAG endpoints (mock LLM/retrieval).
+
+### 3. LLM Instruction/Prompt Generation
+- Add `.llm_instruction(purpose:, fields:)` DSL extension to annotate endpoints with prompt templates or instructions.
+- Auto-generate OpenAPI-based prompts for LLM tool-use (e.g., for OpenAI function calling, LangChain).
+- CLI: `rapitapir generate llm-prompts` to export prompts for all endpoints.
+- Documentation: “How to use RapiTapir endpoints as LLM tools”.
+
+---
+
+## Phase B: Agent & Tooling Ecosystem
+
+### 4. Agent Tool Registration & Orchestration
+- Auto-generate OpenAPI/JSON Schema for agent tool registration (OpenAI, LangChain, etc.).
+- Add endpoints for agent-to-agent communication (e.g., `/agent/invoke`, `/agent/context`).
+- Provide a Ruby module for registering RapiTapir endpoints as agent tools (with function signatures, descriptions, and examples).
+- CLI: `rapitapir agent export-tools` to generate tool schemas for agent frameworks.
+
+### 5. Agent-Driven Workflows
+- Enable endpoints to accept/return LLM-generated instructions, plans, or summaries.
+- Add support for “chain-of-thought” and multi-step agent workflows (e.g., via a `.agent_workflow` DSL helper).
+- Example:
+  ```ruby
+  endpoint = RapiTapir.post("/plan")
+    .in(json_body(:goal => :string))
+    .agent_workflow(steps: [:search, :summarize, :act])
+    .out(json_body(:plan => :string))
+  ```
+
+---
+
+## Phase C: Developer Experience & Documentation
+
+### 6. AI-Enhanced Documentation & Testing
+- Integrate LLMs to auto-generate endpoint summaries, descriptions, and usage examples.
+- Add CLI commands:
+  - `rapitapir ai describe-endpoints` (auto-generate docs)
+  - `rapitapir ai suggest-tests` (auto-generate RSpec tests)
+- Web UI: “Describe this endpoint” and “Suggest test cases” buttons in generated docs.
+
+### 7. AI-Driven Endpoint Design
+- CLI: `rapitapir ai generate-endpoint "A GET /books endpoint that returns a list of books with title and author"`
+  - Uses LLM to generate endpoint DSL, schema, and docs.
+- Web UI: Interactive wizard for natural language to endpoint DSL/code.
+
+### 8. RAG/LLM Example Gallery
+- Add example endpoints and use cases for RAG, MCP, and agent orchestration in `examples/ai/`.
+- Documentation: “AI Patterns with RapiTapir” guide.
+
+---
+
+## Phase D: Ecosystem & Community
+
+### 9. Plugin & Extension System
+- Provide hooks for custom LLMs, retrieval backends, and agent frameworks.
+- Document how to build and share AI plugins/extensions.
+
+### 10. Community & Feedback
+- Launch a feedback program for AI features.
+- Collect and publish community-contributed AI endpoint patterns.
+
+---
+
+## Milestones & Timeline (Suggested)
+
+| Phase | Feature Area                | Timeline   |
+|-------|----------------------------|------------|
+| A     | Core AI Integration        | 4 weeks    |
+| B     | Agent & Tooling Ecosystem  | 3 weeks    |
+| C     | Dev Experience & Docs      | 3 weeks    |
+| D     | Ecosystem & Community      | Ongoing    |
+
+---
+
+## Success Metrics
+
+- MCP/RAG endpoints pass integration tests with LLM/agent frameworks.
+- CLI and docs support AI-driven workflows.
+- Community adoption of AI features and plugins.
+- RapiTapir endpoints can be used as tools by LLM agents with minimal friction.
+
+---
+
+RapiTapir will become a first-class Ruby API platform for AI-native and agent-driven applications, while maintaining its core strengths in type safety, developer experience, and extensibility.