sentinel_rb 0.1.0

data/Rakefile

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+require "rubocop/rake_task"
+
+RuboCop::RakeTask.new
+
+task default: %i[spec rubocop]

data/docs/architecture.md

@@ -0,0 +1,130 @@
+# SentinelRb Architecture
+
+## Overview
+
+SentinelRb is a Ruby gem that provides LLM-driven prompt inspection to automatically detect common antipatterns in prompts that are difficult to catch with static analysis.
+
+## Core Components
+
+### 1. Configuration Management
+- **File**: `.sentinel.yml`
+- **Purpose**: Centralized configuration for thresholds, providers, and dangerous tools
+- **Key Settings**:
+  - `provider`: LLM provider (openai, anthropic, etc.)
+  - `model`: Specific model to use
+  - `relevance_threshold`: Threshold for relevance scoring (default: 0.55)
+  - `divergence_threshold`: Threshold for KL divergence detection (default: 0.25)
+  - `dangerous_tools`: List of tools considered dangerous for auto-execution
+
+### 2. LLM Client Layer
+- **Purpose**: Abstraction layer for different LLM providers
+- **Supported Providers**:
+  - OpenAI GPT models
+  - Anthropic Claude models
+  - Custom API endpoints
+- **Key Methods**:
+  - `similarity(prompt, reference)`: Calculate semantic similarity
+  - `fact_check(statement)`: Verify factual accuracy
+  - `analyze_content(prompt)`: General content analysis
+
+### 3. Analyzer System
+
+#### Base Analyzer
+```ruby
+class SentinelRb::Analyzers::Base
+  def initialize(prompt, config, client)
+    @prompt = prompt
+    @config = config
+    @client = client
+  end
+
+  def call
+    # Returns array of findings: [{id:, level:, message:}, ...]
+  end
+end
+```
+
+#### Implemented Analyzers
+
+##### A1: Irrelevant Information Detector
+- **Purpose**: Detect prompts containing irrelevant or noisy information
+- **Method**: Uses LLM to generate relevance scores
+- **Threshold**: Configurable via `relevance_threshold`
+- **Output**: Warning when average relevance score is below threshold
+
+##### A2: Misinformation & Logical Contradictions
+- **Purpose**: Verify factual accuracy and logical consistency
+- **Method**: RAG-based knowledge base lookup or fact-checking API
+- **Detection**: Cross-references statements against reliable sources
+- **Output**: Error level for confirmed misinformation
+
+##### A3: Few-shot Bias Order Detection
+- **Purpose**: Detect ordering bias in few-shot examples
+- **Method**: Compares example order with canonical examples in YAML
+- **Metric**: KL Divergence calculation
+- **Threshold**: Configurable via `divergence_threshold`
+
+##### A4: Base Model Usage Detection
+- **Purpose**: Flag usage of base models instead of instruction-tuned models
+- **Method**: String matching for '-base' in model names
+- **Output**: Immediate warning for any base model usage
+
+##### A5: Dangerous Automatic Tool Execution
+- **Purpose**: Detect dangerous tools marked for automatic execution
+- **Method**: JSON parsing to find `dangerous:true && auto_execute:true`
+- **Configuration**: Uses `dangerous_tools` list from config
+- **Output**: Critical level warning for security risks
+
+### 4. Reporting System
+- **Formats**: Table, JSON, detailed reports
+- **Integration**: Designed for CI/CD pipeline integration
+- **Output Levels**: info, warn, error, critical
+
+## File Processing Pipeline
+
+1. **Discovery**: Glob pattern matching to find prompt files
+2. **Loading**: Read and parse prompt files (MD, JSON, YAML support)
+3. **Analysis**: Run all enabled analyzers on each prompt
+4. **Aggregation**: Collect results from all analyzers
+5. **Reporting**: Format and output results in specified format
+
+## Extension Points
+
+### Custom Analyzers
+Developers can create custom analyzers by:
+1. Inheriting from `SentinelRb::Analyzers::Base`
+2. Implementing the `call` method
+3. Registering the analyzer in the configuration
+
+### Custom LLM Providers
+Support for additional LLM providers can be added by:
+1. Implementing the client interface
+2. Adding provider-specific configuration
+3. Registering the provider in the client factory
+
+## Security Considerations
+
+- API keys are loaded from environment variables
+- Dangerous tool detection prevents accidental auto-execution
+- No prompt data is persisted by default
+- Configurable rate limiting for API calls
+
+## Performance Optimization
+
+- Parallel processing of multiple prompt files
+- Caching of LLM responses for repeated analysis
+- Configurable batch processing for large prompt sets
+- Optional skip patterns for excluding files
+
+## CI/CD Integration
+
+### GitHub Actions
+Pre-built workflow templates for:
+- Pull request validation
+- Scheduled prompt auditing
+- Release gate checks
+
+### Configuration Examples
+- Development environment setup
+- Production deployment settings
+- Team-specific threshold configurations

data/docs/development.md

@@ -0,0 +1,376 @@
+# SentinelRb Development Guide
+
+## Development Setup
+
+### Prerequisites
+- Ruby >= 3.1.0
+- Bundler
+- Git
+
+### Initial Setup
+```bash
+git clone <repository-url>
+cd sentinel_rb
+bin/setup
+```
+
+### Running Tests
+```bash
+# All tests
+rake spec
+
+# Specific test file
+rspec spec/analyzers/irrelevant_info_spec.rb
+
+# With coverage
+COVERAGE=true rake spec
+```
+
+### Development Console
+```bash
+bin/console
+```
+
+## Project Structure
+
+```
+sentinel_rb/
+├── lib/
+│   ├── sentinel_rb.rb           # Main entry point
+│   └── sentinel_rb/
+│       ├── version.rb           # Version management
+│       ├── config.rb           # Configuration loading
+│       ├── cli.rb              # Command line interface
+│       ├── client/             # LLM client implementations
+│       │   ├── base.rb
+│       │   ├── openai.rb
+│       │   └── anthropic.rb
+│       ├── analyzers/          # Analysis modules
+│       │   ├── base.rb
+│       │   ├── irrelevant_info.rb
+│       │   ├── misinformation.rb
+│       │   ├── few_shot_bias.rb
+│       │   ├── base_model.rb
+│       │   └── dangerous_tools.rb
+│       ├── report/             # Reporting system
+│       │   ├── formatter.rb
+│       │   ├── table.rb
+│       │   ├── json.rb
+│       │   └── detailed.rb
+│       └── utils/              # Utility classes
+│           ├── file_finder.rb
+│           └── prompt_parser.rb
+├── spec/                       # Test files
+├── docs/                       # Documentation
+├── exe/                        # Executable files
+└── bin/                        # Development scripts
+```
+
+## Adding New Analyzers
+
+### 1. Create Analyzer Class
+```ruby
+# lib/sentinel_rb/analyzers/your_analyzer.rb
+module SentinelRb
+  module Analyzers
+    class YourAnalyzer < Base
+      def call
+        # Your analysis logic here
+        findings = []
+
+        if condition_met?
+          findings << {
+            id: 'A6',
+            level: :warn,
+            message: 'Your custom warning message'
+          }
+        end
+
+        findings
+      end
+
+      private
+
+      def condition_met?
+        # Your condition logic
+      end
+    end
+  end
+end
+```
+
+### 2. Register Analyzer
+```ruby
+# lib/sentinel_rb.rb
+require 'sentinel_rb/analyzers/your_analyzer'
+
+module SentinelRb
+  ANALYZERS = {
+    'A1' => Analyzers::IrrelevantInfo,
+    'A2' => Analyzers::Misinformation,
+    'A3' => Analyzers::FewShotBias,
+    'A4' => Analyzers::BaseModel,
+    'A5' => Analyzers::DangerousTools,
+    'A6' => Analyzers::YourAnalyzer  # Add here
+  }.freeze
+end
+```
+
+### 3. Add Tests
+```ruby
+# spec/analyzers/your_analyzer_spec.rb
+require 'spec_helper'
+
+RSpec.describe SentinelRb::Analyzers::YourAnalyzer do
+  let(:config) { { 'your_threshold' => 0.5 } }
+  let(:client) { instance_double(SentinelRb::Client::OpenAI) }
+  let(:analyzer) { described_class.new(prompt, config, client) }
+
+  describe '#call' do
+    context 'when condition is met' do
+      let(:prompt) { 'test prompt that triggers condition' }
+
+      it 'returns warning' do
+        result = analyzer.call
+        expect(result).to include(
+          hash_including(
+            id: 'A6',
+            level: :warn,
+            message: include('Your custom warning')
+          )
+        )
+      end
+    end
+
+    context 'when condition is not met' do
+      let(:prompt) { 'normal prompt' }
+
+      it 'returns no findings' do
+        result = analyzer.call
+        expect(result).to be_empty
+      end
+    end
+  end
+end
+```
+
+## Adding LLM Providers
+
+### 1. Implement Client Interface
+```ruby
+# lib/sentinel_rb/client/your_provider.rb
+module SentinelRb
+  module Client
+    class YourProvider < Base
+      def initialize(config)
+        super
+        @api_key = ENV['YOUR_PROVIDER_API_KEY']
+        @base_url = config['base_url'] || 'https://api.yourprovider.com'
+      end
+
+      def similarity(text1, text2)
+        # Implement similarity calculation
+      end
+
+      def fact_check(statement)
+        # Implement fact checking
+      end
+
+      def analyze_content(prompt)
+        # Implement content analysis
+      end
+
+      private
+
+      def make_request(endpoint, payload)
+        # HTTP request implementation
+      end
+    end
+  end
+end
+```
+
+### 2. Register Provider
+```ruby
+# lib/sentinel_rb/client.rb
+module SentinelRb
+  module Client
+    PROVIDERS = {
+      'openai' => OpenAI,
+      'anthropic' => Anthropic,
+      'your_provider' => YourProvider  # Add here
+    }.freeze
+
+    def self.create(config)
+      provider = config['provider']
+      client_class = PROVIDERS[provider]
+      raise "Unsupported provider: #{provider}" unless client_class
+
+      client_class.new(config)
+    end
+  end
+end
+```
+
+## Testing Guidelines
+
+### Unit Tests
+- Each analyzer should have comprehensive test coverage
+- Mock external API calls
+- Test edge cases and error conditions
+
+### Integration Tests
+- Test CLI commands end-to-end
+- Test configuration loading
+- Test file processing pipeline
+
+### Example Test Structure
+```ruby
+RSpec.describe SentinelRb::Analyzers::IrrelevantInfo do
+  let(:config) { { 'relevance_threshold' => 0.55 } }
+  let(:client) { instance_double(SentinelRb::Client::OpenAI) }
+  let(:analyzer) { described_class.new(prompt, config, client) }
+
+  before do
+    allow(client).to receive(:similarity).and_return(similarity_score)
+  end
+
+  context 'with relevant prompt' do
+    let(:prompt) { 'Clear, focused task description' }
+    let(:similarity_score) { 0.8 }
+
+    it 'returns no findings' do
+      expect(analyzer.call).to be_empty
+    end
+  end
+
+  context 'with irrelevant prompt' do
+    let(:prompt) { 'Off-topic content mixed with task' }
+    let(:similarity_score) { 0.3 }
+
+    it 'returns relevance warning' do
+      result = analyzer.call
+      expect(result).to include(
+        hash_including(
+          id: 'A1',
+          level: :warn,
+          message: include('relevance')
+        )
+      )
+    end
+  end
+end
+```
+
+## Code Style and Conventions
+
+### Ruby Style
+- Follow Ruby community style guide
+- Use RuboCop for style enforcement
+- 2-space indentation
+- Maximum line length: 100 characters
+
+### Naming Conventions
+- Classes: PascalCase
+- Methods: snake_case
+- Constants: SCREAMING_SNAKE_CASE
+- Files: snake_case
+
+### Documentation
+- Use YARD for API documentation
+- Include examples in method documentation
+- Document all public methods and classes
+
+### Example Documentation
+```ruby
+# Analyzes prompts for irrelevant information
+#
+# @param prompt [String] the prompt text to analyze
+# @param config [Hash] configuration options
+# @param client [Client::Base] LLM client instance
+# @return [Array<Hash>] array of findings
+#
+# @example
+#   analyzer = IrrelevantInfo.new(prompt, config, client)
+#   findings = analyzer.call
+#   findings.each { |f| puts f[:message] }
+class IrrelevantInfo < Base
+  # Performs the analysis
+  #
+  # @return [Array<Hash>] findings with :id, :level, :message keys
+  def call
+    # Implementation
+  end
+end
+```
+
+## Performance Considerations
+
+### Caching
+- Implement response caching for repeated API calls
+- Use file-based cache for development
+- Redis cache for production deployments
+
+### Parallel Processing
+- Use thread pools for I/O bound operations
+- Implement batch processing for large prompt sets
+- Consider memory usage with large files
+
+### Rate Limiting
+- Implement respectful rate limiting for API calls
+- Configurable delays between requests
+- Exponential backoff for failed requests
+
+## Release Process
+
+### Version Management
+```bash
+# Update version
+vim lib/sentinel_rb/version.rb
+
+# Update changelog
+vim CHANGELOG.md
+
+# Commit changes
+git add -A
+git commit -m "Release v1.2.3"
+
+# Tag release
+git tag v1.2.3
+git push origin main --tags
+```
+
+### Gem Publishing
+```bash
+# Build gem
+rake build
+
+# Test gem installation
+gem install pkg/sentinel_rb-1.2.3.gem
+
+# Publish to RubyGems
+rake release
+```
+
+## Contributing Guidelines
+
+### Pull Request Process
+1. Fork the repository
+2. Create feature branch
+3. Add tests for new functionality
+4. Ensure all tests pass
+5. Update documentation
+6. Submit pull request
+
+### Code Review Checklist
+- [ ] Tests added for new functionality
+- [ ] Documentation updated
+- [ ] Performance impact considered
+- [ ] Security implications reviewed
+- [ ] Backward compatibility maintained
+
+### Issue Reporting
+- Use issue templates
+- Include reproduction steps
+- Provide relevant configuration
+- Include error messages and logs