sentinel_rb 0.1.0

data/docs/usage.md

@@ -0,0 +1,238 @@
+# SentinelRb Usage Guide
+
+## Quick Start
+
+### Installation
+```bash
+gem install sentinel_rb
+```
+
+### Basic Usage
+```bash
+sentinel --glob "prompts/**/*.{md,json}" --config .sentinel.yml
+```
+
+## Configuration
+
+### Creating Configuration File
+Create `.sentinel.yml` in your project root:
+
+```yaml
+# LLM Provider Configuration
+provider: openai
+model: gpt-4o-mini
+api_key_env: OPENAI_API_KEY
+
+# Analysis Thresholds
+relevance_threshold: 0.55
+divergence_threshold: 0.25
+
+# Security Settings
+dangerous_tools:
+  - delete_file
+  - transfer_funds
+  - system_shutdown
+  - exec_command
+
+# File Processing
+skip_patterns:
+  - "**/.git/**"
+  - "**/node_modules/**"
+  - "**/tmp/**"
+
+# Output Settings
+output_format: table  # table, json, detailed
+log_level: warn      # debug, info, warn, error
+```
+
+### Environment Variables
+```bash
+export OPENAI_API_KEY="your-api-key-here"
+export ANTHROPIC_API_KEY="your-anthropic-key"  # if using Anthropic
+```
+
+## Command Line Interface
+
+### Basic Commands
+```bash
+# Analyze specific files
+sentinel --files prompt1.md prompt2.json
+
+# Use glob patterns
+sentinel --glob "prompts/**/*.md"
+
+# Specify configuration
+sentinel --config custom-config.yml --glob "**/*.prompt"
+
+# Output formats
+sentinel --format json --output results.json
+sentinel --format table
+sentinel --format detailed --output report.txt
+```
+
+### Advanced Options
+```bash
+# Parallel processing
+sentinel --workers 4 --glob "**/*.md"
+
+# Skip specific analyzers
+sentinel --skip A1,A3 --glob "**/*.md"
+
+# Run only specific analyzers
+sentinel --only A2,A4 --glob "**/*.md"
+
+# Verbose output
+sentinel --verbose --glob "**/*.md"
+```
+
+## Integration Examples
+
+### GitHub Actions
+```yaml
+name: Sentinel Prompt QA
+on: [pull_request]
+
+jobs:
+  prompt-check:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: 3.3
+      - run: gem install sentinel_rb
+      - name: Run Sentinel
+        env:
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+        run: |
+          sentinel --glob "prompts/**/*" --config .sentinel.yml --format table
+```
+
+### Pre-commit Hook
+```bash
+#!/bin/sh
+# .git/hooks/pre-commit
+sentinel --glob "prompts/**/*.{md,json}" --config .sentinel.yml --format table
+exit_code=$?
+if [ $exit_code -ne 0 ]; then
+  echo "Sentinel found issues in prompts. Please review and fix."
+  exit 1
+fi
+```
+
+### Ruby Integration
+```ruby
+require 'sentinel_rb'
+
+# Programmatic usage
+config = SentinelRb::Config.load('.sentinel.yml')
+analyzer = SentinelRb::Analyzer.new(config)
+
+results = analyzer.analyze_file('path/to/prompt.md')
+results.each do |finding|
+  puts "#{finding[:level]}: #{finding[:message]}"
+end
+```
+
+## Analyzer-Specific Usage
+
+### A1: Irrelevant Information
+- **Purpose**: Detect noisy or off-topic content
+- **Tuning**: Adjust `relevance_threshold` (0.0-1.0)
+- **Example**: Flag prompts with marketing copy mixed with technical instructions
+
+### A2: Misinformation Detection
+- **Purpose**: Verify factual accuracy
+- **Configuration**: Enable fact-checking API or RAG database
+- **Example**: Detect outdated API documentation or incorrect technical claims
+
+### A3: Few-shot Bias Order
+- **Purpose**: Detect ordering bias in examples
+- **Configuration**: Set `divergence_threshold` for KL divergence
+- **Example**: Flag when examples are always positive-negative-positive pattern
+
+### A4: Base Model Usage
+- **Purpose**: Prevent base model usage in production
+- **Configuration**: Automatically detects '-base' in model names
+- **Example**: Flag `gpt-4-base` usage instead of `gpt-4`
+
+### A5: Dangerous Tool Execution
+- **Purpose**: Prevent auto-execution of dangerous tools
+- **Configuration**: Customize `dangerous_tools` list
+- **Example**: Flag tools that can delete files or transfer money
+
+## Troubleshooting
+
+### Common Issues
+
+#### API Key Issues
+```bash
+# Check environment variable
+echo $OPENAI_API_KEY
+
+# Test API connectivity
+sentinel --test-connection
+```
+
+#### File Permission Issues
+```bash
+# Check file permissions
+ls -la prompts/
+
+# Fix permissions
+chmod 644 prompts/*.md
+```
+
+#### Configuration Issues
+```bash
+# Validate configuration
+sentinel --validate-config .sentinel.yml
+
+# Use default configuration
+sentinel --no-config --glob "**/*.md"
+```
+
+### Performance Tuning
+
+#### For Large Prompt Sets
+```yaml
+# .sentinel.yml
+parallel_workers: 8
+batch_size: 10
+cache_responses: true
+rate_limit: 100  # requests per minute
+```
+
+#### For CI/CD Optimization
+```yaml
+# Faster CI configuration
+provider: openai
+model: gpt-3.5-turbo  # Faster, cheaper model
+cache_responses: true
+skip_patterns:
+  - "**/test/**"
+  - "**/examples/**"
+```
+
+## Best Practices
+
+### Prompt Organization
+```
+prompts/
+├── system/           # System prompts
+├── user/            # User interaction prompts
+├── examples/        # Few-shot examples
+└── templates/       # Reusable templates
+```
+
+### Team Workflow
+1. **Development**: Use lenient thresholds for exploration
+2. **Staging**: Apply production thresholds for validation
+3. **Production**: Strict validation with CI gate checks
+4. **Review**: Regular threshold adjustment based on findings
+
+### Configuration Management
+- Use different configs for different environments
+- Version control your `.sentinel.yml`
+- Document threshold choices and reasoning
+- Regular review and updates of dangerous tools list

data/exe/sentinel_rb

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "sentinel_rb"
+
+SentinelRb::CLI.start(ARGV)

data/lib/sentinel_rb/analyzer.rb

@@ -0,0 +1,140 @@
+# frozen_string_literal: true
+
+require_relative "config"
+require_relative "client"
+require_relative "analyzers/irrelevant_info"
+require_relative "analyzers/misinformation"
+require_relative "analyzers/few_shot_bias"
+require_relative "analyzers/base_model_usage"
+require_relative "analyzers/dangerous_tools"
+
+module SentinelRb
+  # Main analyzer engine that coordinates prompt analysis
+  class Analyzer
+    ANALYZERS = {
+      "A1" => Analyzers::IrrelevantInfo,
+      "A2" => Analyzers::Misinformation,
+      "A3" => Analyzers::FewShotBias,
+      "A4" => Analyzers::BaseModelUsage,
+      "A5" => Analyzers::DangerousTools
+    }.freeze
+
+    def initialize(config = nil)
+      @config = config || Config.load
+      @client = Client::Factory.create(@config)
+    end
+
+    # Analyze a single prompt string
+    # @param prompt [String] The prompt text to analyze
+    # @param analyzer_ids [Array<String>] Specific analyzers to run (default: all)
+    # @return [Array<Hash>] Array of findings
+    def analyze_prompt(prompt, analyzer_ids: nil)
+      return [] if prompt.nil? || prompt.strip.empty?
+
+      analyzers_to_run = determine_analyzers(analyzer_ids)
+      findings = []
+
+      analyzers_to_run.each do |analyzer_class|
+        analyzer = analyzer_class.new(prompt, @config, @client)
+        findings.concat(analyzer.call)
+      rescue StandardError => e
+        # Log error but continue with other analyzers
+        findings << {
+          id: "ERROR",
+          level: :error,
+          message: "Analyzer #{analyzer_class.name} failed: #{e.message}",
+          details: { error_class: e.class.name, backtrace: e.backtrace.first(3) }
+        }
+      end
+
+      findings
+    end
+
+    # Analyze a file containing prompt content
+    # @param file_path [String] Path to the file to analyze
+    # @param analyzer_ids [Array<String>] Specific analyzers to run (default: all)
+    # @return [Hash] Analysis results with file info and findings
+    def analyze_file(file_path, analyzer_ids: nil)
+      unless File.exist?(file_path)
+        return {
+          file: file_path,
+          error: "File not found",
+          findings: []
+        }
+      end
+
+      begin
+        content = File.read(file_path, encoding: "UTF-8")
+        puts "Debug: File size: #{content.length}, Content preview: #{content[0..100].inspect}" if ENV["DEBUG"]
+        findings = analyze_prompt(content, analyzer_ids: analyzer_ids)
+
+        {
+          file: file_path,
+          size: content.length,
+          findings: findings,
+          analyzed_at: Time.now
+        }
+      rescue StandardError => e
+        {
+          file: file_path,
+          error: "Failed to read or analyze file: #{e.message}",
+          findings: []
+        }
+      end
+    end
+
+    # Analyze multiple files matching a glob pattern
+    # @param pattern [String] Glob pattern for files to analyze
+    # @param analyzer_ids [Array<String>] Specific analyzers to run (default: all)
+    # @return [Array<Hash>] Array of file analysis results
+    def analyze_files(pattern, analyzer_ids: nil)
+      files = Dir.glob(pattern).reject { |f| should_skip_file?(f) }
+
+      files.map do |file|
+        analyze_file(file, analyzer_ids: analyzer_ids)
+      end
+    end
+
+    # Get summary statistics for analysis results
+    # @param results [Array<Hash>] Results from analyze_files
+    # @return [Hash] Summary statistics
+    def summarize_results(results)
+      total_files = results.length
+      files_with_issues = results.count { |r| r[:findings]&.any? }
+      total_findings = results.sum { |r| r[:findings]&.length || 0 }
+
+      findings_by_level = results
+                          .flat_map { |r| r[:findings] || [] }
+                          .group_by { |f| f[:level] }
+                          .transform_values(&:count)
+
+      {
+        total_files: total_files,
+        files_with_issues: files_with_issues,
+        total_findings: total_findings,
+        findings_by_level: findings_by_level,
+        pass_rate: total_files.positive? ? ((total_files - files_with_issues).to_f / total_files * 100).round(1) : 100.0
+      }
+    end
+
+    private
+
+    def determine_analyzers(analyzer_ids)
+      if analyzer_ids.nil? || analyzer_ids.empty?
+        ANALYZERS.values
+      else
+        analyzer_ids.map do |id|
+          ANALYZERS[id] or raise ArgumentError, "Unknown analyzer: #{id}"
+        end
+      end
+    end
+
+    def should_skip_file?(file_path)
+      skip_patterns = @config["skip_patterns"] || []
+
+      skip_patterns.any? do |pattern|
+        File.fnmatch(pattern, file_path, File::FNM_PATHNAME | File::FNM_DOTMATCH)
+      end
+    end
+  end
+end

data/lib/sentinel_rb/analyzers/base.rb

@@ -0,0 +1,53 @@
+# frozen_string_literal: true
+
+module SentinelRb
+  module Analyzers
+    # Base class for all prompt analyzers
+    class Base
+      attr_reader :prompt, :config, :client
+
+      def initialize(prompt, config, client)
+        @prompt = prompt
+        @config = config
+        @client = client
+      end
+
+      # Abstract method: Perform analysis on the prompt
+      # @return [Array<Hash>] Array of findings with :id, :level, :message keys
+      def call
+        raise NotImplementedError, "Subclasses must implement #call"
+      end
+
+      protected
+
+      # Helper method to create standardized finding hash
+      # @param id [String] Analyzer ID (e.g., 'A1')
+      # @param level [Symbol] Severity level (:info, :warn, :error, :critical)
+      # @param message [String] Human-readable description of the finding
+      # @param details [Hash] Additional details about the finding
+      # @return [Hash] Standardized finding hash
+      def create_finding(id:, level:, message:, details: {})
+        {
+          id: id,
+          level: level,
+          message: message,
+          details: details,
+          analyzer: self.class.name.split("::").last
+        }
+      end
+
+      # Helper method to check if a threshold is exceeded
+      # @param score [Float] The score to check
+      # @param threshold [Float] The threshold value
+      # @param higher_is_better [Boolean] Whether higher scores are better
+      # @return [Boolean] True if threshold is exceeded
+      def threshold_exceeded?(score, threshold, higher_is_better: true)
+        if higher_is_better
+          score < threshold
+        else
+          score > threshold
+        end
+      end
+    end
+  end
+end

data/lib/sentinel_rb/analyzers/base_model_usage.rb

@@ -0,0 +1,188 @@
+# frozen_string_literal: true
+
+require_relative "base"
+
+module SentinelRb
+  module Analyzers
+    # A4: Base Model Usage Detection
+    # Detects prompts that may be attempting to use base models inappropriately
+    # or trying to bypass safety measures
+    class BaseModelUsage < Base
+      def initialize(prompt, config, client)
+        super
+        @instruction_keywords = config["instruction_keywords"] || default_instruction_keywords
+        @bypass_patterns = config["bypass_patterns"] || default_bypass_patterns
+      end
+
+      def call
+        analyze(@prompt)
+      end
+
+      def analyze(prompt)
+        findings = []
+
+        # Check for base model instruction patterns
+        findings.concat(check_base_model_instructions(prompt))
+
+        # Check for safety bypass attempts
+        findings.concat(check_safety_bypass_patterns(prompt))
+
+        # Check for role-playing that might bypass guidelines
+        findings.concat(check_inappropriate_roleplay(prompt))
+
+        # Check for attempts to access training data
+        findings.concat(check_training_data_access(prompt))
+
+        findings
+      end
+
+      private
+
+      def check_base_model_instructions(prompt)
+        findings = []
+
+        instruction_patterns = [
+          {
+            pattern: /\b(ignore|disregard|forget).{0,20}(instructions|guidelines|rules|constraints)\b/i,
+            message: "Prompt attempts to bypass model instructions",
+            level: :error
+          },
+          {
+            pattern: /\b(act as|pretend to be|you are now).{0,50}(uncensored|unfiltered|unrestricted)\b/i,
+            message: "Prompt attempts to create uncensored behavior",
+            level: :error
+          },
+          {
+            pattern: /\b(jailbreak|prompt injection|escape|hack).{0,20}(model|system|AI)\b/i,
+            message: "Prompt contains explicit jailbreaking language",
+            level: :error
+          }
+        ]
+
+        instruction_patterns.each do |pattern_info|
+          next unless prompt.match?(pattern_info[:pattern])
+
+          findings << create_finding(
+            id: "A4",
+            level: pattern_info[:level],
+            message: pattern_info[:message],
+            details: {
+              pattern_matched: pattern_info[:pattern].source,
+              suggestion: "Remove attempts to bypass model guidelines and use appropriate prompting techniques"
+            }
+          )
+        end
+
+        findings
+      end
+
+      def check_safety_bypass_patterns(prompt)
+        findings = []
+
+        bypass_patterns = [
+          {
+            pattern: /\b(DAN|do anything now|opposite day|evil mode)\b/i,
+            message: "Prompt uses known jailbreak technique",
+            level: :error
+          },
+          {
+            pattern: /\b(hypothetically|theoretically|in fiction|imagine if).{0,30}(illegal|harmful|dangerous)\b/i,
+            message: "Prompt may be attempting to circumvent safety through hypothetical framing",
+            level: :warn
+          },
+          {
+            pattern: /\b(roleplay|pretend).{0,50}(no limits|no restrictions|anything goes)\b/i,
+            message: "Prompt uses roleplay to bypass restrictions",
+            level: :warn
+          }
+        ]
+
+        bypass_patterns.each do |pattern_info|
+          next unless prompt.match?(pattern_info[:pattern])
+
+          findings << create_finding(
+            id: "A4",
+            level: pattern_info[:level],
+            message: pattern_info[:message],
+            details: {
+              pattern_matched: pattern_info[:pattern].source,
+              suggestion: "Use direct, honest prompting instead of attempting to circumvent safety measures"
+            }
+          )
+        end
+
+        findings
+      end
+
+      def check_inappropriate_roleplay(prompt)
+        findings = []
+
+        # Check for potentially problematic roleplay scenarios
+        problematic_roles = [
+          /\b(criminal|terrorist|hacker|nazi|supremacist)\b/i,
+          /\b(drug dealer|arms dealer|human trafficker)\b/i,
+          /\b(dictator|tyrant|oppressor)\b/i
+        ]
+
+        if prompt.match?(/\b(roleplay|act as|pretend to be|you are)\b/i)
+          problematic_roles.each do |role_pattern|
+            next unless prompt.match?(role_pattern)
+
+            findings << create_finding(
+              id: "A4",
+              level: :warn,
+              message: "Prompt involves roleplay as potentially harmful persona",
+              details: {
+                role_pattern: role_pattern.source,
+                suggestion: "Consider using more appropriate roleplay scenarios that don't involve harmful personas"
+              }
+            )
+          end
+        end
+
+        findings
+      end
+
+      def check_training_data_access(prompt)
+        findings = []
+
+        training_access_patterns = [
+          /\b(repeat|recite|reproduce).{0,20}(training data|dataset|examples)\b/i,
+          /\b(what did you learn|show me examples).{0,30}(from training|during training)\b/i,
+          /\b(memorized|remember).{0,20}(from your training|specific examples)\b/i
+        ]
+
+        training_access_patterns.each do |pattern|
+          next unless prompt.match?(pattern)
+
+          findings << create_finding(
+            id: "A4",
+            level: :info,
+            message: "Prompt may be attempting to access specific training data",
+            details: {
+              pattern_matched: pattern.source,
+              suggestion: "Focus on asking for general information rather than specific training examples"
+            }
+          )
+        end
+
+        findings
+      end
+
+      def default_instruction_keywords
+        %w[
+          ignore disregard forget bypass circumvent
+          uncensored unfiltered unrestricted unlimited
+          jailbreak hack override disable
+        ]
+      end
+
+      def default_bypass_patterns
+        [
+          "DAN", "do anything now", "opposite day", "evil mode",
+          "developer mode", "god mode", "admin mode"
+        ]
+      end
+    end
+  end
+end