sqs_processor 0.1.1 → 0.1.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6efefaed76c0681bb284588331b52a371f49642879819faa4e102e2a03b4db7f
-  data.tar.gz: 15192a15f7e5d75f584817733a9c8fc50ee2489c4e06273705fa1abc9f7c2499
+  metadata.gz: c8c84fdb30ca8f133ccf09157614bd1aff65487f384a587abe975c2b883ca338
+  data.tar.gz: f9177b02b2b713e1acbf5ce9fac29f9a13c48eee8454b07778df81dd7472ca09
 SHA512:
-  metadata.gz: 4102296039c1b78b61cf5f7d41ad159465fc45b41678fb8953301ea667220b8a1b656771e7650454a1c3464c6e45fb7ffc6a4402342b78ffb370afde873c43b0
-  data.tar.gz: c497b6a153e6c7a2c4acad647e11180e14f61e6b4037c9591dce3c8f5aca93bd6b9a17c5144c8ac49ae3337a209e4cf3879d8482df1af1fca2417f121f88e035
+  metadata.gz: c0cae0d423be0b4605ab032007df401017836617d020f0b98062b45d8a5d7cb09b89d87111a7217199bb2b20392a29d9ea57e7904b3b72bc616c30ad66d42fcd
+  data.tar.gz: 25fc1ea117532b69f30a34f71b03ca844bbb9ae9c6173c500e0b81ba943db085a76e0ae2425425b90bae5c9ab6dda878618bc3e6116f7ad336d2dd8635f072e8

data/README.md

@@ -9,7 +9,7 @@ A Ruby gem for processing messages from Amazon SQS queues with configurable mess
 - **Error Handling**: Robust error handling with message retention on failure
 - **Customizable**: Extensible message processing logic
 - **Logging**: Comprehensive logging with configurable levels
-- **Command Line Options**: Flexible configuration via command line arguments
+- **Graceful Shutdown**: Handles SIGTERM, SIGINT, and SIGQUIT signals gracefully
 - **Environment Variables**: Support for AWS credentials and configuration via environment variables
 
 ## Prerequisites
@@ -58,37 +58,7 @@ DATA_SYNC_SQS_QUEUE_URL=https://sqs.us-east-1.amazonaws.com/123456789012/your-qu
 
 ## Usage
 
-### Command Line Usage
-
-```bash
-sqs_processor [options]
-```
-
-### Command Line Options
-
-```bash
-sqs_processor [options]
-
-Options:
-  -q, --queue-url URL              SQS Queue URL
-  -r, --region REGION              AWS Region
-  -m, --max-messages NUMBER        Max messages per batch (default: 10)
-  -v, --visibility-timeout SECONDS Visibility timeout in seconds (default: 30)
-  -h, --help                       Show this help message
-```
-
-### Examples
-
-```bash
-# Basic usage with environment variables
-sqs_processor
-
-# Specify queue URL and region
-sqs_processor -q https://sqs.us-west-2.amazonaws.com/123456789012/my-queue -r us-west-2
-
-# Custom batch size and visibility timeout
-sqs_processor -m 5 -v 60
-```
+The gem is designed to be used programmatically in your Ruby applications.
 
 ### Programmatic Usage
 
@@ -97,14 +67,14 @@ sqs_processor -m 5 -v 60
 ```ruby
 require 'sqs_processor'
 
-# Create a processor instance
+# Create a processor instance with AWS credentials
 processor = SQSProcessor::Processor.new(
   queue_url: 'https://sqs.us-east-1.amazonaws.com/123456789012/my-queue',
-  region: 'us-east-1',
-  max_messages: 10,
-  visibility_timeout: 30,
   aws_access_key_id: 'your-access-key',
-  aws_secret_access_key: 'your-secret-key'
+  aws_secret_access_key: 'your-secret-key',
+  aws_region: 'us-east-1',
+  max_messages: 10,
+  visibility_timeout: 30
 )
 
 # Start processing messages
@@ -117,8 +87,7 @@ The gem uses a hook-based approach for message processing. You must implement th
 
 ### Hook Method
 
-The `handle_message(message, body)` method receives:
-- `message`: The SQS message object (contains message_id, receipt_handle, etc.)
+The `handle_message(body)` method receives:
 - `body`: The parsed JSON body of the message
 
 Return `true` if processing was successful (message will be deleted from queue), or `false` if processing failed (message will remain in queue for retry).
@@ -141,8 +110,8 @@ To implement custom message processing logic, create a subclass of `SQSProcessor
 require 'sqs_processor'
 
 class MyCustomProcessor < SQSProcessor::Processor
-  def handle_message(message, body)
-    # This method receives the SQS message object and parsed body
+  def handle_message(body)
+    # This method receives the parsed message body
     # Return true if processing was successful, false otherwise
     
     case body['event_type']
@@ -174,9 +143,9 @@ end
 # Usage
 processor = MyCustomProcessor.new(
   queue_url: 'your-queue-url',
-  region: 'us-east-1',
   aws_access_key_id: 'your-access-key',
-  aws_secret_access_key: 'your-secret-key'
+  aws_secret_access_key: 'your-secret-key',
+  aws_region: 'us-east-1'
 )
 processor.process_messages
 ```
@@ -188,13 +157,12 @@ processor.process_messages
 The `SQSProcessor::Processor.new` method accepts the following parameters:
 
 - `queue_url:` (required) - The SQS queue URL
-- `region:` (optional, default: 'us-east-1') - AWS region
+- `aws_access_key_id:` (required) - AWS access key ID
+- `aws_secret_access_key:` (required) - AWS secret access key
+- `aws_region:` (optional, default: 'us-east-1') - AWS region
+- `aws_session_token:` (optional) - AWS session token for temporary credentials
 - `max_messages:` (optional, default: 10) - Maximum messages per batch
 - `visibility_timeout:` (optional, default: 30) - Message visibility timeout in seconds
-- `aws_access_key_id:` (optional) - AWS access key ID
-- `aws_secret_access_key:` (optional) - AWS secret access key
-- `aws_session_token:` (optional) - AWS session token for temporary credentials
-- `aws_credentials:` (optional) - Pre-configured AWS credentials object
 - `logger:` (optional) - Custom logger instance
 
 ### Environment Variables
@@ -228,6 +196,22 @@ The script supports multiple ways to provide AWS credentials:
 - **Network Errors**: Automatic retry with exponential backoff
 - **Queue Errors**: Comprehensive error logging with stack traces
 
+## Graceful Shutdown
+
+The processor handles termination signals gracefully:
+
+- **SIGTERM**: Standard termination signal (used by container orchestrators)
+- **SIGINT**: Interrupt signal (Ctrl+C)
+- **SIGQUIT**: Quit signal
+
+When a shutdown signal is received:
+1. The processor stops accepting new messages
+2. Completes processing of any current message batch
+3. Logs the shutdown process
+4. Exits cleanly
+
+This ensures that no messages are lost during deployments or process termination.
+
 ## Monitoring
 
 The script provides detailed logging including:
@@ -244,6 +228,7 @@ The script provides detailed logging including:
 3. **Handle errors gracefully**: Return `false` from processing methods to keep messages in queue
 4. **Monitor queue depth**: Use the built-in queue attribute reporting
 5. **Use appropriate batch sizes**: Balance between throughput and memory usage
+6. **Deploy with graceful shutdown**: The processor handles SIGTERM gracefully for container deployments
 
 ## Troubleshooting
 

data/lib/sqs_processor/processor.rb

@@ -11,20 +11,16 @@ module SQSProcessor
   class Processor
     attr_reader :sqs_client, :queue_url, :logger
 
-    def initialize(queue_url:, region: 'us-east-1', max_messages: 10, visibility_timeout: 30,
-                   aws_access_key_id: nil, aws_secret_access_key: nil, aws_session_token: nil,
-                   aws_credentials: nil, logger: nil)
+    def initialize(queue_url:, aws_access_key_id:, aws_secret_access_key:, aws_region: 'us-east-1',
+                   aws_session_token: nil, max_messages: 10, visibility_timeout: 30, logger: nil)
       @queue_url = queue_url
-      @region = region
       @max_messages = max_messages
       @visibility_timeout = visibility_timeout
-      @aws_credentials = aws_credentials
-      @aws_access_key_id = aws_access_key_id
-      @aws_secret_access_key = aws_secret_access_key
-      @aws_session_token = aws_session_token
+      @shutdown_requested = false
 
       setup_logger(logger)
-      setup_sqs_client
+      setup_sqs_client(aws_access_key_id, aws_secret_access_key, aws_session_token, aws_region)
+      setup_signal_handlers
 
       raise 'Queue URL is required.' unless @queue_url
     end
@@ -37,23 +33,31 @@ module SQSProcessor
       end
     end
 
-    def setup_sqs_client
-      credentials = if @aws_credentials
-                      @aws_credentials
-                    elsif @aws_access_key_id && @aws_secret_access_key
-                      Aws::Credentials.new(@aws_access_key_id, @aws_secret_access_key, @aws_session_token)
-                    elsif ENV['DATA_SYNC_AWS_ACCESS_KEY_ID'] && ENV['DATA_SYNC_AWS_SECRET_ACCESS_KEY']
-                      Aws::Credentials.new(ENV['DATA_SYNC_AWS_ACCESS_KEY_ID'], ENV['DATA_SYNC_AWS_SECRET_ACCESS_KEY'])
-                    else
-                      Aws::Credentials.new
-                    end
-
+    def setup_sqs_client(aws_access_key_id, aws_secret_access_key, aws_session_token, aws_region)
+      credentials = Aws::Credentials.new(aws_access_key_id, aws_secret_access_key, aws_session_token)
       @sqs_client = Aws::SQS::Client.new(
-        region: @region,
+        region: aws_region,
         credentials: credentials
       )
     end
 
+    def setup_signal_handlers
+      Signal.trap('TERM') do
+        logger.info 'Received SIGTERM, initiating graceful shutdown...'
+        @shutdown_requested = true
+      end
+
+      Signal.trap('INT') do
+        logger.info 'Received SIGINT, initiating graceful shutdown...'
+        @shutdown_requested = true
+      end
+
+      Signal.trap('QUIT') do
+        logger.info 'Received SIGQUIT, initiating graceful shutdown...'
+        @shutdown_requested = true
+      end
+    end
+
     def process_messages
       logger.info 'Starting SQS message processing...'
       logger.info "Queue URL: #{@queue_url}"
@@ -61,6 +65,8 @@ module SQSProcessor
       logger.info "Visibility timeout: #{@visibility_timeout} seconds"
 
       loop do
+        break if @shutdown_requested
+
         receive_messages
         sleep 1 # Small delay between polling cycles
       rescue StandardError => e
@@ -68,6 +74,8 @@ module SQSProcessor
         logger.error e.backtrace.join("\n")
         sleep 5 # Longer delay on error
       end
+
+      logger.info 'Graceful shutdown completed.'
     end
 
     def receive_messages
@@ -86,6 +94,8 @@ module SQSProcessor
       logger.info "Received #{response.messages.length} message(s)"
 
       response.messages.each do |message|
+        break if @shutdown_requested
+
         process_single_message(message)
       end
     end
@@ -149,5 +159,14 @@ module SQSProcessor
 
       attributes
     end
+
+    def shutdown_requested?
+      @shutdown_requested
+    end
+
+    def request_shutdown
+      logger.info 'Shutdown requested manually...'
+      @shutdown_requested = true
+    end
   end
 end

data/lib/sqs_processor/version.rb

@@ -1,3 +1,3 @@
 module SQSProcessor
-  VERSION = "0.1.1"
+  VERSION = "0.1.3"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: sqs_processor
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.1.3
 platform: ruby
 authors:
 - Unni Tallman
@@ -125,14 +125,12 @@ description: A comprehensive Ruby gem for processing messages from Amazon SQS qu
   with configurable message handling, error recovery, and extensible processing logic.
 email:
 - unni@bigbinary.com
-executables:
-- sqs_processor
+executables: []
 extensions: []
 extra_rdoc_files: []
 files:
 - LICENSE
 - README.md
-- bin/sqs_processor
 - lib/sqs_processor.rb
 - lib/sqs_processor/processor.rb
 - lib/sqs_processor/version.rb

data/bin/sqs_processor

@@ -1,77 +0,0 @@
-#!/usr/bin/env ruby
-
-require 'sqs_processor'
-require 'optparse'
-
-def parse_options
-  options = {
-    queue_url: nil,
-    region: nil,
-    max_messages: 10,
-    visibility_timeout: 30
-  }
-
-  OptionParser.new do |opts|
-    opts.banner = 'Usage: sqs_processor [options]'
-
-    opts.on('-q', '--queue-url URL', 'SQS Queue URL') do |url|
-      options[:queue_url] = url
-    end
-
-    opts.on('-r', '--region REGION', 'AWS Region') do |region|
-      options[:region] = region
-    end
-
-    opts.on('-m', '--max-messages NUMBER', Integer, 'Max messages per batch (default: 10)') do |num|
-      options[:max_messages] = num
-    end
-
-    opts.on('-v', '--visibility-timeout SECONDS', Integer, 'Visibility timeout in seconds (default: 30)') do |timeout|
-      options[:visibility_timeout] = timeout
-    end
-
-    opts.on('-h', '--help', 'Show this help message') do
-      puts opts
-      exit
-    end
-  end.parse!
-
-  options
-end
-
-if __FILE__ == $0
-  options = parse_options
-
-  begin
-    # Load environment variables
-    Dotenv.load
-
-    # Get queue URL from options or environment
-    queue_url = options[:queue_url] || ENV.fetch('DATA_SYNC_SQS_QUEUE_URL', nil)
-
-    unless queue_url
-      puts 'Error: Queue URL is required. Set DATA_SYNC_SQS_QUEUE_URL environment variable or use -q option.'
-      exit 1
-    end
-
-    processor = SQSProcessor::Processor.new(
-      queue_url: queue_url,
-      region: options[:region] || ENV['DATA_SYNC_AWS_REGION'] || 'us-east-1',
-      max_messages: options[:max_messages],
-      visibility_timeout: options[:visibility_timeout],
-      aws_access_key_id: ENV.fetch('DATA_SYNC_AWS_ACCESS_KEY_ID', nil),
-      aws_secret_access_key: ENV.fetch('DATA_SYNC_AWS_SECRET_ACCESS_KEY', nil),
-      aws_session_token: ENV.fetch('DATA_SYNC_AWS_SESSION_TOKEN', nil)
-    )
-
-    # Show queue attributes before starting
-    processor.get_queue_attributes
-
-    # Start processing messages
-    processor.process_messages
-  rescue StandardError => e
-    puts "Error: #{e.message}"
-    puts e.backtrace.join("\n")
-    exit 1
-  end
-end