falqon 0.0.1 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4ba4e39b3f74379b319b0946fd639ecb58fe7306f2f062edbeca42ad7cb3224f
-  data.tar.gz: 91aa2ebc4e44d890ad6343b85960d968aa33b10ef158a393f2119e8561db75d4
+  metadata.gz: 2a6de2c2359a6d2aebf3f7f0531b9dea885d46f579a64aa0b46036c204c7df52
+  data.tar.gz: 8896a1d994cd209f436c8c260c5eba7985c9b3055064f186e26d7ff8452f7407
 SHA512:
-  metadata.gz: 6ebe08c3284edf73c79a37a0fb909dec8ce12cb52790426b0aeaad175000255204b09c1ead7834eac35872893825d43128b18567d18b7e9c3a919aef00e48c40
-  data.tar.gz: a738b12bdcf0776ee1008cf833e87477e08d6a551c6881cbe92945ea9499e0830fd81908beaa6ca184809ca32a14350fc960fda728a47d365b01d43434c52e1a
+  metadata.gz: 465c3dc7415da3ee83118227916a0d21a653fd8c74b2876a73e2510e545e7b210d0a96425d3f014ad997516f9314fea9de819b81deb373ec949162a3b7d6ca15
+  data.tar.gz: 231afe294759e0cc1fe0b0a884a99044808021e0d6d1f7201a8406f7a5b0b0128c6821f25b6d743d9c2475e9397c94dce10fc12ec831fbd123c7875495e067a1

data/CHANGELOG.md

@@ -1,5 +1,30 @@
 # Changelog
 
-## Falqon v0.0.1 (2022-06-22)
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Added
+
+### Changed
+
+### Removed
+
+## Falqon v1.0.0 (2024-12-07)
+
+### Changed
+
+- Use atomic execution (MULTI/EXEC) where possible.
+
+## Falqon v0.1.0 (2024-11-16)
 
 Initial release
+
+## Falqon v0.0.1 (2023-06-22)
+
+[unreleased]: https://github.com/floriandejonckheere/falqon/compare/v1.0.0...HEAD
+[1.0.0]: https://github.com/floriandejonckheere/falqon-pro/releases/tag/v1.0.0
+[0.1.0]: https://github.com/floriandejonckheere/falqon-pro/releases/tag/v0.1.0

data/Gemfile

@@ -5,13 +5,45 @@ source "https://rubygems.org"
 # Specify your gem's dependencies in falqon.gemspec
 gemspec
 
+# Task runner
+gem "rake", "13.2.1", require: false
+
+group :development do
+  # Documentation
+  gem "yard", "0.9.37", require: false
+  gem "yard-sorbet", "0.9.0", require: false
+
+  # Simple server
+  gem "webrick", "1.9.1", require: false
+end
+
 group :development, :test do
-  gem "debug", require: false
-  gem "ffaker", require: false
-  gem "rake", require: false
-  gem "rspec", require: false
-  gem "rubocop", require: false
-  gem "rubocop-performance", require: false
-  gem "rubocop-rspec", require: false
-  gem "timecop", require: false
+  # Debugger
+  gem "debug", "1.9.2", require: false
+
+  # Build objects for tests
+  gem "factory_bot", "6.5.0", require: false
+
+  # Generate fake data
+  gem "ffaker", "2.23.0", require: false
+
+  # Mock Redis server
+  gem "mock_redis", "0.48.1", require: false
+
+  # Behavior-driven test framework
+  gem "rspec", "3.13.0", require: false
+
+  # Linter
+  gem "rubocop", "1.69.0", require: false
+  gem "rubocop-factory_bot", "2.26.1", require: false
+  gem "rubocop-performance", "1.23.0", require: false
+  gem "rubocop-rake", "0.6.0", require: false
+  gem "rubocop-rspec", "3.2.0", require: false
+
+  # Type checker
+  gem "sorbet", "0.5.11672", require: false
+  gem "tapioca", "0.16.5", require: false
+
+  # Time control
+  gem "timecop", "0.9.10", require: false
 end

data/README.md

@@ -1,13 +1,40 @@
 # Falqon
-
-![Continuous Integration](https://github.com/floriandejonckheere/falqon/workflows/Continuous%20Integration/badge.svg)
+[![Continuous Integration](https://github.com/floriandejonckheere/falqon/actions/workflows/ci.yml/badge.svg)](https://github.com/floriandejonckheere/falqon/actions/workflows/ci.yml)
 ![Release](https://img.shields.io/github/v/release/floriandejonckheere/falqon?label=Latest%20release)
 
-Simple, efficient messaging queue for Ruby.
+Simple, efficient, and reliable messaging queue for Ruby.
+
+Falqon is a simple messaging queue implementation, backed by the in-memory Redis key-value store.
+It exposes a simple Ruby API to send and receive messages between different processes, between threads in the same process, or even fibers in the same thread.
+It is perfect when you require a lightweight solution for processing messages, but don't want to deal with the complexity of a full-blown message queue like RabbitMQ or Kafka.
+
+See the [documentation](https://docs.falqon.dev) for more information on how to use Falqon in your application.
+
+## Features
+
+Falqon offers an elegant solution for messaging queues in Ruby.
+
+- Elegant: only two methods to send and receive messages
+- Reliable: no data is lost when a client crashes unexpectedly
+- Fast: Falqon is built on top of Redis, a fast in-memory data store
+- Flexible: tune the behaviour of the queue to your needs
+
+## Get started
+
+- Install Falqon and get working with messaging queues in a heartbeat using the [quickstart guide](#quickstart)
+- Check out the [API documentation](https://docs.falqon.dev/) for more information on how to use Falqon in your application
+- Check out the [CLI documentation](https://docs.falqon.dev/Falqon/CLI.html) for more information on how to manage queues and messages from the command line
+- Read the [architecture documentation](#architecture) to learn more about how Falqon works under the hood
+- Read about [Falqon Pro](https://docs.falqon.dev/pro), the commercial addon for Falqon that offers additional features
+
+## Quickstart
+
+### Requirements
 
-Falqon offers a simple messaging queue implementation backed by Redis.
+Falqon requires a Redis 6+ server to be available.
+Use the [docker-compose.yml](https://github.com/floriandejonckheere/falqon/blob/master/docker-compose.yml) file to quickly spin up a Redis server.
 
-## Installation
+### Installation
 
 Add this line to your application's Gemfile:
 
@@ -23,14 +50,72 @@ Or install it yourself as:
 
     $ gem install falqon
 
-## Usage
+### Configuration
+
+The default configuration works out of the box with the provided `docker-compose.yml` file.
+See [configuration](https://docs.falqon.dev/Falqon/Configuration.html) if you want to adjust the configuration.
+
+### Usage
 
 ```ruby
 require "falqon"
 
-# TODO
+queue = Falqon::Queue.new("my_queue")
+
+# Push a message to the queue
+queue.push("Hello, world!", "Hello, world again!")
+
+# Pop a message from the queue (return style)
+puts queue.pop # => "Hello, world!"
+
+queue.empty? # => false
+
+queue.peek # => "Hello, world again!"
+
+# Pop a message from the queue (block style)
+queue.pop do |message|
+  puts message # => "Hello, world again!"
+  
+  # Raising a Falqon::Error exception will cause the message to be requeued
+  raise Falqon::Error, "Something went wrong"
+end
+
+queue.empty? # => false
+
+puts queue.pop # => "Hello, world again!"
+
+queue.empty? # => true
+
+queue.peek # => nil
 ```
 
+For more comprehensive examples, see the [examples directory](examples/) in the repository.
+
+## Architecture
+
+A queue is identified with a name, which is used as a key prefix.
+Queues are stored in Redis as a list of incrementing integers representing unique message identifiers.
+The messages itself are stored in Redis as strings.
+
+The following Redis keys are used to store data.
+
+- `[{prefix}:]queues`: set of queue names
+
+- `[{prefix}/]{name}`: list of message identifiers on the (pending) queue
+
+- `[{prefix}/]{name}:id`: message identifier sequence
+
+- `[{prefix}/]{name}:processing`: list of message identifiers being processed
+
+- `[{prefix}/]{name}:scheduled`: list of message identifiers scheduled to retry
+
+- `[{prefix}/]{name}:dead`: list of message identifiers that have been discarded
+
+- `[{prefix}/]{name}:data:{id}`: message data for identifier `{id}`
+
+- `[{prefix}/]{name}:metadata`: metadata for the queue
+
+- `[{prefix}/]{name}:metadata:{id}`: metadata for identifier `{id}`
 
 ## Testing
 
@@ -44,10 +129,25 @@ bundle exec rspec
 To release a new version, update the version number in `lib/falqon/version.rb`, update the changelog, commit the files and create a git tag starting with `v`, and push it to the repository.
 Github Actions will automatically run the test suite, build the `.gem` file and push it to [rubygems.org](https://rubygems.org).
 
+## Documentation
+
+The documentation in `docs/` is automatically built by [YARD](https://yardoc.org) and pushed to [docs.falqon.dev](https://docs.falqon.dev) on every push to the `main` branch.
+Locally, you can build the documentation using the following commands:
+
+```sh
+rake yard
+```
+
+In development, you can start a local server to preview the documentation:
+
+```sh
+yard server --reload
+```
+
 ## Contributing
 
 Bug reports and pull requests are welcome on GitHub at [https://github.com/floriandejonckheere/falqon](https://github.com/floriandejonckheere/falqon). 
 
 ## License
 
-The software is available as open source under the terms of the [LGPLv3 License](https://www.gnu.org/licenses/lgpl-3.0.html).
+The software is available as open source under the terms of the [LGPL-3.0 License](https://www.gnu.org/licenses/lgpl-3.0.html).

data/bin/falqon

@@ -0,0 +1,8 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "bundler/setup"
+
+require "falqon"
+
+Falqon::CLI.start(ARGV)

data/config/inflections.rb

@@ -1,3 +1,5 @@
 # frozen_string_literal: true
 
-Falqon.loader.inflector.inflect({})
+Falqon.loader.inflector.inflect({
+                                  "cli" => "CLI",
+                                })

data/lib/falqon/cli/base.rb

@@ -0,0 +1,35 @@
+# frozen_string_literal: true
+
+module Falqon
+  class CLI
+    # @!visibility private
+    class Base
+      attr_reader :options
+
+      def initialize(options = {})
+        @options = options
+      end
+
+      def call
+        validate
+        execute
+      rescue StandardError => e
+        puts e.message
+      end
+
+      protected
+
+      def validate
+        raise NotImplementedError
+      end
+
+      def execute
+        raise NotImplementedError
+      end
+
+      def pluralize(count, singular, plural)
+        "#{count} #{count == 1 ? singular : plural}"
+      end
+    end
+  end
+end

data/lib/falqon/cli/clear.rb

@@ -0,0 +1,86 @@
+# frozen_string_literal: true
+
+module Falqon
+  class CLI
+    # Clear messages from a queue
+    #
+    # Clearing a subqueue removes all messages and their data from the subqueue.
+    #
+    # Usage:
+    #   falqon clear -q, --queue=QUEUE
+    #
+    # Options:
+    #   -q, --queue=QUEUE                                           # Queue name
+    #       [--pending], [--no-pending], [--skip-pending]           # Clear only pending messages
+    #       [--processing], [--no-processing], [--skip-processing]  # Clear only processing messages
+    #       [--dead], [--no-dead], [--skip-dead]                    # Clear only dead messages
+    #
+    # If none of the +--pending+, +--processing+, +--scheduled+, or +--dead+ options are specified, all messages are cleared.
+    #
+    # @example Clear all messages in a queue
+    #   $ falqon clear --queue jobs
+    #   Cleared 3 messages from queue jobs
+    #
+    # @example Clear only pending messages
+    #   $ falqon clear --queue jobs --pending
+    #   Cleared 3 messages from queue jobs
+    #
+    # @example Clear only processing messages
+    #   $ falqon clear --queue jobs --processing
+    #   Cleared 3 messages from queue jobs
+    #
+    # @example Clear only scheduled messages
+    #   $ falqon clear --queue jobs --scheduled
+    #   Cleared 3 messages from queue jobs
+    #
+    # @example Clear only dead messages
+    #   $ falqon clear --queue jobs --dead
+    #   Cleared 3 messages from queue jobs
+    #
+    class Clear < Base
+      # @!visibility private
+      def validate
+        raise "No queue registered with this name: #{options[:queue]}" if options[:queue] && !Falqon::Queue.all.map(&:name).include?(options[:queue])
+        raise "--pending, --processing, --scheduled, and --dead are mutually exclusive" if [options[:pending], options[:processing], options[:scheduled], options[:dead]].count(true) > 1
+      end
+
+      # @!visibility private
+      def execute
+        # Clear messages
+        ids = subqueue.clear
+
+        if options[:pending]
+          puts "Cleared #{pluralize(ids.count, 'pending message', 'pending messages')} from queue #{queue.name}"
+        elsif options[:processing]
+          puts "Cleared #{pluralize(ids.count, 'processing message', 'processing messages')} from queue #{queue.name}"
+        elsif options[:scheduled]
+          puts "Cleared #{pluralize(ids.count, 'scheduled message', 'scheduled messages')} from queue #{queue.name}"
+        elsif options[:dead]
+          puts "Cleared #{pluralize(ids.count, 'dead message', 'dead messages')} from queue #{queue.name}"
+        else
+          puts "Cleared #{pluralize(ids.count, 'message', 'messages')} from queue #{queue.name}"
+        end
+      end
+
+      private
+
+      def queue
+        @queue ||= Falqon::Queue.new(options[:queue])
+      end
+
+      def subqueue
+        @subqueue ||= if options[:pending]
+                        queue.pending
+                      elsif options[:processing]
+                        queue.processing
+                      elsif options[:scheduled]
+                        queue.scheduled
+                      elsif options[:dead]
+                        queue.dead
+                      else
+                        queue
+                      end
+      end
+    end
+  end
+end

data/lib/falqon/cli/delete.rb

@@ -0,0 +1,152 @@
+# frozen_string_literal: true
+
+module Falqon
+  class CLI
+    # Delete messages from a queue
+    #
+    # Deleting a message removes it including its data from the queue.
+    #
+    # Usage:
+    #   falqon delete -q, --queue=QUEUE
+    #
+    # Options:
+    #   -q, --queue=QUEUE                                           # Queue name
+    #       [--pending], [--no-pending], [--skip-pending]           # Delete only pending messages (default)
+    #       [--processing], [--no-processing], [--skip-processing]  # Delete only processing messages
+    #       [--dead], [--no-dead], [--skip-dead]                    # Delete only dead messages
+    #       [--head=N]                                              # Delete N messages from head of queue
+    #       [--tail=N]                                              # Delete N messages from tail of queue
+    #       [--index=N]                                             # Delete message at index N
+    #       [--range=N M]                                           # Delete messages at index N to M
+    #       [--id=N]                                                # Delete message with ID N
+    #
+    # @example Delete all messages in the queue (by default only pending messages are deleted)
+    #   $ falqon delete --queue jobs
+    #   Deleted 10 messages from queue jobs
+    #
+    # @example Delete only pending messages
+    #   $ falqon delete --queue jobs --pending
+    #   Deleted 10 pending messages from queue jobs
+    #
+    # @example Delete only processing messages
+    #   $ falqon delete --queue jobs --processing
+    #   Deleted 1 processing message from queue jobs
+    #
+    # @example Delete only scheduled messages
+    #   $ falqon delete --queue jobs --scheduled
+    #   Deleted 1 scheduled message from queue jobs
+    #
+    # @example Delete only dead messages
+    #   $ falqon delete --queue jobs --dead
+    #   Deleted 5 dead messages from queue jobs
+    #
+    # @example Delete first 5 messages
+    #   $ falqon delete --queue jobs --head 5
+    #   Deleted 5 messages from queue jobs
+    #
+    # @example Delete last 5 messages
+    #   $ falqon delete --queue jobs --tail 5
+    #   Deleted 5 messages from queue jobs
+    #
+    # @example Delete message at index 5
+    #   $ falqon delete --queue jobs --index 3 --index 5
+    #   Deleted 1 message from queue jobs
+    #
+    # @example Delete messages from index 5 to 10
+    #   $ falqon delete --queue jobs --range 5 10
+    #   Deleted 6 messages from queue jobs
+    #
+    # @example Delete message with ID 5
+    #   $ falqon delete --queue jobs --id 5 --id 1
+    #   Deleted 2 messages from queue jobs
+    #
+    class Delete < Base
+      # @!visibility private
+      def validate
+        raise "No queue registered with this name: #{options[:queue]}" if options[:queue] && !Falqon::Queue.all.map(&:name).include?(options[:queue])
+
+        raise "--pending, --processing, --scheduled, and --dead are mutually exclusive" if [options[:pending], options[:processing], options[:scheduled], options[:dead]].count(true) > 1
+
+        raise "--head, --tail, --index, and --range are mutually exclusive" if [options[:head], options[:tail], options[:index], options[:range]].count { |o| o } > 1
+        raise "--range must be specified as two integers" if options[:range] && options[:range].count != 2
+
+        raise "--id is mutually exclusive with --head, --tail, --index, and --range" if options[:id] && [options[:head], options[:tail], options[:index], options[:range]].count { |o| o }.positive?
+      end
+
+      # @!visibility private
+      def execute
+        # Collect identifiers
+        ids = if options[:id]
+                Array(options[:id])
+              elsif options[:index]
+                Array(options[:index]).map do |i|
+                  subqueue.peek(index: i) || raise("No message at index #{i}")
+                end
+              else
+                start, stop = range_options
+
+                subqueue.range(start:, stop:).map(&:to_i)
+              end
+
+        # Transform identifiers to messages
+        messages = ids.map do |id|
+          message = Falqon::Message.new(queue, id: id.to_i)
+
+          raise "No message with ID #{id}" unless message.exists?
+
+          message
+        end
+
+        # Delete messages
+        messages.each(&:delete)
+
+        if options[:processing]
+          puts "Deleted #{pluralize(messages.count, 'processing message', 'processing messages')} from queue #{queue.name}"
+        elsif options[:scheduled]
+          puts "Deleted #{pluralize(messages.count, 'scheduled message', 'scheduled messages')} from queue #{queue.name}"
+        elsif options[:dead]
+          puts "Deleted #{pluralize(messages.count, 'dead message', 'dead messages')} from queue #{queue.name}"
+        else # options[:pending]
+          puts "Deleted #{pluralize(messages.count, 'pending message', 'pending messages')} from queue #{queue.name}"
+        end
+      end
+
+      private
+
+      def queue
+        @queue ||= Falqon::Queue.new(options[:queue])
+      end
+
+      def subqueue
+        @subqueue ||= if options[:processing]
+                        queue.processing
+                      elsif options[:scheduled]
+                        queue.scheduled
+                      elsif options[:dead]
+                        queue.dead
+                      else # options[:pending]
+                        queue.pending
+                      end
+      end
+
+      def range_options
+        if options[:tail]
+          [
+            -options[:tail],
+            -1,
+          ]
+        elsif options[:range]
+          [
+            options[:range].first,
+            options[:range].last,
+          ]
+        else # options[:head]
+          [
+            0,
+            options.fetch(:head, 0) - 1,
+          ]
+        end
+      end
+    end
+  end
+end

data/lib/falqon/cli/kill.rb

@@ -0,0 +1,143 @@
+# frozen_string_literal: true
+
+module Falqon
+  class CLI
+    # Kill messages in a queue
+    #
+    # Killing a message removes it from the pending or scheduled queue, and moves it to the dead queue.
+    #
+    # Usage:
+    #   falqon kill -q, --queue=QUEUE
+    #
+    # Options:
+    #   -q, --queue=QUEUE                                           # Queue name
+    #       [--pending], [--no-pending], [--skip-pending]           # Kill only pending messages (default)
+    #       [--processing], [--no-processing], [--skip-processing]  # Kill only processing messages
+    #       [--head=N]                                              # Kill N messages from head of queue
+    #       [--tail=N]                                              # Kill N messages from tail of queue
+    #       [--index=N]                                             # Kill message at index N
+    #       [--range=N M]                                           # Kill messages at index N to M
+    #       [--id=N]                                                # Kill message with ID N
+    #
+    # @example Kill all messages in the queue (by default only pending messages are killed)
+    #   $ falqon kill --queue jobs
+    #   Killed 10 messages from queue jobs
+    #
+    # @example Kill only pending messages
+    #   $ falqon kill --queue jobs --pending
+    #   Killed 10 pending messages from queue jobs
+    #
+    # @example Kill only processing messages
+    #   $ falqon kill --queue jobs --processing
+    #   Killed 1 processing message from queue jobs
+    #
+    # @example Kill only scheduled messages
+    #   $ falqon kill --queue jobs --scheduled
+    #   Killed 1 scheduled message from queue jobs
+    #
+    # @example Kill first 5 messages
+    #   $ falqon kill --queue jobs --head 5
+    #   Killed 5 messages from queue jobs
+    #
+    # @example Kill last 5 messages
+    #   $ falqon kill --queue jobs --tail 5
+    #   Killed 5 messages from queue jobs
+    #
+    # @example Kill message at index 5
+    #   $ falqon kill --queue jobs --index 3 --index 5
+    #   Killed 1 message from queue jobs
+    #
+    # @example Kill messages from index 5 to 10
+    #   $ falqon kill --queue jobs --range 5 10
+    #   Killed 6 messages from queue jobs
+    #
+    # @example Kill message with ID 5
+    #   $ falqon kill --queue jobs --id 5 --id 1
+    #   Killed 2 messages from queue jobs
+    #
+    class Kill < Base
+      # @!visibility private
+      def validate
+        raise "No queue registered with this name: #{options[:queue]}" if options[:queue] && !Falqon::Queue.all.map(&:name).include?(options[:queue])
+
+        raise "--pending, --scheduled, and --processing are mutually exclusive" if [options[:pending], options[:scheduled], options[:processing]].count(true) > 1
+
+        raise "--head, --tail, --index, and --range are mutually exclusive" if [options[:head], options[:tail], options[:index], options[:range]].count { |o| o } > 1
+        raise "--range must be specified as two integers" if options[:range] && options[:range].count != 2
+
+        raise "--id is mutually exclusive with --head, --tail, --index, and --range" if options[:id] && [options[:head], options[:tail], options[:index], options[:range]].count { |o| o }.positive?
+      end
+
+      # @!visibility private
+      def execute
+        # Collect identifiers
+        ids = if options[:id]
+                Array(options[:id])
+              elsif options[:index]
+                Array(options[:index]).map do |i|
+                  subqueue.peek(index: i) || raise("No message at index #{i}")
+                end
+              else
+                start, stop = range_options
+
+                subqueue.range(start:, stop:).map(&:to_i)
+              end
+
+        # Transform identifiers to messages
+        messages = ids.map do |id|
+          message = Falqon::Message.new(queue, id: id.to_i)
+
+          raise "No message with ID #{id}" unless message.exists?
+
+          message
+        end
+
+        # Kill messages
+        messages.each(&:kill)
+
+        if options[:processing]
+          puts "Killed #{pluralize(messages.count, 'processing message', 'processing messages')} in queue #{queue.name}"
+        elsif options[:scheduled]
+          puts "Killed #{pluralize(messages.count, 'scheduled message', 'scheduled messages')} in queue #{queue.name}"
+        else # options[:pending]
+          puts "Killed #{pluralize(messages.count, 'pending message', 'pending messages')} in queue #{queue.name}"
+        end
+      end
+
+      private
+
+      def queue
+        @queue ||= Falqon::Queue.new(options[:queue])
+      end
+
+      def subqueue
+        @subqueue ||= if options[:processing]
+                        queue.processing
+                      elsif options[:scheduled]
+                        queue.scheduled
+                      else # options[:pending]
+                        queue.pending
+                      end
+      end
+
+      def range_options
+        if options[:tail]
+          [
+            -options[:tail],
+            -1,
+          ]
+        elsif options[:range]
+          [
+            options[:range].first,
+            options[:range].last,
+          ]
+        else # options[:head]
+          [
+            0,
+            options.fetch(:head, 0) - 1,
+          ]
+        end
+      end
+    end
+  end
+end