hackler 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 4ee91a4d7a9915e7b074d7e6f882bfbc020de74875bba7827f018affb4fe4ed5
+  data.tar.gz: 18e8d50fa3213fd06ceeca5a0c2be32a4a062de969b62e61a39632131d0980b0
+SHA512:
+  metadata.gz: ecaabb2b8c2d08d2ed478f5f1ae1e6981d60f94a70e767abe79d56941df5579a958aabfadb12b45433a03900876450b6a8b849dd8a7ac7070f7d5f8f216ea6b3
+  data.tar.gz: 0c1b68a8578d5260c714dfd0772039fbdbbcb8fc28cb2ceae8191152cefe9fae393350db4e15ed51e18ef2b9d59f1d89dadae854e9ea392d1d2083eadb3dab02

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright Jyrki Gadinger
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,149 @@
+# Hackler
+
+> **Hackler**, da (/ˈhaklɐ/)
+>
+> _(Austria, informal) hard worker_
+
+A cursed approach to background jobs.  **Here be dragons.**
+
+Imagine you have a Rails app deployed to a web host that can **only** run Rack
+applications backed by a MySQL database.  There's no possibility to spin up
+any form of containerised application (podman, docker), and no possibility to
+drop into a shell (configure systemd units by hand, run stuff in a tmux/screen
+session), nothing.
+
+As long as you don't need any form of background processing, this is fine.
+
+However, once you want to make use of background jobs you'll run into a few
+challenges:
+
+- Most (if not all) in-process adapters will lose their enqueued jobs when
+  the app is restarted, this is not suitable for production.
+- Pretty much every other adapter requires either another process running
+  that's separate from the main app.  Remember, our web host can only run
+  Rack-based apps!
+- Most Active Job adapters usually require a specific database or message bus
+  too, be it PostgreSQL, Redis, RabbitMQ, beanstalkd, ... but all we have is
+  a basic MySQL.
+- Running the extra worker process on another server requires access to the
+  main database, and the additional database the job library needs, this
+  complicates the configuration of those databases (firewalls, VPNs, ACLs)...
+  and on a shared web host you might not be able to expose the MySQL as you
+  wish.
+- Running the extra worker process on another server requires an exact copy of
+  the app you're currently running, complicating deployment even further.
+- Trying to start the extra worker process as part of the Rack app is even
+  more cursed than whatever this mess is.  Not to mention that restarts are
+  going to get interesting...
+- Let's face it: there are no other background job processors out there that
+  are cooler than Sidekiq.
+- And migrating the app to another host is not a possibility either.
+
+So what can we do?
+
+Of course, we'll let our Rails web app talk to another Rails web app that can
+make use of a real job processor, all over HTTP.  *Obviously*.
+
+## Sounds cursed, I'm in!
+
+In the end you'll end up with two Rails apps.
+
+One is your usual web app that will make use of a new kind of Active Job
+adapter.  Let's call it... `rails-web`.
+
+The other one is a barebones Rails app that can use any other Active Job
+adapter you like (yay, Sidekiq!).  This one does not need access to the code
+or database of `rails-web`, and therefore doesn't really care what jobs you
+throw at it.  I'll call that one `rails-worker`.
+
+Whenever `rails-web` wants to enqueue an Active Job job, the `:hackler`
+adapter will create a new `Hackler::Job` record and perform a HTTP POST
+request to `rails-worker`, which will then enqueue a very basic job in the
+adapter chosen by that app (probably `:sidekiq`).  The basic job only has two
+parameters: a base callback URL and the job ID.
+
+Once the job processor framework used by `rails-worker` decides to enqueue the
+job, the `rails-worker` makes a HTTP POST request to `rails-web`, which
+fetches the stored job information from its database and runs the job.  If
+it's successful that job record is gone from the database.  Should the job
+fail however, the `rails-worker` process will automagically™ capture the
+exception thrown by the job and re-raise it on the `rails-worker` side.
+
+Of course, not everyone can access those HTTP endpoints exposed by Hackler,
+so a shared secret must be defined on both ends.  Some hashed value based on
+that and the job info will be sent with each request.
+
+Is this a good idea?  Probably not, longer running jobs might block some
+incoming requests.  Should be fine for quick jobs though, like sending out
+emails.
+
+Is this extra cursed?  **_Heck yea!_**
+
+## Installation
+
+Add the gem to your Gemfile:
+
+```bash
+$ bundle add hackler
+```
+
+In your main web app, install and configure Hackler:
+
+```bash
+$ bin/rails generate hackler:install
+```
+
+Make sure to edit `shared_secret`, `web_base_url`, and `worker_base_url` in
+`config/initializers/hackler.rb`.
+
+----
+
+To install and configure Hackler in the worker app you first need to configure
+an Active Job adapter.  One that actually processes these jobs.  I can't
+remember which one I liked best, but I think some numbers will do fine as
+well...
+
+```ruby
+# config/application.rb
+
+class Application < Rails::Application
+  # ...
+  config.active_job.queue_adapter = 62060811362.to_s(36).to_sym
+```
+
+Then, install and configure Hackler:
+
+```bash
+$ bin/rails generate hackler:worker_install
+```
+
+Make sure to set `shared_secret` in `config/initializers/hackler.rb` to the
+same value as on the web setup.
+
+----
+
+You are now ready to run the two Rails apps simultaneously!  Use one terminal
+per command:
+
+```bash
+# in your web app directory:
+bin/rails server
+
+# in your worker app directory:
+bin/rails server -p 4000
+bundle exec sidekiq
+```
+
+This repository contains an example Rails app with Sidekiq as its job backend
+at `/hacklerer`.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+
+## That's cool and all, but should I use this?
+
+Probably not.  If you do, please let me know why.
+
+As of now this library is a proof-of-concept.  Or a prototype.  Or an art
+project.  I mean, I didn't even bother with tests yet...

data/Rakefile

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+require "bundler/setup"
+
+APP_RAKEFILE = File.expand_path("test/dummy/Rakefile", __dir__)
+load "rails/tasks/engine.rake"
+
+load "rails/tasks/statistics.rake"
+
+require "bundler/gem_tasks"

data/app/controllers/hackler/application_controller.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module Hackler
+  class ApplicationController < ActionController::API
+    wrap_parameters false
+  end
+end

data/app/controllers/hackler/job_controller.rb

@@ -0,0 +1,58 @@
+# frozen_string_literal: true
+
+module Hackler
+  class JobController < ApplicationController
+    REQUIRED_PARAMS = %i[id base_url].freeze
+
+    before_action :set_secret
+    before_action :set_params
+    before_action :verify_secret!
+
+    def enqueue
+      Hackler::HacklerJob
+        .set(wait_until: Time.zone.at(@params[:timestamp]))
+        .perform_later(@params[:id], @params[:base_url])
+      render head: :no_content
+    end
+
+    def work
+      job = Hackler::Job.find(@params[:id])
+      ActiveJob::Base.execute(JSON.parse(job.data))
+      job.destroy!
+      render head: :no_content
+    rescue Exception => e # rubocop:disable Lint/RescueException
+      # this needs to be a Exception, otherwise we won't catch e.g. `NoMethodError`s
+      render json: {
+        exception: {
+          class:     e.class.name,
+          message:   e.message,
+          backtrace: Hackler.backtrace_cleaner.call(e.backtrace),
+        },
+      }, status: :internal_server_error
+    end
+
+    private
+
+    def set_secret
+      @header_secret = request.headers["x-hackler-secret"]
+      return if @header_secret
+
+      # pretend we're not there if the secret is missing
+      render head: :not_found
+    end
+
+    def set_params
+      @params = params.permit(:id, :base_url, :timestamp).to_h.symbolize_keys
+      return if @params.keys.intersection(REQUIRED_PARAMS).size == 2
+
+      render json: { error: "required parameters are missing" }, status: :unprocessable_content
+    end
+
+    def verify_secret!
+      expected_secret = Hackler.build_secret(**@params)
+      return if @header_secret == expected_secret
+
+      render json: { error: "unauthorised" }, status: :unauthorized
+    end
+  end
+end

data/app/jobs/hackler/application_job.rb

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+module Hackler
+  class ApplicationJob < ActiveJob::Base
+  end
+end

data/app/jobs/hackler/hackler_job.rb

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+module Hackler
+  class HacklerJob < ApplicationJob
+    queue_as Hackler.worker_queue
+
+    def perform(id, base_url)
+      conn = Hackler.connection_for(base_url)
+      parameters = {
+        id:,
+        base_url:,
+      }
+      begin
+        conn.post("work", parameters, { "x-hackler-secret" => Hackler.build_secret(**parameters) })
+      rescue Faraday::ServerError => e
+        json_response = begin
+          JSON.parse(e.response_body, symbolize_names: true)
+        rescue
+          {}
+        end
+        raise Hackler::JobError.from_json(json_response[:exception]) if json_response.key?(:exception)
+
+        raise # reraise other unexpected errors
+      end
+    end
+  end
+end

data/app/models/hackler/application_record.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module Hackler
+  class ApplicationRecord < ActiveRecord::Base
+    self.abstract_class = true
+  end
+end

data/app/models/hackler/job.rb

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+module Hackler
+  class Job < ApplicationRecord
+  end
+end

data/config/routes.rb

@@ -0,0 +1,4 @@
+Hackler::Engine.routes.draw do
+  post "enqueue", to: "job#enqueue", constraints: ->() { Hackler.worker }
+  post "work",    to: "job#work",    constraints: ->() { !Hackler.worker }
+end

data/db/migrate/20241002145356_create_hackler_jobs.rb

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+class CreateHacklerJobs < ActiveRecord::Migration[7.2]
+  def change
+    create_table :hackler_jobs do |t|
+      t.text :data
+
+      t.timestamps
+    end
+  end
+end

data/lib/active_job/queue_adapters/hackler_adapter.rb

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+module ActiveJob
+  module QueueAdapters
+    class HacklerAdapter < AbstractAdapter
+      def enqueue(job)
+        j = Hackler::Job.create(data: JSON.dump(job.serialize))
+        notify_worker(j)
+      end
+
+      def enqueue_at(job, timestamp)
+        j = Hackler::Job.create(data: JSON.dump(job.serialize))
+        notify_worker(j, timestamp)
+      end
+
+      private
+
+      def notify_worker(job, timestamp = nil)
+        conn = Hackler.connection_for(Hackler.worker_base_url)
+        parameters = {
+          id:        job.id,
+          base_url:  Hackler.web_base_url,
+          timestamp:,
+        }
+        conn.post("enqueue", parameters, { "x-hackler-secret" => Hackler.build_secret(**parameters) })
+      end
+    end
+  end
+end

data/lib/generators/hackler/install/USAGE

@@ -0,0 +1,5 @@
+Description:
+    Installs Hackler in a sad web app
+
+Example:
+    bin/rails generate hackler:install

data/lib/generators/hackler/install/install_generator.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+class Hackler::InstallGenerator < Rails::Generators::Base
+  source_root File.expand_path("templates", __dir__)
+
+  def copy_files
+    copy_file "initializer.rb", "config/initializers/hackler.rb"
+  end
+
+  def install_route
+    route %(mount Hackler::Engine => "/hackler")
+  end
+
+  def configure_active_job_adapter
+    gsub_file Pathname(destination_root).join("config/environments/production.rb"),
+              /(# )?config\.active_job\.queue_adapter\s+=.*/,
+              "config.active_job.queue_adapter = :hackler"
+  end
+
+  def add_migrations
+    rails_command "hackler:install:migrations"
+  end
+end

data/lib/generators/hackler/install/templates/initializer.rb

@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+
+Hackler.configure do |config|
+  # change this to an actual secret string
+  config.shared_secret = "hackme"
+
+  # `false` if this is the web app, `true` if this is the Hackler worker
+  config.worker = false
+
+  # set this to the base url where Hackler gets mounted in your app
+  config.web_base_url = "https://webapp.example.com/hackler"
+
+  # set this to where the Hackler worker runs
+  config.worker_base_url = "https://hackler.example.com/hackler"
+
+  # define a custom backtrace cleaner
+  # config.backtrace_cleaner = ->(backtrace) { ::Rails.backtrace_cleaner.clean(backtrace) }
+end

data/lib/generators/hackler/worker_install/USAGE

@@ -0,0 +1,5 @@
+Description:
+    Installs Hackler in `worker` mode
+
+Example:
+    bin/rails generate hackler:worker_install

data/lib/generators/hackler/worker_install/templates/initializer.rb

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+Hackler.configure do |config|
+  # change this to an actual secret string
+  config.shared_secret = "hackme"
+
+  # `false` if this is the web app, `true` if this is the Hackler worker
+  config.worker = true
+
+  # queue name where Hackler jobs will be processed
+  config.worker_queue = :default
+end
+
+return # remove this line if you use Sidekiq and want to have some nice presets:
+
+# record backtraces by default
+Sidekiq.default_job_options = { backtrace: true } unless ENV.key?("SIDEKIQ_DISABLE_BACKTRACES")
+
+Sidekiq.configure_server do |config|
+  # record full backtraces, they are cleaned by the hackler worker
+  config[:backtrace_cleaner] = ->(backtrace) { backtrace }
+end

data/lib/generators/hackler/worker_install/worker_install_generator.rb

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+class Hackler::WorkerInstallGenerator < Rails::Generators::Base
+  source_root File.expand_path("templates", __dir__)
+
+  def copy_files
+    copy_file "initializer.rb", "config/initializers/hackler.rb"
+  end
+
+  def install_route
+    route %(mount Hackler::Engine => "/hackler")
+  end
+end

data/lib/hackler/engine.rb

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+module Hackler
+  class Engine < ::Rails::Engine
+    isolate_namespace Hackler
+    config.generators.api_only = true
+  end
+end

data/lib/hackler/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module Hackler
+  VERSION = "0.1.0"
+end

data/lib/hackler.rb

@@ -0,0 +1,77 @@
+# frozen_string_literal: true
+
+require "hackler/version"
+require "hackler/engine"
+
+require "active_job"
+require "active_job/queue_adapters"
+require "active_record"
+
+require "digest/sha2"
+require "faraday"
+
+require "zeitwerk"
+
+loader = Zeitwerk::Loader.for_gem(warn_on_extra_files: false)
+loader.ignore("#{__dir__}/generators")
+loader.setup
+
+module Hackler
+  mattr_accessor :shared_secret, default: "hackme"
+
+  # set to true if this is the instance that can actually run the jobs
+  mattr_accessor :worker, default: false
+
+  mattr_accessor :backtrace_cleaner, default: ->(backtrace) { ::Rails.backtrace_cleaner.clean(backtrace) }
+
+  mattr_accessor :web_base_url
+
+  mattr_accessor :worker_base_url
+
+  # set this to the queue hackler should use in jobs
+  mattr_accessor :worker_queue, default: :default
+
+  def self.configure = yield self
+
+  def self.build_secret(id:, base_url:, **)
+    Digest::SHA512.base64digest([shared_secret, id, base_url].join(&:to_s))
+  end
+
+  def self.connection_for(url) = Faraday.new(url:) do |builder|
+    builder.request :json
+    builder.response :json
+    builder.response :raise_error
+  end
+
+  # Base class for Hackler errors
+  class Error < StandardError; end
+
+  # Raised when a job failed
+  class JobError < Error
+    def self.from_json(json)
+      message   = json.fetch(:message, "???")
+      klass     = json.fetch(:class, "???")
+      backtrace = json.fetch(:backtrace) { ["???"] * 5 }
+
+      # sick ruby trick: create a new subclass and extend it with a module
+      # which overwrites the exception class' name, in order to make it look
+      # better in e.g. the sidekiq dashboards
+      subclass = Class.new(self).extend(NameExtender.new(klass))
+      subclass.new(message).tap do |exc|
+        exc.set_backtrace backtrace
+      end
+    end
+
+    class NameExtender < Module
+      def initialize(name)
+        super
+
+        define_method :name do
+          name
+        end
+      end
+    end
+
+    private_constant :NameExtender
+  end
+end

data/lib/tasks/hackler_tasks.rake

@@ -0,0 +1,6 @@
+# frozen_string_literal: true
+
+# desc "Explaining what the task does"
+# task :hackler do
+#   # Task goes here
+# end

metadata

@@ -0,0 +1,96 @@
+--- !ruby/object:Gem::Specification
+name: hackler
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Jyrki Gadinger
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2024-10-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: faraday
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.12'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.12'
+- !ruby/object:Gem::Dependency
+  name: rails
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 7.2.1
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 7.2.1
+description: A cursed approach to background jobs.  Here be dragons.
+email:
+- nilsding@nilsding.org
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- MIT-LICENSE
+- README.md
+- Rakefile
+- app/controllers/hackler/application_controller.rb
+- app/controllers/hackler/job_controller.rb
+- app/jobs/hackler/application_job.rb
+- app/jobs/hackler/hackler_job.rb
+- app/models/hackler/application_record.rb
+- app/models/hackler/job.rb
+- config/routes.rb
+- db/migrate/20241002145356_create_hackler_jobs.rb
+- lib/active_job/queue_adapters/hackler_adapter.rb
+- lib/generators/hackler/install/USAGE
+- lib/generators/hackler/install/install_generator.rb
+- lib/generators/hackler/install/templates/initializer.rb
+- lib/generators/hackler/worker_install/USAGE
+- lib/generators/hackler/worker_install/templates/initializer.rb
+- lib/generators/hackler/worker_install/worker_install_generator.rb
+- lib/hackler.rb
+- lib/hackler/engine.rb
+- lib/hackler/version.rb
+- lib/tasks/hackler_tasks.rake
+homepage: https://github.com/nilsding/hackler
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/nilsding/hackler
+  source_code_uri: https://github.com/nilsding/hackler
+  rubygems_mfa_required: 'true'
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 3.2.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.5.9
+signing_key:
+specification_version: 4
+summary: A cursed approach to background jobs
+test_files: []