q4m 0.0.6 → 0.0.7

data/README.rdoc

@@ -2,32 +2,83 @@
 
 Interface to pragmatically setup a queue job for Q4M!
 
+= So... What's Q4M
+
+Q4M is a open-sourced MySQL Engine. that attempts to take away some
+logic from your application by encapsulating it inside of mysql.
+
+The documentation is kinda bad, but the magic that it does in the
+background is sweeeeeet!
+
+You basically store all of your background jobs inside a MySQL table.
+that table's engine should be a *queue* (q4m) engine!
+
+Exampe:
+
+    drop table if exists shell_jobs, ruby_jobs;
+    create table shell_jobs (command text) engine=queue;
+    create table ruby_jobs (command text) engine=queue;
+
+That engine will work under some kind of transaction. When you connect
+to the database the engine will give you a list of all the pending
+jobs. When you start a transaction you'll receive *ONE* job to work
+with. In the mean time the whole table gets locked..
+And if you do a 'select *' all results will return emtpy!
+
+Don't panic!
+
+This is great, you can never access two jobs at the same time.
+if another worker/machine connects they'll get a list of jobs except
+the one is wrapped in the transaction!
+
+So that means that the locking only happens on the connection that is
+perfoming the transaction but any other workers will get the full list
+of jobs except the ones that are being processed by other workers.
+
+When the first transaction gets terminated the engine will
+automatically unlock the table and delete the record.
+
+This is great, because you can have any number of workers working
+concurrently without making a mess!
+
+Meaning that a job can never be called in twice by anyone, and that
+you can grow horizontally, the limit is the sky!
+
+What's next?
+
+The job is done here!
+
+Go do something more interesting like clustering! ;)
+
+Seek & destroy, Seek & destroy!
+
+= So what exactly does this gem do?
+
+Gives you an interface to quickly setup how specificly a job is
+supposed to run!
+
+Basically, you define a class create an method called execute, which
+receives a *job* argument. truly a MySQL record in the form of an
+array, what you do from there is up to you!
+
+if something goes wrong just call the *queue_abort* method and
+everything gets rolled back!
+
 = Installation
 
     kazu@utopia:~$ gem install q4m
 
-= Usage
+= Simple example
 
     require 'rubygems'
     require 'q4m'
 
-    class ShellJob < Q4m::Queue::Job
+    # Specify how shell jobs will run
 
-      # *job* is a mysql record that is retrived from *table_name*
-      def execute job
-        # let the log know if the job was executed correctly!
-        job = job.first
-        return @log.info("Executed: #{job.inspect}") if system job
-
-        # If failed then log the job that failed and rollback the job so
-        # it stays in the queue!
-        @log.error "Aborted job: #{job.inspect}"
-        queue_abort
-      end
+    class ShellJob < ::Q4m::Queue::Job
 
-      # Database q4m table (name) where jobs are stored
-      def table_name
-        'my_queue'
+      def execute job
+        queue_abort unless system(job.first)
       end
 
     end
@@ -37,23 +88,194 @@ Interface to pragmatically setup a queue job for Q4M!
     pass = 'secret'
     database = 'my_db'
 
+    # Instantiate and run a job!
     mysql = Mysql.new host, user, pass, database
     job = ShellJob.new mysql, Logger.new('q4m.log')
 
     job.run
 
+= Quick tip
+
+Note that: The *ShellJob* class will try to connect to the
+'shell_jobs' q4m table by default, if you want to use a different table
+override the *table_name* method and specify one of your liking!
+
+    class ShellJob < ::Q4m::Queue::Job
+
+     def execute job
+        queue_abort unless system(job.first)
+     end
+
+      def table_name
+        'my_cool_table'
+      end
+
+    end
+
+= Run on a specific environment like rails?
+
+Would you like to have an entire rails-app as your worker?
+but dont't want to load the whole env with rake tasks everytime?
+
+That would be sweet right? and you can even reuse/test that code.
+
+No problem!
+
+You can leave only-one instance of rails running
+and tell it to pull jobs from the queue whenever it can?
+
+Yes!!
+
+But let's keep it flexibe so let's run two kinds of jobs, :ruby & :shell
+
+    module Background
+
+      # Specify a shell job. isn't this short & sweet?
+      class ShellJob < ::Q4m::Queue::Job
+
+        def execute job
+          queue_abort unless system(job.first)
+        end
+
+      end
+
+      # Now do the same thing for ruby
+      # and let's encapsulate the whole thing, and say that all ruby
+      # code that goes into the queue will be methods and they must
+      # always return booleans, if true complete the job successfully!
+      # rollback the transaction otherwise!
+      class RubyJob < ::Q4m::Queue::Job
+
+        def execute job
+          queue_abort unless (eval(job.first) == true)
+        end
+
+      end
+
+      # This hack will mimic a very simple deamon that will run in the
+      # background stay alert for any new jobs that get appended to
+      # the queue
+      class Deamon
+
+        def self.run type = :ruby
+          log   = Rails.root + 'log' + 'q4m.log'
+          path  = Rails.root + 'config' + 'database.yml'
+          conf  = YAML::load_file(path)['queues']
+          mysql = Mysql.new conf['host'], conf['username'], conf['password'], conf['database']
+          job   = if type == :ruby
+            Background::RubyJob.new mysql, Logger.new(log)
+          else
+            Background::ShellJob.new mysql, Logger.new(log)
+          end
+          if File.exist? '/tmp/run_queue.txt'
+            job.run
+            sleep 1
+            print '.'
+            mysql.close
+            run
+          end
+        end
+
+      end
+
+    end
+
+    # Now let's create a model that will help us create jobs for ruby
+    class RubyJob < ActiveRecord::Base
+
+      establish_connection :queues
+
+      def self.append command
+        create! :command => command
+      end
+
+    end
+
+    # And for shell
+    # this can be encapsulated into a parent class but whatever!
+    class ShellJob < ActiveRecord::Base
+
+      establish_connection :queues
+
+      def self.append command
+        create! :command => command
+      end
+
+    end
+
+    # take it one step farther and build a helper, after all we are
+    # going to be working with strings!
+    class String
+
+      def jobify type = :ruby
+        return RubyJob.append self if type == :ruby
+        return ShellJob.append self if type == :shell
+        nil
+      end
+
+    end
+
+= How to actually run it (on rails)!
+
+    # With the code above you can do something like this!
+
+    $ touch /tmp/run_queue.txt
+
+    # Run the deamon in a rails console!
+    >> Background::Deamon.run
+
+    # Let's create some background jobs on another console..
+
+    # Create a shell job, send it to the queue for later processing!
+    'RAILS_ENV=production rake -T'.jobify :shell
+
+    # Do the same but with ruby!
+    # NOTE: method inside should return boolean!!
+    'User.create_thumbnails(1)'.jobify
+
+    # Things should start running on the background, look at your logs!
+
+    # To stop the deamon cleanly just delete the file '/tmp/run_queue.txt'
+
+= This is great!
+
+Basically any machine running rails with this code, can pull jobs and
+do some hard work for you! even if they are not part of your cluster!
+
+The only requirement is that they can access the mysql-database.
+
+and no, you don't need to install that engine locally!
+
+= Extensibility!
+
+Go farther just create a new job class and run whatever you like!
+
+= Q4M lib?
+
+This is how I installed it.
+
+    wget http://q4m.kazuhooku.com/dist/old/mysql-5.1.41-linux-x86_64-glibc23-with-fast-mutexes-q4m-0.8.9.tar.gz
+    unzip *.gz
+    tar -xvf mysql-5.1.41-linux-x86_64-glibc23-with-fast-mutexes-q4m-0.8.9.tar
+    cd q4m-0.8.9-linux-x86_64
+    sudo cp libqueue_engine.so /usr/lib/mysql/plugin
+    sudo chmod 777 /usr/lib/mysql/plugin/libqueue_engine.so
+    mysql -u root -p -f mysql < support-files/install.sql
+
+You might wanna go to:
+
+    http://q4m.github.com/
 
+And see tha latest on how to get it installed.
 
-== Note on Patches/Pull Requests
+just thought of posting it, so you'll get an idea.
 
-* Fork the project.
-* Make your feature addition or bug fix.
-* Add tests for it. This is important so I don't break it in a
-  future version unintentionally.
-* Commit, do not mess with rakefile, version, or history.
-  (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
-* Send me a pull request. Bonus points for topic branches.
+Yeah, but what's your distro?
 
-== Copyright
+it's right here!
 
-Copyright (c) 2010 kazuyoshi tlacaelel. See LICENSE for details.
+    deploy@staging> cat /etc/*-release
+    DISTRIB_ID=Ubuntu
+    DISTRIB_RELEASE=10.04
+    DISTRIB_CODENAME=lucid
+    DISTRIB_DESCRIPTION="Ubuntu 10.04 LTS"

data/VERSION

@@ -1 +1 @@
-0.0.6
+0.0.7

data/lib/q4m.rb

@@ -24,12 +24,22 @@ module Q4m
       # - Executes the latest job.
       # - Restores the queue so other worker can execute a process.
       #
-      def run
+      def run terminate = false
         return unless @has_jobs
         queue_wait
         @log.info("Executing: #{latest_job.inspect}")
         execute latest_job
         queue_end
+        shutdown if terminate
+        nil
+      end
+
+      # Shuts down the worker
+      #
+      def shutdown
+        @mysql.close
+        @log.close
+        nil
       end
 
       private

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification 
 name: q4m
 version: !ruby/object:Gem::Version 
-  hash: 19
+  hash: 17
   prerelease: false
   segments: 
   - 0
   - 0
-  - 6
-  version: 0.0.6
+  - 7
+  version: 0.0.7
 platform: ruby
 authors: 
 - kazuyoshi tlacaelel
@@ -15,7 +15,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2010-11-09 00:00:00 -06:00
+date: 2010-11-12 00:00:00 -06:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency