resque-telework 0.2.0

data/Gemfile

@@ -0,0 +1,6 @@
+source "http://rubygems.org"
+
+gem 'resque', '1.19.0'
+gem 'sys-cpu'
+
+

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) Gilles Pirio
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,151 @@
+Resque Telework
+===============
+
+[github.com/gip/resque-telework](https://github.com/gip/resque-telework)
+
+Telework depends on Resque 1.20+ and Redis 2.2+
+
+Description
+-----------
+
+Telework is a [Resque](https://github.com/defunkt/resque) plugin aimed at controlling Resque workers from the web UI. It makes it easy to manage workers on a complex systems that includes several hosts, different queue(s) and an evolving source code that is deployed several times a day. Beyond starting and stopping workers on remote hosts, the plugin makes it easy to switch between code revisions, gives a partial view of each worker's log (stdout and stderr) and maintains a status of each workers.
+
+Telework comes with three main components
+
+* A web interface that smoothly integrates in Resque and adds its own tab
+* A daemon process to be started on each host (`rake telework:start_daemon` starts a new daemon and returns while `rake telework:daemon` runs the daemon interactively)
+* A registration command (`rake telework:register_revision`) to be called by the deployment script when a new revision is added on the host
+
+Note that currently (Telework 0.0.1), the daemon process is included in the main app, which is not really elegant as the full Rails environment needs to be loaded to run the daemon. A light-weight daemon is currently being developed and should be ready in the coming weeks.
+
+Overview of the WebUI
+---------------------
+
+![Main Telework Window](https://github.com/gip/resque-telework/raw/master/doc/screenshots/view_overview.png)
+
+The screenshot above shows the initial version of the Telework main window. The top table shows the active hosts, the different revision and the running workers. The bottom table shows the different status messages received from the hosts. Not that the layout is being improved and will look better soon :)
+
+Installation
+------------
+
+Install as a gem:
+
+```
+gilles@myapphost $ gem install resque-telework
+```
+
+You may also add the following line in the Gemfile
+
+```
+gem 'resque-telework'
+```
+
+Configuration
+-------------
+
+Some external configuration is necessary when working with Telework as the gem needs a way to retrieve information about the code revision being deployed (git hash or SVN revision number), its path, the location for log files and so on.. When Telework rake tasks start (`telework:register_revision`, `telework:start_daemon` or `telework:daemon`), it will try to open the file in the environment variable `TELEWORK_CONFIG_FILE`. If this variable doesn't exist it will try to open the `telework.conf` file in the local directory.
+
+The configuration file should contains information about the revision being deployed in the JSON format. A simple way of achieving this is to add a task in the deployment script. For instance, if you are using [Capistrano](https://github.com/capistrano/capistrano), the new task could look like this:
+
+```ruby
+# ...
+
+namespace :deploy do
+
+    # ... other deployment tasks here
+
+    # Telework registration task (example for github)
+    task :telework_register do
+      repo= 'john/reputedly'                                                         # <<< Change your Github repo name here 
+      github_repo= "https://github.com/#{repo}"
+      log_path= "#{deploy_to}/shared/worker_log"                                     # <<< Change paths to the log files here
+      run "mkdir -p #{log_path}" # Making sure the log directory exists
+      begin 
+        require 'octokit'  # Gem to access the Github API
+        client = Octokit::Client.new(:login => ACCOUNT, :password => PASSWORD )      # <<< Put your Github credentials here
+        commit= client.commit(repo, latest_revision)
+        rev_date= commit['commit']['committer']['date']
+        rev_name= commit['commit']['committer']['name']
+        rev_info= commit['commit']['message']
+      rescue                                                                         # No big deal if there is a problem accessing Github, 
+                                                                                     #   the info fields will just remain empty
+      end
+      cfg= { :revision => latest_revision,                                           # latest_revison, current_release, branch,...
+             :revision_small => latest_revision[0..6],                               #   are defined by Capistrano
+             :revision_path => "#{current_release}",
+             :revision_link => "#{github_repo}/commit/#{latest_revision}",
+             :revision_branch => branch,
+             :revision_date => rev_date,
+             :revision_committer => rev_name,
+             :revision_deployement_date => Time.now,
+             :revision_info => rev_info,
+             :revision_log_path => log_path,
+             :daemon_pooling_interval => 2,
+             :daemon_log_path => deploy_to }
+      
+      # Create the config file
+      require 'json' 
+      put cfg.to_json, "#{deploy_to}/current/telework.conf"
+      
+      # Start the registration rake task
+      run "cd #{deploy_to}/current && bundle exec rake telework:register_revision --trace"
+    end
+    after "deploy:more_symlinks", "deploy:telework_register"                          # <<< Schedule the task at the end of deployment
+
+end
+```
+
+Workflow
+--------
+
+After Telework is installed and the `TeleworkConfig` class modified to match your environment, the code may be deployed to all the relevant hosts. If you're using [Capistrano](https://github.com/capistrano/capistrano) it may look like:
+
+```
+gilles@myapphost $ cap deploy -S servers=myapphost,myworkhost0,myworkhost1,myworkhost2
+```
+
+The code above deploys the code to the main app box (`myapphost`) and all the other 'worker' hosts. On each of these hosts, it is now necessary to register the new revision with Telework and start the Telework daemon. For instance on host0, this is done using the following commands:
+
+```
+gilles@myworkhost0 $ rake telework:register_revision
+gilles@myworkhost0 $ rake telework:start_daemon
+```
+
+The main Telework tab should now show the new box as alive. It is now possible to seamlessly start new workers on these boxes using the new web-based UI.
+
+Going forward, when a new version of the app is deployed on host, it is necessary to register the new revision using the following command:
+
+```
+gilles@myworkhost0 $ rake telework:register_revision
+```
+Note that it is not necessary to stop and restart the daemon. Restarting the daemon is only required when the Telework gem is updated.
+
+Known Issues
+------------
+
+For version 0.2:
+
+* Buttons are not aligned in the web-UI
+* The daemon crashes if any of the log directories do not exist
+
+Bugs
+----
+
+Please report bugs on [github](https://github.com/gip/resque-telework/issues) or directly to [gip.github@gmail.com](gip.github@gmail.com)
+
+Todo
+----
+
+The following features are are being developed and should be available shortly:
+
+* Improved window layout
+
+The following features are planned for future releases:
+
+* Light-weight daemon in Haskell
+* Worker statistics
+
+Thanks
+------
+
+I would like to thank [Entelo](http://www.entelo.com/) for the awesome environment and support to open-source development 

data/lib/resque-telework/global.rb

@@ -0,0 +1,10 @@
+module Resque
+  module Plugins
+    module Telework
+      Name = NAME = 'resque-telework'
+      Nickname = NICKNAME = 'telework'
+      Version = VERSION = '0.2.0'
+      RedisInterfaceVersion = REDIS_INTERFACE_VERSION = '2'
+    end
+  end
+end

data/lib/resque-telework/manager.rb

@@ -0,0 +1,205 @@
+module Resque
+  module Plugins
+    module Telework      
+      class Manager
+        
+        include Resque::Plugins::Telework::Redis
+        
+        def initialize(cfg)
+          @RUN_DAEMON= true
+          @HOST= cfg['hostname']
+          @SLEEP= cfg['daemon_pooling_interval']
+          @WORKERS= {}
+          @STOPPED= []
+        end
+        
+        # The manager (e.g. daemon) main loop
+        def start
+          send_status( 'Info', "Daemon (PID #{Process.pid}) starting on host #{@HOST}" )
+          unless check_redis # Check the Redis interface version
+           err= "Telework: Error: Redis interface version mismatch, exiting"
+           puts err # We can't use send_status() as it relies on Redis so we just show a message
+           raise err
+          end
+          if is_alive(@HOST)  # Only one deamon can be run on a given host at the moment (this may change)
+            send_status( 'Error', "There is already a daemon running on #{@HOST}")
+            send_status( 'Error', "This daemon (PID #{Process.pid}) cannot be started and will terminare now")
+            exit
+          end
+          loop do                                # The main loop
+            while @RUN_DAEMON do                 # If there is no request to stop
+              i_am_alive(health_info)            # Notify the system that the daemon is alive
+              check_processes                    # Check the status of the child processes (to catch zombies)
+              while cmd= cmds_pop( @HOST ) do    # Pop a command in the command queue
+                do_command(cmd)                  # Execute it
+              end
+              sleep @SLEEP                       # Sleep
+            end
+                                                 # A stop request has been received
+            send_status( 'Info', "A stop request has been received and the #{@HOST} daemon will now terminate") if @WORKERS.empty?
+            break if @WORKERS.empty?
+            send_status( 'Error', "A stop request has been received by the #{@HOST} daemon but there are still running worker(s) so it will keep running") unless @WORKERS.empty?
+            @RUN_DAEMON= true
+          end
+        rescue Interrupt         # Control-C
+          send_status( 'Info', "Interruption for #{@HOST} daemon, exiting gracefully") if @WORKERS.empty?
+          send_status( 'Error', "Interruption for #{@HOST} daemon, exiting, running workers may now unexpectedly terminate") unless @WORKERS.empty?
+        rescue SystemExit        # Exit has been called
+          send_status( 'Info', "Exit called in #{@HOST} daemon") if @WORKERS.empty?
+          send_status( 'Error', "Exit called in #{@HOST} daemon but workers are still running") unless @WORKERS.empty?
+        rescue Exception => e    # Other exceptions
+          send_status( 'Error', "Exception #{e.message}")
+          puts "Backtrace: #{e.backtrace}"
+          send_status( 'Error', "Exception should not be raised in the #{@HOST} daemon, please submit a bug report")
+        end
+        
+        # Health info
+        def health_info
+          require "sys/cpu"
+          load= Sys::CPU.load_avg
+          { :cpu_load_1mins => load[0],
+            :cpu_load_5mins => load[1],
+            :cpu_load_15mins => load[2] }
+        rescue
+          {}
+        end
+        
+        # Add a status message on the status queue
+        def send_status( severity, message )
+          puts "Telework: #{severity}: #{message}"
+          info= { 'host'=> @HOST, 'severity' => severity, 'message'=> message,
+                  'date'=> Time.now }
+          status_push(info)
+        end
+        
+        # Execute a command synchronously
+        def do_command( cmd )
+          case cmd['command']
+          when 'start_worker'
+            start_worker( cmd, find_revision(cmd['revision']) )
+          when 'signal_worker'
+            manage_worker( cmd )
+          when 'stop_daemon'
+            @RUN_DAEMON= false
+          when 'kill_daemon'
+            send_status( 'Error', "A kill request has been received, the daemon on #{@HOST} is now brutally terminating by calling exit()")
+            exit # Bye
+          else
+            send_status( 'Error', "Unknown command '#{cmd['command']}'" )
+          end
+        end
+        
+        # Start a task
+        def start_worker( cmd, rev_info )
+          # Retrieving args
+          path= rev_info['revision_path']
+          log_path= rev_info['revision_log_path']
+          log_path||= "."
+          rev= rev_info['revision']
+          id= cmd['worker_id']
+          queuel= cmd['queue'].gsub(/,/, '_').gsub(/\*/, 'STAR')
+          # Starting the job
+          env= {}
+          env["QUEUE"]= cmd['queue']
+          # env["COUNT"]= cmd['worker_count'] if cmd['worker_count']
+          env["RAILS_ENV"]= cmd['rails_env'] if "(default)" != cmd['rails_env']
+          env["BUNDLE_GEMFILE"] = path+"/Gemfile" if ENV["BUNDLE_GEMFILE"]           # To make sure we use the new gems
+          opt= { :in => "/dev/null", 
+                 :out => "#{log_path}/telework_#{id}_#{queuel}_stdout.log", 
+                 :err => "#{log_path}/telework_#{id}_#{queuel}_stderr.log", 
+                 :chdir => path,
+                 :unsetenv_others => false }
+          exec= cmd['exec']
+          pid= spawn( env, exec, opt) # Start it!
+          info= { 'pid' => pid, 'status' => 'RUN', 'environment' => env, 'options' => opt, 'revision_info' => rev_info }
+          # Log snapshot
+          info['log_snapshot_period']= cmd['log_snapshot_period'] if cmd['log_snapshot_period']
+          info['log_snapshort_lines']= cmd['log_snapshot_lines'] if cmd['log_snapshot_lines']
+          @WORKERS[id]= info
+          workers_add( @HOST, id, info )
+          send_status( 'Info', "Starting worker #{id} (PID #{pid})" )
+          # Create an helper file
+          intro = "# Telework: starting worker #{id} on host #{@HOST} at #{Time.now.strftime("%a %b %e %R %Y")}"
+          env.keys.each { |v| intro+= "\n# Telework: environment variable '#{v}' set to '#{env[v]}'" }
+          intro+= "\n# Telework: command line is: #{exec}"
+          intro+= "\n# Telework: path is: #{path}"
+          intro+= "\n# Telework: log file for stdout is: #{opt[:out]}"
+          intro+= "\n# Telework: log file for stderr is: #{opt[:err]}"
+          intro+= "\n# Telework: PID is: #{pid}"
+          intro+= "\n"
+          File.open("#{log_path}/telework_#{id}.log", 'w') { |f| f.write(intro) }
+        end
+
+        def manage_worker ( cmd )
+          id= cmd['worker_id']
+          sig= cmd['action'] # Can be QUIT, KILL, CONT, PAUSE
+          info= @WORKERS[id]
+          send_status( 'Error', "Worker #{id} was not found on this host" ) unless info
+          return unless info
+          status= sig
+          sig= 'USR2' if 'PAUSE'==sig # Pause a Resque worker using USR2 signal
+          status= 'RUN' if status=='CONT'
+          send_status( 'Info', "Signaling worker #{id} (PID #{info['pid']}) using signal #{sig}" )
+          Process.kill( sig, info['pid'] ) # Signaling...
+          @STOPPED << id if 'QUIT'==sig || 'KILL'==sig
+          info['status']= status
+          workers_add( @HOST, id, info )
+          @WORKERS[id]= info
+        end
+                
+        def check_processes
+          #workers_delall( @HOST )
+          @WORKERS.keys.each do |id|
+            remove= false
+            unexpected_death= false
+            begin # Zombie hunt..
+              res= Process.waitpid(@WORKERS[id]['pid'], Process::WNOHANG)
+              remove= true if res 
+            rescue # Not a child.. so the process is already dead (we don't know why, maybe someone did a kill -9)
+              unexpected_death= true
+              remove= true
+            end
+            if remove
+              workers_rem( @HOST, id )
+              if unexpected_death
+                send_status( 'Error', "Worker #{id} (PID #{@WORKERS[id]['pid']}) has unexpectedly ended" )
+              else
+                send_status( 'Info', "Worker #{id} (PID #{@WORKERS[id]['pid']}) has exited" ) if @STOPPED.index(id)
+                send_status( 'Error', "Worker #{id} (PID #{@WORKERS[id]['pid']}) has unexpectedly exited" ) unless @STOPPED.index(id)
+                @STOPPED.delete(id)
+              end
+              @WORKERS.delete(id)
+            else
+              update_log_snapshot(id)
+              workers_add( @HOST, id, @WORKERS[id] )
+            end            
+          end
+        end
+        
+        def update_log_snapshot( id )
+          ls= @WORKERS[id]['log_snapshot_period']
+          return unless ls
+          last= @WORKERS[id]['last_log_snapshot']
+          last||= 0
+          now= Time.now.to_i
+          if now >= last+ls
+            size= @WORKERS[id]['log_snapshot_lines']
+            size||= 20
+            # Getting the logs
+            logerr= get_tail( @WORKERS[id]['options'][:err], size )
+            logout= get_tail( @WORKERS[id]['options'][:out], size )
+            # Write back
+            info= { :date => Time.now, :log_stderr => logerr, :log_stdout => logout }
+            logs_add( @HOST, id, info )
+            @WORKERS[id]['last_log_snapshot']= now
+          end 
+        end
+        
+        def get_tail( f, size )
+          `tail -n #{size} #{f}`
+        end
+      
+      end
+    end
+  end
+end

data/lib/resque-telework/railtie.rb

@@ -0,0 +1,11 @@
+require 'resque-telework'
+require 'rails'
+
+class Railtie < Rails::Railtie
+  railtie_name :telework
+  
+  rake_tasks do
+    load "tasks/telework.rake"
+  end
+end
+  

data/lib/resque-telework/redis.rb

@@ -0,0 +1,354 @@
+module Resque
+  module Plugins
+    module Telework      
+      module Redis
+      
+      def key_prefix
+        "plugins:#{Resque::Plugins::Telework::Nickname}"
+      end
+
+      def redis_interface_key # String
+        "#{key_prefix}:redisif"
+      end
+
+      def ids_key # String
+        "#{key_prefix}:ids"
+      end
+      
+      def hosts_key # Set
+        "#{key_prefix}:hosts"
+      end
+      
+      def revisions_key( h ) # List
+        "#{key_prefix}:host:#{h}:revisions"
+      end
+        
+      def workers_key( h ) # Hash
+        "#{key_prefix}:host:#{h}:workers"
+      end
+
+      def tasks_key( h ) # Hash
+        "#{key_prefix}:host:#{h}:tasks"
+      end
+      
+      def logs_key( h ) # Hash
+        "#{key_prefix}:host:#{h}:logs"        
+      end
+      
+      def cmds_key( h ) # List
+        "#{key_prefix}:host:#{h}:cmds"
+      end
+
+      def acks_key( h ) # List
+        "#{key_prefix}:host:#{h}:acks"
+      end
+      
+      def status_key # List
+        "#{key_prefix}:status"
+      end
+      
+      def alive_key( h ) # String, with TTL
+        "#{key_prefix}:host:#{h}:alive"
+      end
+      
+      def last_seen_key( h ) # String, no TTL
+        "#{key_prefix}:host:#{h}:last_seen"
+      end
+
+      def notes_key # List
+        "#{key_prefix}:notes"
+      end
+  
+      # Checks
+      def check_redis
+        res= true
+        v0= Resque::Plugins::Telework::REDIS_INTERFACE_VERSION
+        v= Resque.redis.get(redis_interface_key)
+        if v!=v0
+          Resque.redis.set(redis_interface_key, v0) unless v
+          res=false if v
+        end
+        res
+      end
+        
+      # Clients (hosts) side
+    
+      def i_am_alive( info= {}, ttl=10 )
+        h= @HOST
+        t= Time.now
+        info= info.merge( { 'date' => t, 'version' => Resque::Plugins::Telework::Version } )
+        k= alive_key(h)
+        hosts_add(h)
+        Resque.redis.set(k, info.to_json )
+        Resque.redis.expire(k, ttl)
+        Resque.redis.set(last_seen_key(h), t)
+      end
+      
+      def register_revision( h, rev, lim=9 )
+        k= revisions_key(h)
+        Resque.redis.ltrim(k, 0, lim-1)
+        rem= []
+        Resque.redis.lrange(k, 0, lim-1).each do |s|
+          info= ActiveSupport::JSON.decode(s)
+          if info['revision']==rev['revision']
+            rem << s
+            puts "Telework: Info: Revision #{rev['revision']} was already registered for this host, so the previous one will be unregistered" 
+          end
+          if info['revision_path']==rev['revision_path']
+            rem << s
+            puts "Telework: Info: Path for revision #{rev['revision']} was already registedred by another revision which will therefore by removed"
+          end
+        end
+        rem.each { |r| Resque.redis.lrem(k, 0, r) }
+        revisions_add( h, rev )
+      end
+      
+      def find_revision( rev )
+        revisions(@HOST).each do |r|
+          return r if rev==r['revision']
+        end
+        nil
+      end
+        
+      def hosts_add( h )
+        Resque.redis.sadd(hosts_key, h)
+      end
+      
+      def revisions_add( h, v )
+        hosts_add(h)
+        k= revisions_key(h)
+        Resque.redis.lpush(k, v.to_json )
+      end
+            
+      def workers_delall( h )
+        Resque.redis.del(workers_key(h))
+      end      
+            
+      def workers_add( h, id, info, ttl=10 )
+        k= workers_key(h)
+        Resque.redis.hset(k, id, info.to_json )
+        Resque.redis.expire(k, ttl)
+      end
+      
+      def workers_rem( h , id )
+        k= workers_key(h)
+        Resque.redis.hdel(k, id)
+      end
+
+      def tasks_add( h, id, info)
+        k= tasks_key(h)
+        Resque.redis.hset(k, id, info.to_json )
+      end
+      
+      def tasks_rem( h , id )
+        k= tasks_key(h)
+        Resque.redis.hdel(k, id)
+      end
+      
+      def cmds_pop( h )
+        info= Resque.redis.rpop(cmds_key(h))
+        info ? ActiveSupport::JSON.decode(info) : nil
+      end
+      
+      def logs_add( h, id, info )
+        k= logs_key(h)
+        Resque.redis.hset(k, id, info.to_json ) 
+      end
+      
+      def acks_push( h, info, lim=10 )
+        Resque.redis.lpush(acks_key(h), info)
+        Resque.redis.ltrim(acks_key(h), 0, lim-1)
+      end
+
+      def status_push( info, lim=100 )
+        Resque.redis.lpush(status_key, info.to_json )
+        Resque.redis.ltrim(status_key, 0, lim-1)
+      end
+      
+      # Server side
+      
+      def daemons_state( clean = 30000000 )
+        alive= []
+        dead= []
+        unknown= []
+        hosts.each do |h|
+          life= is_alive(h)
+          alive << [h, "Alive", life]  if life
+          unless life
+            ls= last_seen(h)
+            dead << [h, "Last seen #{fmt_date(ls, true)}", {} ] if ls
+            unknown << [h, 'Unknown', {} ] unless ls
+          end
+        end
+        alive+dead+unknown
+      end
+      
+      def configuration
+        c= {}
+        hosts.each do |h|
+          c[h]= tasks(h).map{ |id, info| info }
+        end
+        c.to_json
+      end
+      
+      # This function update the status of the tasks depending of what is found in workers
+      # This function must be idempotent
+      def reconcile
+        hosts.each do |h|
+          tasks(h).each do |id, info|
+            statuses= []
+            pids= []
+            tstatus= info['worker_status']              # Task status
+            info['worker_id'].each do |id|
+              worker= workers_by_id( h, id )
+              wstatus= worker ? worker['status'] : 'STOP' # Worker status
+              # wstatus: QUIT, KILL, CONT, PAUSE, RUN, STOP
+              # tstatus: Running, Starting, Stopped, Paused
+              ws= case wstatus
+              when "QUIT"
+                "Quitting"
+              when "KILL"
+                "Killing"
+              when "CONT"
+                "Resuming"
+              when "PAUSE"
+                "Paused"
+              when "RUN"
+                "Running"
+              when "STOP"
+                "Stopped"
+              else
+                "Unknown"
+              end
+              statuses << ws
+              pids << worker['pid'] if worker
+            end
+            ts= statuses.uniq * ","
+            if ts!=tstatus #&& (tstatus!="Starting" || wstatus!="STOP")
+              info['worker_status']= ts
+              info['worker_pid']= pids
+              tasks_add( h, id, info )
+            end
+          end
+        end
+      end
+      
+      def workers( h )
+         Resque.redis.hgetall(workers_key(h)).collect { |id, info| [id,  ActiveSupport::JSON.decode(info)] }
+      end
+      
+      def workers_by_id( h, id )
+        k= workers_key(h)
+        info= Resque.redis.hget(k, id)
+        info ? ActiveSupport::JSON.decode(info) : nil
+      end
+
+      def tasks( h )
+         Resque.redis.hgetall(tasks_key(h)).collect { |id, info| [id,  ActiveSupport::JSON.decode(info)] }
+      end
+      
+      def tasks_by_id( h, id )
+        k= tasks_key(h)
+        info= Resque.redis.hget(k, id)
+        info ? ActiveSupport::JSON.decode(info) : nil
+      end      
+      
+      def logs_by_id( h, id )
+        k= logs_key(h)
+        info= Resque.redis.hget(k, id)
+        info ? ActiveSupport::JSON.decode(info) : nil      
+      end
+      
+      def unique_id
+        Resque.redis.incr(ids_key)
+      end
+      
+      def cmds_push( h, info, ttl=300 )
+        k= cmds_key(h)
+        Resque.redis.lpush(k, info.to_json)
+        Resque.redis.expire(k, ttl)
+      end
+
+      def notes_push( info )
+        Resque.redis.lpush(notes_key, info.to_json)
+      end
+      
+      def notes_pop ( lim= 100 )
+        Resque.redis.lrange(notes_key, 0, lim-1).collect { |s| ActiveSupport::JSON.decode(s) }
+      end
+      
+      def notes_del( id )
+        info= Resque.redis.lindex(notes_key, id)
+        Resque.redis.lrem(notes_key, 0, info)
+      end
+
+      def acks_pop( h )
+        Resque.redis.rpop(acks_key(h))
+      end
+
+      def statuses( lim=100 )
+        Resque.redis.lrange(status_key, 0, lim-1).collect { |s| ActiveSupport::JSON.decode(s) }
+      end
+      
+      def hosts_rem( h )
+        [ revisions_key(h), workers_key(h),
+          cmds_key(h), alive_key(h), last_seen_key(h) ].each do |k|
+          Resque.redis.del(k)
+        end
+        Resque.redis.srem( hosts_key, h )
+      end
+         
+      def hosts
+        Resque.redis.smembers(hosts_key)
+      end
+
+      def revisions( h, lim=30 )
+        k= revisions_key(h)
+        Resque.redis.ltrim(k, 0, lim-1)
+        Resque.redis.lrange(k, 0, lim-1).map { |s| ActiveSupport::JSON.decode(s) }
+      end
+      
+      def is_alive( h )
+        v= Resque.redis.get(alive_key(h))
+        return nil unless v
+        begin
+          ActiveSupport::JSON.decode(v)
+        rescue
+          {}
+        end
+      end
+      
+      def last_seen( h )
+        Resque.redis.get(last_seen_key(h))
+      end
+      
+      def nb_keys
+        Resque.redis.keys("#{key_prefix}:*").length
+      end
+      
+      def fmt_date( t, rel=false ) # This is not redis-specific and should be moved to another class!
+        begin
+          if rel
+            "#{time_ago_in_words(Time.parse(t))} ago"
+          else
+            Time.parse(t).strftime("%a %b %e %R %Y")
+          end
+        rescue
+          "(unknown date)"
+        end
+      end
+      
+      def text_to_html(s)
+        return "" unless s
+        ss= s.gsub(/\n/, '<br>')
+      end
+        
+      end
+    end
+  end
+end
+
+class TeleworkRedis
+  include ActionView::Helpers::DateHelper
+  include Resque::Plugins::Telework::Redis
+end