fire_and_forget 0.1.2 → 0.2.0

data/Gemfile.lock

@@ -1,10 +1,3 @@
-PATH
-  remote: .
-  specs:
-    fire_and_forget (0.1.0)
-      daemons (~> 1.1.0)
-      json (~> 1.4.6)
-
 GEM
   remote: http://rubygems.org/
   specs:
@@ -26,7 +19,6 @@ PLATFORMS
 DEPENDENCIES
   bundler (~> 1.0.0)
   daemons (~> 1.1.0)
-  fire_and_forget!
   jeweler (~> 1.5.1)
   jnunemaker-matchy (~> 0.4)
   json (~> 1.4.6)

data/README.rdoc

@@ -14,23 +14,26 @@ What it doesn't:
 
 - Have any kind of queue mechanism. If you fire the same task twice then that task will be running twice
 - Have any persistant state
+- Work in non-UNIX environments
 
 == Quick Start
 
+  gem install fire_and_forget
 
-FAF works by running a simple TCP socket server so before we want to use it we have to start this server.
+FAF works by running a simple UNIX socket server so before we want to use it we
+have to start this server.
 
-  $ faf
-  FAF process 16235 listening on 127.0.0.1:3001...
+  $ fire_forget
+  FAF process 16235 listening on /tmp/fire_and_forget.sock ...
 
-The service defaults to listening to port 3001 on localhost. You can change these values thus:
+To change the path of the socket file use the '-s' parameter:
 
-  $ faf -a 192.168.1.22 -p 9090
+  $ fire_forget -s /path/to/socket
+  FAF process 16240 listening on /path/to/socket ...
 
-If you do this you will need to set both the web-app and the tasks to use the right values using
+If you do this you will need to set the web-app to use the right values:
 
-  FireAndForget.port = 9090
-  FireAndForget.bind_address = "192.168.1.22"
+  FireAndForget.socket = "/path/to/socket"
 
 Now inside our Ruby app:
 
@@ -39,13 +42,16 @@ Now inside our Ruby app:
 
   # First set register our task by giving it a name and a path to a script file
   FireAndForget.add_task(:long_running_task, "/path/to/script")
+  # => #<FireAndForget::TaskDescription @name=:long_running_task ...>
 
-  # when we want to call the script we simple call #fire passing the name of the task and the options
-  # we want to pass
+  # when we want to call the script we simple call #fire passing the name of the
+  # task and the options we want to pass
   FireAndForget.fire(:long_running_task, {:param3 => "value3"})
+  # => "OK"
 
   # Or, use the name as a method:
   FireAndForget.long_running_task({:param3 => "value3"})
+  # => "OK"
 
 This will result in the following command being exec'd in an independent process:
 
@@ -53,6 +59,38 @@ This will result in the following command being exec'd in an independent process
 
 It up to the script to parse and deal with the command line options.
 
+The options for the #add_task call are as follows:
+
+  task_name: A symbol giving the (unique) name for this task
+  path_to_binary: The path on disk of the executable to run when this task is launched
+  niceness: an integer in the range 0-20 giving the 'niceness' of the task process.
+  default_params: Command line parameters to pass to each invocation of this task
+  env: settings to add to this task's environment
+
+The higher the nice value, the lower the priority of the task. Setting a high 'nice'
+allows you to launch intensive background tasks without affecting the
+performance of your front line HTTP services.
+
+For example:
+
+  FireAndForget.add_task(:long_running_task, "/path/to/script", 12,
+    {:param1 => "value1", :param2 => "value2", :param3 => "value3"},
+    {"ENVIRONMENT_SETTING" => "Something"}
+  )
+
+this will add some default parameters to all invocations of the task, so now the call
+
+  FireAndForget.fire(:long_running_task, {:param3 => "newvalue3"})
+
+will issue the following command in the background
+
+  /path/to/script --param1="value1" --param2="value2" --param3="newvalue3"
+
+as well as setting it's nice value to 12 and updating ENV with the extra parameter "ENVIRONMENT_SETTING".
+
+Parameter values are JSON encoded so you can pass arrays or hashes. Again, it is
+up to the script to decode these values.
+
 Interprocess communication is relatively easy. Take the following as the source of the script "/path/to/script"
 
   #!/usr/bin/env ruby
@@ -60,9 +98,9 @@ Interprocess communication is relatively easy. Take the following as the source
   require 'rubygems'
   require 'fire_and_forget'
 
-  # this will mix in the comms methods and map this task to the :long_running_task label
-  # used in the calling script
-  include FireAndForget::Daemon[:long_running_task]
+  # this will mix in the F&F status methods
+  # the F&F configuration (socket and task-name) is calculated automatically from ENV parameters
+  include FireAndForget::Daemon
 
   30.times do |i|
     # update our status. What you put here is up to you, but it should be a String
@@ -83,10 +121,33 @@ If we decided we've had enough then we can kill it:
   # Or send any signal (see the Process.kill documentation)
   FireAndForget.kill(:long_running_task, "HUP")
 
+== Supporting multiple users
+
+Tasks will attempt to run as the same user as the originating process. To enable
+this, and hence enable a single F&F server to launch tasks for any number of
+separate web-apps, simply run the F&F server as root. See the following section on
+Security for the implications of this and how to stop unlimited access to the service.
+
 == Security
 
-F&F is intended to be run in a relatively trusted environment and by default only binds to localhost to stop external access. But to stop unintended access to system commands, only scripts belonging to the same user as the F&F server process are executed. This might not be enough so in the future I might add a limit on the locations of runnable scripts.
+F&F is intended to be run in a relatively trusted environment. You the UNIX
+socket file has its permissions set so that only the process owner and a defined
+group can write to the file. To change the UNIX group that has access pass the
+group name as a parameter to the server:
+
+  $ fire_forget -s /path/to/socket -g webapp_group
+
+Only members of the 'webapp_group' will be able to launch processes through
+the server. Additionally, only scripts belonging to the same user as the F&F task description
+are executed.
+
+In the meantime, as I say, this is defined to be run in a trusted environment, for instance on a
+single non-shared server (potentially running many sites).
+
+== TODO
 
+- Improve security restrictions (perhaps by limiting the tasks that each user can launch)
+- Improve the tests so they are closer to testing the full stack
 
 == Contributing to fire_and_forget
 

data/bin/fire_forget

@@ -6,49 +6,72 @@ Dir.chdir(File.join(File.dirname(__FILE__), '..'))
 faf_lib_path = File.expand_path(File.dirname(__FILE__) + "/../lib")
 $:.unshift(faf_lib_path) unless $:.include?(faf_lib_path)
 
-require 'rubygems' unless defined?(Gem)
+# include dependencies without using rubygems (to save memory)
+$:.unshift('vendor/json-1.5.0/lib')
+$:.unshift('vendor/daemons-1.1.0/lib')
+
+require 'vendor/json-1.5.0/lib/json/pure'
+require 'vendor/daemons-1.1.0/lib/daemons'
+
 
 require 'ostruct'
 require 'optparse'
 require 'socket'
+require 'etc'
 require 'fire_and_forget'
 
 options = OpenStruct.new
-options.bind_address = "127.0.0.1"
-options.port = FAF::DEFAULT_PORT
+options.socket = FireAndForget::DEFAULT_SOCKET
+options.gid = Process.egid
 
 OptionParser.new do |opts|
-  opts.on("-a", "--bind-address ADDRESS", "Bind Address") { |v| options.bind_address = v }
-  opts.on("-p", "--port PORT", Integer, "Port") { |p| options.port = p }
+  opts.on("-s", "--socket SOCKET", "Socket") { |v| options.socket = v }
+  opts.on("-g", "--socket-group GROUPNAME", "Socket owning group") { |v| options.gid = nil; options.group_name = v }
 end.parse!
 
+raise ArgumentError, "You must specify a socket file using -s" unless options.socket
+unless options.gid
+  if options.group_name
+    options.gid = Etc.getgrnam(options.group_name).gid
+  end
+end
+
 server = nil
 
 begin
-  server = TCPServer.new(options.bind_address, options.port)
+  raise Errno::EADDRINUSE if File.exists?(options.socket) and File.socket?(options.socket)
+  server = UNIXServer.new(options.socket)
+  File.chown(nil, options.gid, options.socket)
+  # make the socket group writable
+  File.chmod(0770, options.socket)
 rescue Errno::EADDRINUSE
-  puts "FAF unable to bind to #{options.bind_address}:#{options.port}"
+  puts "FAF unable to create socket #{options.socket}: file exists"
   exit(1)
 end
 
 run = true
 
 server_thread = Thread.new do
-  while run and (session = server.accept)
-    request = response = ""
-
-    while l = session.gets
-      request << l
-    end
-    session.close_read
-
-    begin
-      response = FAF::Server.parse(request)
-    rescue  => e
-      response = "ERROR #{e}"
+  begin
+    while run and (session = server.accept)
+      request = response = ""
+
+      while l = session.gets
+        request << l
+      end
+      session.close_read
+
+      begin
+        response = FAF::Server.parse(request)
+      rescue  => e
+        response = "ERROR #{e}"
+      end
+      session.write(response)
+      session.close
     end
-    session.write(response)
-    session.close
+  ensure
+    server.close
+    File.unlink(options.socket) if File.exists?(options.socket)
   end
 end
 
@@ -59,7 +82,7 @@ end
   end
 end
 
-puts "Fire&Forget process #{$$} listening on #{options.bind_address}:#{options.port}..."
+puts "Fire&Forget process #{$$} listening on #{options.socket} ..."
 
 server_thread.join
 

data/examples/long_task

@@ -2,33 +2,44 @@
 
 Dir.chdir(File.join(File.dirname(__FILE__), '..'))
 
-$:.unshift(File.dirname(__FILE__) + "/../lib")
+File.open(File.join(File.dirname(__FILE__), "../../long.out"), 'w+') do |file|
+  file.sync = true
+  STDOUT.reopen(file)
+  STDERR.reopen(file)
 
-require 'rubygems'
-require 'bundler'
+  $:.unshift(File.dirname(__FILE__) + "/../lib")
 
-begin
-  Bundler.setup(:default)
-rescue Bundler::BundlerError => e
-  $stderr.puts e.message
-  $stderr.puts "Run `bundle install` to install missing gems"
-  exit e.status_code
-end
+  require 'etc'
+  require 'rubygems'
+  require 'bundler'
 
-require 'fire_and_forget'
+  begin
+    Bundler.setup(:default)
+  rescue Bundler::BundlerError => e
+    $stderr.puts e.message
+    $stderr.puts "Run `bundle install` to install missing gems"
+    exit e.status_code
+  end
 
-include FAF::Daemon[:publish]
+  require 'fire_and_forget'
 
+  include FAF::Daemon
 
-File.open(File.join(File.dirname(__FILE__), "../../long.out"), 'w') do |file|
-  file.sync = true
+  file.write(("="*40)+"\n")
   file.write("PID: #{$$}\n")
+  file.write("User: #{Etc.getlogin}\n")
+  file.write("UID: #{Process.uid}\n")
   file.write(`ps -xO nice | grep  '^#{$$}'`)
   file.write("\n")
-  60.times do |i|
+  ENV.keys.sort.each do | k |
+    file.write(" #{k}: #{ENV[k].inspect}\n")
+  end
+  file.write("\n")
+
+  30.times do |i|
     file.write("#{i}\n")
     begin
-      set_status(i)
+      set_task_status(i)
     rescue Exception => e
       file.write(e.to_s + "\n")
     end

data/fire_and_forget.gemspec

@@ -5,11 +5,11 @@
 
 Gem::Specification.new do |s|
   s.name = %q{fire_and_forget}
-  s.version = "0.1.2"
+  s.version = "0.2.0"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Garry Hill"]
-  s.date = %q{2010-12-15}
+  s.date = %q{2011-01-19}
   s.default_executable = %q{fire_forget}
   s.email = %q{garry@magnetised.info}
   s.executables = ["fire_forget"]
@@ -31,19 +31,56 @@ Gem::Specification.new do |s|
     "lib/fire_and_forget/client.rb",
     "lib/fire_and_forget/command.rb",
     "lib/fire_and_forget/command/fire.rb",
+    "lib/fire_and_forget/command/get_pid.rb",
     "lib/fire_and_forget/command/get_status.rb",
     "lib/fire_and_forget/command/kill.rb",
     "lib/fire_and_forget/command/set_pid.rb",
     "lib/fire_and_forget/command/set_status.rb",
     "lib/fire_and_forget/config.rb",
     "lib/fire_and_forget/daemon.rb",
+    "lib/fire_and_forget/errors.rb",
     "lib/fire_and_forget/launcher.rb",
     "lib/fire_and_forget/server.rb",
-    "lib/fire_and_forget/task.rb",
+    "lib/fire_and_forget/task_description.rb",
     "lib/fire_and_forget/utilities.rb",
     "lib/fire_and_forget/version.rb",
     "test/helper.rb",
-    "test/test_fire_and_forget.rb"
+    "test/test_fire_and_forget.rb",
+    "vendor/daemons-1.1.0/LICENSE",
+    "vendor/daemons-1.1.0/README",
+    "vendor/daemons-1.1.0/Rakefile",
+    "vendor/daemons-1.1.0/Releases",
+    "vendor/daemons-1.1.0/TODO",
+    "vendor/daemons-1.1.0/lib/daemons.rb",
+    "vendor/daemons-1.1.0/lib/daemons/application.rb",
+    "vendor/daemons-1.1.0/lib/daemons/application_group.rb",
+    "vendor/daemons-1.1.0/lib/daemons/change_privilege.rb",
+    "vendor/daemons-1.1.0/lib/daemons/cmdline.rb",
+    "vendor/daemons-1.1.0/lib/daemons/controller.rb",
+    "vendor/daemons-1.1.0/lib/daemons/daemonize.rb",
+    "vendor/daemons-1.1.0/lib/daemons/etc_extension.rb",
+    "vendor/daemons-1.1.0/lib/daemons/exceptions.rb",
+    "vendor/daemons-1.1.0/lib/daemons/monitor.rb",
+    "vendor/daemons-1.1.0/lib/daemons/pid.rb",
+    "vendor/daemons-1.1.0/lib/daemons/pidfile.rb",
+    "vendor/daemons-1.1.0/lib/daemons/pidmem.rb",
+    "vendor/daemons-1.1.0/setup.rb",
+    "vendor/json-1.5.0/COPYING",
+    "vendor/json-1.5.0/GPL",
+    "vendor/json-1.5.0/README",
+    "vendor/json-1.5.0/README-json-jruby.markdown",
+    "vendor/json-1.5.0/Rakefile",
+    "vendor/json-1.5.0/TODO",
+    "vendor/json-1.5.0/VERSION",
+    "vendor/json-1.5.0/lib/json.rb",
+    "vendor/json-1.5.0/lib/json/add/core.rb",
+    "vendor/json-1.5.0/lib/json/add/rails.rb",
+    "vendor/json-1.5.0/lib/json/common.rb",
+    "vendor/json-1.5.0/lib/json/editor.rb",
+    "vendor/json-1.5.0/lib/json/pure.rb",
+    "vendor/json-1.5.0/lib/json/pure/generator.rb",
+    "vendor/json-1.5.0/lib/json/pure/parser.rb",
+    "vendor/json-1.5.0/lib/json/version.rb"
   ]
   s.homepage = %q{http://github.com/magnetised/fire_and_forget}
   s.licenses = ["MIT"]

data/lib/fire_and_forget/client.rb

@@ -13,7 +13,7 @@ module FireAndForget
       def open_connection
         connection = result = nil
         begin
-          connection = TCPSocket.open(FireAndForget.bind_address, FireAndForget.port)
+          connection = UNIXSocket.open(FireAndForget.socket)
           yield(connection)
           connection.flush
           connection.close_write

data/lib/fire_and_forget/command/fire.rb

@@ -4,7 +4,13 @@ require 'daemons'
 module FireAndForget
   module Command
     class Fire < CommandBase
-      attr_reader :niceness
+      attr_reader :niceness, :task_uid, :task_gid
+
+      def initialize(task, params={})
+        super
+        @task_uid = Process.euid
+        @task_gid = Process.egid
+      end
 
       def niceness
         @task.niceness
@@ -23,24 +29,37 @@ module FireAndForget
       end
 
       def permitted?
-        raise Errno::EACCES.new("'#{binary}' does not belong to user '#{ENV["USER"]}'") unless File.owned?(binary)
+        raise PermissionsError, "'#{binary}' does not belong to user '#{ENV["USER"]}'" unless File.stat(binary).uid == task_uid
         true
       end
 
       def exists?
-        raise Errno::ENOENT.new("'#{binary}'") unless File.exists?(binary)
+        raise FileNotFoundError, "'#{binary}'" unless File.exists?(binary)
         true
       end
 
+      def env
+        @task.env.merge({
+          FireAndForget::ENV_SOCKET => FireAndForget.socket,
+          FireAndForget::ENV_TASK_NAME => @task.name.to_s
+        })
+      end
+
       def run
         if valid?
           pid = fork do
+            # set up the environment so that the task can access the F&F server
+            env.each { | k, v | ENV[k] = v }
             Daemons.daemonize(:backtrace => true)
             Process.setpriority(Process::PRIO_PROCESS, 0, niceness) if niceness > 0
+            # change to the UID of the originating thread if necessary
+            Process::UID.change_privilege(task_uid) unless Process.euid == task_uid
             exec(cmd)
           end
           Process.detach(pid) if pid
-          pid
+          # don't return the PID as it's actually wrong (the daemonize call forks again so our original
+          # PID is at least 1 out)
+          "OK"
         end
       end
     end

data/lib/fire_and_forget/command/get_pid.rb

@@ -0,0 +1,20 @@
+module FireAndForget
+  module Command
+    class GetPid < CommandBase
+
+      attr_reader :task_name, :pid
+
+      def initialize(task_name)
+        @task_name = task_name.to_sym
+      end
+
+      def run
+        FireAndForget::Server.pids[@task_name]
+      end
+    end
+  end
+end
+
+
+
+

data/lib/fire_and_forget/command/set_pid.rb

@@ -15,5 +15,3 @@ module FireAndForget
   end
 end
 
-
-

data/lib/fire_and_forget/command/set_status.rb

@@ -9,7 +9,7 @@ module FireAndForget
 
       def run
         FireAndForget::Server.set_pid(@task_name, @pid)
-        FireAndForget::Server.status[@task_name] = @status_value
+        FireAndForget::Server.status[@task_name] = @status_value.to_s
       end
     end
   end

data/lib/fire_and_forget/command.rb

@@ -7,6 +7,16 @@ module FireAndForget
       Marshal.load(command)
     end
 
+    def self.allowed?(cmd)
+      allowed_commands.include?(cmd.class)
+    end
+
+    def self.allowed_commands
+      @allowed_commands ||= self.constants.map { |c| self.const_get(c) }.select do |k|
+        k.respond_to?(:ancestors) && k.ancestors.include?(CommandBase)
+      end
+    end
+
     class CommandBase
       attr_reader :tag, :cmd, :params, :task
 
@@ -39,5 +49,6 @@ module FireAndForget
     autoload :SetStatus, "fire_and_forget/command/set_status"
     autoload :GetStatus, "fire_and_forget/command/get_status"
     autoload :SetPid, "fire_and_forget/command/set_pid"
+    autoload :GetPid, "fire_and_forget/command/get_pid"
   end
 end

data/lib/fire_and_forget/config.rb

@@ -1,14 +1,9 @@
 module FireAndForget
   module Config
-    attr_accessor :port
-    attr_accessor :bind_address
+    attr_accessor :socket
 
-    def port
-      @port ||= FireAndForget::DEFAULT_PORT
-    end
-
-    def bind_address
-      @bind_address ||= "127.0.0.1"
+    def socket
+      @socket ||= (ENV[FireAndForget::ENV_SOCKET] || FireAndForget::DEFAULT_SOCKET)
     end
   end
 end

data/lib/fire_and_forget/daemon.rb

@@ -1,31 +1,22 @@
 module FireAndForget
-  module Daemon # need better name!
+  module Daemon
 
-    def self.[](task_name)
-      m = Module.new do
-        def self.included(klass)
-          FireAndForget.map_pid(self.task_name, $$)
-        rescue Errno::ECONNREFUSED
-          # server isn't running but we don't want this to stop our script
-        end
+    def self.task_name
+      ENV[FireAndForget::ENV_TASK_NAME]
+    end
 
-        def self.task_name=(task_name)
-          @@task_name = task_name
-        end
+    def self.included(klass)
+      FireAndForget.set_pid(self.task_name, $$)
+    rescue Errno::ECONNREFUSED
+      # server isn't running but we don't want this to stop our script
+    end
 
-        def self.task_name
-          @@task_name
-        end
 
-        def set_task_status(status)
-          FireAndForget.set_status(@@task_name, status)
-        rescue Errno::ECONNREFUSED
-          # server isn't running but we don't want this to stop our script
-        end
-      end
-      m.task_name = task_name
-      m
+    def set_task_status(status)
+      FireAndForget.set_status(FireAndForget::Daemon.task_name, status)
+    rescue Errno::ECONNREFUSED
+      # server isn't running but we don't want this to stop our script
     end
-
   end
 end
+

data/lib/fire_and_forget/errors.rb

@@ -0,0 +1,8 @@
+
+
+module FireAndForget
+  class Error < ::StandardError; end
+  class PermissionsError < Error; end
+  class FileNotFoundError < Error; end
+end
+