murder 0.0.0.pre → 0.1.0

data/README

@@ -1,6 +1,7 @@
-Murder by Larry Gadea and Matt Freels
+Murder by Larry Gadea <lg@twitter.com> and Matt Freels <freels@twitter.com>
 Copyright 2010 Twitter Inc.
 
+
 DESCRIPTION
 -----------
 
@@ -11,6 +12,23 @@ centralized distribution systems wouldn't otherwise function. A "Murder" is
 normally used to refer to a flock of crows, which in this case applies to a
 bunch of servers doing something.
 
+
+QUICK START
+-----------
+
+For the impatient, `gem install murder` and add these lines to your Capfile:
+
+    require 'murder'
+
+    set :deploy_via, :murder
+    after 'deploy:setup', 'murder:distribute_files'
+    before 'murder:start_seeding', 'murder:start_tracker'
+    after 'murder:stop_seeding', 'murder:stop_tracker'
+
+
+HOW IT WORKS
+------------
+
 In order to do a Murder transfer, there are several components required to be
 set up beforehand -- many the result of BitTorrent nature of the system. Murder
 is based on BitTornado.
@@ -23,68 +41,99 @@ acceptable. To keep things simple tracker-less distribution (DHT) is currently
 not supported. The tracker is actually just a mini-httpd that hosts a
 /announce path which the Bittorrent clients update their state onto.
 
-- A seeder. This is the server which has the files that you'd like to deploy
-onto all other servers. For Twitter, this is the server that did the git diff.
-The files are placed into a directory that a torrent gets created from. Murder
-will tgz up the directory and create a .torrent file (a very small file
-containing basic hash information about the tgz file). This .torrent file lets
-the peers know what they're downloading. The tracker keeps track of which
-.torrent files are currently being distributed. Once a Murder transfer is
-started, the seeder will be the first server many machines go to to get
-pieces. These pieces will then be distributed in a tree-fashion to the rest of
-the network, but without necessarily getting the parts from the seeder.
+- A seeder. This is the server which has the files that you'd like to
+deploy onto all other servers. The files are placed into a directory
+that a torrent gets created from. Murder will tgz up the directory and
+create a .torrent file (a very small file containing basic hash
+information about the tgz file). This .torrent file lets the peers
+know what they're downloading. The tracker keeps track of which
+.torrent files are currently being distributed. Once a Murder transfer
+is started, the seeder will be the first server many machines go to to
+get pieces. These pieces will then be distributed in a tree-fashion to
+the rest of the network, but without necessarily getting the parts
+from the seeder.
 
 - Peers. This is the group of servers (hundreds to tens of thousands) which
 will be receiving the files and distributing the pieces amongst themselves.
 Once a peer is done downloading the entire tgz file, it will continue seeding
 for a while to prevent a hotspot effect on the seeder.
 
-MURDER TRANSFER PROCESS
+
+CONFIGURATION AND USAGE
 -----------------------
 
-1. Configure the list of servers and general settings in config.rb. (one time)
-2. Distribute the Murder files to all your servers: (one time)
+Murder integrates with Capistrano. The most simple way to use it is as
+a deploy strategy, by setting `:deploy_via` to `:murder`. By default,
+murder makes the same assumptions that cap makes. All servers without
+`:no_release => true` will act as peers. Additionally, murder will
+automatically use the first peer as both tracker and seeder. you may
+redefine the `tracker`, `seeder` and `peer` roles yourself to change
+these defaults, for instance, if you want to set up a dedicated
+tracker.
+
+All involved servers must have python installed and the related murder
+support files (BitTornado, etc.). To upload the support files to the
+tracker, seeder, and peers, run:
+
     cap murder:distribute_files
-3. Start the tracker: (one time)
+
+By default, these will go in `shared/murder` in your apps deploy
+directory. Override this by setting the variable
+`remote_murder_path`. For convenience, you can add an after hook to
+run this on `deploy:setup`:
+
+    after 'deploy:setup', 'murder:distribute_files'
+
+Before deploying, you must start the tracker:
+
     cap murder:start_tracker
-4. Create a torrent file from a remote directory of files (on seeder):
-    cap murder:create_torrent tag="Deploy20100101" files_path="~/files"
-5. Start seeding the files:
-    cap murder:start_seeding tag="Deploy20100101"
-6. Distribute the files to all servers:
-    cap murder:peer tag="Deploy20100101" destination_path="/tmp/out"
 
-Once completed, all files will be in /tmp/out/Deploy20091015/ on all servers.
+To have this happen automatically during a deploy, add the following hooks:
 
-EXAMPLE DEPLOY
---------------
+    before 'murder:start_seeding', 'murder:start_tracker'
+    after 'murder:stop_seeding', 'murder:stop_tracker'
+
+At this point you should be able to deploy normally:
+
+    cap deploy
+
+
+MANUAL USAGE (murder without a deploy strategy)
+-----------------------------------------------
+
+Murder can also be used as a general mechanism to distribute files
+across a generic set of servers. To do so create a Capfile, require
+murder, and manually define roles:
+
+    require 'rubygems'
+    require 'murder'
+
+    set :remote_murder_path, '/opt/local/murder' # or some other directory
+
+    role :peer, 'host1', 'host2', 'host3', 'host4', 'host5', host6', host7'
+    role :seeder, 'host1'
+    role :tracker, 'host1'
+
+To distribute a directory of files, first make sure that murder is set
+up on all hosts, then manually run the murder cap tasks:
+
+1. Start the tracker:
+    cap murder:start_tracker
+
+2. Create a torrent from a directory of files on the seeder, and start seeding:
+    scp -r ./files host1:~/files
+    cap murder:create_torrent tag="Deploy1" files_path="~/files"
+    cap murder:start_seeding tag="Deploy1"
+
+3. Distribute the torrent to all peers:
+    cap murder:peer tag="Deploy1" destination_path="/tmp"
+
+4. Stop the seeder and tracker:
+    cap murder:stop_seeding
+    cap murder:stop_tracker
+
+When this finishes, all peers will have the files in /tmp/Deploy1
 
-cap murder:distribute_files
-cap murder:start_tracker
-cap murder:create_torrent tag="Deploy20100101"
-  files_path="/usr/local/twitter/production/current"
-cap murder:start_seeding tag="Deploy20100101"
-time cap murder:peer tag="Deploy20100101"
-  destination_path="/usr/local/twitter/releases"
-
-All the files have been transferred to all servers at this point. To clean up
-use:
-
-cap murder:stop_seeding
-cap murder:stop_tracker
-
-HOW TO LEARN MURDER INCREMENTALLY
----------------------------------
-
-- Skim dist/*.py to see what Murder does behind the scenes
-- Experiment with the python scripts
-- Read Capfile
-- Read murder_config.rb
-- Read murder_admin.rb
-- Read murder.rb
-- Read the reference below
-- Experiment with the Capistrano deploy methods
-- Read and experiment with the Murder Capistrano strategy
 
 TASK REFERENCE
 --------------
@@ -151,15 +200,11 @@ stop_all_peering:
 command will forcibly kill all "murder_client.py peer" commands that are
 running.
 
-CONFIG REFERENCE:
-
-user:
-  Provided by Capistrano to specify if to use a different username when sshing
-into machines.
+CONFIG REFERENCE
+----------------
 
-host_suffix:
-  For the tracker_host, seeder_host and peers servers, this suffix will always
-be added to the end when trying to connect to them.
+Variables
+---------
 
 default_tag:
   A tag name to use by default such that a tag parameter doesn't need to be
@@ -174,51 +219,19 @@ default_destination_path:
   A path on the peers' file system where the files that were distributed
 should be decompressed into.
 
-tracker_host:
-  The hostname of a single tracker server. If you're using a host_suffix, do
-not specify a suffix here.
+remote_murder_path:
+ A path where murder will look for its support files on each host. `cap
+murder:distribute_files` will upload murder support files here.
+
+
+Roles
+-----
 
-tracker_port:
-  This is the port on which the mini-web-server tracker that the Bittorrent
-libraries run is hosted on. Must be 8998 for now.
+tracker:
+  Host on which to run the BitTorrent tracker
 
-seeder_host:
-  The hostname of a single seeder server. This server is the one that has all
-the files that the peers want. If you're using a host_suffix, do not specify a
-suffix here.
+seeder:
+  Host which will be the source of the files to be distributed via BitTorrent
 
 peers:
-  A list of peers which will be receiving the files and distributing them
-amongst themselves. If you're using a host_suffix, do not specify a suffix
-here.
-
-CAPISTRANO COPY STRATEGY
-------------------------
-
-In addition to being usable from both the commandline and Capistrano commands,
-an optional Capistrano copy strategy is included to help easily retrofit
-existing deploy environments you might already have.
-
-Requirements to use build.rb
-  - Add a require line for build.rb in your Capfile. This will override the
-    default build strategy. There are some other parameters you must now
-    specify though.
-  - To enable it, add the following to your Capfile:
-    load 'deploy'
-    set :strategy, Capistrano::Deploy::Strategy::Build.new(self)
-  - Add "set :murder, true" to your Capfile
-  - Make sure your release_name is correct (should be already)
-  - If you're distribution is a file instead of a directory,
-    Add "distribution_is_a_file" to your Capfile
-  - Add the following to your Capfile:
-    set :build_task do
-      "true"
-    end
-    set :copy_compression, :gz
-    set :package_name do
-      "#{release_name}-#{real_revision[0, 8]}.tar.#{copy_compression}"
-    end
-    set :branch, variables[:branch] || 'deploy'
-
-The requirements are a bit hefty for now, though hopefully this will get easier
-in the future.
+  All hosts to which files should be distributed

data/Rakefile

@@ -9,7 +9,7 @@ begin
     gem.description = %Q{Large scale server deploys using BitTorrent and the BitTornado library}
     gem.email = "lg@twitter.com"
     gem.homepage = "http://github.com/lg/murder"
-    gem.authors = ["Larry Gadea"]
+    gem.authors = ["Larry Gadea", "Matt Freels"]
     # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
   end
   Jeweler::GemcutterTasks.new

data/VERSION

@@ -1 +1 @@
-0.0.0.pre
+0.1.0

data/dist/murder_client.py

@@ -1,5 +1,6 @@
 # Copyright 2010 Twitter, Inc.
 # Copyright 2010 Larry Gadea <lg@twitter.com>
+# Copyright 2010 Matt Freels <freels@twitter.com>
 #
 # Licensed under the Apache License, Version 2.0 (the "License"); you may
 # not use this file except in compliance with the License. You may obtain
@@ -27,7 +28,7 @@ if PSYCO.psyco:
         psyco.full()
     except:
         pass
-    
+
 from BitTornado.download_bt1 import BT1Download, defaults, parse_params, get_usage, get_response
 from BitTornado.RawServer import RawServer, UPnP_ERROR
 from random import seed
@@ -91,17 +92,17 @@ class HeadlessDisplayer:
 
     def finished(self):
         global doneFlag
-      
+
         self.done = True
         self.percentDone = '100'
         self.timeEst = 'Download Succeeded!'
         self.downRate = ''
         #self.display()
-        
+
         global isPeer
-        
+
         print "done and done"
-        
+
         if isPeer:
           if os.fork():
             os._exit(0)
@@ -118,7 +119,7 @@ class HeadlessDisplayer:
 
           t = threading.Timer(30.0, ok_close_now)
           t.start()
-        
+
     def failed(self):
         self.done = True
         self.percentDone = '0'
@@ -135,12 +136,12 @@ class HeadlessDisplayer:
         print errormsg
         doneFlag.set()
 
-    def display(self, dpflag = Event(), fractionDone = None, timeEst = None, 
+    def display(self, dpflag = Event(), fractionDone = None, timeEst = None,
             downRate = None, upRate = None, activity = None,
             statistics = None,  **kws):
         if self.last_update_time + 0.1 > clock() and fractionDone not in (0.0, 1.0) and activity is not None:
             return
-        self.last_update_time = clock()        
+        self.last_update_time = clock()
         if fractionDone is not None:
             self.percentDone = str(float(int(fractionDone * 1000)) / 10)
         if timeEst is not None:
@@ -174,7 +175,7 @@ class HeadlessDisplayer:
         #print 'seed status:   ', self.seedStatus
         #print 'peer status:   ', self.peerStatus
         #stdout.flush()
-        dpflag.set()        
+        dpflag.set()
 
     def chooseFile(self, default, size, saveas, dir):
         self.file = '%s (%.1f MB)' % (default, float(size) / (1 << 20))
@@ -212,7 +213,7 @@ def run(params):
 
         myid = createPeerID()
         seed(myid)
-        
+
         global doneFlag
         doneFlag = Event()
         def disp_exception(text):
@@ -245,7 +246,7 @@ def run(params):
         dow = BT1Download(h.display, h.finished, h.error, disp_exception, doneFlag,
                         config, response, infohash, myid, rawserver, listen_port,
                         configdir)
-        
+
         if not dow.saveAs(h.chooseFile, h.newpath):
             break
 
@@ -283,9 +284,9 @@ if __name__ == '__main__':
     sys.exit(1)
 
   argv = ["--responsefile", sys.argv[2],
-          "--saveas", sys.argv[3], 
+          "--saveas", sys.argv[3],
           "--ip", sys.argv[4]]
-  
+
   isPeer = sys.argv[1] == "peer"
-  
+
   run(argv[1:])

data/dist/murder_make_torrent.py

@@ -1,5 +1,6 @@
 # Copyright 2010 Twitter, Inc.
 # Copyright 2010 Larry Gadea <lg@twitter.com>
+# Copyright 2010 Matt Freels <freels@twitter.com>
 #
 # Licensed under the Apache License, Version 2.0 (the "License"); you may
 # not use this file except in compliance with the License. You may obtain

data/dist/murder_tracker.py

@@ -1,5 +1,6 @@
 # Copyright 2010 Twitter, Inc.
 # Copyright 2010 Larry Gadea <lg@twitter.com>
+# Copyright 2010 Matt Freels <freels@twitter.com>
 #
 # Licensed under the Apache License, Version 2.0 (the "License"); you may
 # not use this file except in compliance with the License. You may obtain

data/lib/capistrano/recipes/deploy/strategy/murder.rb

@@ -1,5 +1,6 @@
 # Copyright 2010 Twitter, Inc.
 # Copyright 2010 Larry Gadea <lg@twitter.com>
+# Copyright 2010 Matt Freels <freels@twitter.com>
 #
 # Licensed under the Apache License, Version 2.0 (the "License"); you may
 # not use this file except in compliance with the License. You may obtain

data/lib/murder.rb

@@ -1,5 +1,6 @@
 # Copyright 2010 Twitter, Inc.
 # Copyright 2010 Larry Gadea <lg@twitter.com>
+# Copyright 2010 Matt Freels <freels@twitter.com>
 #
 # Licensed under the Apache License, Version 2.0 (the "License"); you may
 # not use this file except in compliance with the License. You may obtain

data/lib/murder/admin.rb

@@ -1,5 +1,6 @@
 # Copyright 2010 Twitter, Inc.
 # Copyright 2010 Larry Gadea <lg@twitter.com>
+# Copyright 2010 Matt Freels <freels@twitter.com>
 #
 # Licensed under the Apache License, Version 2.0 (the "License"); you may
 # not use this file except in compliance with the License. You may obtain
@@ -14,6 +15,9 @@
 # limitations under the License.
 
 namespace :murder do
+  desc <<-DESC
+  SCPs a compressed version of all files from ./dist (the python Bittorrent library and custom scripts) to all server. The entire directory is sent, regardless of the role of each individual server. The path on the server is specified by remote_murder_path and will be cleared prior to transferring files over.
+  DESC
   task :distribute_files, :roles => [:tracker, :seeder, :peer] do
     dist_path = File.expand_path('../../dist', __FILE__)
 
@@ -29,18 +33,22 @@ namespace :murder do
     system "rm /tmp/murder_dist.tgz"
   end
 
+  desc "Starts the Bittorrent tracker (essentially a mini-web-server) listening on port 8998."
   task :start_tracker, :roles => :tracker do
     run("screen -dms murder_tracker python #{remote_murder_path}/murder_tracker.py && sleep 0.2", :pty => true)
   end
 
+  desc "If the Bittorrent tracker is running, this will kill the process. Note that if it is not running you will receive an error."
   task :stop_tracker, :roles => :tracker do
     run("pkill -f 'SCREEN.*murder_tracker.py'")
   end
 
+  desc "Identical to stop_seeding, except this will kill all seeding processes. No 'tag' argument is needed."
   task :stop_all_seeding, :roles => :seeder do
     run("pkill -f \"SCREEN.*seeder-\"")
   end
 
+  desc 'Sometimes peers can go on forever (usually because of an error). This command will forcibly kill all "murder_client.py peer" commands that are running.'
   task :stop_all_peering, :roles => :peer do
     run("pkill -f \"murder_client.py peer\"")
   end

data/lib/murder/murder.rb

@@ -1,5 +1,6 @@
 # Copyright 2010 Twitter, Inc.
 # Copyright 2010 Larry Gadea <lg@twitter.com>
+# Copyright 2010 Matt Freels <freels@twitter.com>
 #
 # Licensed under the Apache License, Version 2.0 (the "License"); you may
 # not use this file except in compliance with the License. You may obtain
@@ -13,15 +14,10 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# no default defaults...
-set :default_tag, ''
-set :default_seeder_files_path, ''
-set :default_destination_path, ''
-
-# default remote dist path in app shared directory
-set(:remote_murder_path) { "#{shared_path}/murder" }
-
 namespace :murder do
+  desc <<-DESC
+  Compresses the directory specified by the passed-in argument 'files_path' and creates a .torrent file identified by the 'tag' argument. Be sure to use the same 'tag' value with any following commands. Any .git directories will be skipped. Once completed, the .torrent will be downloaded to your local /tmp/TAG.tgz.torrent.
+  DESC
   task :create_torrent, :roles => :seeder do
     require_tag
     if !(seeder_files_path = (default_seeder_files_path if default_seeder_files_path != "") || ENV['files_path'])
@@ -44,21 +40,33 @@ namespace :murder do
     download_torrent unless ENV['do_not_download_torrent']
   end
 
+  desc <<-DESC
+  Although not necessary to run, if the file from create_torrent was lost, you can redownload it from the seeder using this task. You must specify a valid 'tag' argument.
+  DESC
   task :download_torrent, :roles => :seeder do
     require_tag
     download("#{filename}.torrent", "#{filename}.torrent", :via => :scp)
   end
 
+  desc <<-DESC
+  Will cause the seeder machine to connect to the tracker and start seeding. The ip address returned by the 'host' bash command will be announced to the tracker. The server will not stop seeding until the stop_seeding task is called. You must specify a valid 'tag' argument (which identifies the .torrent in /tmp to use)
+  DESC
   task :start_seeding, :roles => :seeder do
     require_tag
     run "screen -dms 'seeder-#{tag}' python #{remote_murder_path}/murder_client.py seeder '#{filename}.torrent' '#{filename}' `host $HOSTNAME | awk '{print $4}'`"
   end
 
+  desc <<-DESC
+  If the seeder is currently seeding, this will kill the process. Note that if it is not running, you will receive an error. If a peer was downloading from this seed, the peer will find another host to receive any remaining data. You must specify a valid 'tag' argument.
+  DESC
   task :stop_seeding, :roles => :seeder do
     require_tag
     run("pkill -f \"SCREEN.*seeder-#{tag}\"")
   end
 
+  desc <<-DESC
+  Instructs all the peer servers to connect to the tracker and start download and spreading pieces and files amongst themselves. You must specify a valid 'tag' argument. Once the download is complete on a server, that server will fork the download process and seed for 30 seconds while returning control to Capistrano. Cap will then extract the files to the passed in 'destination_path' argument to destination_path/TAG/*. To not create this tag named directory, pass in the 'no_tag_directory=1' argument. If the directory is empty, this command will fail. To clean it, pass in the 'unsafe_please_delete=1' argument. The compressed tgz in /tmp is never removed. When this task completes, all files have been transferred and moved into the requested directory.
+  DESC
   task :peer, :roles => :peer do
     require_tag
 

data/murder.gemspec

@@ -5,11 +5,11 @@
 
 Gem::Specification.new do |s|
   s.name = %q{murder}
-  s.version = "0.0.0.pre"
+  s.version = "0.1.0"
 
-  s.required_rubygems_version = Gem::Requirement.new("> 1.3.1") if s.respond_to? :required_rubygems_version=
-  s.authors = ["Larry Gadea"]
-  s.date = %q{2010-03-15}
+  s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
+  s.authors = ["Larry Gadea", "Matt Freels"]
+  s.date = %q{2010-03-17}
   s.description = %q{Large scale server deploys using BitTorrent and the BitTornado library}
   s.email = %q{lg@twitter.com}
   s.extra_rdoc_files = [

metadata

@@ -1,21 +1,21 @@
 --- !ruby/object:Gem::Specification 
 name: murder
 version: !ruby/object:Gem::Version 
-  prerelease: true
+  prerelease: false
   segments: 
   - 0
+  - 1
   - 0
-  - 0
-  - pre
-  version: 0.0.0.pre
+  version: 0.1.0
 platform: ruby
 authors: 
 - Larry Gadea
+- Matt Freels
 autorequire: 
 bindir: bin
 cert_chain: []
 
-date: 2010-03-15 00:00:00 -07:00
+date: 2010-03-17 00:00:00 -07:00
 default_executable: 
 dependencies: []
 
@@ -111,13 +111,11 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: "0"
 required_rubygems_version: !ruby/object:Gem::Requirement 
   requirements: 
-  - - ">"
+  - - ">="
     - !ruby/object:Gem::Version 
       segments: 
-      - 1
-      - 3
-      - 1
-      version: 1.3.1
+      - 0
+      version: "0"
 requirements: []
 
 rubyforge_project: