murder 0.0.0.pre

data/.gitignore

@@ -0,0 +1 @@
+pkg/*

data/LICENSE

@@ -0,0 +1,17 @@
+This software is licensed under the Apache 2 license, quoted below.
+
+Copyright 2010 Twitter, Inc.
+Copyright 2010 Larry Gadea <lg@twitter.com>
+Copyright 2010 Matt Freels <freels@twitter.com>
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not
+use this file except in compliance with the License. You may obtain a copy of
+the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+License for the specific language governing permissions and limitations under
+the License.

data/README

@@ -0,0 +1,224 @@
+Murder by Larry Gadea and Matt Freels
+Copyright 2010 Twitter Inc.
+
+DESCRIPTION
+-----------
+
+Murder is a method of using Bittorrent to distribute files to a large amount
+of servers within a production environment. This allows for scaleable and fast
+deploys in environments of hundreds to tens of thousands of servers where
+centralized distribution systems wouldn't otherwise function. A "Murder" is
+normally used to refer to a flock of crows, which in this case applies to a
+bunch of servers doing something.
+
+In order to do a Murder transfer, there are several components required to be
+set up beforehand -- many the result of BitTorrent nature of the system. Murder
+is based on BitTornado.
+
+- A torrent tracker. This tracker, started by running the 'murder_tracker.py'
+script, runs a self-contained server on one machine. Although technically this
+is still a centralized system (everyone relying on this tracker), the
+communication between this server and the rest is minimal and normally
+acceptable. To keep things simple tracker-less distribution (DHT) is currently
+not supported. The tracker is actually just a mini-httpd that hosts a
+/announce path which the Bittorrent clients update their state onto.
+
+- A seeder. This is the server which has the files that you'd like to deploy
+onto all other servers. For Twitter, this is the server that did the git diff.
+The files are placed into a directory that a torrent gets created from. Murder
+will tgz up the directory and create a .torrent file (a very small file
+containing basic hash information about the tgz file). This .torrent file lets
+the peers know what they're downloading. The tracker keeps track of which
+.torrent files are currently being distributed. Once a Murder transfer is
+started, the seeder will be the first server many machines go to to get
+pieces. These pieces will then be distributed in a tree-fashion to the rest of
+the network, but without necessarily getting the parts from the seeder.
+
+- Peers. This is the group of servers (hundreds to tens of thousands) which
+will be receiving the files and distributing the pieces amongst themselves.
+Once a peer is done downloading the entire tgz file, it will continue seeding
+for a while to prevent a hotspot effect on the seeder.
+
+MURDER TRANSFER PROCESS
+-----------------------
+
+1. Configure the list of servers and general settings in config.rb. (one time)
+2. Distribute the Murder files to all your servers: (one time)
+    cap murder:distribute_files
+3. Start the tracker: (one time)
+    cap murder:start_tracker
+4. Create a torrent file from a remote directory of files (on seeder):
+    cap murder:create_torrent tag="Deploy20100101" files_path="~/files"
+5. Start seeding the files:
+    cap murder:start_seeding tag="Deploy20100101"
+6. Distribute the files to all servers:
+    cap murder:peer tag="Deploy20100101" destination_path="/tmp/out"
+
+Once completed, all files will be in /tmp/out/Deploy20091015/ on all servers.
+
+EXAMPLE DEPLOY
+--------------
+
+cap murder:distribute_files
+cap murder:start_tracker
+cap murder:create_torrent tag="Deploy20100101"
+  files_path="/usr/local/twitter/production/current"
+cap murder:start_seeding tag="Deploy20100101"
+time cap murder:peer tag="Deploy20100101"
+  destination_path="/usr/local/twitter/releases"
+
+All the files have been transferred to all servers at this point. To clean up
+use:
+
+cap murder:stop_seeding
+cap murder:stop_tracker
+
+HOW TO LEARN MURDER INCREMENTALLY
+---------------------------------
+
+- Skim dist/*.py to see what Murder does behind the scenes
+- Experiment with the python scripts
+- Read Capfile
+- Read murder_config.rb
+- Read murder_admin.rb
+- Read murder.rb
+- Read the reference below
+- Experiment with the Capistrano deploy methods
+- Read and experiment with the Murder Capistrano strategy
+
+TASK REFERENCE
+--------------
+
+distribute_files:
+  SCPs a compressed version of all files from ./dist (the python Bittorrent
+library and custom scripts) to all server. The entire directory is sent,
+regardless of the role of each individual server. The path on the server is
+specified by remote_murder_path and will be cleared prior to transferring
+files over.
+
+start_tracker:
+  Starts the Bittorrent tracker (essentially a mini-web-server) listening on
+port 8998.
+
+stop_tracker:
+  If the Bittorrent tracker is running, this will kill the process. Note that
+if it is not running you will receive an error.
+
+create_torrent:
+  Compresses the directory specified by the passed-in argument 'files_path'
+and creates a .torrent file identified by the 'tag' argument. Be sure to use
+the same 'tag' value with any following commands. Any .git directories will be
+skipped. Once completed, the .torrent will be downloaded to your local
+/tmp/TAG.tgz.torrent.
+
+download_torrent:
+  Although not necessary to run, if the file from create_torrent was lost, you
+can redownload it from the seeder using this task. You must specify a valid
+'tag' argument.
+
+start_seeding:
+  Will cause the seeder machine to connect to the tracker and start seeding.
+The ip address returned by the 'host' bash command will be announced to the
+tracker. The server will not stop seeding until the stop_seeding task is
+called. You must specify a valid 'tag' argument (which identifies the .torrent
+in /tmp to use)
+
+stop_seeding:
+  If the seeder is currently seeding, this will kill the process. Note that if
+it is not running, you will receive an error. If a peer was downloading from
+this seed, the peer will find another host to receive any remaining data. You
+must specify a valid 'tag' argument.
+
+stop_all_seeding:
+  Identical to stop_seeding, except this will kill all seeding processes. No
+'tag' argument is needed.
+
+peer:
+  Instructs all the peer servers to connect to the tracker and start download
+and spreading pieces and files amongst themselves. You must specify a valid
+'tag' argument. Once the download is complete on a server, that server will
+fork the download process and seed for 30 seconds while returning control to
+Capistrano. Cap will then extract the files to the passed in
+'destination_path' argument to destination_path/TAG/*. To not create this tag
+named directory, pass in the 'no_tag_directory=1' argument. If the directory
+is empty, this command will fail. To clean it, pass in the
+'unsafe_please_delete=1' argument. The compressed tgz in /tmp is never
+removed. When this task completes, all files have been transferred and moved
+into the requested directory.
+
+stop_all_peering:
+  Sometimes peers can go on forever (usually because of an error). This
+command will forcibly kill all "murder_client.py peer" commands that are
+running.
+
+CONFIG REFERENCE:
+
+user:
+  Provided by Capistrano to specify if to use a different username when sshing
+into machines.
+
+host_suffix:
+  For the tracker_host, seeder_host and peers servers, this suffix will always
+be added to the end when trying to connect to them.
+
+default_tag:
+  A tag name to use by default such that a tag parameter doesn't need to be
+manually entered on every task. Not recommended to be used since files will be
+overwritten.
+
+default_seeder_files_path:
+  A path on the seeder's file system where the files to be distributed are
+stored.
+
+default_destination_path:
+  A path on the peers' file system where the files that were distributed
+should be decompressed into.
+
+tracker_host:
+  The hostname of a single tracker server. If you're using a host_suffix, do
+not specify a suffix here.
+
+tracker_port:
+  This is the port on which the mini-web-server tracker that the Bittorrent
+libraries run is hosted on. Must be 8998 for now.
+
+seeder_host:
+  The hostname of a single seeder server. This server is the one that has all
+the files that the peers want. If you're using a host_suffix, do not specify a
+suffix here.
+
+peers:
+  A list of peers which will be receiving the files and distributing them
+amongst themselves. If you're using a host_suffix, do not specify a suffix
+here.
+
+CAPISTRANO COPY STRATEGY
+------------------------
+
+In addition to being usable from both the commandline and Capistrano commands,
+an optional Capistrano copy strategy is included to help easily retrofit
+existing deploy environments you might already have.
+
+Requirements to use build.rb
+  - Add a require line for build.rb in your Capfile. This will override the
+    default build strategy. There are some other parameters you must now
+    specify though.
+  - To enable it, add the following to your Capfile:
+    load 'deploy'
+    set :strategy, Capistrano::Deploy::Strategy::Build.new(self)
+  - Add "set :murder, true" to your Capfile
+  - Make sure your release_name is correct (should be already)
+  - If you're distribution is a file instead of a directory,
+    Add "distribution_is_a_file" to your Capfile
+  - Add the following to your Capfile:
+    set :build_task do
+      "true"
+    end
+    set :copy_compression, :gz
+    set :package_name do
+      "#{release_name}-#{real_revision[0, 8]}.tar.#{copy_compression}"
+    end
+    set :branch, variables[:branch] || 'deploy'
+
+The requirements are a bit hefty for now, though hopefully this will get easier
+in the future.

data/Rakefile

@@ -0,0 +1,52 @@
+require 'rubygems'
+require 'rake'
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |gem|
+    gem.name = "murder"
+    gem.summary = %Q{Large scale server deploys using BitTorrent and the BitTornado library}
+    gem.description = %Q{Large scale server deploys using BitTorrent and the BitTornado library}
+    gem.email = "lg@twitter.com"
+    gem.homepage = "http://github.com/lg/murder"
+    gem.authors = ["Larry Gadea"]
+    # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
+  end
+  Jeweler::GemcutterTasks.new
+rescue LoadError
+  puts "Jeweler (or a dependency) not available. Install it with: gem install jeweler"
+end
+
+# require 'rake/testtask'
+# Rake::TestTask.new(:test) do |test|
+#   test.libs << 'lib' << 'test'
+#   test.pattern = 'test/**/test_*.rb'
+#   test.verbose = true
+# end
+
+# begin
+#   require 'rcov/rcovtask'
+#   Rcov::RcovTask.new do |test|
+#     test.libs << 'test'
+#     test.pattern = 'test/**/test_*.rb'
+#     test.verbose = true
+#   end
+# rescue LoadError
+#   task :rcov do
+#     abort "RCov is not available. In order to run rcov, you must: sudo gem install spicycode-rcov"
+#   end
+# end
+
+# task :test => :check_dependencies
+
+# task :default => :test
+
+# require 'rake/rdoctask'
+# Rake::RDocTask.new do |rdoc|
+#   version = File.exist?('VERSION') ? File.read('VERSION') : ""
+
+#   rdoc.rdoc_dir = 'rdoc'
+#   rdoc.title = "murder #{version}"
+#   rdoc.rdoc_files.include('README*')
+#   rdoc.rdoc_files.include('lib/**/*.rb')
+# end

data/VERSION

@@ -0,0 +1 @@
+0.0.0.pre

data/dist/BitTornado/BT1/Choker.py

@@ -0,0 +1,128 @@
+# Written by Bram Cohen
+# see LICENSE.txt for license information
+
+from random import randrange, shuffle
+from BitTornado.clock import clock
+try:
+    True
+except:
+    True = 1
+    False = 0
+
+class Choker:
+    def __init__(self, config, schedule, picker, done = lambda: False):
+        self.config = config
+        self.round_robin_period = config['round_robin_period']
+        self.schedule = schedule
+        self.picker = picker
+        self.connections = []
+        self.last_preferred = 0
+        self.last_round_robin = clock()
+        self.done = done
+        self.super_seed = False
+        self.paused = False
+        schedule(self._round_robin, 5)
+
+    def set_round_robin_period(self, x):
+        self.round_robin_period = x
+
+    def _round_robin(self):
+        self.schedule(self._round_robin, 5)
+        if self.super_seed:
+            cons = range(len(self.connections))
+            to_close = []
+            count = self.config['min_uploads']-self.last_preferred
+            if count > 0:   # optimization
+                shuffle(cons)
+            for c in cons:
+                i = self.picker.next_have(self.connections[c], count > 0)
+                if i is None:
+                    continue
+                if i < 0:
+                    to_close.append(self.connections[c])
+                    continue
+                self.connections[c].send_have(i)
+                count -= 1
+            for c in to_close:
+                c.close()
+        if self.last_round_robin + self.round_robin_period < clock():
+            self.last_round_robin = clock()
+            for i in xrange(1, len(self.connections)):
+                c = self.connections[i]
+                u = c.get_upload()
+                if u.is_choked() and u.is_interested():
+                    self.connections = self.connections[i:] + self.connections[:i]
+                    break
+        self._rechoke()
+
+    def _rechoke(self):
+        preferred = []
+        maxuploads = self.config['max_uploads']
+        if self.paused:
+            for c in self.connections:
+                c.get_upload().choke()
+            return
+        if maxuploads > 1:
+            for c in self.connections:
+                u = c.get_upload()
+                if not u.is_interested():
+                    continue
+                if self.done():
+                    r = u.get_rate()
+                else:
+                    d = c.get_download()
+                    r = d.get_rate()
+                    if r < 1000 or d.is_snubbed():
+                        continue
+                preferred.append((-r, c))
+            self.last_preferred = len(preferred)
+            preferred.sort()
+            del preferred[maxuploads-1:]
+            preferred = [x[1] for x in preferred]
+        count = len(preferred)
+        hit = False
+        to_unchoke = []
+        for c in self.connections:
+            u = c.get_upload()
+            if c in preferred:
+                to_unchoke.append(u)
+            else:
+                if count < maxuploads or not hit:
+                    to_unchoke.append(u)
+                    if u.is_interested():
+                        count += 1
+                        hit = True
+                else:
+                    u.choke()
+        for u in to_unchoke:
+            u.unchoke()
+
+    def connection_made(self, connection, p = None):
+        if p is None:
+            p = randrange(-2, len(self.connections) + 1)
+        self.connections.insert(max(p, 0), connection)
+        self._rechoke()
+
+    def connection_lost(self, connection):
+        self.connections.remove(connection)
+        self.picker.lost_peer(connection)
+        if connection.get_upload().is_interested() and not connection.get_upload().is_choked():
+            self._rechoke()
+
+    def interested(self, connection):
+        if not connection.get_upload().is_choked():
+            self._rechoke()
+
+    def not_interested(self, connection):
+        if not connection.get_upload().is_choked():
+            self._rechoke()
+
+    def set_super_seed(self):
+        while self.connections:             # close all connections
+            self.connections[0].close()
+        self.picker.set_superseed()
+        self.super_seed = True
+
+    def pause(self, flag):
+        self.paused = flag
+        self._rechoke()

data/dist/BitTornado/BT1/Connecter.py

@@ -0,0 +1,288 @@
+# Written by Bram Cohen
+# see LICENSE.txt for license information
+
+from BitTornado.bitfield import Bitfield
+from BitTornado.clock import clock
+from binascii import b2a_hex
+
+try:
+    True
+except:
+    True = 1
+    False = 0
+
+DEBUG = False
+
+def toint(s):
+    return long(b2a_hex(s), 16)
+
+def tobinary(i):
+    return (chr(i >> 24) + chr((i >> 16) & 0xFF) + 
+        chr((i >> 8) & 0xFF) + chr(i & 0xFF))
+
+CHOKE = chr(0)
+UNCHOKE = chr(1)
+INTERESTED = chr(2)
+NOT_INTERESTED = chr(3)
+# index
+HAVE = chr(4)
+# index, bitfield
+BITFIELD = chr(5)
+# index, begin, length
+REQUEST = chr(6)
+# index, begin, piece
+PIECE = chr(7)
+# index, begin, piece
+CANCEL = chr(8)
+
+class Connection:
+    def __init__(self, connection, connecter):
+        self.connection = connection
+        self.connecter = connecter
+        self.got_anything = False
+        self.next_upload = None
+        self.outqueue = []
+        self.partial_message = None
+        self.download = None
+        self.send_choke_queued = False
+        self.just_unchoked = None
+
+    def get_ip(self, real=False):
+        return self.connection.get_ip(real)
+
+    def get_id(self):
+        return self.connection.get_id()
+
+    def get_readable_id(self):
+        return self.connection.get_readable_id()
+
+    def close(self):
+        if DEBUG:
+            print 'connection closed'
+        self.connection.close()
+
+    def is_locally_initiated(self):
+        return self.connection.is_locally_initiated()
+
+    def send_interested(self):
+        self._send_message(INTERESTED)
+
+    def send_not_interested(self):
+        self._send_message(NOT_INTERESTED)
+
+    def send_choke(self):
+        if self.partial_message:
+            self.send_choke_queued = True
+        else:
+            self._send_message(CHOKE)
+            self.upload.choke_sent()
+            self.just_unchoked = 0
+
+    def send_unchoke(self):
+        if self.send_choke_queued:
+            self.send_choke_queued = False
+            if DEBUG:
+                print 'CHOKE SUPPRESSED'
+        else:
+            self._send_message(UNCHOKE)
+            if ( self.partial_message or self.just_unchoked is None
+                 or not self.upload.interested or self.download.active_requests ):
+                self.just_unchoked = 0
+            else:
+                self.just_unchoked = clock()
+
+    def send_request(self, index, begin, length):
+        self._send_message(REQUEST + tobinary(index) + 
+            tobinary(begin) + tobinary(length))
+        if DEBUG:
+            print 'sent request: '+str(index)+': '+str(begin)+'-'+str(begin+length)
+
+    def send_cancel(self, index, begin, length):
+        self._send_message(CANCEL + tobinary(index) + 
+            tobinary(begin) + tobinary(length))
+        if DEBUG:
+            print 'sent cancel: '+str(index)+': '+str(begin)+'-'+str(begin+length)
+
+    def send_bitfield(self, bitfield):
+        self._send_message(BITFIELD + bitfield)
+
+    def send_have(self, index):
+        self._send_message(HAVE + tobinary(index))
+
+    def send_keepalive(self):
+        self._send_message('')
+
+    def _send_message(self, s):
+        s = tobinary(len(s))+s
+        if self.partial_message:
+            self.outqueue.append(s)
+        else:
+            self.connection.send_message_raw(s)
+
+    def send_partial(self, bytes):
+        if self.connection.closed:
+            return 0
+        if self.partial_message is None:
+            s = self.upload.get_upload_chunk()
+            if s is None:
+                return 0
+            index, begin, piece = s
+            self.partial_message = ''.join((
+                            tobinary(len(piece) + 9), PIECE,
+                            tobinary(index), tobinary(begin), piece.tostring() ))
+            if DEBUG:
+                print 'sending chunk: '+str(index)+': '+str(begin)+'-'+str(begin+len(piece))
+
+        if bytes < len(self.partial_message):
+            self.connection.send_message_raw(self.partial_message[:bytes])
+            self.partial_message = self.partial_message[bytes:]
+            return bytes
+
+        q = [self.partial_message]
+        self.partial_message = None
+        if self.send_choke_queued:
+            self.send_choke_queued = False
+            self.outqueue.append(tobinary(1)+CHOKE)
+            self.upload.choke_sent()
+            self.just_unchoked = 0
+        q.extend(self.outqueue)
+        self.outqueue = []
+        q = ''.join(q)
+        self.connection.send_message_raw(q)
+        return len(q)
+
+    def get_upload(self):
+        return self.upload
+
+    def get_download(self):
+        return self.download
+
+    def set_download(self, download):
+        self.download = download
+
+    def backlogged(self):
+        return not self.connection.is_flushed()
+
+    def got_request(self, i, p, l):
+        self.upload.got_request(i, p, l)
+        if self.just_unchoked:
+            self.connecter.ratelimiter.ping(clock() - self.just_unchoked)
+            self.just_unchoked = 0
+    
+
+
+
+class Connecter:
+    def __init__(self, make_upload, downloader, choker, numpieces,
+            totalup, config, ratelimiter, sched = None):
+        self.downloader = downloader
+        self.make_upload = make_upload
+        self.choker = choker
+        self.numpieces = numpieces
+        self.config = config
+        self.ratelimiter = ratelimiter
+        self.rate_capped = False
+        self.sched = sched
+        self.totalup = totalup
+        self.rate_capped = False
+        self.connections = {}
+        self.external_connection_made = 0
+
+    def how_many_connections(self):
+        return len(self.connections)
+
+    def connection_made(self, connection):
+        c = Connection(connection, self)
+        self.connections[connection] = c
+        c.upload = self.make_upload(c, self.ratelimiter, self.totalup)
+        c.download = self.downloader.make_download(c)
+        self.choker.connection_made(c)
+        return c
+
+    def connection_lost(self, connection):
+        c = self.connections[connection]
+        del self.connections[connection]
+        if c.download:
+            c.download.disconnected()
+        self.choker.connection_lost(c)
+
+    def connection_flushed(self, connection):
+        conn = self.connections[connection]
+        if conn.next_upload is None and (conn.partial_message is not None
+               or len(conn.upload.buffer) > 0):
+            self.ratelimiter.queue(conn)
+            
+    def got_piece(self, i):
+        for co in self.connections.values():
+            co.send_have(i)
+
+    def got_message(self, connection, message):
+        c = self.connections[connection]
+        t = message[0]
+        if t == BITFIELD and c.got_anything:
+            connection.close()
+            return
+        c.got_anything = True
+        if (t in [CHOKE, UNCHOKE, INTERESTED, NOT_INTERESTED] and 
+                len(message) != 1):
+            connection.close()
+            return
+        if t == CHOKE:
+            c.download.got_choke()
+        elif t == UNCHOKE:
+            c.download.got_unchoke()
+        elif t == INTERESTED:
+            if not c.download.have.complete():
+                c.upload.got_interested()
+        elif t == NOT_INTERESTED:
+            c.upload.got_not_interested()
+        elif t == HAVE:
+            if len(message) != 5:
+                connection.close()
+                return
+            i = toint(message[1:])
+            if i >= self.numpieces:
+                connection.close()
+                return
+            if c.download.got_have(i):
+                c.upload.got_not_interested()
+        elif t == BITFIELD:
+            try:
+                b = Bitfield(self.numpieces, message[1:])
+            except ValueError:
+                connection.close()
+                return
+            if c.download.got_have_bitfield(b):
+                c.upload.got_not_interested()
+        elif t == REQUEST:
+            if len(message) != 13:
+                connection.close()
+                return
+            i = toint(message[1:5])
+            if i >= self.numpieces:
+                connection.close()
+                return
+            c.got_request(i, toint(message[5:9]), 
+                toint(message[9:]))
+        elif t == CANCEL:
+            if len(message) != 13:
+                connection.close()
+                return
+            i = toint(message[1:5])
+            if i >= self.numpieces:
+                connection.close()
+                return
+            c.upload.got_cancel(i, toint(message[5:9]), 
+                toint(message[9:]))
+        elif t == PIECE:
+            if len(message) <= 9:
+                connection.close()
+                return
+            i = toint(message[1:5])
+            if i >= self.numpieces:
+                connection.close()
+                return
+            if c.download.got_piece(i, toint(message[5:9]), message[9:]):
+                self.got_piece(i)
+        else:
+            connection.close()