distribustream 0.1.0

data/README

@@ -0,0 +1,107 @@
+Welcome to DistribuStream!
+
+DistribuStream is a fully open peercasting system which allows on-demand
+or live streaming media to be delivered at a fraction of the normal cost. 
+
+This README covers the initial public release, known issues, and a general
+development roadmap.
+
+--
+
+Usage:
+
+The DistribuStream gem includes three config files that can be located in
+the conf directory of the gem:
+
+example.yml  - A standard example file, configuring for 5kB chunks
+debug.yml    - Same as example.yml, but configured for additional debug info
+bigchunk.yml - An example config file, using 250kB chunks
+
+Chunk size controls how large the segments which are exchanged between peers
+are.  Larger chunk sizes reduce the amount of control traffic the server
+sends, but increase the probability of errors.
+
+To begin, copy one of these config file and edit it to your choosing.  Be
+sure to set the host address to bind to and optionally the vhost you wish
+to identify as.
+
+Next, start the DistribuStream server:
+
+distribustream --conf myconfig.yml
+
+The DistribuStream server manages traffic on the peer network.  It also handles
+the checksumming of files.  
+
+You can see what's going on through the web interface, which runs one port
+above the port you configured the server to listen on.  If the server is
+listening on the default port of 6086, you can reach the web interface at:
+
+http://myserver.url:6087/
+
+To populate the server with files, you need to attach the DistribuStream
+seed.  This works off of the same config file as the server, and must
+be run on the same system:
+
+dsseed --conf myconfig.yml
+
+At this point your server is ready to go.
+
+To test your server, use the DistribuStream client:
+
+dsclient --url pdtp://myserver.url/file.ext
+
+This will download file.ext from your DistribuStream server.
+
+While you can't control the output filename at this point, the client supports
+non-seekable output such as pipes.  To play streaming media as it downloads,
+you can:
+
+mkfifo file.ext
+dsclient --url pdtp://myserver.url/file.ext &
+mediaplayer file.ext
+
+--
+
+Known Issues:
+
+The client presently stores incoming data in a memory buffer.  This causes
+the client to consume massive amounts of memory as the file downloads.
+Subsequent releases will fix this by improving the design of the memory
+buffer, moving to a disk-backed buffer and/or discarding some of the
+downloaded data after it's been played back.
+
+The protocol facilitates allowing clients to have a moving window of data
+in a stream, so they need not retain data which has already been displayed
+to the user.
+
+Seeds are presently not authenticated in any way, thus anyone can attach
+a seed and populate the server with any files of their choosing.  However,
+since file checksumming is done by the server itself, this means that only
+seeds running on the same system as the server will actually work.
+
+This will be resolved by either incorporating the seed directly into the
+DistribuStream server, or adding both authentication and commands for
+checksumming to the server <-> seed protocol.
+
+--
+
+Development Roadmap:
+
+The immediate goal is to improve the performance of the client, which presently
+consumes far too much RAM for practical use with large media files.  Another
+immediate goal is solving the above problems with seeds.
+
+DistribuStream uses an assemblage of various tools which do not work together
+particularly well.  These include the EventMachine Ruby gem, which provides
+the I/O layer for the DistribuStream server, and the Mongrel web server, which
+runs independently of EventMachine and uses threads.
+
+Initial work will focus on converting the existing implementation to a fully
+EventMachine-based approach which eliminates the use of threads.
+
+Subsequent work will focus on improving the APIs provided by the various
+components so that the client and server can both
+
+Long-term goals include a move to UDP to reduce protocol latency and overhead
+as well as encrypting all traffic to ensure privacy and security of the
+data being transmitted.

data/Rakefile

@@ -0,0 +1,44 @@
+require 'rake'
+require 'rake/rdoctask'
+require 'rake/gempackagetask'
+load 'distribustream.gemspec'
+
+# Default Rake task
+task :default => :rdoc
+
+# RDoc
+Rake::RDocTask.new(:rdoc) do |task|
+  task.rdoc_dir = 'doc'
+  task.title    = 'DistribuStream'
+  task.rdoc_files.include('bin/**/*.rb')
+  task.rdoc_files.include('lib/**/*.rb')
+  task.rdoc_files.include('simulation/**/*.rb')
+  task.rdoc_files.include('test/**/*.rb')
+end
+
+# Gem
+Rake::GemPackageTask.new(GEMSPEC) do |pkg|
+  pkg.need_tar = true
+end
+
+# RSpec
+begin
+require 'spec/rake/spectask'
+
+Spec::Rake::SpecTask.new(:spec) do |task|
+  task.spec_files = FileList['**/*_spec.rb']
+end
+
+Spec::Rake::SpecTask.new(:specfs) do |task|
+  task.spec_files= FileList['**/*_spec.rb']
+  task.spec_opts="-f s".split
+end
+
+Spec::Rake::SpecTask.new(:spec_coverage) do |task|
+  task.spec_files = FileList['**/*_spec.rb']
+  task.rcov = true
+end
+rescue LoadError
+end
+
+

data/bin/distribustream

@@ -0,0 +1,60 @@
+#!/usr/bin/env ruby
+#--
+# Copyright (C) 2006-07 ClickCaster, Inc. (info@clickcaster.com)
+# All rights reserved.  See COPYING for permissions.
+# 
+# This source file is distributed as part of the 
+# DistribuStream file transfer system.
+#
+# See http://distribustream.rubyforge.org/
+#++
+
+require 'rubygems'
+require 'eventmachine'
+require 'optparse'
+require 'logger'
+require 'mongrel'
+
+require File.dirname(__FILE__) + '/../lib/pdtp/server'
+
+common_init $0
+
+server = PDTP::Server.new
+server.file_service = PDTP::Server::FileService.new
+PDTP::Protocol.listener = server
+
+#set up the mongrel server for serving the stats page
+class MongrelServerHandler< Mongrel::HttpHandler
+  def initialize(server)
+    @server = server
+  end
+
+  def process(request,response)
+    response.start(200) do |head, out|
+      out.write begin
+        outstr = @server.generate_html_stats
+      rescue Exception=>e
+        outstr = "Exception: #{e}\n#{e.backtrace.join("\n")}"
+      end
+    end    
+  end
+end
+
+#run the mongrel server
+mongrel_server = Mongrel::HttpServer.new '0.0.0.0', @@config[:port] + 1
+@@log.info "Mongrel server listening on port: #{@@config[:port] + 1}"
+mongrel_server.register '/', MongrelServerHandler.new(server)
+mongrel_server.run
+
+#set root directory
+server.file_service.root = @@config[:file_root]
+server.file_service.default_chunk_size = @@config[:chunk_size]
+
+EventMachine::run do
+  host, port = '0.0.0.0', @@config[:port]
+  EventMachine::start_server host, port, PDTP::Protocol
+  @@log.info "accepting connections with ev=#{EventMachine::VERSION}"
+  @@log.info "host=#{host}  port=#{port}"
+
+  EventMachine::add_periodic_timer(2) { server.clear_all_stalled_transfers }
+end

data/bin/dsclient

@@ -0,0 +1,43 @@
+#!/usr/bin/env ruby
+#--
+# Copyright (C) 2006-07 ClickCaster, Inc. (info@clickcaster.com)
+# All rights reserved.  See COPYING for permissions.
+# 
+# This source file is distributed as part of the 
+# DistribuStream file transfer system.
+#
+# See http://distribustream.rubyforge.org/
+#++
+
+require 'rubygems'
+require 'eventmachine'
+require 'mongrel'
+require 'optparse'
+require 'uri'
+
+require File.dirname(__FILE__) + '/../lib/pdtp/client'
+
+uri = nil
+listen_port = 8000
+
+OptionParser.new do |opts|
+  opts.banner = "Usage: #{$0} [options]"
+  opts.on("--url URL", "Fetch from the specified URL") do |u|
+    uri = URI.parse u
+  end
+  opts.on("--help", "Prints this usage info.") do
+    puts opts
+    exit
+  end
+  opts.on("--listen PORT", "Port to listen on") do |l|
+    listen_port = l.to_i
+  end
+end.parse!
+
+raise "Please specify a URL in the form --url URL" unless uri
+raise "Only pdtp:// URLs are supported" unless uri.scheme == 'pdtp'
+
+options = { :listen_port => listen_port }
+options[:port] = uri.port unless uri.port.nil?
+
+PDTP::Client.get uri.host, uri.path, options

data/bin/dsseed

@@ -0,0 +1,103 @@
+#!/usr/bin/env ruby
+#--
+# Copyright (C) 2006-07 ClickCaster, Inc. (info@clickcaster.com)
+# All rights reserved.  See COPYING for permissions.
+# 
+# This source file is distributed as part of the 
+# DistribuStream file transfer system.
+#
+# See http://distribustream.rubyforge.org/
+#++
+
+require 'optparse'
+require 'rubygems'
+require 'eventmachine'
+require 'mongrel'
+
+require File.dirname(__FILE__) + '/../lib/pdtp/client'
+
+common_init $0
+
+# Fine all suitable files in the give path
+def find_files(base_path)
+  require 'find'
+
+  found = []
+  excludes = %w{.svn CVS}
+  base_full = File.expand_path(base_path)
+
+  Find.find(base_full) do |path|
+    if FileTest.directory?(path)
+      next unless excludes.include?(File.basename(path))
+      Find.prune
+    else
+      filename = path[(base_path.size - path.size + 1)..-1] #the entire file path after the base_path
+      found << filename
+    end
+  end
+
+  found
+end
+
+# Implements the file service for the pdtp protocol
+class FileServiceProtocol < PDTP::Protocol
+  def initialize(*args)
+    super
+  end
+
+  # Called after a connection to the server has been established
+  def connection_completed
+    begin
+      listen_port = @@config[:listen_port]
+
+      #create the client
+      client = PDTP::Client.new
+      PDTP::Protocol.listener = client
+      client.server_connection = self
+      client.generate_client_id listen_port
+
+      # Start a mongrel server on the specified port.  If it isnt available, keep trying higher ports
+      begin
+        mongrel_server=Mongrel::HttpServer.new '0.0.0.0', listen_port
+      rescue Exception=>e
+        listen_port+=1
+        retry
+      end
+
+      @@log.info "listening on port #{listen_port}"
+      mongrel_server.register "/", client
+      mongrel_server.run
+
+      # Tell the server a little bit about ourself
+      send_message :client_info, :listen_port => listen_port, :client_id => client.my_id
+
+      @@log.info 'This client is providing'
+      sfs = PDTP::Server::FileService.new
+      sfs.root = @@config[:file_root]
+      client.file_service = sfs #give this client access to all data
+
+      hostname = @@config[:vhost]
+
+      # Provide all the files in the root directory
+      files = find_files @@config[:file_root]
+      files.each { |file| send_message :provide, :url => "http://#{hostname}/#{file}" }
+    rescue Exception => e
+      puts "Exception in connection_completed: #{e}"
+      puts e.backtrace.join("\n")
+      exit
+    end
+  end  
+
+  def unbind
+    super
+    puts "Disconnected from PDTP server."
+  end
+end
+
+# Run the EventMachine reactor loop
+EventMachine::run do
+  host, port, listen_port = @@config[:host], @@config[:port], @@config[:listen_port]
+  connection = EventMachine::connect host, port, FileServiceProtocol
+  @@log.info "connecting with ev=#{EventMachine::VERSION}"
+  @@log.info "host= #{host}  port=#{port}"
+end

data/conf/bigchunk.yml

@@ -0,0 +1,18 @@
+---
+# Virtual hostname server identifies itself as
+:vhost: example.com
+
+# Hostname or IP address to bind to
+:host: 0.0.0.0
+
+# Base port the server listens on
+:port: 6086
+
+# Run in quiet mode
+:quiet: true
+
+# Base directory from which files are served
+:file_root: /Users/tony/dstest/files
+
+# Size of segments to be distributed (in bytes)
+:chunk_size: 250000

data/conf/debug.yml

@@ -0,0 +1,18 @@
+---
+# Virtual hostname server identifies itself as
+:vhost: example.com
+
+# Hostname or IP address to bind to
+:host: 0.0.0.0
+
+# Base port the server listens on
+:port: 6086
+
+# Run in quiet mode
+:quiet: false
+
+# Base directory from which files are served
+:file_root: /Users/tony/dstest/files
+
+# Size of segments to be distributed (in bytes)
+:chunk_size: 5000

data/conf/example.yml

@@ -0,0 +1,18 @@
+---
+# Virtual hostname server identifies itself as
+:vhost: example.com
+
+# Hostname or IP address to bind to
+:host: 0.0.0.0
+
+# Base port the server listens on
+:port: 6086
+
+# Run in quiet mode
+:quiet: true
+
+# Base directory from which files are served
+:file_root: /Users/tony/dstest/files
+
+# Size of segments to be distributed (in bytes)
+:chunk_size: 5000

data/distribustream.gemspec

@@ -0,0 +1,20 @@
+require 'rubygems'
+
+GEMSPEC = Gem::Specification.new do |s|
+  s.name = "distribustream"
+  s.version = "0.1.0"
+  s.date = "2008-10-11"
+  s.summary = "DistribuStream is a fully open peercasting system allowing on-demand or live streaming media to be delivered at a fraction of the normal cost"
+  s.email = "tony@clickcaster.com"
+  s.homepage = "http://distribustream.rubyforge.org"
+  s.rubyforge_project = "distribustream"
+  s.has_rdoc = true
+  s.rdoc_options = ["--exclude", "definitions", "--exclude", "indexes"]
+  s.extra_rdoc_files = ["COPYING", "README", "CHANGES"]
+  s.authors = ["Tony Arcieri", "Ashvin Mysore", "Galen Pahlke", "James Sanders", "Tom Stapleton"]
+  s.files = Dir.glob("{bin,lib,conf}/**/*") + ['Rakefile', 'distribustream.gemspec']
+  s.executables = %w{distribustream dsseed dsclient}
+  s.add_dependency("eventmachine", ">= 0.9.0")
+  s.add_dependency("mongrel", ">= 1.0.1")
+  s.add_dependency("json", ">= 1.1.0")
+end

data/lib/pdtp/client.rb

@@ -0,0 +1,195 @@
+#--
+# Copyright (C) 2006-07 ClickCaster, Inc. (info@clickcaster.com)
+# All rights reserved.  See COPYING for permissions.
+# 
+# This source file is distributed as part of the 
+# DistribuStream file transfer system.
+#
+# See http://distribustream.rubyforge.org/
+#++
+
+require 'rubygems'
+require 'eventmachine'
+require 'mongrel'
+require 'net/http'
+require 'thread'
+require 'digest/md5'
+
+require File.dirname(__FILE__) + '/common/common_init'
+require File.dirname(__FILE__) + '/common/protocol'
+require File.dirname(__FILE__) + '/client/protocol'
+require File.dirname(__FILE__) + '/client/file_service'
+require File.dirname(__FILE__) + '/client/transfer'
+require File.dirname(__FILE__) + '/server/file_service'
+
+module PDTP
+  # This is the main driver for the client-side implementation
+  # of PDTP. It maintains a single connection to a server and 
+  # all the necessary connections to peers. It is responsible
+  # for handling all messages corresponding to these connections.
+
+  # Client inherits from Mongrel::HttpHandler in order to handle
+  # incoming HTTP connections
+  class Client < Mongrel::HttpHandler
+    # Accessor for a client file service instance
+    attr_accessor :file_service
+    attr_accessor :server_connection
+    attr_accessor :my_id
+    
+    def self.get(host, path, options = {})
+      path = '/' + path unless path[0] == ?/
+      
+      opts = {
+        :host => host,
+        :port => 6086,
+        :file_root => '.',
+        :quiet => true,
+        :listen_port => 8000,
+        :request_url => "http://#{host}#{path}"
+      }.merge(options)
+      
+      common_init $0, opts
+      
+      # Run the EventMachine reactor loop
+      EventMachine::run do
+        connection = EventMachine::connect host, opts[:port], Client::Protocol
+        @@log.info "connecting with ev=#{EventMachine::VERSION}"
+        @@log.info "host= #{host} port=#{opts[:port]}"
+      end
+    end
+
+    def initialize
+      @transfers = []
+      @mutex = Mutex.new
+    end
+
+    # This method is called after a connection to the server
+    # has been successfully established.
+    def connection_created(connection)
+      @@log.debug("[mongrel] Opened connection...");
+    end
+
+    # This method is called when the server connection is destroyed
+    def connection_destroyed(connection)
+      @@log.debug("[mongrel] Closed connection...")
+    end
+
+    # Returns a transfer object if the given connection is a peer associated with 
+    # that transfer. Otherwise returns nil.
+    def get_transfer(connection)
+      @transfers.each { |t| return t if t.peer == connection }
+      nil
+    end
+
+    # This method is called when an HTTP request is received. It is called in 
+    # a separate thread, one for each request.
+    def process(request,response)
+      begin
+        @@log.debug "Creating Transfer::Listener"
+        transfer = Transfer::Listener.new(
+          request, 
+          response, 
+          @server_connection, 
+          @file_service, 
+          self
+        )
+
+        #Needs to be locked because multiple threads could attempt to append a transfer at once
+        @mutex.synchronize { @transfers << transfer }
+        transfer.handle_header 
+      rescue Exception=>e
+        transfer.write_http_exception(e)
+      end
+
+      transfer.send_completed_message transfer.hash
+    end
+
+    # Returns true if the given message refers to the given transfer  
+    def transfer_matches?(transfer, message)
+      transfer.peer       == message["peer"] and 
+      transfer.url        == message["url"] and
+      transfer.byte_range == message["range"] and
+      transfer.peer_id    == message["peer_id"]
+    end
+
+    # Called when any server message is received. This is the brains of
+    # the client's protocol handling.
+    def dispatch_message(command, message, connection)
+      case command
+      when "tell_info" # Receive and store information for this url  
+        info = FileInfo.new message["url"].split('/').last
+        info.file_size = message["size"]
+        info.base_chunk_size = message["chunk_size"]
+        info.streaming = message["streaming"]
+        @file_service.set_info(message["url"], info)   
+      when "transfer" # Begin a transfer as a connector
+        transfer = Transfer::Connector.new(message,@server_connection,@file_service,self)
+
+        @@log.debug "TRANSFER STARTING"
+
+        # Run each transfer in its own thread and notify the server upon completion
+        Thread.new(transfer) do |t|
+          begin
+            t.run
+          rescue Exception=>e
+            @@log.info("Exception in dispatch_message: " + e.exception + "\n" + e.backtrace.join("\n")) 
+          end
+          t.send_completed_message(t.hash)
+        end
+      when "tell_verify"
+        # We are a listener, and asked for verification of a transfer from a server.
+        # After asking for verification, we stopped running, and must be restarted
+        # if verification is successful
+        
+        found=false
+        @transfers.each do |t|
+          if t.matches_message?(message)
+            finished(t)
+            t.tell_verify(message["is_authorized"])
+            found=true
+            break
+          end
+        end
+
+        unless found
+          puts "BUG: Tell verify sent for an unknown transfer"
+          exit!
+        end
+      when "hash_verify"
+        @@log.debug "Hash verified for url=#{message["url"]} range=#{message["range"]} hash_ok=#{message["hash_ok"]}"
+      when "protocol_error", "protocol_warn" #ignore
+      else raise "Server sent an unknown message type: #{command} "
+      end
+    end
+
+    #Prints the number of transfers associated with this client
+    def print_stats
+      @@log.debug "client:  num_transfers=#{@transfers.size}"
+    end
+
+    #Provides a threadsafe mechanism for transfers to report themselves finished
+    def finished(transfer)
+      @mutex.synchronize do
+        @transfers.delete(transfer)
+      end
+    end
+
+    # Generate and set the client ID for an instance
+    def generate_client_id(port = 0)
+      @my_id = Client.generate_client_id port
+    end
+
+    # Client ID generator routine
+    def self.generate_client_id(port = 0)
+      md5 = Digest::MD5::new
+      now = Time::now
+      md5.update now.to_s
+      md5.update String(now.usec)
+      md5.update String(rand(0))
+      md5.update String($$)
+
+      #return md5.hexdigest+":#{port}" # long id
+      return md5.hexdigest[0..5] # short id
+    end
+  end
+end