distribustream 0.2.1 → 0.3.0

data/CHANGES

@@ -1,3 +1,21 @@
+Version 0.3.0
+* dsseed now factored into server.rb.  Remote seeds will not be supported until
+  protocol changes can be made to support authenticating remote seeds, remote
+  file lookup, and remote hash checking.
+
+* Switch to Kirk Haines' Evented Mongrel and remove all usage of threads from
+  the entire application
+
+* Stats page relocated to /stats.  This allows a single mongrel to operate for
+  both the file service and for the stats page.
+
+* Remove common_init.rb and any dependency on the @@log and @@config variables
+
+* Add logging facility to the server application to replace @@log.  Similar
+  facilities should be added for clients.
+
+* Refactor PDTP::Client and PDTP::Server APIs to support the above changes.
+
 Version 0.2.1
 * Switch HTTP client to use an EventMachine-based version, rather than net/http
 

data/README

@@ -3,14 +3,6 @@ Welcome to DistribuStream!
 DistribuStream is a fully open peercasting system which allows on-demand
 or live streaming media to be delivered at a fraction of the normal cost. 
 
-This README covers the initial public release, known issues, and a general
-development roadmap.
-
-Contents:
-1. Usage
-2. Known Issues
-3. Development Roadmap
-
 --
 
 Usage:
@@ -34,22 +26,15 @@ Next, start the DistribuStream server:
 
 dstream --conf myconfig.yml
 
-The DistribuStream server manages traffic on the peer network.  It also handles
-the checksumming of files.  
+You can think of the DistribuStream server like a regular web or FTP server.
+It looks through the specified directory and makes the files underneath it
+available to the peer-to-peer network.
 
 You can see what's going on through the web interface, which runs one port
 above the port you configured the server to listen on.  If the server is
 listening on the default port of 6086, you can reach the web interface at:
 
-http://myserver.url:6087/
-
-To populate the server with files, you need to attach the DistribuStream
-seed.  This works off of the same config file as the server, and must
-be run on the same system:
-
-dsseed --conf myconfig.yml
-
-At this point your server is ready to go.
+http://myserver.url:6087/stats
 
 To test your server, use the DistribuStream client:
 
@@ -64,30 +49,9 @@ dsclient -o pdtp://myserver.url/file.ext | mediaplayer -
 
 --
 
-Known Issues:
-
-Seeds are presently not authenticated in any way, thus anyone can attach
-a seed and populate the server with any files of their choosing.  However,
-since file checksumming is done by the server itself, this means that only
-seeds running on the same system as the server will actually work.
-
-This will be resolved by either incorporating the seed directly into the
-DistribuStream server, or adding both authentication and commands for
-checksumming to the server <-> seed protocol.
-
---
-
 Development Roadmap:
 
-DistribuStream uses an assemblage of various tools which do not work together
-particularly well.  These include the EventMachine Ruby gem, which provides
-the I/O layer for the DistribuStream server, and the Mongrel web server, which
-runs independently of EventMachine and uses threads.
-
-Initial work will focus on converting the existing implementation to a fully
-EventMachine-based approach which eliminates the use of threads.
-
-Mid-term work will focus on improving the efficiency of peer-to-peer traffic
+Short-term goals focus on improving the efficiency of peer-to-peer traffic
 routing, by incorporating all of the following constraints:
 
 1. Does the potential "giver" peer have chunks the "taker" peer is requesting?

data/bin/dsclient

@@ -21,12 +21,24 @@
 
 #--
 # Copyright (C) 2006-07 ClickCaster, Inc. (info@clickcaster.com)
-# All rights reserved.  See COPYING for permissions.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+# 
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
 # 
 # This source file is distributed as part of the 
 # DistribuStream file transfer system.
 #
-# See http://distribustream.rubyforge.org/
+# See http://distribustream.org/
 #++
 
 require 'uri'
@@ -86,22 +98,23 @@ class Callbacks < PDTP::Client::Callbacks
   attr_accessor :uri
   attr_accessor :output
   
-  def connected
+  def initialize(uri, output)
+    @uri, @output = uri, output
+  end
+  
+  def connected(client)
     STDERR.write "Connected to #{uri.host}\n"
     STDERR.write "Beginning transfer...\n"
     
     # Begin fetching the file once we've connected
-    client.get uri.path, output
+    client.get @uri.path, @output
   end
   
-  def disconnected
+  def disconnected(client)
     STDERR.write "Disconnected from #{uri.host}\n"
     client.stop
   end
 end
-
-@@log = Logger.new(STDERR)
-@@log.level=Logger::INFO
   
-@client = PDTP::Client.new uri.host, uri.port || 6086, :listen_port => listen_port
-@client.connect(Callbacks) { |c| c.uri, c.output = uri, output }
+client = PDTP::Client.new uri.host, uri.port || 6086, :listen_port => listen_port
+client.connect Callbacks, uri, output

data/bin/dstream

@@ -1,20 +1,83 @@
 #!/usr/bin/env ruby
 #--
 # Copyright (C) 2006-07 ClickCaster, Inc. (info@clickcaster.com)
-# All rights reserved.  See COPYING for permissions.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+# 
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
 # 
 # This source file is distributed as part of the 
 # DistribuStream file transfer system.
 #
-# See http://distribustream.rubyforge.org/
+# See http://distribustream.org/
 #++
 
-require File.dirname(__FILE__) + '/../lib/pdtp/common/common_init'
+require 'optparse'
 require File.dirname(__FILE__) + '/../lib/pdtp/server'
 
-common_init File.basename($0)
+program_name = File.basename($0)
+config_filename = nil
+
+OptionParser.new do |opts|
+  opts.banner = "Usage: #{program_name} [options]"
+  opts.on("--config CONFIGFILE", "Load specified config file.") do |c|
+    config_filename = c
+  end
+  opts.on("--help", "Prints this usage info.") do
+    puts opts
+    exit
+  end
+end.parse!      
+
+config = {
+  :host             => '0.0.0.0',
+  :port             => 6086, #server port
+  :listen_port      => 8000, #client listen port
+  :file_root        => '.',
+  :chunk_size       => 5000,
+  :quiet            => true
+}
+
+if config_filename.nil?
+  puts "No config file specified. Using defaults."
+else
+  begin
+    confstr = File.read(config_filename)
+  rescue
+    puts "Unable to open config file: #{config_filename}"
+    exit
+  end
+  
+  begin
+    new_config = YAML.load confstr
+    config.merge!(new_config)
+    config[:vhost] ||= config[:host] # Use host as vhost unless specified
+  rescue Exception => e
+    puts "Error parsing config file: #{config_filename}"
+    puts e
+    exit
+  end
+end
+
+puts "#{program_name} starting. Run '#{program_name} --help' for more info."
+
+server = PDTP::Server.new config[:host], config[:port]
+
+STDERR.sync = true
+logger = Logger.new STDERR
+logger.datetime_format = "%H:%M:%S "
+logger.level = Logger::INFO if config[:quiet]  
 
-server = PDTP::Server.new @@config[:host], @@config[:port]
-server.enable_stats_service
-server.enable_file_service @@config[:file_root], :chunk_size => @@config[:chunk_size]
+server.enable_logging logger
+server.enable_status_page
+server.enable_file_service config[:file_root], :chunk_size => config[:chunk_size]
 server.run

data/distribustream.gemspec

@@ -2,19 +2,19 @@ require 'rubygems'
 
 GEMSPEC = Gem::Specification.new do |s|
   s.name = "distribustream"
-  s.version = "0.2.1"
-  s.date = "2008-10-22"
+  s.version = "0.3.0"
+  s.date = "2008-11-09"
   s.summary = "DistribuStream is a fully open peercasting system allowing on-demand or live streaming media to be delivered at a fraction of the normal cost"
   s.email = "tony@clickcaster.com"
-  s.homepage = "http://distribustream.rubyforge.org"
+  s.homepage = "http://distribustream.org"
   s.rubyforge_project = "distribustream"
   s.has_rdoc = true
   s.rdoc_options = ["--exclude", "definitions", "--exclude", "indexes"]
-  s.extra_rdoc_files = ["COPYING", "README", "CHANGES"]
+  s.extra_rdoc_files = ["COPYING", "README", "CHANGES", "pdtp-specification.xml"]
   s.authors = ["Tony Arcieri", "Ashvin Mysore", "Galen Pahlke", "James Sanders", "Tom Stapleton"]
   s.files = Dir.glob("{bin,lib,conf}/**/*") + ['Rakefile', 'distribustream.gemspec']
-  s.executables = %w{dstream dsseed dsclient}
+  s.executables = %w{dstream dsclient}
   s.add_dependency("eventmachine", ">= 0.9.0")
-  s.add_dependency("mongrel", ">= 1.0.1")
+  s.add_dependency("mongrel", ">= 1.0.2")
   s.add_dependency("json", ">= 1.1.0")
 end

data/lib/pdtp/client.rb

@@ -1,19 +1,31 @@
 #--
 # Copyright (C) 2006-07 ClickCaster, Inc. (info@clickcaster.com)
-# All rights reserved.  See COPYING for permissions.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+# 
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
 # 
 # This source file is distributed as part of the 
 # DistribuStream file transfer system.
 #
-# See http://distribustream.rubyforge.org/
+# See http://distribustream.org/
 #++
 
 require 'rubygems'
 require 'eventmachine'
-require 'thread'
 require 'digest/md5'
 
 require File.dirname(__FILE__) + '/common'
+require File.dirname(__FILE__) + '/common/http_server'
 require File.dirname(__FILE__) + '/client/connection'
 require File.dirname(__FILE__) + '/client/callbacks'
 require File.dirname(__FILE__) + '/client/file_service'
@@ -23,10 +35,10 @@ module PDTP
   # PDTP::Client provides an interface for accessing resources on PDTP servers
   class Client
     # None of these should be publically accessible, and will be factored away in time
-    attr_reader :connection, :client_id, :listen_port, :file_service, :transfers, :lock
+    attr_reader :connection, :client_id, :listen_port, :file_service, :transfers
 
     # Create a PDTP::Client object for accessing the PDTP server at the given host and port
-    def initialize(host, port = 6086, options = {})
+    def initialize(host, port = PDTP::DEFAULT_PORT, options = {})
       @host, @port = host, port
             
       @listen_addr = options[:listen_addr] || '0.0.0.0'
@@ -34,45 +46,49 @@ module PDTP
       
       @file_service = PDTP::Client::FileService.new
       @transfers = []
-      @lock = Mutex.new
 
       # Start a Mongrel server on the specified port.  If it isnt available, keep trying higher ports
-      begin
+      # FIXME: Evented Mongrel won't throw an exception if the port is unavailable.  This needs to
+      # be dealt with through EventMachine
+      #begin
         @http_server = Mongrel::HttpServer.new @listen_addr, @listen_port
-      rescue Exception => e
-        @listen_port += 1
-        retry
-      end
+      #rescue
+      #  @listen_port += 1
+      #  retry
+      #end
 
       @client_id = Digest::MD5.hexdigest "#{Time.now.to_f}#{$$}"
 
       #@@log.info "listening on port #{@listen_port}"
       @http_handler = HttpHandler.new(self)
       @http_server.register '/', @http_handler
-      @http_server.run
     end
 
     # Connect to the PDTP server. This is a blocking call which runs the client event loop
-    def connect(callbacks = nil)
+    def connect(callbacks = nil, *args)
       klass = if(callbacks and callbacks.is_a?(Class))
         callbacks
       else
         Class.new(Callbacks) {callbacks and include callbacks}
       end
       
+      arity = klass.instance_method(:initialize).arity
+      expected = arity >= 0 ? arity : -(arity + 1)
+      if (arity >= 0 and args.size != expected) or (arity < 0 and args.size < expected)
+        raise ArgumentError, "wrong number of arguments for #{klass}#initialize (#{args.size} for #{expected})" 
+      end
+          
       # Run the EventMachine reactor loop
       EventMachine.run do
+        @http_server.run_evented
+        
         EventMachine.connect(@host, @port, Connection) do |c|
           # Store reference to the connection
           @connection = c
           
           # Set connection variables and configure callbacks
           c.client = self
-          c.callbacks = klass.new(self)
-          c.callbacks.client = self
-          
-          # Allow users to populate instance variables in their callback class
-          yield c.callbacks if block_given?
+          c.callbacks = klass.new *args
         end
 
         #@@log.info "connecting with ev=#{EventMachine::VERSION}"
@@ -110,4 +126,4 @@ module PDTP
       EventMachine.stop_event_loop
     end
   end
-end
+end

data/lib/pdtp/client/callbacks.rb

@@ -1,27 +1,38 @@
 #--
 # Copyright (C) 2006-07 ClickCaster, Inc. (info@clickcaster.com)
-# All rights reserved.  See COPYING for permissions.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+# 
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
 # 
 # This source file is distributed as part of the 
 # DistribuStream file transfer system.
 #
-# See http://distribustream.rubyforge.org/
+# See http://distribustream.org/
 #++
 
 module PDTP
   class Client
-    # Callbacks 
+    # Callbacks which can be overridden by clients to handle events
     class Callbacks
       attr_accessor :client
       
-      def initialize(connection)
-        @connection = connection
+      def initialize(*args)
       end
       
-      def connected
+      def connected(client)
       end
       
-      def disconnected
+      def disconnected(client)
         client.stop
       end
     end

data/lib/pdtp/client/connection.rb

@@ -1,11 +1,23 @@
 #--
 # Copyright (C) 2006-07 ClickCaster, Inc. (info@clickcaster.com)
-# All rights reserved.  See COPYING for permissions.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+# 
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
 # 
 # This source file is distributed as part of the 
 # DistribuStream file transfer system.
 #
-# See http://distribustream.rubyforge.org/
+# See http://distribustream.org/
 #++
 
 require 'rubygems'
@@ -32,13 +44,22 @@ module PDTP
             :client_id => @client.client_id
           )
           
-          callbacks.connected
+          callbacks.connected client
         rescue Exception => e
           puts "Exception in connection_completed: #{e}"
           puts e.backtrace.join("\n")
           exit
         end
-      end      
+      end
+      
+      # Called when the connection to the server closes
+      def unbind
+        if callbacks
+          callbacks.disconnected client
+        else
+          puts "Disconnected from PDTP server"
+        end
+      end
       
       # Called when any server message is received. This is the brains of
       # the client's protocol handling.
@@ -60,17 +81,7 @@ module PDTP
           
           transfer.run
 
-          #@@log.debug "TRANSFER STARTING"
-
-          # Run each transfer in its own thread and notify the server upon completion
-          #Thread.new(transfer) do |t|
-          #  begin
-          #    t.run
-          #  rescue Exception=>e
-          #    @@log.info("Exception in dispatch_message: " + e.exception + "\n" + e.backtrace.join("\n")) 
-          #  end
-          #  t.send_completed_message(t.hash)
-          #end
+          ##@@log.debug "TRANSFER STARTING"
         when "tell_verify"
           # We are a listener, and asked for verification of a transfer from a server.
           # After asking for verification, we stopped running, and must be restarted
@@ -91,25 +102,20 @@ module PDTP
             exit!
           end
         when "hash_verify"
-          @@log.debug "Hash verified for url=#{message["url"]} range=#{message["range"]} hash_ok=#{message["hash_ok"]}"
+          #@@log.debug "Hash verified for url=#{message["url"]} range=#{message["range"]} hash_ok=#{message["hash_ok"]}"
         when "protocol_error", "protocol_warn" #ignore
         else raise "Server sent an unknown message type: #{command} "
         end
       end
-
-      def unbind
-        super
-        puts 'Disconnected from PDTP server.'
-      end
       
-      #Prints the number of transfers associated with this client
+      # Prints the number of transfers associated with this client
       def print_stats
-        @@log.debug "client:  num_transfers=#{@client.transfers.size}"
+        #@@log.debug "client:  num_transfers=#{@client.transfers.size}"
       end
 
-      #Provides a threadsafe mechanism for transfers to report themselves finished
+      # Transfers to report themselves finished
       def finished(transfer)
-        @client.lock.synchronize { @client.transfers.delete(transfer) }
+        @client.transfers.delete(transfer)
       end
     end
   end