distribustream 0.2.0 → 0.2.1

data/CHANGES

@@ -1,3 +1,8 @@
+Version 0.2.1
+* Switch HTTP client to use an EventMachine-based version, rather than net/http
+
+* Added PDTP.version for retrieving version of the library
+
 Version 0.2.0
 * PDTP::Server and PDTP::Client now expose APIs intended for public consumption
   They may change slightly down the road, but will remain largely the same

data/README

@@ -6,6 +6,11 @@ or live streaming media to be delivered at a fraction of the normal cost.
 This README covers the initial public release, known issues, and a general
 development roadmap.
 
+Contents:
+1. Usage
+2. Known Issues
+3. Development Roadmap
+
 --
 
 Usage:
@@ -27,7 +32,7 @@ to identify as.
 
 Next, start the DistribuStream server:
 
-distribustream --conf myconfig.yml
+dstream --conf myconfig.yml
 
 The DistribuStream server manages traffic on the peer network.  It also handles
 the checksumming of files.  
@@ -48,32 +53,19 @@ At this point your server is ready to go.
 
 To test your server, use the DistribuStream client:
 
-dsclient --url pdtp://myserver.url/file.ext
+dsclient pdtp://myserver.url/file.ext
 
 This will download file.ext from your DistribuStream server.
 
-While you can't control the output filename at this point, the client supports
-non-seekable output such as pipes.  To play streaming media as it downloads,
-you can:
+The client also supports non-seekable output such as pipes.  To play streaming 
+media as it downloads, you can:
 
-mkfifo file.ext
-dsclient --url pdtp://myserver.url/file.ext &
-mediaplayer file.ext
+dsclient -o pdtp://myserver.url/file.ext | mediaplayer -
 
 --
 
 Known Issues:
 
-The client presently stores incoming data in a memory buffer.  This causes
-the client to consume massive amounts of memory as the file downloads.
-Subsequent releases will fix this by improving the design of the memory
-buffer, moving to a disk-backed buffer and/or discarding some of the
-downloaded data after it's been played back.
-
-The protocol facilitates allowing clients to have a moving window of data
-in a stream, so they need not retain data which has already been displayed
-to the user.
-
 Seeds are presently not authenticated in any way, thus anyone can attach
 a seed and populate the server with any files of their choosing.  However,
 since file checksumming is done by the server itself, this means that only
@@ -87,10 +79,6 @@ checksumming to the server <-> seed protocol.
 
 Development Roadmap:
 
-The immediate goal is to improve the performance of the client, which presently
-consumes far too much RAM for practical use with large media files.  Another
-immediate goal is solving the above problems with seeds.
-
 DistribuStream uses an assemblage of various tools which do not work together
 particularly well.  These include the EventMachine Ruby gem, which provides
 the I/O layer for the DistribuStream server, and the Mongrel web server, which
@@ -99,8 +87,17 @@ runs independently of EventMachine and uses threads.
 Initial work will focus on converting the existing implementation to a fully
 EventMachine-based approach which eliminates the use of threads.
 
-Subsequent work will focus on improving the APIs provided by the various
-components so that the client and server can both
+Mid-term work will focus on improving the efficiency of peer-to-peer traffic
+routing, by incorporating all of the following constraints:
+
+1. Does the potential "giver" peer have chunks the "taker" peer is requesting?
+2. Are the peers in the same prefix? (at least the same /16, if not the same /24)
+3. Are the peers firewalled? (the server could do a connect test as soon as a 
+   client registers their listen port)
+4. What is the trust between peers? (the "random" connecting of untrusted peers 
+   could be replaced by something that operates on the above two bits of data)
+5. Does the "giver" have enough bandwidth? (modeled over time)
+6. Are total inbound and outbound links minimized? (to improve TCP congestion)
 
 Long-term goals include a move to UDP to reduce protocol latency and overhead
 as well as encrypting all traffic to ensure privacy and security of the

data/distribustream.gemspec

@@ -2,7 +2,7 @@ require 'rubygems'
 
 GEMSPEC = Gem::Specification.new do |s|
   s.name = "distribustream"
-  s.version = "0.2.0"
+  s.version = "0.2.1"
   s.date = "2008-10-22"
   s.summary = "DistribuStream is a fully open peercasting system allowing on-demand or live streaming media to be delivered at a fraction of the normal cost"
   s.email = "tony@clickcaster.com"

data/lib/pdtp/client.rb

@@ -13,6 +13,7 @@ require 'eventmachine'
 require 'thread'
 require 'digest/md5'
 
+require File.dirname(__FILE__) + '/common'
 require File.dirname(__FILE__) + '/client/connection'
 require File.dirname(__FILE__) + '/client/callbacks'
 require File.dirname(__FILE__) + '/client/file_service'

data/lib/pdtp/client/connection.rb

@@ -57,18 +57,20 @@ module PDTP
             message,
             @client.file_service
           )
+          
+          transfer.run
 
-          @@log.debug "TRANSFER STARTING"
+          #@@log.debug "TRANSFER STARTING"
 
           # Run each transfer in its own thread and notify the server upon completion
-          Thread.new(transfer) do |t|
-            begin
-              t.run
-            rescue Exception=>e
-              @@log.info("Exception in dispatch_message: " + e.exception + "\n" + e.backtrace.join("\n")) 
-            end
-            t.send_completed_message(t.hash)
-          end
+          #Thread.new(transfer) do |t|
+          #  begin
+          #    t.run
+          #  rescue Exception=>e
+          #    @@log.info("Exception in dispatch_message: " + e.exception + "\n" + e.backtrace.join("\n")) 
+          #  end
+          #  t.send_completed_message(t.hash)
+          #end
         when "tell_verify"
           # We are a listener, and asked for verification of a transfer from a server.
           # After asking for verification, we stopped running, and must be restarted

data/lib/pdtp/client/http_client.rb

@@ -0,0 +1,114 @@
+# $Id: httpclient.rb 518 2007-08-30 10:17:02Z blackhedd $
+#
+# Author:: Francis Cianfrocca (gmail: blackhedd)
+# Homepage::  http://rubyeventmachine.com
+# Date:: 16 July 2006
+# 
+# See EventMachine and EventMachine::Connection for documentation and
+# usage examples.
+#
+#----------------------------------------------------------------------------
+#
+# Copyright (C) 2006-07 by Francis Cianfrocca. All Rights Reserved.
+# Gmail: blackhedd
+# 
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of either: 1) the GNU General Public License
+# as published by the Free Software Foundation; either version 2 of the
+# License, or (at your option) any later version; or 2) Ruby's License.
+# 
+# See the file COPYING for complete licensing information.
+#
+#---------------------------------------------------------------------------
+#
+# 
+
+# This version of HttpClient has been modified for use in DistribuStream
+# Notable changes:
+# - Moved into PDTP::Client namespace
+# - HTTP/1.1 switched to HTTP/1.0
+# - Introduced support for HTTP ranges
+# - Support for X headers
+
+module PDTP
+  class Client
+    class HttpClient < EventMachine::Protocols::HttpClient
+      # Override send_request to support the additional features we need
+      def send_request args
+        args[:verb] ||= args[:method] # Support :method as an alternative to :verb.
+        args[:verb] ||= :get # IS THIS A GOOD IDEA, to default to GET if nothing was specified?
+
+        verb = args[:verb].to_s.upcase
+        unless ["GET", "POST", "PUT", "DELETE", "HEAD"].include?(verb)
+          set_deferred_status :failed, {:status => 0} # TODO, not signalling the error type
+          return # NOTE THE EARLY RETURN, we're not sending any data.
+        end
+
+        request = args[:request] || "/"
+        unless request[0,1] == "/"
+          request = "/" + request
+        end
+
+        qs = args[:query_string] || ""
+        if qs.length > 0 and qs[0,1] != '?'
+          qs = "?" + qs
+        end
+
+        # Allow an override for the host header if it's not the connect-string.
+        port = args[:port]
+        host = args[:host_header] || "#{args[:host]}#{port}" || "_"
+        
+        # POST items.
+        postcontenttype = args[:contenttype] || "application/octet-stream"
+        postcontent = args[:content] || ""
+        raise "oversized content in HTTP POST" if postcontent.length > MaxPostContentLength
+
+        # ESSENTIAL for the request's line-endings to be CRLF, not LF. Some servers misbehave otherwise.
+        # TODO: We ASSUME the caller wants to send a 1.1 request. May not be a good assumption.
+        req = [
+          "#{verb} #{request}#{qs} HTTP/1.1",
+          "Host: #{host}",
+          "User-Agent: #{args[:agent] || 'Ruby EventMachine'}"
+        ]
+
+        if verb == "POST" || verb == "PUT"
+          req << "Content-Type: #{postcontenttype}"
+          req << "Content-Length: #{postcontent.length}"
+        end
+        
+        if args[:range]
+          # Convert the range to an array if it isn't one already
+          args[:range] = [args[:range]] unless args[:range].is_a?(Array)
+
+          # Transform it to a text string
+          range = args[:range].map { |v| v.is_a?(Range) ? "#{v.min}-#{v.max}" : (v >= 0 ? "#{v}-" : v.to_s) }.join(',')
+          req << "Range: bytes=#{range}"
+        end
+        
+        if args[:headers]
+          headers = case args[:headers]
+          when Array then args[:headers]
+          when String then [args[:headers]]
+          else raise ArgumentError, "headers must be an array or string"
+          end
+          
+          req += headers
+        end
+
+        # TODO, this cookie handler assumes it's getting a single, semicolon-delimited string.
+        # Eventually we will want to deal intelligently with arrays and hashes.
+        if args[:cookie]
+          req << "Cookie: #{args[:cookie]}"
+        end
+
+        req << ""
+        reqstring = req.map {|l| "#{l}\r\n"}.join
+        send_data reqstring
+
+        if verb == "POST" || verb == "PUT"
+          send_data postcontent
+        end
+      end
+    end
+  end
+end

data/lib/pdtp/client/transfer.rb

@@ -8,12 +8,15 @@
 # See http://distribustream.rubyforge.org/
 #++
 
-require File.dirname(__FILE__) + '/file_service'
 require "thread"
-require "net/http"
+#require "net/http"
 require "uri"
 require "digest/sha2"
 
+require File.dirname(__FILE__) + '/../common'
+require File.dirname(__FILE__) + '/file_service'
+require File.dirname(__FILE__) + '/http_client'
+
 module PDTP
   class Client
     module Transfer
@@ -33,10 +36,10 @@ module PDTP
 
         # Returns true if a server message matches this transfer
         def matches_message?(message)
-          @peer == message["peer"] and
-          @url == message["url"] and
+          @peer       == message["peer"] and
+          @url        == message["url"] and
           @byte_range == message["range"] and
-          @peer_id == message["peer_id"] 
+          @peer_id    == message["peer_id"]
         end
 
         # Takes an HTTP range and returns a ruby Range object
@@ -177,50 +180,57 @@ module PDTP
       # Implements http transfer between two peers from the connector's (client) perspective
       class Connector < Base
         def initialize(connection, message, file_service)
-          @file_service=file_service
-          @peer,@port=message["host"],message["port"]
-          @method = message["method"]
-          @url=message["url"]
-          @byte_range=message["range"]
-          @peer_id=message["peer_id"]
-          @connection=connection
+          @connection   = connection
+          @file_service = file_service
+          @peer         = message["host"]
+          @port         = message["port"]
+          @method       = message["method"]
+          @url          = message["url"]
+          @byte_range   = message["range"]
+          @peer_id      = message["peer_id"]
         end
 
         # Perform the transfer
         def run
-          hash=nil
-
-          info=@file_service.get_info(@url)
+          info = @file_service.get_info(@url)
 
           #compute the vhost and path
           #FIXME work with ports
-          uri=URI.split(@url)
-          path=uri[5]
-          vhost=uri[2]   
-
-          if @method == "get"
-            req = Net::HTTP::Get.new(path)
-            body = nil
-          elsif @method == "put"
-            req = Net::HTTP::Put.new(path)
-            body = info.read(@byte_range)
-          else
-            raise HTTPException.new(405,"Invalid method: #{@method}")
+          uri = URI.parse(@url)
+          
+          options = {
+            :method => @method,
+            :host => @peer,
+            :host_header => uri.host,
+            :port => @port,
+            :request => uri.path,
+            :range => @byte_range,
+            :agent => "DistribuStream #{PDTP::VERSION}",
+            :headers => "X-PDTP-Peer-Id: #{@connection.client.client_id}"
+          }
+          
+          case @method
+          when 'get'
+          when 'put' then options[:content] = info.read(@byte_range)
+          else raise HTTPException.new(405, "Invalid method: #{@method}")
+          end
+          
+          http = HttpClient.request options          
+          http.callback do |response|
+            hash = nil
+            
+            if @method == 'get' and response[:status] == 206
+              @@log.debug("Body Downloaded: url=#{@url} range=#{@byte_range} peer=#{@peer}:#{@port}")
+              info.write(@byte_range.first,response[:content])
+              hash = Digest::SHA256.hexdigest(response[:content]) rescue nil
+            elsif @method == 'put' and response[:status] == 200
+              @@log.debug("Body Uploaded: url=#{@url} range=#{@byte_range} peer=#{@peer}:#{@port}")
+            else
+              raise RuntimeError, "Invalid method/status combination: #{@method} #{response[:status]}"
+            end
+            
+            send_completed_message hash
           end
-
-          req.add_field("Range", "bytes=#{@byte_range.begin}-#{@byte_range.end}")
-          req.add_field("Host",vhost)
-          req.add_field("X-PDTP-Peer-Id", @connection.client.client_id)
-          res = Net::HTTP.start(@peer,@port) {|http| http.request(req,body) }
-
-          if res.code == '206' and @method == 'get'
-            #we are the taker
-            @@log.debug("Body Downloaded: url=#{@url} range=#{@byte_range} peer=#{@peer}:#{@port}")
-            info.write(@byte_range.first,res.body)
-            @hash=Digest::SHA256.hexdigest(res.body) rescue nil
-          else
-            raise "HTTP RESPONSE: code=#{res.code} body=#{res.body}"
-          end  
         end
       end
     end

data/lib/pdtp/common.rb

@@ -0,0 +1,15 @@
+#--
+# Copyright (C) 2006-07 ClickCaster, Inc. (info@clickcaster.com)
+# All rights reserved.  See COPYING for permissions.
+# 
+# This source file is distributed as part of the 
+# DistribuStream file transfer system.
+#
+# See http://distribustream.rubyforge.org/
+#++
+
+# Namespace for all PDTP components
+module PDTP
+  PDTP::VERSION = '0.2.1' unless defined? PDTP::VERSION
+  def self.version() VERSION end
+end

data/lib/pdtp/common/common_init.rb

@@ -30,7 +30,7 @@ CONFIG_TYPES = {
   :quiet            => :bool,
   :chunk_size       => :int,
   :request_url      => :string
-}
+} unless defined? CONFIG_TYPES
 
 #prints banner and loads config file
 def common_init(program_name, config = nil)
@@ -119,4 +119,4 @@ end
 #responds to config options that are used by both client and server
 def handle_config_options
   @@log.level=Logger::INFO if @@config[:quiet]  
-end
+end

data/lib/pdtp/common/{packet.rb → length_prefix_protocol.rb}

@@ -12,58 +12,71 @@ require 'rubygems'
 require 'eventmachine'
 
 module PDTP
-  # EventMachine connection adapter for sending and receiving frames with 16-bit length prefixes
-  class Packet < EventMachine::Connection
-    # Class for processing length prefixes in packet frames
+  # EventMachine connection adapter for length prefix framing
+  class LengthPrefixProtocol < EventMachine::Connection
+    # Class for processing size prefixes in packet frames
     class Prefix
-      attr_reader :length
+      attr_reader :size, :data
       
-      def initialize
-        # Constant at 2 bytes, but expandable if we desire
-        @length = 2
+      def initialize(size = 2)
+        unless size == 2 or size == 4
+          raise ArgumentError, 'only 2 or 4 byte prefixes are supported' 
+        end
+        
+        @size = size
         reset!
       end
       
+      # Has the entire prefix been read yet?
       def read?
-        @length == @read
+        @size == @read
       end
       
+      # Append data to the prefix and return any extra
       def append(data)
-        toread = @length - @read
+        toread = @size - @read
         new_data = data[0..(toread - 1)]
         
         @data << new_data        
-        @read += new_data.size
-        return nil unless @read == @length
+        @read += new_data.length
+        return nil unless @read == @size
         
-        @size = @data.unpack('n').first
-        result = data[toread..data.size]
+        @length = @data.unpack(@size == 2 ? 'n' : 'N').first
+        result = data[toread..data.length]
         
         return nil if result.nil? or result.empty?
         result
       end
       
-      def size
-        return false unless read?
-        @size
+      # Length of the payload extracted from the prefix
+      def length
+        raise RuntimeError, 'length called before prefix extracted' unless read?
+        @length
       end
       
       def reset!
-        @size = nil
+        @length = nil
         @read = 0
         @data = ''        
       end
     end
     
-    def initialize(*args)
+    def initialize *args
       super
+      
       @prefix = Prefix.new
       @buffer = ''
     end
     
+    def prefix_size=(size)
+      data = @prefix.data << @buffer
+      @prefix = Prefix.new size
+      receive_data data
+    end
+    
     # Callback for processing incoming frames
     def receive_data(data)
-      # Read data and append it to the length prefix unless it's already been read
+      # Read data and append it to the size prefix unless it's already been read
       data = @prefix.append(data) unless @prefix.read?
       return if data.nil?
       
@@ -71,14 +84,16 @@ module PDTP
       @buffer << data
       
       # Don't do anything until we receive the specified amount of data
-      return unless @buffer.size >= @prefix.size
+      return unless @buffer.length >= @prefix.length
       
       # Extract the specified amount of data and process it
-      data = @buffer[0..(@prefix.size - 1)]      
-      receive_packet data
+      data = @buffer[0..(@prefix.length - 1)]
       
       # Store any remaining data
-      remainder = @buffer[@prefix.size..@buffer.size]
+      remainder = @buffer[@prefix.length..@buffer.length]
+      
+      # Invoke receive_packet and allow the user to process the data
+      receive_packet data
       
       # Reset the prefix and buffer since we've received a whole frame
       @prefix.reset!
@@ -95,12 +110,10 @@ module PDTP
     def receive_packet(packet)
     end
     
-    # Send a packet with a specified length prefix
+    # Send a packet with a specified size prefix
     def send_packet(data)
-      length = data.size
-      raise ArgumentError, 'oversize packet' if length >= 256**@prefix.length
-      
-      send_data [length].pack('n') << data
+      raise ArgumentError, 'packet too long for prefix length' if data.size >= 256**@prefix.size
+      send_data [data.size].pack(@prefix.size == 2 ? 'n' : 'N') << data
     end
   end
-end
+end

data/lib/pdtp/common/protocol.rb

@@ -19,10 +19,10 @@ rescue LoadError
   require 'json'
 end
 
-require File.dirname(__FILE__) + '/packet.rb'
+require File.dirname(__FILE__) + '/length_prefix_protocol.rb'
 
 module PDTP
-  PROTOCOL_DEBUG=true
+  PDTP::PROTOCOL_DEBUG = true unless defined? PDTP::PROTOCOL_DEBUG
 
   class ProtocolError < Exception
   end
@@ -31,7 +31,7 @@ module PDTP
   end
 
   # EventMachine handler class for the PDTP protocol
-  class Protocol < PDTP::Packet
+  class Protocol < PDTP::LengthPrefixProtocol
     @@num_connections = 0
     @@message_params = nil
     @connection_open = false
@@ -40,10 +40,10 @@ module PDTP
       @connection_open
     end
 
-    def initialize(data)
+    def initialize *args
       user_data = nil
       @mutex = Mutex.new
-      super data
+      super
     end
 
     #called by EventMachine after a connection has been established
@@ -67,8 +67,8 @@ module PDTP
     #close a connection, but first send the specified error message
     def error_close_connection(error) 
       if PROTOCOL_DEBUG
-        send_message :protocol_error, :message => msg 
-        close_connection(true) # close after writing
+        send_message :protocol_error, :message => error
+        close_connection true # close after writing
       else
         close_connection
       end
@@ -105,7 +105,7 @@ module PDTP
       end
     end
 
-    RANGENAMES = %w{chunk_range range byte_range}
+    PDTP::RANGENAMES = %w{chunk_range range byte_range} unless defined? PDTP::RANGENAMES
 
     #converts Ruby Range classes in the message to PDTP protocol hashes with min and max
     # 0..-1 => nil  (entire file)
@@ -333,4 +333,4 @@ module PDTP
       mp
     end
   end
-end
+end

data/lib/pdtp/server.rb

@@ -12,6 +12,7 @@ require 'rubygems'
 require 'eventmachine'
 require 'mongrel'
 
+require File.dirname(__FILE__) + '/common'
 require File.dirname(__FILE__) + '/server/dispatcher'
 require File.dirname(__FILE__) + '/server/file_service'
 require File.dirname(__FILE__) + '/server/connection'

data/lib/pdtp/server/file_service_protocol.rb

@@ -24,14 +24,14 @@ module PDTP
       class Protocol < PDTP::Client::Connection
         attr_reader :client, :connection, :file_service, :client_id, :transfers, :lock
         
-        def initialize(data)
+        def initialize *args
           @transfers = []
           @lock = Mutex.new
           @client = self
           @connection = self
           @client_id = Digest::MD5.hexdigest "#{Time.now.to_f}#{$$}"
           
-          super data
+          super
         end
         
         # Called after a connection to the server has been established
@@ -113,4 +113,4 @@ module PDTP
       end
     end
   end
-end
+end

metadata

@@ -1,9 +1,9 @@
 --- !ruby/object:Gem::Specification 
-rubygems_version: 0.9.0
+rubygems_version: 0.9.4
 specification_version: 1
 name: distribustream
 version: !ruby/object:Gem::Version 
-  version: 0.2.0
+  version: 0.2.1
 date: 2008-10-22 00:00:00 -06:00
 summary: DistribuStream is a fully open peercasting system allowing on-demand or live streaming media to be delivered at a fraction of the normal cost
 require_paths: 
@@ -38,20 +38,21 @@ files:
 - bin/dstream
 - lib/pdtp
 - lib/pdtp/client
-- lib/pdtp/client.rb
-- lib/pdtp/common
-- lib/pdtp/server
-- lib/pdtp/server.rb
 - lib/pdtp/client/callbacks.rb
 - lib/pdtp/client/connection.rb
 - lib/pdtp/client/file_buffer.rb
 - lib/pdtp/client/file_service.rb
+- lib/pdtp/client/http_client.rb
 - lib/pdtp/client/http_handler.rb
 - lib/pdtp/client/transfer.rb
+- lib/pdtp/client.rb
+- lib/pdtp/common
 - lib/pdtp/common/common_init.rb
 - lib/pdtp/common/file_service.rb
-- lib/pdtp/common/packet.rb
+- lib/pdtp/common/length_prefix_protocol.rb
 - lib/pdtp/common/protocol.rb
+- lib/pdtp/common.rb
+- lib/pdtp/server
 - lib/pdtp/server/client_info.rb
 - lib/pdtp/server/connection.rb
 - lib/pdtp/server/dispatcher.rb
@@ -60,6 +61,7 @@ files:
 - lib/pdtp/server/stats_handler.rb
 - lib/pdtp/server/transfer.rb
 - lib/pdtp/server/trust.rb
+- lib/pdtp/server.rb
 - conf/bigchunk.yml
 - conf/debug.yml
 - conf/example.yml