distribustream 0.2.1 → 0.3.0

Sign up to get free protection for your applications and to get access to all the features.
data/CHANGES CHANGED
@@ -1,3 +1,21 @@
1
+ Version 0.3.0
2
+ * dsseed now factored into server.rb. Remote seeds will not be supported until
3
+ protocol changes can be made to support authenticating remote seeds, remote
4
+ file lookup, and remote hash checking.
5
+
6
+ * Switch to Kirk Haines' Evented Mongrel and remove all usage of threads from
7
+ the entire application
8
+
9
+ * Stats page relocated to /stats. This allows a single mongrel to operate for
10
+ both the file service and for the stats page.
11
+
12
+ * Remove common_init.rb and any dependency on the @@log and @@config variables
13
+
14
+ * Add logging facility to the server application to replace @@log. Similar
15
+ facilities should be added for clients.
16
+
17
+ * Refactor PDTP::Client and PDTP::Server APIs to support the above changes.
18
+
1
19
  Version 0.2.1
2
20
  * Switch HTTP client to use an EventMachine-based version, rather than net/http
3
21
 
data/README CHANGED
@@ -3,14 +3,6 @@ Welcome to DistribuStream!
3
3
  DistribuStream is a fully open peercasting system which allows on-demand
4
4
  or live streaming media to be delivered at a fraction of the normal cost.
5
5
 
6
- This README covers the initial public release, known issues, and a general
7
- development roadmap.
8
-
9
- Contents:
10
- 1. Usage
11
- 2. Known Issues
12
- 3. Development Roadmap
13
-
14
6
  --
15
7
 
16
8
  Usage:
@@ -34,22 +26,15 @@ Next, start the DistribuStream server:
34
26
 
35
27
  dstream --conf myconfig.yml
36
28
 
37
- The DistribuStream server manages traffic on the peer network. It also handles
38
- the checksumming of files.
29
+ You can think of the DistribuStream server like a regular web or FTP server.
30
+ It looks through the specified directory and makes the files underneath it
31
+ available to the peer-to-peer network.
39
32
 
40
33
  You can see what's going on through the web interface, which runs one port
41
34
  above the port you configured the server to listen on. If the server is
42
35
  listening on the default port of 6086, you can reach the web interface at:
43
36
 
44
- http://myserver.url:6087/
45
-
46
- To populate the server with files, you need to attach the DistribuStream
47
- seed. This works off of the same config file as the server, and must
48
- be run on the same system:
49
-
50
- dsseed --conf myconfig.yml
51
-
52
- At this point your server is ready to go.
37
+ http://myserver.url:6087/stats
53
38
 
54
39
  To test your server, use the DistribuStream client:
55
40
 
@@ -64,30 +49,9 @@ dsclient -o pdtp://myserver.url/file.ext | mediaplayer -
64
49
 
65
50
  --
66
51
 
67
- Known Issues:
68
-
69
- Seeds are presently not authenticated in any way, thus anyone can attach
70
- a seed and populate the server with any files of their choosing. However,
71
- since file checksumming is done by the server itself, this means that only
72
- seeds running on the same system as the server will actually work.
73
-
74
- This will be resolved by either incorporating the seed directly into the
75
- DistribuStream server, or adding both authentication and commands for
76
- checksumming to the server <-> seed protocol.
77
-
78
- --
79
-
80
52
  Development Roadmap:
81
53
 
82
- DistribuStream uses an assemblage of various tools which do not work together
83
- particularly well. These include the EventMachine Ruby gem, which provides
84
- the I/O layer for the DistribuStream server, and the Mongrel web server, which
85
- runs independently of EventMachine and uses threads.
86
-
87
- Initial work will focus on converting the existing implementation to a fully
88
- EventMachine-based approach which eliminates the use of threads.
89
-
90
- Mid-term work will focus on improving the efficiency of peer-to-peer traffic
54
+ Short-term goals focus on improving the efficiency of peer-to-peer traffic
91
55
  routing, by incorporating all of the following constraints:
92
56
 
93
57
  1. Does the potential "giver" peer have chunks the "taker" peer is requesting?
@@ -21,12 +21,24 @@
21
21
 
22
22
  #--
23
23
  # Copyright (C) 2006-07 ClickCaster, Inc. (info@clickcaster.com)
24
- # All rights reserved. See COPYING for permissions.
24
+ #
25
+ # This program is free software: you can redistribute it and/or modify
26
+ # it under the terms of the GNU General Public License as published by
27
+ # the Free Software Foundation, either version 3 of the License, or
28
+ # (at your option) any later version.
29
+ #
30
+ # This program is distributed in the hope that it will be useful,
31
+ # but WITHOUT ANY WARRANTY; without even the implied warranty of
32
+ # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
33
+ # GNU General Public License for more details.
34
+ #
35
+ # You should have received a copy of the GNU General Public License
36
+ # along with this program. If not, see <http://www.gnu.org/licenses/>.
25
37
  #
26
38
  # This source file is distributed as part of the
27
39
  # DistribuStream file transfer system.
28
40
  #
29
- # See http://distribustream.rubyforge.org/
41
+ # See http://distribustream.org/
30
42
  #++
31
43
 
32
44
  require 'uri'
@@ -86,22 +98,23 @@ class Callbacks < PDTP::Client::Callbacks
86
98
  attr_accessor :uri
87
99
  attr_accessor :output
88
100
 
89
- def connected
101
+ def initialize(uri, output)
102
+ @uri, @output = uri, output
103
+ end
104
+
105
+ def connected(client)
90
106
  STDERR.write "Connected to #{uri.host}\n"
91
107
  STDERR.write "Beginning transfer...\n"
92
108
 
93
109
  # Begin fetching the file once we've connected
94
- client.get uri.path, output
110
+ client.get @uri.path, @output
95
111
  end
96
112
 
97
- def disconnected
113
+ def disconnected(client)
98
114
  STDERR.write "Disconnected from #{uri.host}\n"
99
115
  client.stop
100
116
  end
101
117
  end
102
-
103
- @@log = Logger.new(STDERR)
104
- @@log.level=Logger::INFO
105
118
 
106
- @client = PDTP::Client.new uri.host, uri.port || 6086, :listen_port => listen_port
107
- @client.connect(Callbacks) { |c| c.uri, c.output = uri, output }
119
+ client = PDTP::Client.new uri.host, uri.port || 6086, :listen_port => listen_port
120
+ client.connect Callbacks, uri, output
@@ -1,20 +1,83 @@
1
1
  #!/usr/bin/env ruby
2
2
  #--
3
3
  # Copyright (C) 2006-07 ClickCaster, Inc. (info@clickcaster.com)
4
- # All rights reserved. See COPYING for permissions.
4
+ #
5
+ # This program is free software: you can redistribute it and/or modify
6
+ # it under the terms of the GNU General Public License as published by
7
+ # the Free Software Foundation, either version 3 of the License, or
8
+ # (at your option) any later version.
9
+ #
10
+ # This program is distributed in the hope that it will be useful,
11
+ # but WITHOUT ANY WARRANTY; without even the implied warranty of
12
+ # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
13
+ # GNU General Public License for more details.
14
+ #
15
+ # You should have received a copy of the GNU General Public License
16
+ # along with this program. If not, see <http://www.gnu.org/licenses/>.
5
17
  #
6
18
  # This source file is distributed as part of the
7
19
  # DistribuStream file transfer system.
8
20
  #
9
- # See http://distribustream.rubyforge.org/
21
+ # See http://distribustream.org/
10
22
  #++
11
23
 
12
- require File.dirname(__FILE__) + '/../lib/pdtp/common/common_init'
24
+ require 'optparse'
13
25
  require File.dirname(__FILE__) + '/../lib/pdtp/server'
14
26
 
15
- common_init File.basename($0)
27
+ program_name = File.basename($0)
28
+ config_filename = nil
29
+
30
+ OptionParser.new do |opts|
31
+ opts.banner = "Usage: #{program_name} [options]"
32
+ opts.on("--config CONFIGFILE", "Load specified config file.") do |c|
33
+ config_filename = c
34
+ end
35
+ opts.on("--help", "Prints this usage info.") do
36
+ puts opts
37
+ exit
38
+ end
39
+ end.parse!
40
+
41
+ config = {
42
+ :host => '0.0.0.0',
43
+ :port => 6086, #server port
44
+ :listen_port => 8000, #client listen port
45
+ :file_root => '.',
46
+ :chunk_size => 5000,
47
+ :quiet => true
48
+ }
49
+
50
+ if config_filename.nil?
51
+ puts "No config file specified. Using defaults."
52
+ else
53
+ begin
54
+ confstr = File.read(config_filename)
55
+ rescue
56
+ puts "Unable to open config file: #{config_filename}"
57
+ exit
58
+ end
59
+
60
+ begin
61
+ new_config = YAML.load confstr
62
+ config.merge!(new_config)
63
+ config[:vhost] ||= config[:host] # Use host as vhost unless specified
64
+ rescue Exception => e
65
+ puts "Error parsing config file: #{config_filename}"
66
+ puts e
67
+ exit
68
+ end
69
+ end
70
+
71
+ puts "#{program_name} starting. Run '#{program_name} --help' for more info."
72
+
73
+ server = PDTP::Server.new config[:host], config[:port]
74
+
75
+ STDERR.sync = true
76
+ logger = Logger.new STDERR
77
+ logger.datetime_format = "%H:%M:%S "
78
+ logger.level = Logger::INFO if config[:quiet]
16
79
 
17
- server = PDTP::Server.new @@config[:host], @@config[:port]
18
- server.enable_stats_service
19
- server.enable_file_service @@config[:file_root], :chunk_size => @@config[:chunk_size]
80
+ server.enable_logging logger
81
+ server.enable_status_page
82
+ server.enable_file_service config[:file_root], :chunk_size => config[:chunk_size]
20
83
  server.run
@@ -2,19 +2,19 @@ require 'rubygems'
2
2
 
3
3
  GEMSPEC = Gem::Specification.new do |s|
4
4
  s.name = "distribustream"
5
- s.version = "0.2.1"
6
- s.date = "2008-10-22"
5
+ s.version = "0.3.0"
6
+ s.date = "2008-11-09"
7
7
  s.summary = "DistribuStream is a fully open peercasting system allowing on-demand or live streaming media to be delivered at a fraction of the normal cost"
8
8
  s.email = "tony@clickcaster.com"
9
- s.homepage = "http://distribustream.rubyforge.org"
9
+ s.homepage = "http://distribustream.org"
10
10
  s.rubyforge_project = "distribustream"
11
11
  s.has_rdoc = true
12
12
  s.rdoc_options = ["--exclude", "definitions", "--exclude", "indexes"]
13
- s.extra_rdoc_files = ["COPYING", "README", "CHANGES"]
13
+ s.extra_rdoc_files = ["COPYING", "README", "CHANGES", "pdtp-specification.xml"]
14
14
  s.authors = ["Tony Arcieri", "Ashvin Mysore", "Galen Pahlke", "James Sanders", "Tom Stapleton"]
15
15
  s.files = Dir.glob("{bin,lib,conf}/**/*") + ['Rakefile', 'distribustream.gemspec']
16
- s.executables = %w{dstream dsseed dsclient}
16
+ s.executables = %w{dstream dsclient}
17
17
  s.add_dependency("eventmachine", ">= 0.9.0")
18
- s.add_dependency("mongrel", ">= 1.0.1")
18
+ s.add_dependency("mongrel", ">= 1.0.2")
19
19
  s.add_dependency("json", ">= 1.1.0")
20
20
  end
@@ -1,19 +1,31 @@
1
1
  #--
2
2
  # Copyright (C) 2006-07 ClickCaster, Inc. (info@clickcaster.com)
3
- # All rights reserved. See COPYING for permissions.
3
+ #
4
+ # This program is free software: you can redistribute it and/or modify
5
+ # it under the terms of the GNU General Public License as published by
6
+ # the Free Software Foundation, either version 3 of the License, or
7
+ # (at your option) any later version.
8
+ #
9
+ # This program is distributed in the hope that it will be useful,
10
+ # but WITHOUT ANY WARRANTY; without even the implied warranty of
11
+ # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12
+ # GNU General Public License for more details.
13
+ #
14
+ # You should have received a copy of the GNU General Public License
15
+ # along with this program. If not, see <http://www.gnu.org/licenses/>.
4
16
  #
5
17
  # This source file is distributed as part of the
6
18
  # DistribuStream file transfer system.
7
19
  #
8
- # See http://distribustream.rubyforge.org/
20
+ # See http://distribustream.org/
9
21
  #++
10
22
 
11
23
  require 'rubygems'
12
24
  require 'eventmachine'
13
- require 'thread'
14
25
  require 'digest/md5'
15
26
 
16
27
  require File.dirname(__FILE__) + '/common'
28
+ require File.dirname(__FILE__) + '/common/http_server'
17
29
  require File.dirname(__FILE__) + '/client/connection'
18
30
  require File.dirname(__FILE__) + '/client/callbacks'
19
31
  require File.dirname(__FILE__) + '/client/file_service'
@@ -23,10 +35,10 @@ module PDTP
23
35
  # PDTP::Client provides an interface for accessing resources on PDTP servers
24
36
  class Client
25
37
  # None of these should be publically accessible, and will be factored away in time
26
- attr_reader :connection, :client_id, :listen_port, :file_service, :transfers, :lock
38
+ attr_reader :connection, :client_id, :listen_port, :file_service, :transfers
27
39
 
28
40
  # Create a PDTP::Client object for accessing the PDTP server at the given host and port
29
- def initialize(host, port = 6086, options = {})
41
+ def initialize(host, port = PDTP::DEFAULT_PORT, options = {})
30
42
  @host, @port = host, port
31
43
 
32
44
  @listen_addr = options[:listen_addr] || '0.0.0.0'
@@ -34,45 +46,49 @@ module PDTP
34
46
 
35
47
  @file_service = PDTP::Client::FileService.new
36
48
  @transfers = []
37
- @lock = Mutex.new
38
49
 
39
50
  # Start a Mongrel server on the specified port. If it isnt available, keep trying higher ports
40
- begin
51
+ # FIXME: Evented Mongrel won't throw an exception if the port is unavailable. This needs to
52
+ # be dealt with through EventMachine
53
+ #begin
41
54
  @http_server = Mongrel::HttpServer.new @listen_addr, @listen_port
42
- rescue Exception => e
43
- @listen_port += 1
44
- retry
45
- end
55
+ #rescue
56
+ # @listen_port += 1
57
+ # retry
58
+ #end
46
59
 
47
60
  @client_id = Digest::MD5.hexdigest "#{Time.now.to_f}#{$$}"
48
61
 
49
62
  #@@log.info "listening on port #{@listen_port}"
50
63
  @http_handler = HttpHandler.new(self)
51
64
  @http_server.register '/', @http_handler
52
- @http_server.run
53
65
  end
54
66
 
55
67
  # Connect to the PDTP server. This is a blocking call which runs the client event loop
56
- def connect(callbacks = nil)
68
+ def connect(callbacks = nil, *args)
57
69
  klass = if(callbacks and callbacks.is_a?(Class))
58
70
  callbacks
59
71
  else
60
72
  Class.new(Callbacks) {callbacks and include callbacks}
61
73
  end
62
74
 
75
+ arity = klass.instance_method(:initialize).arity
76
+ expected = arity >= 0 ? arity : -(arity + 1)
77
+ if (arity >= 0 and args.size != expected) or (arity < 0 and args.size < expected)
78
+ raise ArgumentError, "wrong number of arguments for #{klass}#initialize (#{args.size} for #{expected})"
79
+ end
80
+
63
81
  # Run the EventMachine reactor loop
64
82
  EventMachine.run do
83
+ @http_server.run_evented
84
+
65
85
  EventMachine.connect(@host, @port, Connection) do |c|
66
86
  # Store reference to the connection
67
87
  @connection = c
68
88
 
69
89
  # Set connection variables and configure callbacks
70
90
  c.client = self
71
- c.callbacks = klass.new(self)
72
- c.callbacks.client = self
73
-
74
- # Allow users to populate instance variables in their callback class
75
- yield c.callbacks if block_given?
91
+ c.callbacks = klass.new *args
76
92
  end
77
93
 
78
94
  #@@log.info "connecting with ev=#{EventMachine::VERSION}"
@@ -110,4 +126,4 @@ module PDTP
110
126
  EventMachine.stop_event_loop
111
127
  end
112
128
  end
113
- end
129
+ end
@@ -1,27 +1,38 @@
1
1
  #--
2
2
  # Copyright (C) 2006-07 ClickCaster, Inc. (info@clickcaster.com)
3
- # All rights reserved. See COPYING for permissions.
3
+ #
4
+ # This program is free software: you can redistribute it and/or modify
5
+ # it under the terms of the GNU General Public License as published by
6
+ # the Free Software Foundation, either version 3 of the License, or
7
+ # (at your option) any later version.
8
+ #
9
+ # This program is distributed in the hope that it will be useful,
10
+ # but WITHOUT ANY WARRANTY; without even the implied warranty of
11
+ # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12
+ # GNU General Public License for more details.
13
+ #
14
+ # You should have received a copy of the GNU General Public License
15
+ # along with this program. If not, see <http://www.gnu.org/licenses/>.
4
16
  #
5
17
  # This source file is distributed as part of the
6
18
  # DistribuStream file transfer system.
7
19
  #
8
- # See http://distribustream.rubyforge.org/
20
+ # See http://distribustream.org/
9
21
  #++
10
22
 
11
23
  module PDTP
12
24
  class Client
13
- # Callbacks
25
+ # Callbacks which can be overridden by clients to handle events
14
26
  class Callbacks
15
27
  attr_accessor :client
16
28
 
17
- def initialize(connection)
18
- @connection = connection
29
+ def initialize(*args)
19
30
  end
20
31
 
21
- def connected
32
+ def connected(client)
22
33
  end
23
34
 
24
- def disconnected
35
+ def disconnected(client)
25
36
  client.stop
26
37
  end
27
38
  end
@@ -1,11 +1,23 @@
1
1
  #--
2
2
  # Copyright (C) 2006-07 ClickCaster, Inc. (info@clickcaster.com)
3
- # All rights reserved. See COPYING for permissions.
3
+ #
4
+ # This program is free software: you can redistribute it and/or modify
5
+ # it under the terms of the GNU General Public License as published by
6
+ # the Free Software Foundation, either version 3 of the License, or
7
+ # (at your option) any later version.
8
+ #
9
+ # This program is distributed in the hope that it will be useful,
10
+ # but WITHOUT ANY WARRANTY; without even the implied warranty of
11
+ # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12
+ # GNU General Public License for more details.
13
+ #
14
+ # You should have received a copy of the GNU General Public License
15
+ # along with this program. If not, see <http://www.gnu.org/licenses/>.
4
16
  #
5
17
  # This source file is distributed as part of the
6
18
  # DistribuStream file transfer system.
7
19
  #
8
- # See http://distribustream.rubyforge.org/
20
+ # See http://distribustream.org/
9
21
  #++
10
22
 
11
23
  require 'rubygems'
@@ -32,13 +44,22 @@ module PDTP
32
44
  :client_id => @client.client_id
33
45
  )
34
46
 
35
- callbacks.connected
47
+ callbacks.connected client
36
48
  rescue Exception => e
37
49
  puts "Exception in connection_completed: #{e}"
38
50
  puts e.backtrace.join("\n")
39
51
  exit
40
52
  end
41
- end
53
+ end
54
+
55
+ # Called when the connection to the server closes
56
+ def unbind
57
+ if callbacks
58
+ callbacks.disconnected client
59
+ else
60
+ puts "Disconnected from PDTP server"
61
+ end
62
+ end
42
63
 
43
64
  # Called when any server message is received. This is the brains of
44
65
  # the client's protocol handling.
@@ -60,17 +81,7 @@ module PDTP
60
81
 
61
82
  transfer.run
62
83
 
63
- #@@log.debug "TRANSFER STARTING"
64
-
65
- # Run each transfer in its own thread and notify the server upon completion
66
- #Thread.new(transfer) do |t|
67
- # begin
68
- # t.run
69
- # rescue Exception=>e
70
- # @@log.info("Exception in dispatch_message: " + e.exception + "\n" + e.backtrace.join("\n"))
71
- # end
72
- # t.send_completed_message(t.hash)
73
- #end
84
+ ##@@log.debug "TRANSFER STARTING"
74
85
  when "tell_verify"
75
86
  # We are a listener, and asked for verification of a transfer from a server.
76
87
  # After asking for verification, we stopped running, and must be restarted
@@ -91,25 +102,20 @@ module PDTP
91
102
  exit!
92
103
  end
93
104
  when "hash_verify"
94
- @@log.debug "Hash verified for url=#{message["url"]} range=#{message["range"]} hash_ok=#{message["hash_ok"]}"
105
+ #@@log.debug "Hash verified for url=#{message["url"]} range=#{message["range"]} hash_ok=#{message["hash_ok"]}"
95
106
  when "protocol_error", "protocol_warn" #ignore
96
107
  else raise "Server sent an unknown message type: #{command} "
97
108
  end
98
109
  end
99
-
100
- def unbind
101
- super
102
- puts 'Disconnected from PDTP server.'
103
- end
104
110
 
105
- #Prints the number of transfers associated with this client
111
+ # Prints the number of transfers associated with this client
106
112
  def print_stats
107
- @@log.debug "client: num_transfers=#{@client.transfers.size}"
113
+ #@@log.debug "client: num_transfers=#{@client.transfers.size}"
108
114
  end
109
115
 
110
- #Provides a threadsafe mechanism for transfers to report themselves finished
116
+ # Transfers to report themselves finished
111
117
  def finished(transfer)
112
- @client.lock.synchronize { @client.transfers.delete(transfer) }
118
+ @client.transfers.delete(transfer)
113
119
  end
114
120
  end
115
121
  end