adhd 0.0.1 → 0.1.0

data/README.rdoc

@@ -1,9 +1,68 @@
 = adhd
 
-Description goes here.
+Adhd is an asynchronous, distributed hard drive.  Actually we're not sure if it's really asynchronous or even what asynchronicity would mean in the context of a hard drive, but it is definitely distributed.  Adhd is essentially a management layer (written using Ruby and eventmachine) which controls clusters of CouchDB databases to replicate files across disparate machines.  Unlike most clustering storage solutions, adhd assumes that machines in the cluster may have different amounts of bandwidth available and is designed to work both inside and outside the data centre.
+
+== Installation
+
+Don't use this software yet.  It is experimental and may eat your mother.
+
+Having said that, <pre>sudo gem install adhd</pre> may do the job.
+
+== How it works
+
+=== External API
+
+The user issues an HTTP PUT request to a ADHD url.  This request can go to any machine in the ADHD cluster.  A file is attached to the body of the request, and ADHD stores the file on a number of different servers.  At a later time, the user can issue a GET request to the same url and get back their file.
+
+
+=== Under the Hood
+
+==== Overview
+
+Multiple server nodes are part of an ADHD network.  Each node has a full copy of two CouchDB databases: the node management database, and the shard range database.
+
+The node management database keeps a record of all existing nodes, their location (IP), and current status.  This database is replicated across all nodes in the cluster;  any node is in theory capable of becoming a management node if the current management nodes become unavailable.
+
+The shard range database is also replicated everywhere.  It allows any node to find out which storage nodes are responsible for holding a particular piece of content.  Management nodes are in charge of maintaining the shard database and pushing changes to non-management nodes.
+
+Storage nodes hold one or more shards in addition to the management and shard range databases.  These shards contain the content and some metadata.  A shard consists of a CouchDB database which holds Couch documents - one attached file per document.
+
+==== PUT requests
+
+A PUT request reaches any ADHD node.  The first line of the request is parsed, and the ADHD node figures out the name, MIME type and content length of the file based on the incoming HTTP request headers.  Based on the MD5 hash of the filename, we extract a 20-byte internal ID string for the file and use this string to figure out which shard range the file will be assigned to.
+
+Once we have a shard range, we ask the shard range database which node(s) are in charge of storage for the given shard and are currently available. One of these nodes is chosen (starting with a master node and falling back to other nodes as necessary).  The file metadata is written as a CouchDB document to the chosen node, followed by the file as an attachment.
+
+==== GET requests
+
+The GET request is basically the same as the PUT request, except that the file is retrieved from the CouchDB shard instead of being created on a shard.
+
+==== Replication
+
+Replication happens differently for different databases.  The node management database and the shard range database are periodically synced to and from the management nodes.  The content shards are replicated when they get updated (i.e. when a file is added to a node which is responsible for a given shard, the file will be replicated to all other nodes which are responsible for the same shard).
+
+== Rationale
+
+We can have multiple redundant nodes storing the same files, getting rid of the need for backups.  Not every node needs to store every file, so we can scale up storage capacity by adding more nodes. The design also provides load balancing as it means that we can serve files from multiple nodes.
+
+Because all admin information is shared by every node, the unavailability of any node does not jeopardize the operation of the system.  As soon as a node becomes unavailable, any other node can perform its functions (although we will lose storage capacity throughout the cluster if we lose multiple large storage nodes).
+
+=== Some vapourware objectives
+
+The design also allows for different capabilities in storage nodes.  Shards can be assigned based on specific properties (available bandwidth, available processing power, etc) so that we can store files in the most efficient possible way.  It might be desirable, for example, to put all video files on machines with fast processors for doing video encoding, whereas audio and photos wouldn't need any post-processing and could go somewhere else.  Or we could store new and popular content in shard ranges which are on fast servers with high throughput, while putting less popular content on servers with less bandwidth (and lower storage costs).  Ideally we would like to get to a point where low-demand files can be stored on relatively low-bandwidth home or office network connections and still be available in the cluster.
+
+
+== Fictional use cases (no one has ever used this software in real life)
+
+Wikipedia: photos of Cheryl Cole and Robbie Williams are unfortunately wildly popular at present and will be requested often.  Photos of the millions of singers with more talent and less marketing budgets will not be requested very often, but it's still good if they are in the cluster.  If by some stroke of good luck Kevin Quain suddenly gets the recognition he deserves, his photo will be shifted from the "dial-up" node class to something with more stomp.
+
+Archive.org: those sex-ed videos and Bert the turtle from the Prelinger Archive are awesome and everybody wants to watch them, so they should be on beefy video-storage nodes with high bandwidth.  Documentaries on the yellow-bellied sapsucker may be requested only once in a while.
+
+BBC News: material going up on the site right now will need a lot of bandwidth, but by next week most of this media will be out of the news cycle and can be consigned to a set of servers with much less bandwidth.  By next year, it is unlikely that this media will be viewed even a few times a day, so it could be smart to trade storage costs for speed and put old media on cheaper, lower-bandwidth boxes with huge hard drives.
+
 
 == Note on Patches/Pull Requests
- 
+
 * Fork the project.
 * Make your feature addition or bug fix.
 * Add tests for it. This is important so I don't break it in a
@@ -16,3 +75,4 @@ Description goes here.
 == Copyright
 
 Copyright (c) 2009 dave@netbook. See LICENSE for details.
+

data/VERSION

@@ -1 +1 @@
-0.0.1
+0.1.0

data/adhd.gemspec

@@ -5,15 +5,14 @@
 
 Gem::Specification.new do |s|
   s.name = %q{adhd}
-  s.version = "0.0.1"
+  s.version = "0.1.0"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["dave.hrycyszyn@headlondon.com"]
-  s.date = %q{2009-12-19}
-  s.default_executable = %q{adhd}
+  s.date = %q{2009-12-27}
   s.description = %q{More to say when something works! Do not bother installing this! }
   s.email = %q{dave.hrycyszyn@headlondon.com}
-  s.executables = ["adhd"]
+  s.executables = ["adhd", "adhd_cleanup"]
   s.extra_rdoc_files = [
     "LICENSE",
      "README.rdoc"
@@ -27,11 +26,15 @@ Gem::Specification.new do |s|
      "VERSION",
      "adhd.gemspec",
      "bin/adhd",
+     "bin/adhd_cleanup",
      "doc/adhd.xmi",
-     "lib/adhd.rb",
+     "lib/adhd/adhd_rest_server.rb",
      "lib/adhd/config.yml",
-     "lib/adhd/models.rb",
-     "lib/adhd/node.rb",
+     "lib/adhd/models/content_doc.rb",
+     "lib/adhd/models/content_shard.rb",
+     "lib/adhd/models/node_doc.rb",
+     "lib/adhd/models/shard_range.rb",
+     "lib/adhd/node_manager.rb",
      "lib/adhd/reactor.rb",
      "lib/ext/hash_to_openstruct.rb",
      "lib/public/images/img01.jpg",
@@ -53,7 +56,6 @@ Gem::Specification.new do |s|
      "lib/public/style.css",
      "lib/views/index.erb",
      "lib/views/layout.erb",
-     "models.rb",
      "test/helper.rb",
      "test/test_adhd.rb"
   ]

data/bin/adhd

@@ -1,33 +1,48 @@
 #!/usr/bin/env ruby
 require File.dirname(__FILE__) + '/../lib/ext/hash_to_openstruct'
-require File.dirname(__FILE__) + '/../lib/adhd/node'
+require File.dirname(__FILE__) + '/../lib/adhd/node_manager'
 require File.dirname(__FILE__) + '/../lib/adhd/reactor'
+require File.dirname(__FILE__) + '/../lib/adhd/adhd_rest_server'
 
 require 'optparse'
 require 'ftools'
 require 'yaml'
-require 'socket'
+# require 'socket'
 
 def parse_config(file)
   @config = YAML.load_openstruct(File.read(file))
 end
 
-@command = ARGV.shift
+# @command = ARGV.shift
 
-options = {}
+#options = {}
 
 opts = OptionParser.new do |opts|
-  opts.on("-C", "--config C", "YAML config file") do |n|
+  opts.on("-c", "--config C", "YAML config file") do |n|
+    puts "Parsing config file #{n}"
     parse_config(n)
   end
 end
 
 opts.parse! ARGV
 
-@node = Adhd::Node.new(@config)
+@node_manager = Adhd::NodeManager.new(@config)
+
+
+# Start the Thin server within the reactor loop
 
 EM.run {
-  puts "Starting EventMachine reactor loop..."
-  EM.connect @config.node_url, @config.couchdb_server_port, Adhd::DbUpdateReactor, @node
+  # puts "Starting EventMachine reactor loop..."
+  # EM.connect @config.node_url, @config.couchdb_server_port, Adhd::DbUpdateNotifier, @node_manager
+  timer = EventMachine::PeriodicTimer.new(5) do
+    # puts "Sync Admin"
+    @node_manager.sync_admin
+    @node_manager.run
+  end
+  
+  # Start the server
+  EventMachine::start_server @config.node_url, @config.couchdb_server_port + 1, AdhdRESTServer, @node_manager
+
+  
 }
 

data/bin/adhd_cleanup

@@ -0,0 +1,57 @@
+#!/usr/bin/env ruby
+
+# Usage: adhd_cleanup nodename
+#        Deletes all databases associated with this adhd node
+
+require 'rubygems'
+require 'couchrest'
+require 'optparse'
+
+node_names = []
+delete_all = false
+db_url = ""
+
+opts = OptionParser.new do |opts|
+  opts.on("-u", "--url U", "node name") do |u|
+    puts "Database URL is #{db_url}"
+    # parse_config(n)
+    db_url = u
+  end
+
+
+  opts.on("-n", "--node N", "node name") do |n|
+    puts "Deleting databases of node #{n}"
+    # parse_config(n)
+    node_names << n
+  end
+  
+  opts.on("-d", "--delete", "Delete databases") do |n|
+    delete_all = true
+  end
+end
+
+opts.parse! ARGV
+
+if delete_all
+  puts "DELETE all databases in #{db_url}"
+else
+  puts "LIST all databases in #{db_url}"
+end
+
+puts node_names.join(' ,')
+
+
+# Find all databases we want
+server = CouchRest::Server.new(db_url)
+server.databases.each do |db_name|
+  node_names.each do |node_name|
+    if db_name.split('_')[0] == node_name
+      puts db_name   
+      if delete_all
+        db = server.database(db_name)
+        db.delete!
+      end 
+    end
+  end
+
+end

data/lib/adhd/adhd_rest_server.rb

@@ -0,0 +1,229 @@
+require 'eventmachine'
+require 'uri'
+require 'net/http'
+require 'webrick'  
+
+ module ProxyToServer
+  # This implements the connection that proxies an incoming file to to the 
+  # respective CouchDB instance, as an attachment.
+  
+  def initialize our_client_conn, init_request
+    @our_client_conn = our_client_conn    
+    @init_request = init_request
+    our_client_conn.proxy_conn = self
+  end
+  
+  def post_init
+     # We have opened a connection to the DB server, so now it is time
+     # to send the initial Couch request, using HTTP 1.0.  
+     puts "Send request: #{@init_request}"
+     send_data @init_request
+  end
+  
+  def receive_data data
+    @our_client_conn.proxy_receive_data data
+  end
+
+  def unbind
+    @our_client_conn.proxy_unbind
+  end  
+  
+ end
+
+ module AdhdRESTServer
+  attr_accessor :proxy_conn
+ 
+  def initialize node_manager
+    @node_manager = node_manager
+    @buffer = ""
+    @status = :header
+  end
+ 
+   #def post_init
+   #  puts "-- someone connected to the echo server!"
+   #end
+
+  # The format we are expecting:
+  #
+  # PUT somedatabase/document/attachment?rev=123 HTTP/1.0
+  # Content-Length: 245
+  # Content-Type: image/jpeg
+  #
+  # <JPEG data>
+
+   def receive_data data
+
+     # First we get all the headers in to find out which resource 
+     # we are looking for.
+     
+     if @status == :header
+       @buffer += data
+              
+       if data =~ /\r\n\r\n/
+
+        # Detected end of headers
+        header_data = @buffer[0...($~.begin(0))]
+       
+        # Try the webrick parser
+        @req = WEBrick::HTTPRequest.new(WEBrick::Config::HTTP)
+
+        StringIO.open(header_data, 'rb') do |socket|
+          @req.parse(socket)
+        end
+        
+        # The rest of the incomming connection        
+        @buffer = @buffer[($~.end(0))..-1]
+        
+        # Compute the ID of the sought resource
+        if @req.path =~ /\/adhd\/(.*)/
+          @req.header["Filename"] = $1
+          @req.header["ID"] = MD5.new($1).to_s
+        else
+          # Throw an error          
+        end
+       
+        # Change the status once headers are found
+        @status = :find_node 
+       else
+        # Avoid DoS via buffer filling
+        close_connection if @buffer.length > 1000
+       end
+       
+     end
+     
+     # Now we have the headers, but maybe not the full body, and we are looking
+     # for the right node in our network to handle the call.
+     if @status == :find_node
+        pause # We want to tell the remote host to wait a bit
+              # This would allow us to defer the execution of the calls to find 
+              # the right nodes, and extract the doc.
+        
+        # TODO: We need to push all the chit-chat with the remote servers to 
+        #       A deferable object, or some other connection, not to block.
+        #       Right now we are blocking and it sucks.  
+              
+        # Now get or write the document associated with this file
+        if @req.request_method == "GET"
+          
+          @our_doc = @node_manager.srdb.get_doc_directly(@req.header["ID"])
+          
+          # TODO: handle erros if file does not exist
+          if @our_doc[:ok]
+            @status == :get
+            handle_get
+          else
+            send_data "Problem"
+          end          
+        end
+        
+        if @req.request_method == "PUT"
+          # Define a Doc with the data so far
+          @our_doc = ContentDoc.new
+          
+          @our_doc._id = @req.header["ID"]
+          @our_doc.internal_id = @req.header["ID"]
+          @our_doc.size_bytes = @req.content_length
+          @our_doc.filename = @req.header["Filename"]
+          @our_doc.mime_type = @req.content_type
+          
+          # Write to the right node
+          @our_doc = @node_manager.srdb.write_doc_directly(@our_doc)
+          
+          # TODO: if an error is returned here, we cannot execute the query
+          if @our_doc[:ok]
+            @status = :put
+            handle_put
+          else          
+            send_data "Problem"
+          end
+        end
+      
+        # Now send the reply as an HTTP1.0 reponse
+      
+        # HTTP/1.0 200 OK
+        # Date: Fri, 08 Aug 2003 08:12:31 GMT
+        # Server: Apache/1.3.27 (Unix)
+        # MIME-version: 1.0
+        # Last-Modified: Fri, 01 Aug 2003 12:45:26 GMT
+        # Content-Type: text/html
+        # Content-Length: 2345
+        # ** a blank line *
+        # <HTML> ...
+      
+      
+      
+      # response = @our_doc.to_s
+      # 
+      # send_data "HTTP/1.0 200 OK\r\n"
+      # send_data "Content-Type: text/plain\r\n"
+      # send_data "Content-Length: #{response.length}\r\n"
+      # send_data "\r\n"
+      # send_data response
+      #
+      # # Close the connection
+      # close_connection_after_writing      
+
+     end
+     
+     # We have the header and the node, and now we execute the request
+     if @status == :execute_request
+     
+     end
+     
+   end
+
+  def handle_get  
+    resume
+    # We need to connect to the right server and build a header
+    server_uri = URI.parse(@our_doc[:db].server.uri)
+    server_addr = server_uri.host
+    server_port = server_uri.port
+    
+    docid = @our_doc[:doc]._id
+    dbname = @our_doc[:db].name
+    request = "GET /#{dbname}/#{docid}/#{@our_doc[:doc].filename} HTTP/1.0\r\n\r\n"
+    #send_data request
+    #close_connection_after_writing
+    puts "Connect to #{server_addr} port #{server_port}"
+    conn = EM::connect server_addr, server_port, ProxyToServer, self, request
+    EM::enable_proxy proxy_conn, self, 1024    
+  end
+  
+  def proxy_unbind
+    # Our cpnnection to the CouchDB has just been torn down
+    close_connection_after_writing
+  end
+  
+  def proxy_receive_data data
+    # Response to a PUT request only
+    send_data data
+  end
+  
+  
+  def handle_put
+    resume
+    
+    # We need to connect to the right server and build a header
+    server_uri = URI.parse(@our_doc[:db].server.uri)
+    server_addr = server_uri.host
+    server_port = server_uri.port
+    
+    docid = @our_doc[:doc]._id
+    dbname = @our_doc[:db].name
+    request = "PUT /#{dbname}/#{docid}/#{@our_doc[:doc].filename}?rev=#{@our_doc[:doc]["_rev"]} HTTP/1.0\r\n"
+    request += "Content-Type: #{@our_doc[:doc].mime_type}\r\n"
+    request += "Content-Length: #{@our_doc[:doc].size_bytes}\r\n"
+    request += "\r\n"
+    request += @buffer
+    #send_data request
+    #close_connection_after_writing
+    puts "Connect to #{server_addr} port #{server_port}"
+    conn = EM::connect server_addr, server_port, ProxyToServer, self, request
+    EM::enable_proxy self, proxy_conn, 1024  
+  end
+
+   def unbind
+     puts "-- someone disconnected from the echo server!"
+   end
+ end
+