lwac 0.2.0

data/import_config.md

@@ -0,0 +1,29 @@
+Import Tool Configuration
+=========================
+The import tool is responsible for creating (or adding links into) the server's corpus.  Its configuration file is thus very simple---all options for string processing and storage remain the preserve of the [server's config file](server_config.html).
+
+
+Config
+------
+The import tool uses the server configuration to access a corpus, and loads it as if it were a server.
+
+ * `server_config`  --- The path to the server configuration file.  The importer will use this for storage properties and string sanitisation settings.
+
+Other than that its configuration options apply to output and file handling:
+
+ * `schemata_path`  --- The path where `.sql` files may be found for creating a database.  Leave this blank to use defaults that come bundled with LWAC.
+ * `create_db`      --- Boolean.  Set to `true` to make the import script attempt to create the database if it doesn't exist.
+ * `notify`         --- How many lines should be import before the UI updates and tells the user.
+
+For example:
+
+    :server_config: example_config/server.yml
+    :schemata_path:                             # use defaults
+    :notify: 12345
+    :create_db: true
+
+
+Logging
+-------
+The logging system is the same for client, server, import, and export tools and shares a configuration with them.  For details, see [configuring logging](log_config.html)
+

data/index.md

@@ -0,0 +1,49 @@
+Index
+=====
+Welcome to the LWAC user guide.  The purpose of this document is to explain some of the design concepts used, provide recipes for common tasks, and explain some of the technical capacities/limitations of LWAC.
+
+Installation and Dependencies
+-----------------------------
+LWAC's dependencies and installation are documented [here](install.html)
+
+Concepts
+--------
+The [concepts](concepts.html) used in the model are explained here.
+
+Tools
+-----
+LWAC consists of a number of [tools](tools.html), and these are explained individually here.
+
+Workflows
+---------
+LWAC's [workflow](workflows.html) was designed around longitudinal sampling.  Some common methods are explained here.
+
+Process Monitoring
+------------------
+LWAC is designed for longitudinal sampling, and as such a process monitor is a good idea to ensure that you are notified in case of any problems. [Monitoring and maintenance](monitoring.html) has its own section.
+
+Limits and Performance
+----------------------
+See some rough indications on what [limits LWAC's performance](limits.html)
+
+Bugs and Suggestions
+--------------------
+Please get in touch by reporting bugs to [my issue tracker](http://stephenwattam.com/issues/), or [email me](http://stephenwattam.com/contact/).  Please do this if you notice something---I relish the opportunity to fix it.
+
+If you can write code, feel free to submit a patch or contact me for your own branch in the git repository.  There's a space in the authors listing for your name :-)
+
+License and Usage Conditions
+----------------------------
+LWAC is licensed under a [Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License](http://creativecommons.org/licenses/by-nc-sa/3.0/), a copy of which may be found in the `LICENSE` file in the root of the distribution.
+
+In addition to the creative commons terms above, I wish to mandate the following conditions: You must...
+
+ * Not profit from directly selling the code (including as part of something else);
+ * Not profit from selling corpora built using LWAC.  (Note that I will probably grant permission if contacted, I merely wish to stop people turning it into a commercial service);
+ * Credit LWAC and provide a link to [http://stephenwattam.com/project/LWAC/](http://stephenwattam.com/project/LWAC/) or [ucrel.lancs.ac.uk/LWAC/](http://ucrel.lancs.ac.uk/LWAC/) in any publications (You can also cite the paper pending for WaC8, once it's published);
+ * Not use it to DDOS things.
+
+Other than that, knock yourself out.  The code's fairly clean, and fairly well documented.
+
+It's worth noting here that the data you download from the web is probably copyright to some third party who wouldn't necessarily consent to its distribution.  Laws on the use of web data vary wildly and are a grey area in many juristrictions, so _caveat emptor_ (or _caveat corpus aedificator_).
+

data/install.md

@@ -0,0 +1,29 @@
+Installation
+============
+LWAC is distributed as a gem, and this should handle management of include paths and dependencies:
+
+ * [Ruby](http://www.ruby-lang.org/en/) v1.9.3 or above
+ * [libcurl](http://curl.haxx.se/libcurl/) (client only)
+ * [sqlite3](http://www.sqlite.org/) or [mysql](http://www.mysql.com/) or [mariaDB](https://mariadb.org/) (server only)
+ * Some supporting gems:
+   * simplerpc
+   * sqlite3 or mysql2 (server only)
+   * curb (client only)
+
+To install, simply run:
+
+    $ gem install lwac
+
+to install the latest version, and then:
+
+ * If you are only ever going to run the client, `gem install curb`
+ * If you are only ever going to run the server, `gem install sqlite3` _or_ `gem install mysql2`
+ * If you're going to run both clients and servers, install both of the above.
+
+Git
+---
+If you wish to run LWAC straight from the git repository, this is possible by adding a single item to ruby's `$LOAD_PATH`.  To do this, run:
+
+    ruby -I ./lib ./bin/lwac [commands]
+
+This is particularly useful when modifying the code.  If you make some modifications, please don't hesitate to get in touch and I'll do my best to integrate them upstream.

data/lib/lwac.rb

@@ -0,0 +1,17 @@
+
+# LWAC namespace
+module LWAC
+  # Overall LWAC version, as in the git tags
+  VERSION = '0.2.0'
+  # Date of last significant edit
+  DATE    = '20-07-13'
+
+  # Authors
+  AUTHORS = [
+              {:name => "Stephen Wattam", :contact => "http://stephenwattam.com"},
+             #{:name => "", :contact => ""}  # Add yourself here (and in the gemspec) if you contribute to LWAC
+            ]
+
+  # Location of resources
+  RESOURCE_DIR = File.join( File.dirname( File.expand_path(__FILE__) ), '../resources/') 
+end

data/lib/lwac/client.rb

@@ -0,0 +1,354 @@
+require 'lwac/client/file_cache'
+require 'lwac/client/storage'
+
+require 'lwac/shared/multilog'
+require 'lwac/shared/identity'
+require 'lwac/shared/serialiser'
+
+require 'timeout'
+require 'digest/md5'
+
+
+require 'simplerpc/client'
+require 'lwac/shared/data_types'  # for serialisation
+require 'blat'
+
+require 'curb'
+
+module LWAC
+
+  class DownloadClient
+
+    # Construct a new DownloadClient
+    def initialize(config)
+      # Save the config
+      @config = config
+
+      # Generate a unique identifier for this host
+      @uuid = generate_uuid
+
+      # Fire up el RPC client...
+      @rpc_client = SimpleRPC::Client.new(@config[:server])
+
+      # Don't RPC again until...
+      @rpc_delay    = Time.now
+
+      # Construct a new multi-curl thingy
+      $log.info "Starting download engine..."
+      @dl = Blat::Queue.new(@config[:client][:simultaneous_workers])
+
+      # List of links pending, cached.
+      @links        = []
+      @cache        = new_cache
+      @cache_bytes  = 0
+
+      # Mutices for link access
+      @link_mx      = Mutex.new
+      @cache_mx     = Mutex.new
+
+      # Don't try to acquire more data until...
+      @checkout_delay = Time.now
+
+      # Start the log with UUID info.
+      $log.info "Client started with UUID: #{@uuid}"
+
+      # ping for helpfulness
+      ping
+    end
+
+    # Poll the server and download from the web, maintaining throughput
+    # to the web by downloading batches of links.
+    def work
+
+      loop do
+        @dl.perform do
+
+          sleep(@config[:client][:monitor_rate])
+
+          # Keep the download queue topped up
+          while @dl.request_count < @config[:client][:simultaneous_workers] && (new_link = get_curl)
+            @dl.add(new_link)
+          end
+
+          # Read things safely using a mutex
+          link_len          = @link_mx.synchronize { @links.length }
+          active_requests   = @dl.request_count
+          cache_len, bytes  = @cache_mx.synchronize { [@cache.length, @cache_bytes] }
+
+          # Print nice progress output for folks
+          if @config[:client][:announce_progress] && (link_len > 0 || cache_len > 0 || active_requests > 0)
+            progress_mb      = bytes.to_f / 1024 / 1024
+            limit_mb         = @config[:client][:cache_limit].to_f / 1024 / 1024
+            pc_progress      = (progress_mb / limit_mb) * 100
+
+            str =  "#{progress_bar(@config[:client][:cache_limit], bytes)} #{pc_progress.round}%"
+            str += " #{progress_mb.round(2)}/#{limit_mb.round(2)}MB"
+            str += " (#{link_len} pend, #{active_requests} active, #{cache_len} done)"
+            str += " #{(link_len == 0 && Time.now < @checkout_delay) ? "[waiting #{(@checkout_delay - Time.now).round}s]" : ''}"
+            
+            $log.info(str)
+          end
+
+          # Run out of links
+          if link_len <= 0 && Time.now > @checkout_delay 
+            acquire_links
+          end
+
+          # Downloaded enough data already
+          if (@dl.idle? || bytes > @config[:client][:cache_limit]) && cache_len > 0
+            # (@pool.cache_size.to_f / 1024.0 / 1024.0) > @config[:client][:cache_limit] then
+            # Send completed points back to the server
+            send_cache
+          end
+        end
+
+        $log.debug "Downloader is idle."
+
+      end
+
+    rescue SignalException => se
+      $log.fatal "Caught signal - #{se}"
+    ensure
+      # Cancel web requests
+      cancelled = @dl.request_count
+      if @dl.request_count > 0
+        $log.info "Cancelling #{@dl.request_count} web requests..."
+        @dl.cancel
+      else
+        $log.info "No web requests active."
+      end
+
+      # Tell the server we're dying
+      if @links.length > 0 || @cache.length > 0 || @dl.request_count > 0
+        $log.info "Releasing lock on approx. #{@links.length + @cache.length + cancelled} links..."
+        rpc(5) do |s|
+          s.cancel(LWAC::VERSION, @uuid)
+        end
+      else
+        $log.info "No links to clean up."
+      end
+      
+      # Quit
+      $log.info "Done.  Client has closed cleanly."
+    end
+
+  private
+
+    # Pings the server to test RPC methods
+    def ping
+      $log.info "Pinging server..."
+      nonce = Random.rand(82349849)
+      reply = rpc(1) do |s|
+        s.ping(LWAC::VERSION, @uuid, nonce)
+      end
+      
+      unless nonce == reply
+        $log.warn "Failed to ping server!  Please check your network properties."
+      else
+        $log.info "Your network setup seems to work, that's good news :-)"
+      end
+    end
+
+    # Returns a cURL::Easy object for downloading
+    def get_curl
+      link = @link_mx.synchronize { @links.pop }
+
+      return nil unless link
+
+      # Construct new curl from the link
+      curl      = Curl::Easy.new(link[:link].uri)
+
+      # configure curl using config
+      link[:config][:curl_workers].each do |k, v|
+        if v.is_a?(Array)
+          curl.send(k.to_s + '=', *v)
+        else
+          curl.send(k.to_s + '=', v)
+        end
+      end
+
+      # Set completion handler
+      curl.on_complete do |res|
+        datapoint = nil
+        begin
+          datapoint = LWAC::DataPoint.from_request(link[:config], link[:link], res, @uuid, nil) # TODO: set error if needed.
+        rescue StandardError => e
+          $log.error "Error during request standardisation: #{e}"
+          $log.debug "#{e.backtrace.join("\n")}"
+        
+          # Insert error if the above failed
+          datapoint = LWAC::DataPoint.new(link[:link], {}, '', '', {}, @uuid, e) if !datapoint
+        ensure
+          @cache_mx.synchronize do
+            @cache[link[:link].id] = datapoint
+            @cache_bytes += res.downloaded_bytes
+          end
+        end
+        $log.debug "Link #{link[:link].id} downloaded."
+      end
+
+      $log.debug "Link #{link[:link].id} sent for download."
+
+      # Return curl
+      return curl
+    end
+
+    # Acquire links from the server.
+    def acquire_links
+
+      $log.info "Requesting #{@config[:client][:batch_capacity]} links..."
+
+      loop do
+        ret = rpc do |s|
+          s.check_out(LWAC::VERSION, @uuid, @config[:client][:batch_capacity])
+        end
+
+        # If the server tells us to back off, so do.
+        if ret.class == Fixnum
+          $log.info "Server says to ask again at #{Time.now + ret}"
+          @checkout_delay = Time.now + [ret, @config[:network][:maximum_reconnect_time]].min
+          return 
+        elsif ret.class == Array && ret.length == 2
+
+          # Load the worker config into the list
+          policy, links = ret
+          @link_mx.synchronize do
+            links.each do |l|
+              @links << {:link => l, :config => policy}
+            end
+          end
+
+          $log.info "Received #{links.length}/#{@config[:client][:batch_capacity]} links from server."
+          return links.length
+        else
+          $log.warn "Received unrecognised return from server of type: #{ret.class}.  Retrying..."
+          $log.debug "Server said: '#{ret}'"
+        end
+      end
+    end
+
+
+    def send_cache
+
+      # Create a new cache and keep the old one for uploading
+      cache_to_send, bytes_to_send = nil, nil
+      @cache_mx.synchronize do
+        # Retain handles to old ones
+        cache_to_send = @cache
+        bytes_to_send = @cache_bytes
+
+        # And create new ones
+        @cache          = new_cache
+        @cache_bytes    = 0
+      end
+
+      while cache_to_send.length > 0 do
+
+        # Take them out of the pstore up to a given size limit, then send that chunk
+        pending = []
+        pending_size = 0.0
+        $log.debug "Counting data for upload..."
+        while(cache_to_send.length > 0 and pending_size < @config[:client][:check_in_size]) do
+          key             = cache_to_send.keys[0]
+          dp              = cache_to_send[key]
+          pending_size    += dp.response_properties[:downloaded_bytes].to_f
+          pending         << dp
+          cache_to_send.delete_from_index(key)
+        end
+
+        # send datapoints
+        $log.info "Sending #{pending.length} datapoints (~#{(pending_size.to_f / 1024 / 1024).round(2)}MB) to server..."
+        ret = rpc do |s|
+          s.check_in(LWAC::VERSION, @uuid, pending)
+        end
+        if ret.is_a?(Array)
+          $log.info "Done (server reported #{ret[0]} failures)"
+          $log.info "Server reports work rate as #{ret[1].to_f.round(2)} links/s" if ret[1]
+        else
+          $log.warn "Server returned something unexpected when checking in."
+        end
+      end
+
+      # Here cache_to_send.length == 0, so wipe the cache
+      cache_to_send.delete_all
+    end
+
+    # Yields to perform RPC tasks with a backoff
+    def rpc(retries = -1)
+      $log.debug "Connecting to server #{@rpc_client.hostname}:#{@rpc_client.port}..."
+      # TODO: update this to match new SimpleRPC exception format (as soon as one is implemented)
+      
+      failed = true
+      ret    = nil
+      rpc_delay_increment = @config[:network][:minimum_reconnect_time]
+      while (retries -= 1) != -1 && failed do
+
+        # Delay until the point we were asked to
+        if @rpc_delay > Time.now
+          $log.info "Rate limit: delaying for #{(@rpc_delay - Time.now).round}s until #{@rpc_delay}..."
+          sleep(@rpc_delay - Time.now) + 0.1 
+        end
+
+        begin
+
+          ret = yield(@rpc_client.get_proxy)
+          failed = false
+
+        # This looks funny, and is, but I double-catch in order
+        # to handle remote exceptions, which extend exception but not
+        # standarderror
+        rescue SignalException => se
+          raise se
+        rescue Exception => e
+
+          if e.is_a?(SimpleRPC::RemoteException)
+            $log.error "Server reported error: #{e}"
+          else
+            $log.error "Local error during RPC call: #{e}"
+          end
+
+          $log.debug "#{e.backtrace.join("\n")}"
+          failed = true
+
+          $log.warn "#{retries} retries remaining before disconnection..." if retries > 0
+
+          # Delay longer on failure
+          @rpc_delay = Time.now + [rpc_delay_increment, @config[:network][:maximum_reconnect_time]].min
+          rpc_delay_increment += @config[:network][:connect_failure_penalty]
+        end
+      end
+
+      return ret
+    end
+
+    # Create this client's ID.
+    # Must be persistent across instances, but not across machines.
+    def generate_uuid
+      require 'socket'
+      # TODO: make this ID based more on IP address, and/or make it more readable
+      @config[:client][:uuid_salt] + "_" + Digest::MD5.hexdigest("#{Socket.gethostname}#{@config[:client][:uuid_salt]}").to_s 
+    end
+
+    # Get a new cache to replace the one in the pool
+    def new_cache
+      # Create cache in a random filename in the dir specified
+      filename = nil
+      filename = File.join(@config[:client][:cache_dir], rand.hash.abs.to_s) if @config[:client][:cache_dir]
+
+      $log.debug "Creating cache #{(@config[:client][:cache_dir] == nil) ? "in RAM" : "at #{filename}"}..."
+      return Store.new(filename)
+    end
+
+    # Returns a string progress bar for use in output
+    def progress_bar(total, progress, length=25)
+      bar_len = ((progress.to_f / total.to_f) * length).round
+      str = '['
+      str += '=' * [bar_len, length].min
+      str[-1] = '>' if bar_len > length
+      str += ' ' * (length - [bar_len, length].min)
+      str += ']'
+    end
+
+  end
+
+end